rsnapshot is a great tool that uses rsync to automatically take snapshots of your system and store those snapshots efficiently (using hardlinks). It allows you to take periodic snapshots of your files with each snapshot being an independent set of files with common file versions being hardlinked. This essentially means that the backup folder (holding the snapshots) will increase by only the file sizes of the files that have changed between snapshots.
Furthermore, it can be fully setup and configured via ssh to your server.
This guide covers installing and setting up rsnapshot on a Linux server (this example uses Ubuntu server 16.04).
Install rsnapshot
We'll install rsnapshot using apt-get (note if you're running another Linux variant you'll need to use it's own package manager):
sudo apt-get install rsnapshot
Configure rsnapshot
Once you've installed rsnapshot we'll need to configure it. We will configure it in two steps:
- modify /etc/rsnapshot.conf to suit your desired snapshot settings (e.g. where to store snapshots, what to backup, what to exclude);
- enable cron job for required snapshot frequency.
Modify /etc/rsnapshot.conf
We now want to modify rsnapshot settings. We do this by modifying the rsnapshot.conf file to suit our requirements:
sudo nano /etc/rsnapshot.conf
This will open the configuration file in nano (my favourite text editor on Linux). This is quite a long file. Below I've just added the lines I modified, and a few comments on those modifications:
Note that where you see spaces between arguments they ARE actually tabs. Note that if you copy/paste into a text editor you will likely need to replace spaces with tabs. E.g:
snapshot_root<--TAB-->/backup/rsnapshot/
# This option designates where the snapshots will reside in your file system. I've put them in a /backup/rsnapshot folder snapshot_root /backup/rsnapshot/ # The retain arguments define the number of snapshots to retain at different levels. These levels correspond to cron jobs that we will enable in step 2. # I'm going to run cron job beta daily (so below will keep 7 daily snapshots), and cron job gamma weekly (so will keep 4 weekly snapshots). retain beta 7 retain gamma 4 # The below defines what folders I want excluded from the snapshots (note that their parent/ancestor folder are included) exclude /var/atlassian/application-data/confluence/backups exclude /var/backups exclude /var/tmp exclude /var/lib exclude /var/opt exclude /var/cache exclude /home/jay/installs # Below defines what folders I want included in the snapshots. backup /home/ localhost/ backup /etc/ localhost/ backup /var/ localhost/ backup /usr/local/ localhost/
You can test your configuration file to make sure it doesn't contain errors:
rsnapshot configtest
Enabling rsnapshot cron jobs to execute the snapshots
The above only configures rsnapshot but does not actually execute snapshots. For this we will setup cron jobs. Thankfully, rsnapshot installs a file with the relevant cron jobs. Let's edit this:
sudo nano /etc/cron.d/rsnapshot
Here we want to uncomment the lines associate with the retain levels we defined in the rsnaphot.conf file. Below is an example of my rsnapshot.conf file:
# This is a sample cron file for rsnapshot. # The values used correspond to the examples in /etc/rsnapshot.conf. # There you can also set the backup points and many other things. # # To activate this cron file you have to uncomment the lines below. # Feel free to adapt it to your needs. # 0 */4 * * * root /usr/bin/rsnapshot alpha 30 3 * * * root /usr/bin/rsnapshot beta 0 3 * * 6 root /usr/bin/rsnapshot gamma # 30 2 1 * * root /usr/bin/rsnapshot delta
Note that I've set the gamma to '0 3 * * 6' which mean at 3am every Saturday (0=sunday, 1=monday, ...).
Tarball snapshot for transfer
All snapshots will live in the folder we defined in the rsnapshot.conf file. I often then tarball (compress) a snapshot and copy to a location where I can transfer/download it for archive purposes.
Below will tarball the snapshot to my home folder ready for copying and archiving:
sudo tar cvpzf ~/home-server-rsnapshot-`(date +%Y%m%d)`.tar.gz /backup/rsnapshot/beta.0/
WARNING: snapshots backup sensitive files, such as SSH keys and the like. Once copied for archiving you are strongly advised to encrypt the snapshot tar.gz with something like 7zip.
Backuping up mysql databases
I run several applications which use a mysql database. I want to backup these as well when I take my snapshots. Luckily, we can add a script to rsnapshot.conf which will dump (using mysqldump) all databases and include that dump in our snapshot. Even better is that rsnapshot comes with a script which you can copy and integrate with your rsnapshot.conf file.
Let's copy the script to /usr/local/bin and set appropriate permissions:
sudo cp /usr/share/doc/rsnapshot/examples/utils/backup_mysql.sh /usr/local/bin/ sudo chown root:root /usr/local/bin/backup_mysql.sh sudo chmod 755 /usr/local/bin/backup_mysql.sh
For this script to work, we will need to provide it with mysql credentials that have access to all databases. This generally means root and the associated password. For this we're going to create a text file in /root and will be locked down to only root.
Please note that there are security concerns to be aware of here as anyone with sudo access (e.g. anyone in sudoers) will be able to view the the /root/.my.cnf file and hence the root mysql password. Hence, this approach should only be used when the server is secured and where sudo access is given is only those who should have it.
Right, let's create said file first:
sudo nano /root/.my.cnf
Add the following into the file:
[client] user = root password = <PASSWORD> host = localhost [mysqldump] max_allowed_packet = 512M
where <PASSWORD> is the mysql root password.
Now, exit nano, and let's lock down this file to be only viewable as root:
chmod 600 /root/.my.cnf
With the our .my.cnf file now created, we can finally modify our /etc/rsnapshot.conf file to run the mysqldump script as part of the snapshot process:
# backup script for mysql databases backup_script /usr/local/bin/backup_mysql.sh mysql/
Backuping up postgresql databases
We can't forget postgresql databases. Several of the applications I run on my server use postgresql (i.e. Gitlab, Odoo, etc.). We want to backup these in a similar fashion as our mysql data backups.
We'll start by creating a backup script for our postgresql databases:
sudo nano /usr/local/bin/backup_postgresql.sh
Add (and modify) the below:
#!/bin/sh # login as postgres system user and run pg_dumpall (as that user) sudo -u postgres -i -H sh -c "pg_dumpall > postgres-all-databases.sql" # now move backup file to working directory mv /var/lib/postgresql/postgres-all-databases.sql . # change owner:group and make readable by root only /bin/chown root:root postgres-all-databases.sql /bin/chmod 600 postgres-all-databases.sql
Now set appropriate permissions to our created backup script:
sudo chmod 755 /usr/local/bin/backup_postgresql.sh
Finally, let's add this script to run during our rsnapshot backups. We need to modify our /etc/rsnapshot.conf file to add the follwoing:
# backup script for postgresql backup_script /usr/local/bin/backup_postgresql.sh postgresql/
Backing up GitLab with rsnapshot
The methods above cover most of the applications I run on my server (for example, confluence, Jira since I use mysql for them, and odoo since uses postgresql). One server application I also use is the wonderful GitLab. Although we can dump the postgresql database with pg_dump (see section above), the below method uses GitLabs built in backup and executes it for inclusion in our rsnapshot snapshot. This guide assumes you have GitLab installed (omibus... not build from source) and are using default settings for the backup location.
Let's start by creating a backup script that will do the heavy lifting for us:
sudo nano /usr/local/bin/backup_gitlab.sh
Add (and modify) the below:
#!/bin/sh
# Run gitlab built-in backup (will backup to /var/opt/gitlab/backups)
gitlab-rake gitlab:backup:create
# save current working directory (will copy backup here)
workingDir=$(pwd)
# now copy latest created file (the backup) to the saved working dir
cd /var/opt/gitlab/backups && ls -1t | head -n +1 | xargs -I{} cp -p {} $workingDir
# now delete backups in default gitlab backup folder
rm -f /var/opt/gitlab/backups/*
Now set appropriate permissions to our created backup script:
sudo chmod 755 /usr/local/bin/backup_gitlab.sh
Finally, let's add this script to run during our rsnapshot backups. We need to modify our /etc/rsnapshot.conf file to add the follwoing:
# backup script for gitlab backup_script /usr/local/bin/backup_gitlab.sh gitlab/
References
- https://community.time4vps.eu/discussion/159/how-to-install-and-use-rsnapshot-making-servers-backups
- https://unix.stackexchange.com/questions/1067/what-directories-do-i-need-to-back-up
- https://www.howtoforge.com/set-up-rsnapshot-archiving-of-snapshots-and-backup-of-mysql-databases-on-debian
- https://www.postgresql.org/docs/9.2/static/app-pg-dumpall.html
- https://docs.gitlab.com/ee/raketasks/backup_restore.html
Related articles