Incremental backups

  Last modified on Friday 12 February 2016

PROJECT: Home Server #4 .: Make incremental and automatic backups of a home folder using rsnapshot + cron (and manual backups via public transit).

Let's go!

Backup strategy Version 0 for my home folder was rsync + portable + subway. That is, I would simply make a periodic, manual sync of files using rsync from my primary computer to a portable encrypted USB hard drive. Then ride the subway to a family/friend's place and swap drives (offsite backup).

Version 1 expanded to cron + rsync + Raspberry Pi + portable + subway. One of the advantages of setting up a Raspberry Pi home server is providing a 24/7 uptime location to automatically (using cron) mirror my laptop's home folder over the local area network (LAN).

These backups are snapshots of home at a particular date. Pi server has a backup of 24 hours or less, the USB drive has another snapshot a few weeks old, the offsite another snapshot from a month ago. But there is no ordered progression of backups from Day 2 to Day 3, Week 4, Month 5, etc. One alternative is to every day stash an entire backup of my home folder but its a sub-optimal use of resources and would quickly fill a hard drive.

A much better solution is to use rsnapshot to make incremental backups and my backup strategy Version 2 now incorporates:

0. Rsnapshot

Install rsnapshot, make a directory to store backups, and make a copy of the default config file:

$ sudo apt-get install rsnapshot
$ mkdir /my/backup/location
$ sudo cp /etc/rsnapshot.conf /etc/rsnapshot.conf.default

Modify /etc/rsnapshot.conf (important to separate fields with TABS not spaces). Example of a few tweaks:

snapshot_root   /my/backup/location

cmd_cp          /bin/cp
cmd_rm          /bin/rm
cmd_rsync       /usr/bin/rsync
cmd_du          /usr/bin/du
cmd_rsnapshot_diff      /usr/bin/rsnapshot-diff

retain  daily   7
retain  weekly  4
retain  monthly 12

exclude_file    /home/USER/.rsyncExclude  # ...and create this file with list of things to exclude from backup

link_dest       1

sync_first      1  # allows better recovery in the event that rsnapshot is interrupted (see: ``man rsnapshot``)

backup  /home/          localhost/
backup  /etc/           localhost/

Check config syntax and run backup test:

$ sudo rsnapshot configtest
$ sudo rsnapshot -t sync

If everything checks out OK go ahead and run:

$ sudo rsnapshot sync && sudo rsnapshot daily && sudo rsnapshot du

Automate backups by modifying the sample cron file provided in /etc/cron.d/rsnapshot and running jobs as root. Example config:

# m h  dom mon dow   command
50 23 * * *     root    /usr/bin/rsnapshot sync && /usr/bin/rsnapshot daily
40 22 * * 6     root    /usr/bin/rsnapshot weekly
30 21 1 * *     root    /usr/bin/rsnapshot monthly

1. Backup the backup

Rsnapshot operates as a pull program: it pulls in backups from local and remote devices. Instead of juggling access permissions to allow the rsnapshot server to talk with other devices I decided to limit rsnapshot to making backups on localhost and use my already-configured SSH key setup to push a snapshot of the backup to my Raspberry Pi for remote storage.

1.1 On the Pi

Create a directory to store the backup:

$ mkdir /path/to/backup

1.2 On localhost

Set the ip address and hostname of the Pi server in /etc/hosts:    raspberry.server

Test synching /my/backup/location/ on localhost to raspberry.server:/path/to/backup/ with the rsync --dry-run option (I exclude /etc/ from the backup):

rsync --dry-run --archive --hard-links --numeric-ids --delete --exclude=etc/ --verbose /my/backup/location/ raspberry.server:/path/to/backup/

If everything checks out OK drop --dry-run and re-run the command to make a proper backup.

I use keychain to manage SSH keys for password-less logins to the Pi. Create a shell script and place in ~/bin:

. ${HOME}/.keychain/${HOSTNAME}-sh
rsync --archive --hard-links --numeric-ids --delete --exclude=etc/ /my/backup/location/ $1

Automate the backups by creating a cron job (example that runs daily at 04:50):

# m h  dom mon dow   command
50 4 * * *  /home/USERNAME/bin/ raspberry.server:/path/to/backup/

2. External drive and offsite storage

I connect my USB drive and sync the backup to the device:

$ /home/USERNAME/bin/ /media/USB/path/to/backup/

... and take my hard drive for a ride on the subway to say hello to my offsite storage!

Happy hacking!

More • linux