Server backups: the 3-2-1 rule and how to restore

The 3-2-1 rule says: keep three copies of your important data (the original and two backups), on two different kinds of storage, with at least one copy kept somewhere else. On a VPS, that means having at least one copy outside the server, in addition to the data on the server itself.

The same rule is recommended in the Data Backup Options paper prepared for US-CERT and published on the CISA website. Its purpose is to make sure that one failure, mistake or attack can't destroy every copy at once.

A backup, a snapshot and disk redundancy are not the same thing

• Backup: a separate copy of your data, usually with earlier versions. It brings back a file that was deleted or damaged by mistake.
• Snapshot: records the state of the server at that moment. It is good for a quick way back before a big update. But it depends on the original disk, so it doesn't replace a backup.
• Disk redundancy: the same data written to two disks at once. It protects you when one disk fails. A file you delete is deleted from both disks, and data encrypted by ransomware is encrypted on both. So it isn't a backup.

How to apply 3-2-1 on a VPS

You can count the panel's daily backup as one of your copies. But the copy that meets the "somewhere else" condition is one you set up and control yourself. Seturack's terms of service also ask you to keep a copy of your important data separately.

What to back up

Copying the files of a running database can give you an inconsistent backup. Take a dump first. These commands run as root, as they would in a cron script:

mysqldump --single-transaction --all-databases | gzip > /var/backups/mysql-$(date +%F).sql.gz

sudo -u postgres pg_dumpall | gzip > /var/backups/pg-$(date +%F).sql.gz

For files, this is enough: /etc, your website files, your application's data folder, Docker volumes if you use Docker, and .env files. Don't back up what you can download again, such as packages and Docker images. If your application has an encryption key, store it separately as well. For some applications, a backup without the key is useless.

How often you back up depends on how much data you can afford to lose. One backup a day means losing up to a day of data in the worst case. For an application that records orders or payments, it is safer to dump the database every hour and send it off the server.

An off-server backup with restic

restic is an open source backup tool that encrypts your backups and stores repeated data once. The example below writes the backup to another server reached over SFTP. The target needs passwordless (key-based) SSH login, because an automated backup can't answer a password prompt. Create the password file (/root/.restic-password) in a text editor and run chmod 600 on it.

sudo apt install -y restic

export RESTIC_REPOSITORY=sftp:[email protected]:/backups/vps

export RESTIC_PASSWORD_FILE=/root/.restic-password

restic init

restic backup /etc /var/www /var/backups

restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

The restic documentation warns that if you lose the password, your data is gone for good. Keep the password in a password manager too. Put the two export lines, the database dumps, restic backup and restic forget in a script such as /usr/local/bin/backup.sh, in that order, and make it executable with chmod +x. Don't forget to add a line at the end that deletes old dumps, find /var/backups -name '*.sql.gz' -mtime +2 -delete, or the dump files will fill the disk. Then run the script every night, for example from /etc/cron.d/backup with this line:

30 3 * * * root /usr/local/bin/backup.sh

borg is a similar tool. For plain file copies you can use rsync, and for sending data to object storage you can use rclone.

The real test: restoring

A backup you can't restore is worthless. Try it right after you set things up, and then at regular intervals:

• With restic, extract the latest backup into an empty folder and check the files: restic restore latest --target /tmp/restore-test
• Load a database dump into a temporary database and look at the tables.
• In the panel, click the Recover files from it icon next to a backup, browse inside and download one file. A full restore replaces the entire current disk, so recovering a single file is the safer way to test.

Choose how long to keep backups by how late you might notice a problem. The panel backup keeps the last 7 days, and you can't go back further than that. If you may need older versions, keep weekly and monthly versions in your own copy. For the details of the panel backup, see the daily backups article.

Common mistakes

• Writing the backup to the same server's disk. If the disk fills up or the server is lost, the backup goes with it.
• Keeping the backup password or key in the same place as the backup.
• Not noticing that the script is failing. Add a notification to the script, or check the date of the latest backup now and then with restic snapshots.
• Not watching the backup target fill up. A full target means backups that fail silently.