Testing a restore

What goes wrong, in order of how often we see it

1. The backup was empty. A path changed, a database dump failed on a permission, and the job reported success because nobody checked the exit code.
2. The encryption key is on the machine that died. Encrypted backups are correct; keeping the only copy of the key on the source is not.
3. The restore takes three days. Nobody measured how long pulling two terabytes back actually takes.
4. The dump is inconsistent. Files copied while the database was writing restore into something that will not start.
5. Something was never in scope. Certificates, cron jobs, firewall rules, the contents of /etc. The data came back and the machine still did not work.

Practise it quarterly

Put a recurring half hour in the calendar and do a real restore, not a thought experiment:

1. Take the most recent backup, not a special one made for the test.
2. Restore it somewhere isolated — a spare machine, a virtual machine, a container.
3. Start the application against the restored data.
4. Check a specific record you know should be there, and the date of the newest row.
5. Write down how long the whole thing took.

That last number is your real recovery time. It is usually several times the one people quote from memory.

Database dumps, consistently

For MySQL and MariaDB, use a single transaction so the dump is a consistent snapshot:

mysqldump --single-transaction --routines --triggers --events --all-databases | zstd > dump.sql.zst

For PostgreSQL:

pg_dump -Fc mydb > mydb.dump

Copying the data directory of a running database is not a backup unless you use filesystem snapshots — see ZFS — or a tool built for it such as mariabackup or pg_basebackup.

Verify every run, not just at test time

• Check the exit code of the backup job and alert when it is non-zero. See cron jobs and timers.
• Alert when the backup is smaller than expected, not only when it is missing. A 4 KB dump where yesterday's was 4 GB is a silent failure.
• Alert when the newest backup is older than it should be.
• With restic or borg, run the repository check regularly: restic check --read-data-subset=5%

Keep a copy off the machine

A backup on the same server protects against a mistake, not against the machine. A second copy somewhere else — another of your servers, a storage box, another site — is what protects against fire, theft and ransomware. Keep at least one copy that the production machine cannot delete, because ransomware deletes the backups it can reach. See backups for the strategy and a compromised server for why the immutable copy matters.

---

Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base