Server backups: why a restore test is the real check

When I take over a server, the first thing I ask is: "Do you have backups?" The answer is almost always "yes". The next question — "When did you last restore anything from them?" — is usually met with silence.

That is where backups usually break. Not because nobody makes copies, but because nobody knows whether anything can actually be brought back from them.

Short answer: a backup only exists once something has been successfully restored from it. Keep at least three copies on two different media, with one copy away from the server and out of an attacker's reach. Back up databases with the database's own tools, not as a file copy. And test restores regularly — not only after something is already lost.

Why "we have backups" is often not true

Typical situations:

• The backup script stopped running months ago — after a password change or a full disk, for example — and nobody noticed, because the error message went nowhere.
• The database was copied as files while it was running. The copy exists, but it captured a half-finished write, and the database may not start from it.
• The backup sits on the same server. When the disk dies or ransomware encrypts the server, the copy disappears along with the original.
• The copy exists, but there is no restore procedure. Recovery takes two days instead of an hour, because nobody remembers which version, which folder and in what order.

The 3-2-1 rule in plain words

• 3 copies of the data (the original plus two backups),
• on 2 different media or systems,
• 1 copy somewhere else — another data centre or another provider.

Today I add one more requirement: at least one copy must be immutable or otherwise protected from an attacker. Ransomware deliberately looks for backups too. If the server has both write and delete rights to the backup storage, an attacker can wipe the backups as well. So the server pushes copies out, but cannot delete them itself.

What to back up — and what not

Back up:

• Databases (PostgreSQL, MySQL/MariaDB) with a proper dump or a backup method that gives a consistent state. Not a blind file copy of a running database.
• Application files created by users: uploads, documents, images.
• Configuration: web server config (Nginx, Apache2), application settings, scheduled jobs, Docker compose files and environment variables.
• Certificates and keys — encrypted.

Don't waste space on:

• operating system files that can simply be reinstalled;
• caches and temporary files;
• Docker images that can be pulled again from a registry (back up the data volumes, not the containers).

If a server is well documented, rebuilding the system from scratch is often faster than restoring a whole disk image. So above all, back up what cannot be recreated.

How long to keep copies

A single copy from yesterday does not help if the error happened a week ago and has already found its way into every copy. A typical rotation might be: daily copies for two weeks, weekly copies for a couple of months, monthly copies for a year. The right rhythm depends on how fast your data changes and what your accounting or contracts require.

Backups must be encrypted — especially when they live on someone else's infrastructure. But the encryption key has to be stored somewhere other than the server: if the key disappears with the server, the encrypted copy is useless.

Restore testing: two questions that need answers

Before anything happens, agree on two numbers:

• How much data can you afford to lose? If you back up once a day, in the worst case you lose a full day of work. If that is too much, you need more frequent backups.
• How long can the system be down? Does recovery have to fit into an hour, or can it take a day?

A restore test tells you whether those numbers are actually achievable. My routine is simple:

1. Pick a random backup — not always the latest one.
2. Restore it into a separate test environment, not onto the live server.
3. Check that the database starts, the application opens and the most recent data is there.
4. Measure how long it took, and write down everything that did not go the way the procedure said.

And one automatic safeguard: backups need an alert when a copy is missed or is suspiciously small. A silent failure is the most expensive failure in backups.

Backups and restore tests are part of my server management — what else belongs in a maintenance contract is covered in Linux server maintenance: what the contract covers. If you are moving to a new server, a fresh restore test is also the best check before you upgrade or migrate an old Ubuntu server.

Frequently asked questions

Isn't my hosting provider's backup enough?

It is a good extra layer, but rarely enough on its own. Retention is often short, you can only restore the whole server at once, and the copy lives with the same provider. If the problem is the provider or your account, both copies become unreachable at the same time.

How often should I test a restore?

At least a couple of times a year, and always after a bigger change: a new server, a new database version, a modified backup script. More often for critical systems.

Is RAID a backup?

No. RAID protects against a single disk failing, but a deleted file or ransomware-encrypted data is deleted or encrypted on every disk at once.

Are applications running in Docker backed up differently?

The principle is the same: back up the data and the configuration, not the container. A database in a container is still backed up with a proper database dump, not a raw copy of the volume while it is running.

Are your backups actually verified?

If you don't know for sure when anything was last restored from your server's backups, write to me. I will review what is backed up, where it goes and how long it is kept, and run the first restore test together with you. More about how my server management works is on the Linux server management page.