Rebuild a destroyed VPS against a stopwatch

An untested backup is a belief, not a capability. The only way to know whether you can recover is to destroy something on purpose and rebuild it while timing yourself.

This is the drill. It assumes you already take backups. If you do not, start with encrypted off-site backups with Restic and PostgreSQL backups that pass a restore drill, then come back and rehearse the whole machine rather than one component.

Decide what you are actually promising

Two numbers define recovery, and you should write them down before measuring anything:

RPO is decided by backup frequency. Nightly backups mean a worst case of nearly 24 hours of lost work, no matter how fast you restore. If that is unacceptable, the fix is more frequent backups or continuous archiving, not a faster restore.

RTO is decided by the procedure, and that is what the drill measures.

Set both deliberately. "As fast as possible" is not a target, and it gives you no way to tell whether the drill passed.

Write the runbook before you need it

The recovery procedure must be written down somewhere that survives the loss of the server. A runbook stored only on the machine it describes is not a runbook.

Keep it short and ordered, with the exact commands, and store it in a repository plus somewhere readable offline. The minimum contents:

1. Where the backups are, and how to authenticate to them.
2. Where the repository password or key lives, which must not be only on the lost server.
3. The plan and location the replacement needs.
4. Base packages and versions.
5. Restore steps, data first or config first, in the order that works.
6. DNS and any addresses that must move.
7. How to verify the service is genuinely serving, not just running.

The credentials point deserves emphasis: if the only copy of your backup passphrase was on the machine that died, you do not have backups. Put a copy in a password manager, and a second copy somewhere physically separate.

Run the drill on a scratch server

Do not rehearse on production and do not rehearse in your head.

Deploy a fresh VPS in the same location and on the same plan as the original. Same plan matters, because a restore that needs more memory than the real machine has is a restore that will fail during the real incident.

Start a timer. Then work only from the runbook. If you find yourself remembering a step that is not written down, stop, add it to the runbook, and note that the drill has already paid for itself.

date -u +%FT%TZ    # start of drill

Restore the data

Fetch the backup from off-site, as the real incident would:

export RESTIC_REPOSITORY=sftp:backup@example.net:/vps-01
export RESTIC_PASSWORD_FILE=/root/.config/restic/password

restic snapshots
restic restore latest --target /

Check what arrived before assuming it is complete:

restic stats latest
find /srv -newer /etc/hostname | head

Restore the database separately

Database files copied from a running server are not a backup. Restore from a logical dump or a proper base backup:

sudo -u postgres createdb app
sudo -u postgres pg_restore -d app --clean --if-exists /srv/backups/app.dump

Then check the data is actually there, with a query that would fail if the restore were partial:

sudo -u postgres psql -d app -c "select count(*) from users;"
sudo -u postgres psql -d app -c "select max(created_at) from orders;"

That second query is the one that tells you your real RPO. The gap between that timestamp and the moment of the simulated failure is exactly how much data you lost.

Bring the service up

sudo systemctl daemon-reload
sudo systemctl enable --now my-service
sudo systemctl status my-service

Verify from outside

A service that starts is not a service that works. Test it the way a user would, from somewhere else:

curl -sS -o /dev/null -w '%{http_code}\n' https://example.com/healthz
curl -sS https://example.com/api/orders/recent | head

Stop the timer here, not when the process started.

date -u +%FT%TZ    # end of drill

What the drill will find

It will find something. These are the usual ones, and each is cheaper to discover now:

• A file nobody backed up. Usually something in /etc that was edited by hand, a TLS key, or a cron entry.
• A package version mismatch. The database refuses a dump from a newer major version. Record versions in the runbook.
• A credential that only existed on the dead machine. API tokens, the backup passphrase itself, an SSH key an automated job used.
• DNS TTL longer than your RTO. If the record has a 24-hour TTL, users cannot reach the new address inside a 2-hour target no matter how fast you rebuilt. Lower TTLs in advance of any planned move.
• A restore that needs more disk than the plan has. Restoring alongside existing data can need close to double the space at the moment of the swap.
• Steps in the wrong order. Config restored after the service started, so it ran with defaults and wrote bad state.

Each finding becomes a runbook edit. That is the actual output of the drill, more than the elapsed time.

Record the result

Keep a short log, because the trend matters more than any single run:

A failed drill is a successful drill. Failing at a time you chose, on a scratch server, is the entire point.

Make it a habit

Rehearse on a schedule and after any significant change to the stack. Quarterly suits most small deployments; monthly if the system changes often.

Automate the parts that can be automated, particularly the boring verification that backups are still arriving and still readable:

restic check --read-data-subset=5%

That reads a sample of the actual data rather than only the metadata, which catches corruption that a plain check will not. Run it on a timer and alert if it fails. A backup job that has been silently failing for six weeks is the most common way this all goes wrong.

Also alert on the absence of a fresh snapshot. Monitoring that a job succeeded is good; monitoring that a recent backup exists is better, because it catches the case where the job stopped running at all.

Checklist

• RTO and RPO are written down and agreed.
• The runbook exists outside the server it describes.
• Backup credentials exist in at least two places, neither of them the server.
• The drill runs on a fresh VPS of the same plan and location.
• You worked only from the runbook, and edited it wherever it fell short.
• Recovery was verified from outside, not from the box.
• Measured RPO came from a real timestamp in the restored data.
• Elapsed time is recorded and compared against the target.
• restic check --read-data-subset runs on a timer and alerts.
• The next drill is scheduled.

The first drill is always slower than the target and always finds something missing. That is not a failure of preparation, it is the reason to do it before the day when the machine is gone and the clock is real.