Kingsfordweg 151, 1043GR Amsterdam, The Netherlands ● 24/7 support — support@novogara.com
Novogara BV
Knowledge base

Major OS upgrades

A major version upgrade on a production machine is one of the few operations where the safest answer is usually not to do it.

Consider rebuilding instead

Build a new machine on the new version, move the workload, switch, keep the old one for a week. You get a clean system with none of the accumulated cruft, you test the new version before it matters, and the rollback is “point it back”.

On a monthly dedicated server that costs you one extra month — see moving to another server and planning a migration window. Compared with an in-place upgrade that goes wrong at 40% through, it is cheap.

In-place is reasonable when the machine is simple, when you cannot get a second one, or when the upgrade path is well trodden and your software is all from the distribution.

If you do it in place

Before anything:

  1. A full backup, and a tested restore
  2. An LVM snapshot of the root volume if you have the space — this is the fastest rollback there is
  3. Confirm remote KVM works now, because SSH will stop during the upgrade and the console is your only way back
  4. Note your PHP, database and language runtime versions and check your application supports the new ones
  5. Remove third-party repositories and pinned packages — they are the single most common cause of a broken upgrade

Run it on the console, not over SSH

If you must use SSH, run it inside tmux or screen so a dropped connection does not kill a half-finished dpkg run. Debian's do-release-upgrade starts a second sshd on another port for exactly this reason; let it.

Answer the configuration file prompts deliberately. “Keep the local version” is usually right for files you have edited and wrong for files you have not — and the diff it offers is worth reading.

Afterwards

  • systemctl --failed — what did not come back
  • Check your services answer, from outside, not just that they are running
  • apt autoremove --purge and check /boot has room
  • Re-run your configuration management; it will tell you what drifted
  • Certificates, cron jobs and firewall rules — the usual three that come back subtly different

Do not skip versions

Debian 12 to 13, not 11 to 13. Skipping is unsupported and the failures are creative. Two sequential upgrades with a reboot and a check in between is slower and far more likely to end well.

Know the end-of-life dates

Running past end of life means no security updates, which quietly undoes everything in automatic security updates. Put the date in the calendar when you build the machine, not when the repositories stop answering.

Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base