Rescue mode: fixing a server that will not boot
A bad fstab entry, a half-finished kernel upgrade, a forgotten root password. None of these need a reinstall, and none of them need us.
What rescue mode is
Your server boots a small Linux system over the network instead of from its own disks. Your disks are untouched and unmounted, and you get root on the rescue system over SSH with a temporary password shown in the portal. From there you can mount, inspect and repair what is on the disks.
Start it from the portal, or by asking support. It is not destructive and you can go in and out of it as often as you like. When you are done, reboot back to normal and the machine boots from its own disks again.
Finding your disks
Once in, look at what is there:
lsblk -f
That shows every disk, every partition, the filesystem on it and any label. If you use software RAID, assemble it first:
mdadm --assemble --scan && cat /proc/mdstat
For LVM:
vgscan && vgchange -ay && lvs
Mounting and chrooting in
Mount the root filesystem, then the things a working system expects underneath it:
mount /dev/md2 /mnt
mount /dev/md1 /mnt/boot — if /boot is separate
for d in dev proc sys run; do mount --bind /$d /mnt/$d; done
chroot /mnt /bin/bash
You are now inside your own installation, as root, with your own packages and configuration. Almost everything you would normally do works from here.
The three repairs people actually need
A bad /etc/fstab
The classic: a line added for a disk that is not there, and the machine drops to an
emergency prompt on boot. Edit /etc/fstab, comment the offending line,
reboot. Use UUID= rather than /dev/sdb1 in future —
device names move between boots, UUIDs do not.
A broken bootloader
After a failed kernel upgrade or a disk replacement, reinstall GRUB from inside the chroot, onto every disk that can boot:
grub-install /dev/sda && grub-install /dev/sdb
update-grub
On a RAID 1 pair, installing to only one disk is how you discover months later that the machine cannot boot from the surviving half.
A lost root password
From inside the chroot, passwd root. Set it, exit, reboot. This is
also the moment to notice that you had password login enabled at all — see
securing a new server.
Rescuing data instead of the system
If the installation is beyond saving, rescue mode is still how you get your data out before reinstalling. Mount the disk, copy what matters to somewhere else over SSH, and only then wipe. A reinstall destroys everything on the target disks and we cannot undo it.
If the disk itself is failing rather than the system, stop and read disk health first. Repeated reads from a dying drive can turn a recoverable situation into an unrecoverable one.
When to call us instead
If the machine does not come up in rescue mode either, the problem is below the operating system and it is ours. Open a ticket with the machine identifier and what you have tried; we will put eyes on the console and, if it is hardware, hands on the machine.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base