When the machine panics

First: was it a panic at all?

Three different events look identical from outside:

• A kernel panic — software, and usually leaves a trace in the console.
• A hardware fault — a machine check exception, bad memory, a dying power supply. Check the BMC event log, which survives reboots: ipmitool sel list. See IPMI and the BMC.
• The OOM killer — not a crash at all, but it feels like one. dmesg -T | grep -i 'out of memory', and see swap and the OOM killer.

Start with the previous boot's journal:

journalctl -k -b -1 -e

If that ends abruptly with nothing unusual, the machine died faster than it could write — which is itself a clue pointing at hardware or a hard lockup.

Set up kdump before you need it

kdump reserves a slice of memory holding a second kernel. When the first one panics, the second boots and writes the memory image to disk.

apt install kdump-tools or dnf install kexec-tools, then add crashkernel=512M to the kernel command line and reboot.

Check it is armed: kdump-config status or systemctl status kdump. Dumps land in /var/crash, so make sure there is room — a dump can be large, and a full disk means no dump.

Test it deliberately during a maintenance window:

echo c > /proc/sysrq-trigger

That panics the machine on purpose. It is the only way to know the configuration works, exactly like testing a restore.

Reading a dump

Install crash and the matching debug symbols, then:

crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/.../dump

bt for the backtrace, log for the kernel ring buffer at the moment of death, ps for what was running.

You are usually looking for one line: which module was on the stack. A third-party or out-of-tree driver near the top is the answer more often than not.

Watch the console instead

If kdump is not set up, remote KVM is your other window. Leave the console open on a machine that panics repeatedly and you will see the trace that never reached the disk. Photograph or copy it — the first few lines are the useful part.

The usual causes

• Bad memory — check ECC counters first, always
• A storage controller or NIC driver, particularly after a kernel upgrade
• Overheating — see temperature and cooling
• Out-of-tree modules: ZFS, proprietary drivers, anything DKMS rebuilds
• Genuine kernel bugs, which do exist and are the least likely explanation

Tell us

If the trace points below your operating system — machine check exceptions, memory errors, a controller resetting — open a ticket with the machine identifier and the output. That is our hardware, and it is replaced from stock on site. See hardware failure.

---

Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base