ECC memory and memory errors
Memory fails quietly, and a machine with bad memory behaves like a machine with a hundred different bugs. ECC is what turns that into something you can see.
What ECC actually does
Error-correcting memory stores extra bits alongside your data. A single flipped bit is detected and corrected on the fly; a double-bit error is detected but cannot be corrected, and the machine will normally halt rather than carry on with data it knows is wrong.
Without ECC, both cases happen anyway — you just do not find out. You get a corrupted file, a database that fails its own consistency check, or a process that segfaults once a week in a different place each time. Every machine we build has ECC. It is not an option worth saving money on.
Reading the counters
On Linux, the EDAC subsystem exposes what the memory controller has seen:
edac-util -v
Or straight from the kernel:
grep . /sys/devices/system/edac/mc/mc*/csrow*/ce_count
Also check the system log for mce and EDAC lines:
journalctl -k | grep -iE 'edac|mce|memory error'
And the hardware's own record, which survives reboots and reinstalls:
ipmitool sel list
How to read what you find
- A handful of correctable errors over months. Cosmic rays are real. Note it, watch it, do nothing.
- Correctable errors climbing steadily, on the same DIMM. That module is failing. Replace it before it produces something uncorrectable.
- A sudden burst after a hardware change. Often a seating problem rather than a dead module — worth reseating before replacing.
- An uncorrectable error. Stop. That module comes out. If the machine halted, that is the system doing the right thing.
Which slot
EDAC reports a channel and slot; the BMC event log usually names the DIMM directly, which is more useful. Either way, give us whatever identifier you have when you open the ticket. It saves us pulling every module to find the right one.
Testing properly
If you suspect memory but the counters are clean, a long memtest run during a maintenance window is the honest test. Anything less than a full pass proves nothing — memory faults are often address-specific and only show up when the whole module has been walked. We can run this for you on a machine you take out of service; ask in a ticket.
What we do
Memory is our hardware, so a failing module is our problem. Open a ticket with what you have seen and we schedule a replacement — see hardware failure for how that runs and what notice to expect. The module is replaced from stock in Amsterdam, by our own engineers, not ordered in.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base