Replacing a failed disk
The dangerous moment in a disk replacement is not the failure. It is pulling the wrong drive out of a degraded array.
First, confirm which disk
For Linux software RAID:
cat /proc/mdstat — the failed member is marked
(F)
mdadm --detail /dev/md0
Then map that device to a physical drive, which is the part that matters:
lsblk -o NAME,SERIAL,MODEL,SIZE
The serial number is what identifies the drive in the chassis. A
device name is not enough — /dev/sdb can be a different disk after
a reboot.
Confirm its health while you are there:
smartctl -a /dev/sdb | head -30
See disk health for what the attributes mean.
Make the LED blink if you can
On hardware with a backplane that supports it:
ledctl locate=/dev/sdb
and afterwards ledctl locate_off=/dev/sdb. That turns an identification
problem into a visual one, and it is the single best thing you can do before asking
anyone to touch the machine.
Tell us properly
Open a ticket with:
- The machine identifier — our label, from the chassis or the portal
- The serial number of the failed drive
- Its size and model
- Whether the array is still running degraded or the machine is down
- Whether you have already marked it failed and removed it from the array
With the serial, an engineer walks to the rack and pulls the right drive. Without it, we are matching by position and that is how a second disk gets removed from an array that could only afford to lose one.
Prepare the array first
If the disk is still present but failing, take it out of the array deliberately rather than letting it be yanked:
mdadm --manage /dev/md0 --fail /dev/sdb1
mdadm --manage /dev/md0 --remove /dev/sdb1
After the swap
Copy the partition table from a surviving disk, then add the new member:
sfdisk -d /dev/sda | sfdisk /dev/sdb — GPT: use
sgdisk -R /dev/sdb /dev/sda && sgdisk -G /dev/sdb
mdadm --manage /dev/md0 --add /dev/sdb1
watch cat /proc/mdstat — watch it rebuild
Install the bootloader on the new disk. This is the step everyone forgets, and you discover it months later when the surviving disk fails and nothing boots:
grub-install /dev/sdb
The rebuild window is the risky part
A rebuild reads every block of the surviving disks. If one of them has a latent bad sector, you find out now. That is why RAID 5 on large modern drives is a poor bet and why RAID and disks recommends mirrors or raidz2.
Do not run heavy workloads during a rebuild if you can avoid it, and do not start a second replacement until the first has finished.
Our side
Replacement drives are held in Amsterdam and fitted by our own engineers. You choose the window; we do not reboot your machine to suit our schedule. See hardware failure and, if you colocate, remote hands.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base