Anime-Sanctuary

Software RAID and SMART checks

Software RAID with mdadm and SMART monitoring: two tools so a failing disk never catches you by surprise. You learn to keep the server running through a failure, then to see it coming.

Course
Tutorial 32 of 35
Time
35 min to read
Tested with
Debian 13.7 (trixie), mdadm 4.4, smartmontools 7.4
What you will be able to do
  • Say what RAID protects, and what it doesn't.
  • Build a RAID 1 with mdadm, simulate a failure and rebuild, with no risk to your real disks.
  • Get an email when a disk in the RAID fails.
  • Read a disk's health with smartctl, schedule tests with smartd and know when to replace it.

Before you start: Read your logs with journalctl, Your first backup with rsync.

Contents
  1. What RAID protects, and what it doesn't
  2. RAID 1, RAID 5, RAID 6: which one to pick
  3. 1. Prepare fake disks
  4. 2. Create the RAID 1 and read it
  5. 3. Simulate a failure
  6. 4. Replace the disk and rebuild
  7. 5. Get an email when a disk fails
  8. 6. Clean up the experiment
  9. 7. Read a disk's health with smartctl
  10. 8. Run a short or long test
  11. 9. Let smartd keep watch
  12. When to replace a disk
  13. If it fails
  14. Going further
  15. What's next
  16. Sources

What RAID protects, and what it doesn't

RAID protects against one thing: a disk failure. A hard disk is a part that spins and wears out, so one day it will break. RAID groups several disks so that the failure of one doesn't stop the server.

An opened hard disk: the spinning silver platter and the arm that carries the read head, above the black circuit board.

It is software RAID when Linux does the grouping, with a tool called mdadm, instead of a special card. That is the case in this tutorial. Here is what it covers, and above all what it doesn't.

  • Covered: a dying disk. The server keeps answering, and you swap the part whenever you like.
  • Not covered: your own mistakes. A deleted file is deleted on every disk at once, immediately.
  • Not covered: ransomware. This program encrypts your files to demand money. It encrypts them on every disk of the RAID.
  • Not covered: fire, theft, power failure. All the disks are in the same machine.

So RAID is a parachute against one single risk, to add to your backups and not to replace them. There are several ways to group disks, called levels. Let's look at the three main ones.

RAID 1, RAID 5, RAID 6: which one to pick

Each form of RAID is called a level, and the level sets the price of safety in disks. RAID 1 is a mirror: two disks hold exactly the same data. RAID 5 and RAID 6 spread the data across several disks, with extra parity. Parity is a piece of control information that lets the RAID recompute the content of a lost disk.

The three levels, with 4 TB disks
LevelMinimum disksUsable spaceFailures survived
RAID 12One disk's worth: 2 disks give 4 TB1
RAID 53All disks but one: 3 disks give 8 TB1
RAID 64All disks but two: 4 disks give 8 TB2

The rest of the article follows RAID 1, the easiest to read. To learn it without touching your real disks, we will build fake ones.

1Prepare fake disks

A file can play the part of a disk: Linux can present it as a device, called a loop device. We create 200 MB files, which are pretend disks. Only these files are affected from here on.

First install the tool, then build two fake disks. The truncate command creates a file of the requested size, and losetup plugs it in as a disk.

On the server
sudo apt install -y mdadm
mkdir -p ~/essai-raid
cd ~/essai-raid
truncate -s 200M disque1.img disque2.img
D1=$(sudo losetup --find --show disque1.img)
D2=$(sudo losetup --find --show disque2.img)
echo "$D1 $D2"

The last line prints two names like /dev/loop0, and these names may differ on your machine. The variables D1 and D2, small named memories of the terminal, keep these two names. Keep this terminal open until the end of the experiment. The disks exist, now to join them into a RAID.

2Create the RAID 1 and read it

The mdadm --create command joins the two fake disks into a single disk, /dev/md9, where every piece of data is written twice. Three settings describe it.

  • --level=1 picks the mirror.
  • --raid-devices=2 announces two disks.
  • --run skips the confirmation question about starting.
On the server
sudo mdadm --create /dev/md9 --level=1 --raid-devices=2 --run "$D1" "$D2"
sudo mdadm --wait /dev/md9
cat /proc/mdstat
sudo mdadm --detail /dev/md9

The creation prints a note about booting the machine (/boot): it doesn't concern us here. Right after, the RAID copies one disk onto the other for the first time, and --wait waits for that copy to finish. The /proc/mdstat file is the RAID dashboard: the kernel writes the state of each set there. Here is what you will read for a healthy RAID 1.

Reading /proc/mdstat
Personalities : [raid1]
md9 : active raid1 loop1[1] loop0[0]
      203776 blocks super 1.2 [2/2] [UU]

unused devices: <none>

Remember the end of the line of numbers: [2/2] says two disks are expected and two are there, and each U is a disk that is up. The --detail command gives the same thing in sentences. Look for State : clean (the RAID is in good shape) and Failed Devices : 0 (no failed disk). Now let's put this RAID to the test.

3Simulate a failure

To see the RAID hold up, we declare a disk failed and check that the data stays readable. First let's store a file on the RAID. That needs a filesystem, the way a disk organizes its files.

On the server
sudo mkfs.ext4 -q /dev/md9
sudo mkdir -p /mnt/essai
sudo mount /dev/md9 /mnt/essai
echo "contenu precieux" | sudo tee /mnt/essai/important.txt

Here is what comes next as a diagram: one disk drops, the other answers alone, a third replaces it.

One disk fails, another replaces it Three situations. First, disks A and B hold the same data. Then A fails, crossed out, and B answers alone. Finally A is removed, a new disk C replaces it and a bar shows the copy from B to C. A B 1. A and B hold the same data A B 2. A fails, B answers alone A B C 3. C replaces A, B is copied to C
Text version: at first, two disks A and B hold the same data. When A fails, B answers alone and the server keeps going. A is removed, a new disk C is plugged in, and the RAID copies the data from B to C.

On to the failure itself. The --fail option declares the disk failed, and --remove takes it out of the RAID, as you would pull it from the machine. Then read the file: it should still answer.

On the server
sudo mdadm /dev/md9 --fail "$D1"
cat /proc/mdstat
cat /mnt/essai/important.txt
sudo mdadm /dev/md9 --remove "$D1"

In /proc/mdstat, the failed disk carries the mark (F), and the line of numbers becomes [2/1] [_U]. The underscore marks the missing disk, and the RAID is then called degraded: it works, but with no safety net. Your file still answers: cat should print contenu precieux. Now to give the RAID its safety net back.

4Replace the disk and rebuild

Replacing a disk means plugging in a new one and asking the RAID to fill it. Let's build a third fake disk, then add it with --add. Copying the data from B to C is called the rebuild.

On the server
truncate -s 200M disque3.img
D3=$(sudo losetup --find --show disque3.img)
sudo mdadm /dev/md9 --add "$D3"
cat /proc/mdstat
sudo mdadm --wait /dev/md9
cat /proc/mdstat
sudo mdadm --detail /dev/md9

During the copy, the first /proc/mdstat contains a recovery line with a percentage. On our 200 MB fake disks, it is very short. On a real disk of several terabytes, it takes hours, and the server stays usable. The --wait command waits for the end, then you get [2/2] [UU] and State : clean back. The RAID is repaired. Next time, you want to be told without looking yourself.

5Get an email when a disk fails

mdadm watches the RAIDs in the background and writes to the address on the MAILADDR line of its settings file. On Debian, this line points to root at first, the server's administrator account. Let's replace it with alice, the server's own mailbox, as in Simple monitoring. The watching service, mdmonitor, only rereads this file at startup, so we restart it.

On the server
sudo sed -i 's/^MAILADDR .*/MAILADDR alice/' /etc/mdadm/mdadm.conf
grep MAILADDR /etc/mdadm/mdadm.conf
sudo systemctl restart mdmonitor
systemctl is-active mdmonitor
Expected output
MAILADDR alice
active

To check that the message goes out, mdadm can send a test email for each RAID it finds. The message lands in /var/mail/alice, and the grep command prints its subject.

On the server
sudo mdadm --monitor --scan --oneshot --test
grep ^Subject /var/mail/alice
Expected output
Subject: TestMessage event on /dev/md9:mon-serveur

If /var/mail/alice doesn't exist, the mail command is missing: install it as in Simple monitoring. In a real failure, a message whose subject starts with DegradedArray event arrives within a minute. The experiment is over: let's clean up so we don't leave fake disks behind.

6Clean up the experiment

Clean up in the reverse order of creation: unmount, stop the RAID, unplug the fake disks, delete the files.

On the server
sudo umount /mnt/essai
sudo mdadm --stop /dev/md9
sudo losetup -d "$D1" "$D2" "$D3"
cd ~
rm -r ~/essai-raid
sudo rmdir /mnt/essai

Your machine is back to its earlier state. On real disks, the steps are the same, with different names.

On real disks

First find the disks with lsblk, then check their names twice. The create command erases both disks it names. Here they are /dev/sdb and /dev/sdc, two empty disks you no longer need.

On the server
lsblk -o NAME,SIZE,MODEL,SERIAL
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
sudo mkfs.ext4 /dev/md0
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
sudo update-initramfs -u

The creation asks two questions. The first offers an option to speed up rebuilds: answer n. The second, Continue creating array, asks for confirmation: answer y. The last two lines record the RAID in the configuration, so that it reassembles at reboot. Then mount /dev/md0 wherever you want to keep your data.

RAID can repair a failure that arrives suddenly. But a disk often warns you before it breaks, if you know how to listen. That is SMART's job.

7Read a disk's health with smartctl

SMART is a health record that every disk keeps about itself, and the smartctl command reads it. The smartmontools package provides it. Find your disk's name with lsblk, then read its record. Here we take /dev/sda.

On the server
sudo apt install -y smartmontools
lsblk -d -o NAME,SIZE,MODEL
sudo smartctl -a /dev/sda

The output is long, but two places matter: the verdict at the top and the attribute table below it. Each line of the table is a counter kept by the disk. Here is an excerpt for a healthy disk, with sample values.

Sample output of smartctl -a
SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   087   087   000    Old_age   Always       -       11874
194 Temperature_Celsius     0x0022   064   052   000    Old_age   Always       -       36
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

Look only at the last column, RAW_VALUE: it is the real counter, with no manufacturer calculation. Four attributes deserve your attention.

The attributes that matter
AttributeWhat it countsIf the counter goes up
Reallocated_Sector_CtDamaged areas replaced by spare areasThe disk is wearing out: prepare to replace it
Current_Pending_SectorDoubtful areas it can no longer readKeep watching: if it stays above zero, replace
Offline_UncorrectableUnreadable areas found during a testSame answer as the previous one
UDMA_CRC_Error_CountTransmission errors between the disk and the motherboardIt isn't the disk: change the cable first

A large online backup service tracked five counters across nearly 70,000 disks. The first three in the table are among them. For that service, a counter above zero deserves an investigation. Above zero, 76.7% of the dead disks showed it, against 4.2% of the disks still healthy. The other 23.3% died without warning: a health record helps, but it guarantees nothing. That is one of the reasons to keep backups.

What about an NVMe SSD?

An NVMe SSD doesn't have the same attributes. The command is the same, with the NVMe disk's name, for example sudo smartctl -a /dev/nvme0. Look for the wear percentage and the data integrity error counter, which are the equivalent of the counters above.

Reading the record is a snapshot. To make it complete, the disk must also test itself.

8Run a short or long test

A disk can test itself without leaving service, and there are two tests. The short test takes a few minutes and checks the essentials. The long test reads the whole surface of the disk and takes several hours on a big disk. Start the short one: the command announces the duration, for example Please wait 2 minutes for test to complete. Wait that long, then read the test log and the verdict.

On the server
sudo smartctl -t short /dev/sda
sleep 150
sudo smartctl -l selftest /dev/sda
sudo smartctl -H /dev/sda

Here is the shape of the log, with sample values. The test line should read Completed without error.

Sample output of smartctl -l selftest
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     11874         -

A Completed: read failure line instead means the disk couldn't reread an area: take it seriously. For the long test, replace short with long and let it work. Running these tests by hand every week is quickly forgotten. A service can do it for you.

9Let smartd keep watch

The smartd service rereads the counters in the background, runs the tests at the scheduled times and writes to you when a counter gets worse. On Debian, it starts with the package and reads the /etc/smartd.conf file. Its DEVICESCAN line watches every disk it finds. We keep that line and add three settings to it.

  • -a: watch the disk's health, counters and error logs.
  • -s (S/../.././02|L/../../6/03): a short test every day at 2 AM, a long test every Saturday at 3 AM (the 6 means Saturday).
  • -m alice: the mailbox that receives the alerts, instead of root.
On the server
sudo sed -i 's#^DEVICESCAN.*#DEVICESCAN -a -d removable -n standby -s (S/../.././02|L/../../6/03) -m alice -M exec /usr/share/smartmontools/smartd-runner#' /etc/smartd.conf
grep '^DEVICESCAN' /etc/smartd.conf
sudo systemctl restart smartmontools
systemctl status smartmontools --no-pager

The line printed should be the one you just wrote, and the service should be active (running) on a physical machine. To check that a message goes out, temporarily add -M test at the end of the line: smartd then sends a test email at every start. Remove it afterwards. The alerts arrive, and what is left is knowing what to do with them.

When to replace a disk

Replace a disk as soon as a serious sign appears, without waiting for the complete failure. Here are the signs that matter.

  • A counter going up: Reallocated_Sector_Ct, Current_Pending_Sector or Offline_Uncorrectable increases from one day to the next.
  • A failed long test: Completed: read failure in the test log.
  • A negative verdict: smartctl -H prints FAILED.
  • A RAID that kicks out the same disk: mdadm marks it as failed a second time.

One exception: if only UDMA_CRC_Error_Count goes up, change the cable before the disk. My advice: a disk that has started to reallocate doesn't heal, and it costs less than a failed rebuild.

If it fails

  • Smartctl open device: /dev/sdz failed: No such device: the disk name is wrong, or the disk is virtual. Reread the name with lsblk -d.
  • losetup: disque1.img: failed to set up loop device: No such file or directory: the file doesn't exist where you are. Move into ~/essai-raid with cd ~/essai-raid, or redo step 1.
  • mdadm: cannot open /dev/loop9: No such file or directory: the fake disk name doesn't exist. Redo step 1 and check with echo "$D1 $D2".

Going further

Regular RAID checks and a spare disk

Debian provides mdcheck timers, tasks that systemd runs at set times, which reread the whole RAID to find hidden errors. See what is active on your machine with systemctl list-timers 'mdcheck*'. You can also add one more disk to a healthy RAID with --add: it stays in reserve and takes the place of the first disk that fails.

Other levels and other tools

RAID 5 and RAID 6 are created like RAID 1, with --level=5 or --level=6 and the right number of disks. Other filesystems, like Btrfs or ZFS, handle the grouping themselves. That is a different choice, with its own rules.

What's next

Your disk is now doubled and watched, and you know when to change it. What remains is the case where everything is healthy but the server crawls, because the disk is too busy. The next article, Diagnose a saturated disk, teaches you to measure that saturation. It comes now because you can only diagnose a disk well once you know it is healthy.