Anime-Sanctuary

Diagnose a saturated disk

Slow server while the processor is idle: how do you diagnose a saturated disk with iostat and PSI, find the guilty program, then calm it down?

Course
Tutorial 33 of 35
Time
35 min to read
Tested with
Debian 13.7 (trixie), sysstat 12.7.5, iotop 0.6, util-linux 2.41.5
What you will be able to do
  • Recognize a saturated disk from the I/O wait that top shows.
  • Read the output of iostat -x and the /proc/pressure/io file.
  • Find the guilty program with pidstat, iotop and lsof.
  • Calm it down with ionice, then clean up the experiment.

Before you start: Read your logs with journalctl, Simple monitoring.

Contents
  1. The disk, one checkout for everyone
  2. 1. Set up a controlled overload
  3. 2. Spot the symptom: wait and load
  4. 3. iostat: the state of the disk
  5. 4. PSI: what the programs suffer
  6. 5. Find the culprit
  7. 6. Calm the culprit with ionice
  8. 7. When ionice changes nothing: the scheduler
  9. 8. Clean up the experiment
  10. If it fails
  11. Going further
  12. What's next

The disk, one checkout for everyone

A slow server with an idle processor often has one cause: the disk is saturated. A disk handles only a limited number of requests per second. Each read or write is an I/O operation (input/output), and the ones that arrive too fast wait in a queue, like at a supermarket checkout.

Your website, which wants to read a page, then finds itself behind a big file transfer. It did nothing wrong: it is waiting its turn. We will build this situation in miniature, measure it three ways, then undo it. All the numbers in this article are examples: yours will be different, but shaped the same way.

The queue of a quiet disk and of a saturated disk Two queues of eight slots leading to a disk. Quiet disk: the queue is empty, and a read goes through at once in 5 milliseconds. Saturated disk: seven writes wait in the queue, the read arrives behind them and waits 80 milliseconds. Quiet disk disk A read goes through at once: 5 ms. Saturated disk disk The read waits behindseven writes: 80 ms. Grey: writes. Blue: read.
Text version: on a quiet disk, a read goes through alone and takes 5 milliseconds. On a saturated disk, seven writes fill the queue and the same read waits behind them, 80 milliseconds. Durations are examples.

This picture explains why a saturated disk doesn't show up in the processor. The processor computes little, because it is waiting. To study that wait, we first need a disk that suffers from it.

1Set up a controlled overload

We will build the overload with dd, a command that copies data from one place to another. Here, it will write zeros into a 256 MB file, in a loop, without touching the rest of the disk.

Before the load, let's prepare a probe: a small 4 KB file whose read time we will measure. It plays a single customer at the checkout. Its time to get through tells you how long the queue is. First, create the file:

On the server
dd if=/dev/zero of=/var/tmp/sonde.bin bs=4k count=1 oflag=direct status=none
  • if=/dev/zero ("input file") is the source: a special file that supplies zeros on demand.
  • of=/var/tmp/sonde.bin ("output file") is the file to write.
  • bs=4k count=1: one 4 KB block, copied once.
  • oflag=direct writes without going through the cache, the memory where Linux keeps recent files. Without it, the disk wouldn't really work.
  • status=none hides the messages from dd.

This command prints nothing. Next, we read the file, and this second command prints the time it took:

On the server
dd if=/var/tmp/sonde.bin of=/dev/null bs=4k count=1 iflag=direct
Expected output
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000312 s, 13.1 MB/s

Only the time matters, the number before "s" (seconds). Here 0.000312 s, or 0.3 ms (milliseconds): a tiny time, which is the disk at rest. Now let's start the load, which repeats the same command in a loop, with 1 MB blocks copied 256 times. It has four extra pieces:

  • while true; do ...; done is a loop: "while true" never becomes false, so it repeats dd forever.
  • sh -c '...' starts a small command interpreter (sh) that runs the text between the quotes. This detour is needed because timeout can only watch one program, not a loop.
  • timeout 600 cuts everything off after 600 seconds, which is 10 minutes.
  • & runs the command in the background and gives you the prompt back right away.
On the server
timeout 600 sh -c 'while true; do dd if=/dev/zero of=/var/tmp/charge.bin bs=1M count=256 oflag=direct status=none; done' &

The terminal, the window where you type your commands, prints a line like [1] 4242, then gives you the prompt back. The 4242 is the PID (identification number) of timeout, the first program started. The one for dd will be different, and step 5 shows it to you. If, in the next step, reading the probe becomes fast again, the 10 minutes are over: run the timeout 600 command above again. The disk is working, so let's see what that changes from the outside.

2Spot the symptom: wait and load

You recognize a saturated disk by a processor that waits instead of computing, and top shows it on the %Cpu(s) line, under the name wa. This iowait is the share of time the processor has nothing to do, because it is waiting for an answer from the disk. Let's ask top for a single photo, with -b (plain output, no interactive screen) and -n1 (one reading only). The vertical bar | is a pipe, which passes the output of the left command to the one on the right. Here, head -3 keeps only the first three lines.

On the server
top -bn1 | head -3
Expected output
top - 14:02:11 up 3 days,  4:17,  1 user,  load average: 5.12, 3.40, 1.85
Tasks: 104 total,   2 running, 102 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.3 us,  1.9 sy,  0.0 ni, 13.6 id, 82.0 wa,  0.0 hi,  0.2 si,  0.0 st

Two signs matter here. The wa is very high, 82%, while us (the work done by programs) stays at 2.3%. And the load average is climbing. These are three numbers, for 1, 5 and 15 minutes, and they also count programs stuck waiting for the disk.

Now let's measure what a single customer lives through: read the probe again while the load is running.

On the server
dd if=/var/tmp/sonde.bin of=/dev/null bs=4k count=1 iflag=direct
Expected output
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0821 s, 49.9 kB/s

The same read, which took 0.000312 s (0.3 ms), now takes 0.0821 s (82 ms): over 200 times more. The gap varies from one disk to another. One thing is missing: wa proves the processor is waiting on a disk, without saying which one or how overloaded it is. That is the job of iostat.

3iostat: the state of the disk

The iostat -x command shows, for each disk, how many requests it receives, how long they wait and how busy it is. It comes from the sysstat package, which you know how to install since Updates and packages with apt. The word sudo, placed in front of a command, runs it with administrator rights.

On the server
sudo apt install sysstat

Run it with five settings, which make the output readable:

  • -d shows disks only, without the processor.
  • -x adds the detailed columns (the "x" of "extended").
  • -z hides disks with no activity.
  • -y skips the first reading, an average since boot that says nothing about the present moment.
  • 2 3 asks for a reading every 2 seconds, three times.
On the server
iostat -dxzy 2 3
Expected output
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda             22.50     90.00     0.00   0.00   78.40     4.00   74.00  75776.00     0.00   0.00   11.90  1024.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.64   99.80

You will see three similar blocks, and the name of your disk (often sda or nvme0n1); we show only one. Out of the twenty or so columns, six are enough:

  • r/s and w/s: reads and writes requested per second.
  • r_await and w_await: the average time, in milliseconds, between the request and its answer, queue wait included.
  • aqu-sz: the average number of requests waiting, the length of the queue.
  • %util: the share of time the disk had work to do.

Read await first, comparing it with the quiet state: it is what the waiting program feels. The table puts a quiet disk next to ours. The quiet disk's values are an example: no command in this article produces them.

ColumnQuiet diskSaturated disk
r/s422.5
w/s374
r_await1.3 ms78.4 ms
w_await2.0 ms11.9 ms
aqu-sz0.012.64
%util1.2%99.8%

iostat tells the disk's side of the story. The programs\' side is missing: how much time do they really lose waiting? The kernel has a second counter for that.

4PSI: what the programs suffer

PSI (Pressure Stall Information) is a counter in the kernel, the core of Linux. It tells you what share of the time your programs spend blocked, waiting for the disk. You read it from a plain file, with nothing to install.

On the server
cat /proc/pressure/io
Expected output
some avg10=61.20 avg60=48.35 avg300=22.10 total=1834567890
full avg10=55.80 avg60=43.02 avg300=19.77 total=1650234567

The file has two lines and four numbers per line:

  • some: the share of time when at least one program was waiting for the disk.
  • full: the share of time when all the active programs were waiting at once. The processor then produces nothing useful.
  • avg10, avg60, avg300: that share as a percentage, over the last 10, 60 and 300 seconds.
  • total: the waiting time added up since boot, in microseconds.

Here, some avg10=61.20 means that for 61% of the last 10 seconds, at least one program was waiting. A disk at rest shows values close to 0. Compare avg10 with avg300: if it is much higher, the pressure has just gone up.

iostat or PSI: which one to look at

The two counters complement each other. The table compares them.

CriterioniostatPSI
Point of viewThe diskThe programs
Tells you whetherthe disk is overloadedthe programs are waiting
DetailPer diskTotal
PackagesysstatNone

We know the disk is overloaded and that the programs feel it. The useful question remains: who is making it do all this work?

5Find the culprit

The pidstat -d command lists what each program reads and writes on the disk. It is in the same package as iostat, and sudo lets it see the programs of all users. Here, 2 1 means one reading of 2 seconds.

On the server
sudo pidstat -d 2 1
Expected output
14:03:20      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
14:03:22        0      4242      0.00  75776.00      0.00       0  dd

The PID is the number that identifies a running program. The kB_wr/s column gives the kilobytes written per second: the culprit is the top line, here dd with 75,776 kB/s. The other columns can wait.

The culprit is known: dd. Killing it isn't always a good idea: that program may be your backup. It is better to calm it down. The fold below, opened by clicking its title, shows how to see the file it is aiming at.

Watch live with iotop, and find the file with lsof

The iotop tool does the same thing as pidstat live, like top, but it needs administrator rights. Four options make it readable at a glance:

  • -b: plain output.
  • -o: active programs only.
  • -P: one program per line.
  • -n 2: two readings, the first one is at zero.
On the server
sudo apt install iotop
sudo iotop -boPn 2

You find dd at the top, with its full command line in the last column. Knowing who writes doesn't yet tell you what: lsof ("list open files") answers that. With -p and the PID, it lists the files that program has open. pgrep -nx dd finds the PID of the newest dd, and the $( ) replace the text with the result of the command inside them. If pgrep answers 4242, the command becomes sudo lsof -p 4242. grep charge keeps only the lines about our file.

On the server
sudo lsof -p "$(pgrep -nx dd)" | grep charge
Expected output
dd      4242 root    1w      REG               0,45 134217728   262151 /var/tmp/charge.bin

The 1w is the program's output, open for writing, and the last column gives the file: /var/tmp/charge.bin. Your numbers will be your own.

6Calm the culprit with ionice

The ionice command gives a program a low I/O priority, so it lets the others go ahead at the checkout. A class is a level of politeness. The idle class (-c 3) is the most polite: the program gets the disk only when nobody else wants it.

First stop the running load, then restart it with ionice. pkill stops programs by name, and -f searches the whole text of the command: the one for our loop contains charge.bin. Without this stop, two loads would run at the same time.

On the server
pkill -f charge.bin
ionice -c 3 timeout 600 sh -c 'while true; do dd if=/dev/zero of=/var/tmp/charge.bin bs=1M count=256 oflag=direct status=none; done' &

Then read the probe again with the command from step 2: dd if=/var/tmp/sonde.bin of=/dev/null bs=4k count=1 iflag=direct. Its time should drop, because the load makes room for the others. If it doesn't drop, the likely cause is the disk scheduler, covered right after.

The other classes, nice, and a program that is already running

Open this fold to change the priority of a program that is already running. Two other classes exist:

  • Best-effort (-c 2): the default class, with a level from 0 (highest priority) to 7 (lowest).
  • Realtime (-c 1): always goes first. Avoid it: a single program set this way can starve the whole machine of disk.

The nice command sets the processor access priority, from -20 (highest priority) to 19 (the most polite). According to the documentation, it also shifts the I/O priority: nice -n 19 gives level 7, the lowest of best-effort. For a program that is already running, there is no need to restart it: sudo ionice -c 3 -p 4242 changes its priority in place, with -p followed by its PID (here 4242).

7When ionice changes nothing: the scheduler

The I/O scheduler is the kernel software that decides in which order the requests in the queue go to the disk. It decides whether ionice is heard: some schedulers ignore it, others respect it. According to the kernel documentation, bfq and mq-deadline respect ionice, and none ignores it. We haven't measured this effect on your machine: check it with the probe. Open the fold below, by clicking its title, to read and change your scheduler.

Read and change the scheduler

First find your disk's name and type with lsblk (list of disks), then read its setting.

On the server
lsblk -d -o NAME,ROTA
cat /sys/block/sda/queue/scheduler
Expected output
NAME ROTA
sda     1
[mq-deadline] none bfq

The ROTA column is 1 for a spinning hard disk, 0 for an SSD. Replace sda with your disk's name. The name in brackets is the active scheduler. The three common names:

  • none: no sorting, requests go through in the order they arrive. It ignores ionice; it is the usual choice for fast SSDs.
  • mq-deadline: gives each request a deadline and tends to serve reads before writes. It respects ionice.
  • bfq: shares the disk fairly between programs, so the machine stays responsive. It also respects ionice.

To change the setting, you write the name into the same file. A redirection is the > sign, which sends text into a file. It is run by your terminal, with your rights, and not by sudo. That is why sudo echo bfq > file is refused. The tee command fixes the problem: it writes into the file itself whatever the pipe hands it, and sudo applies to it.

On the server
echo bfq | sudo tee /sys/block/sda/queue/scheduler

Read the file again with the cat command above: bfq should be in brackets. The change applies right away and disappears at reboot. To go back before that, repeat the tee command with the old name, here mq-deadline.

The diagnosis is done: saturated disk, culprit found, priority lowered. All that is left is cleaning up the experiment.

8Clean up the experiment

Let's stop the load, then erase its two test files with rm (remove), whose -f option avoids any message if a file is missing.

On the server
pkill -f charge.bin
rm -f /var/tmp/charge.bin /var/tmp/sonde.bin

Let's check that nothing is writing anymore. The first command looks for a dd that is still active, the second lists the remaining test files.

On the server
pgrep -x dd || echo "plus de dd"
ls /var/tmp/*.bin 2>/dev/null || echo "plus de fichier de test"
Expected output
plus de dd
plus de fichier de test

If you changed the scheduler in step 7, put the old name back with tee, otherwise it stays until the next reboot. The installed packages stay too, but nothing runs in the background: no service, no scheduled task.

If it fails

  • iostat: command not found (or pidstat): the sysstat package is missing. Install it as in step 3 (sudo apt install sysstat).
  • dd: failed to open '/dev/shm/charge.bin': Invalid argument: the /dev/shm folder lives in memory, and the direct option doesn't work there. Write to the real disk, as with /var/tmp.
  • Netlink error: Operation not permitted (1): iotop was started without administrator rights (Netlink is the channel it uses to question the kernel). Put sudo in front.
  • The probe stays fast: the load stopped after 10 minutes, or your disk is very fast. Run the timeout 600 command from step 1 again.
  • No /proc/pressure/io file: your kernel doesn't have PSI. iostat and pidstat are then enough for the diagnosis.

Going further

Simulate a site that reads at random

A website rarely reads a big file in one go: it picks small pieces here and there. The fio tool imitates this for 15 seconds, with 16 reads of 4 KB in parallel, on a 256 MB file. Its options read in groups:

  • --name, --filename, --size: the name of the test, the file used and its size.
  • --rw=randread, --bs=4k: random reads, in 4 KB blocks.
  • --direct=1: without going through the cache, like direct for dd.
  • --runtime=15, --time_based: a duration of 15 seconds, whatever happens.
  • --iodepth=16, --ioengine=libaio: 16 requests at the same time, using Linux's libaio method.
On the server
sudo apt install fio
fio --name=aleatoire --filename=/var/tmp/fio.bin --size=256M --rw=randread --bs=4k --direct=1 --runtime=15 --time_based --iodepth=16 --ioengine=libaio
rm /var/tmp/fio.bin

Open a second terminal window meanwhile and run iostat -dxzy 2 3 there. You will see r/s go up, with reads much smaller than those of dd.

Pressure on the processor and memory

PSI doesn't stop at the disk. The files /proc/pressure/cpu and /proc/pressure/memory have the same format: they tell you whether programs are waiting for a processor core or for memory. The kernel documentation also explains how to get a signal when the pressure passes a threshold you set.

What's next

You can now recognize a saturated disk, find who is saturating it and calm it down by hand. That hand remains the weak link, though: a service that restarts comes back without your settings. The next article, Harden a service with systemd, shows how to fix a service's rules in its own file, once and for all.