Diagnose a saturated disk
Slow server while the processor is idle: how do you diagnose a saturated disk with iostat and PSI, find the guilty program, then calm it down?
- Course
- Tutorial 33 of 35
- Time
- 35 min to read
- Tested with
- Debian 13.7 (trixie), sysstat 12.7.5, iotop 0.6, util-linux 2.41.5
What you will be able to do
- Recognize a saturated disk from the I/O wait that top shows.
- Read the output of iostat -x and the /proc/pressure/io file.
- Find the guilty program with pidstat, iotop and lsof.
- Calm it down with ionice, then clean up the experiment.
Before you start: Read your logs with journalctl, Simple monitoring.
Contents
- The disk, one checkout for everyone
- 1. Set up a controlled overload
- 2. Spot the symptom: wait and load
- 3. iostat: the state of the disk
- 4. PSI: what the programs suffer
- 5. Find the culprit
- 6. Calm the culprit with ionice
- 7. When ionice changes nothing: the scheduler
- 8. Clean up the experiment
- If it fails
- Going further
- What's next
The disk, one checkout for everyone
A slow server with an idle processor often has one cause: the disk is saturated. A disk handles only a limited number of requests per second. Each read or write is an I/O operation (input/output), and the ones that arrive too fast wait in a queue, like at a supermarket checkout.
Your website, which wants to read a page, then finds itself behind a big file transfer. It did nothing wrong: it is waiting its turn. We will build this situation in miniature, measure it three ways, then undo it. All the numbers in this article are examples: yours will be different, but shaped the same way.
This picture explains why a saturated disk doesn't show up in the processor. The processor computes little, because it is waiting. To study that wait, we first need a disk that suffers from it.
1Set up a controlled overload
We will build the overload with dd, a command that copies data from one place to another. Here, it will write zeros into a 256 MB file, in a loop, without touching the rest of the disk.
Before the load, let's prepare a probe: a small 4 KB file whose read time we will measure. It plays a single customer at the checkout. Its time to get through tells you how long the queue is. First, create the file:
dd if=/dev/zero of=/var/tmp/sonde.bin bs=4k count=1 oflag=direct status=noneif=/dev/zero("input file") is the source: a special file that supplies zeros on demand.of=/var/tmp/sonde.bin("output file") is the file to write.bs=4k count=1: one 4 KB block, copied once.oflag=directwrites without going through the cache, the memory where Linux keeps recent files. Without it, the disk wouldn't really work.status=nonehides the messages fromdd.
This command prints nothing. Next, we read the file, and this second command prints the time it took:
dd if=/var/tmp/sonde.bin of=/dev/null bs=4k count=1 iflag=direct1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000312 s, 13.1 MB/sOnly the time matters, the number before "s" (seconds). Here 0.000312 s, or 0.3 ms (milliseconds): a tiny time, which is the disk at rest. Now let's start the load, which repeats the same command in a loop, with 1 MB blocks copied 256 times. It has four extra pieces:
while true; do ...; doneis a loop: "while true" never becomes false, so it repeatsddforever.sh -c '...'starts a small command interpreter (sh) that runs the text between the quotes. This detour is needed becausetimeoutcan only watch one program, not a loop.timeout 600cuts everything off after 600 seconds, which is 10 minutes.&runs the command in the background and gives you the prompt back right away.
timeout 600 sh -c 'while true; do dd if=/dev/zero of=/var/tmp/charge.bin bs=1M count=256 oflag=direct status=none; done' &The terminal, the window where you type your commands, prints a line like [1] 4242, then gives you the prompt back. The 4242 is the PID (identification number) of timeout, the first program started. The one for dd will be different, and step 5 shows it to you. If, in the next step, reading the probe becomes fast again, the 10 minutes are over: run the timeout 600 command above again. The disk is working, so let's see what that changes from the outside.
2Spot the symptom: wait and load
You recognize a saturated disk by a processor that waits instead of computing, and top shows it on the %Cpu(s) line, under the name wa. This iowait is the share of time the processor has nothing to do, because it is waiting for an answer from the disk. Let's ask top for a single photo, with -b (plain output, no interactive screen) and -n1 (one reading only). The vertical bar | is a pipe, which passes the output of the left command to the one on the right. Here, head -3 keeps only the first three lines.
top -bn1 | head -3top - 14:02:11 up 3 days, 4:17, 1 user, load average: 5.12, 3.40, 1.85
Tasks: 104 total, 2 running, 102 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.3 us, 1.9 sy, 0.0 ni, 13.6 id, 82.0 wa, 0.0 hi, 0.2 si, 0.0 stTwo signs matter here. The wa is very high, 82%, while us (the work done by programs) stays at 2.3%. And the load average is climbing. These are three numbers, for 1, 5 and 15 minutes, and they also count programs stuck waiting for the disk.
Now let's measure what a single customer lives through: read the probe again while the load is running.
dd if=/var/tmp/sonde.bin of=/dev/null bs=4k count=1 iflag=direct1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0821 s, 49.9 kB/sThe same read, which took 0.000312 s (0.3 ms), now takes 0.0821 s (82 ms): over 200 times more. The gap varies from one disk to another. One thing is missing: wa proves the processor is waiting on a disk, without saying which one or how overloaded it is. That is the job of iostat.
3iostat: the state of the disk
The iostat -x command shows, for each disk, how many requests it receives, how long they wait and how busy it is. It comes from the sysstat package, which you know how to install since Updates and packages with apt. The word sudo, placed in front of a command, runs it with administrator rights.
sudo apt install sysstatRun it with five settings, which make the output readable:
-dshows disks only, without the processor.-xadds the detailed columns (the "x" of "extended").-zhides disks with no activity.-yskips the first reading, an average since boot that says nothing about the present moment.2 3asks for a reading every 2 seconds, three times.
iostat -dxzy 2 3Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 22.50 90.00 0.00 0.00 78.40 4.00 74.00 75776.00 0.00 0.00 11.90 1024.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.64 99.80You will see three similar blocks, and the name of your disk (often sda or nvme0n1); we show only one. Out of the twenty or so columns, six are enough:
r/sandw/s: reads and writes requested per second.r_awaitandw_await: the average time, in milliseconds, between the request and its answer, queue wait included.aqu-sz: the average number of requests waiting, the length of the queue.%util: the share of time the disk had work to do.
Read await first, comparing it with the quiet state: it is what the waiting program feels. The table puts a quiet disk next to ours. The quiet disk's values are an example: no command in this article produces them.
| Column | Quiet disk | Saturated disk |
|---|---|---|
r/s | 4 | 22.5 |
w/s | 3 | 74 |
r_await | 1.3 ms | 78.4 ms |
w_await | 2.0 ms | 11.9 ms |
aqu-sz | 0.01 | 2.64 |
%util | 1.2% | 99.8% |
iostat tells the disk's side of the story. The programs\' side is missing: how much time do they really lose waiting? The kernel has a second counter for that.
4PSI: what the programs suffer
PSI (Pressure Stall Information) is a counter in the kernel, the core of Linux. It tells you what share of the time your programs spend blocked, waiting for the disk. You read it from a plain file, with nothing to install.
cat /proc/pressure/iosome avg10=61.20 avg60=48.35 avg300=22.10 total=1834567890
full avg10=55.80 avg60=43.02 avg300=19.77 total=1650234567The file has two lines and four numbers per line:
some: the share of time when at least one program was waiting for the disk.full: the share of time when all the active programs were waiting at once. The processor then produces nothing useful.avg10,avg60,avg300: that share as a percentage, over the last 10, 60 and 300 seconds.total: the waiting time added up since boot, in microseconds.
Here, some avg10=61.20 means that for 61% of the last 10 seconds, at least one program was waiting. A disk at rest shows values close to 0. Compare avg10 with avg300: if it is much higher, the pressure has just gone up.
iostat or PSI: which one to look at
The two counters complement each other. The table compares them.
| Criterion | iostat | PSI |
|---|---|---|
| Point of view | The disk | The programs |
| Tells you whether | the disk is overloaded | the programs are waiting |
| Detail | Per disk | Total |
| Package | sysstat | None |
We know the disk is overloaded and that the programs feel it. The useful question remains: who is making it do all this work?
5Find the culprit
The pidstat -d command lists what each program reads and writes on the disk. It is in the same package as iostat, and sudo lets it see the programs of all users. Here, 2 1 means one reading of 2 seconds.
sudo pidstat -d 2 114:03:20 UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
14:03:22 0 4242 0.00 75776.00 0.00 0 ddThe PID is the number that identifies a running program. The kB_wr/s column gives the kilobytes written per second: the culprit is the top line, here dd with 75,776 kB/s. The other columns can wait.
The culprit is known: dd. Killing it isn't always a good idea: that program may be your backup. It is better to calm it down. The fold below, opened by clicking its title, shows how to see the file it is aiming at.
Watch live with iotop, and find the file with lsof
The iotop tool does the same thing as pidstat live, like top, but it needs administrator rights. Four options make it readable at a glance:
-b: plain output.-o: active programs only.-P: one program per line.-n 2: two readings, the first one is at zero.
sudo apt install iotop
sudo iotop -boPn 2You find dd at the top, with its full command line in the last column. Knowing who writes doesn't yet tell you what: lsof ("list open files") answers that. With -p and the PID, it lists the files that program has open. pgrep -nx dd finds the PID of the newest dd, and the $( ) replace the text with the result of the command inside them. If pgrep answers 4242, the command becomes sudo lsof -p 4242. grep charge keeps only the lines about our file.
sudo lsof -p "$(pgrep -nx dd)" | grep chargedd 4242 root 1w REG 0,45 134217728 262151 /var/tmp/charge.binThe 1w is the program's output, open for writing, and the last column gives the file: /var/tmp/charge.bin. Your numbers will be your own.
6Calm the culprit with ionice
The ionice command gives a program a low I/O priority, so it lets the others go ahead at the checkout. A class is a level of politeness. The idle class (-c 3) is the most polite: the program gets the disk only when nobody else wants it.
First stop the running load, then restart it with ionice. pkill stops programs by name, and -f searches the whole text of the command: the one for our loop contains charge.bin. Without this stop, two loads would run at the same time.
pkill -f charge.bin
ionice -c 3 timeout 600 sh -c 'while true; do dd if=/dev/zero of=/var/tmp/charge.bin bs=1M count=256 oflag=direct status=none; done' &Then read the probe again with the command from step 2: dd if=/var/tmp/sonde.bin of=/dev/null bs=4k count=1 iflag=direct. Its time should drop, because the load makes room for the others. If it doesn't drop, the likely cause is the disk scheduler, covered right after.
The other classes, nice, and a program that is already running
Open this fold to change the priority of a program that is already running. Two other classes exist:
- Best-effort (
-c 2): the default class, with a level from 0 (highest priority) to 7 (lowest). - Realtime (
-c 1): always goes first. Avoid it: a single program set this way can starve the whole machine of disk.
The nice command sets the processor access priority, from -20 (highest priority) to 19 (the most polite). According to the documentation, it also shifts the I/O priority: nice -n 19 gives level 7, the lowest of best-effort. For a program that is already running, there is no need to restart it: sudo ionice -c 3 -p 4242 changes its priority in place, with -p followed by its PID (here 4242).
7When ionice changes nothing: the scheduler
The I/O scheduler is the kernel software that decides in which order the requests in the queue go to the disk. It decides whether ionice is heard: some schedulers ignore it, others respect it. According to the kernel documentation, bfq and mq-deadline respect ionice, and none ignores it. We haven't measured this effect on your machine: check it with the probe. Open the fold below, by clicking its title, to read and change your scheduler.
Read and change the scheduler
First find your disk's name and type with lsblk (list of disks), then read its setting.
lsblk -d -o NAME,ROTA
cat /sys/block/sda/queue/schedulerNAME ROTA
sda 1
[mq-deadline] none bfqThe ROTA column is 1 for a spinning hard disk, 0 for an SSD. Replace sda with your disk's name. The name in brackets is the active scheduler. The three common names:
none: no sorting, requests go through in the order they arrive. It ignoresionice; it is the usual choice for fast SSDs.mq-deadline: gives each request a deadline and tends to serve reads before writes. It respectsionice.bfq: shares the disk fairly between programs, so the machine stays responsive. It also respectsionice.
To change the setting, you write the name into the same file. A redirection is the > sign, which sends text into a file. It is run by your terminal, with your rights, and not by sudo. That is why sudo echo bfq > file is refused. The tee command fixes the problem: it writes into the file itself whatever the pipe hands it, and sudo applies to it.
echo bfq | sudo tee /sys/block/sda/queue/schedulerRead the file again with the cat command above: bfq should be in brackets. The change applies right away and disappears at reboot. To go back before that, repeat the tee command with the old name, here mq-deadline.
The diagnosis is done: saturated disk, culprit found, priority lowered. All that is left is cleaning up the experiment.
8Clean up the experiment
Let's stop the load, then erase its two test files with rm (remove), whose -f option avoids any message if a file is missing.
pkill -f charge.bin
rm -f /var/tmp/charge.bin /var/tmp/sonde.binLet's check that nothing is writing anymore. The first command looks for a dd that is still active, the second lists the remaining test files.
pgrep -x dd || echo "plus de dd"
ls /var/tmp/*.bin 2>/dev/null || echo "plus de fichier de test"plus de dd
plus de fichier de testIf you changed the scheduler in step 7, put the old name back with tee, otherwise it stays until the next reboot. The installed packages stay too, but nothing runs in the background: no service, no scheduled task.
If it fails
iostat: command not found(orpidstat): thesysstatpackage is missing. Install it as in step 3 (sudo apt install sysstat).dd: failed to open '/dev/shm/charge.bin': Invalid argument: the/dev/shmfolder lives in memory, and thedirectoption doesn't work there. Write to the real disk, as with/var/tmp.Netlink error: Operation not permitted (1):iotopwas started without administrator rights (Netlink is the channel it uses to question the kernel). Putsudoin front.- The probe stays fast: the load stopped after 10 minutes, or your disk is very fast. Run the
timeout 600command from step 1 again. - No
/proc/pressure/iofile: your kernel doesn't have PSI.iostatandpidstatare then enough for the diagnosis.
Going further
Simulate a site that reads at random
A website rarely reads a big file in one go: it picks small pieces here and there. The fio tool imitates this for 15 seconds, with 16 reads of 4 KB in parallel, on a 256 MB file. Its options read in groups:
--name,--filename,--size: the name of the test, the file used and its size.--rw=randread,--bs=4k: random reads, in 4 KB blocks.--direct=1: without going through the cache, likedirectfordd.--runtime=15,--time_based: a duration of 15 seconds, whatever happens.--iodepth=16,--ioengine=libaio: 16 requests at the same time, using Linux'slibaiomethod.
sudo apt install fio
fio --name=aleatoire --filename=/var/tmp/fio.bin --size=256M --rw=randread --bs=4k --direct=1 --runtime=15 --time_based --iodepth=16 --ioengine=libaio
rm /var/tmp/fio.binOpen a second terminal window meanwhile and run iostat -dxzy 2 3 there. You will see r/s go up, with reads much smaller than those of dd.
Pressure on the processor and memory
PSI doesn't stop at the disk. The files /proc/pressure/cpu and /proc/pressure/memory have the same format: they tell you whether programs are waiting for a processor core or for memory. The kernel documentation also explains how to get a signal when the pressure passes a threshold you set.
What's next
You can now recognize a saturated disk, find who is saturating it and calm it down by hand. That hand remains the weak link, though: a service that restarts comes back without your settings. The next article, Harden a service with systemd, shows how to fix a service's rules in its own file, once and for all.