Anime-Sanctuary

Simple monitoring

How do you monitor a Linux server without spending your evenings staring at graphs? Five checks, one small script, and a message in the server's mailbox only when something changes.

Course
Tutorial 30 of 35
Time
40 min to read
Tested with
Debian 13.7 (trixie), systemd 257, OpenSSL 3.5.7, cron 3.0pl1
What you will be able to do
  • Read the load, memory, disk and failed services with five commands.
  • Know when an HTTPS certificate is about to expire.
  • Install a script that checks all of this and writes to the server's mailbox only when the situation changes.
  • Run it every 5 minutes with cron.

Before you start: Read your logs with journalctl, Your first backup with rsync, Updates and packages with apt.

Contents
  1. Five questions to ask your server
  2. 1. The load
  3. 2. Memory and disk
  4. 3. Failed services
  5. 4. Certificates
  6. 5. Put it all in one script
  7. 6. Alert only when something changes
  8. 7. Run it every 5 minutes
  9. If it fails
  10. Going further
  11. What's next

Five questions to ask your server

To monitor a Linux server the simple way, five questions are enough. Is it overloaded? Does it have memory left? Disk space? Any broken services? Any certificates about to expire? A server that answers "no" five times is healthy. Your server has been running for several articles now, and nobody warns you when it goes down. You find out the day a site stops opening.

First we will ask each question by hand, with one command. Then we will put all five in a small script that the server runs by itself. It speaks up only when an answer changes, not every five minutes to say all is well. Its messages land in the server's own mailbox: a file you read after logging in, not your usual email. Step 6 shows how to read it. Receiving them elsewhere needs a setting we haven't tested, so it sits in a folded section of step 6, to be taken as a lead.

1The load

The load tells you whether the processor is overloaded, and the uptime command shows it along with the time since the last boot. The processor is the chip that runs programs, and each of its cores handles one job at a time. Think of supermarket checkouts: two cores, two checkouts.

On the server
uptime
Expected output
 10:12:01 up 12 days,  3:04,  1 user,  load average: 0.15, 0.10, 0.08

The three numbers after load average are the load: the average number of jobs using a checkout or waiting in line, over the last minute, the last 5 and the last 15. Here, 0.15 is almost nothing. To know whether a number is big, compare it with the number of cores.

On the server
nproc
Expected output
2

With 2 cores, a load of 2 means every checkout is busy and nobody is waiting. Above 4, double that, the line really starts to grow. That is our alert threshold, and step 5 shows how to change it. Memory and disk remain, and they fill up more quietly.

2Memory and disk

You check memory and disk with free and df, and the right column is the one that tells you what is left. The memory (RAM) is the workspace of the running programs. It empties when the machine stops, unlike the disk, which keeps your files.

On the server
free -h
Expected output
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       512Mi       2.4Gi       1.0Mi       1.1Gi       3.3Gi
Swap:          1.0Gi          0B       1.0Gi

Look at available, the memory programs can still get. Linux keeps recently read files in reserve (buff/cache) and hands that memory back as soon as it is needed. So free always looks small, and that isn't a problem. Here, 3.3 Gi out of 3.8 Gi are available. We will alert below 10% of the total.

On the server
df -h /
Expected output
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G  9.8G   28G  26% /

The Use% column gives the share of the disk in use: 26% here. A full disk is the sneakiest failure, because programs stop writing without a word. We will alert at 85%, which leaves time to clean up. Your numbers will differ from the example, and that's fine. The machine itself is checked. Now let's look at the programs it runs.

3Failed services

A service that stopped on an error is listed by systemctl --failed. A service is a program that Debian starts and watches for you, like Apache or ssh. When one crashes and doesn't come back, it moves to "failed". On a healthy server, the list is empty.

On the server
systemctl --failed
Expected output
  UNIT LOAD ACTIVE SUB DESCRIPTION

0 loaded units listed.

To see what a failure looks like, let's make one. The next command starts a demo service, demo-panne, that runs /bin/false, a program whose only job is to fail.

On the server
sudo systemd-run --unit=demo-panne /bin/false
Expected output
Running as unit: demo-panne.service; invocation ID: c498f0594eac4b93aa70299f854f189f

The ID changes on every run. Now ask for the list in compact form, without the header, using --no-legend and --plain. That is the form our script will read.

On the server
systemctl --failed --no-legend --plain
Expected output
demo-panne.service loaded failed failed [systemd-run] /bin/false

The first word is the name of the failed service, and it is the one the script will keep. Leave demo-panne in place: we will use it to test the alert. One last subject depends on the calendar, not the machine: the HTTPS certificate.

4Certificates

An expired certificate shows your visitors a big red warning screen, and the openssl command reads its end date. The certificate is the file that proves your site's identity over HTTPS. The one from HTTPS with Let's Encrypt lives in /etc/letsencrypt/live/, in a folder named after your domain, and lasts 90 days. In our commands, exemple.fr is a sample name: replace it with yours. To find it, list that folder.

On the server
sudo ls /etc/letsencrypt/live/
Expected output
exemple.fr

Now ask the certificate for its end date.

On the server
sudo openssl x509 -enddate -noout -in /etc/letsencrypt/live/exemple.fr/fullchain.pem
Expected output
notAfter=Feb  3 07:22:47 2027 GMT

Your date will be your own. So the script doesn't have to count days, the -checkend option asks OpenSSL a question: "will this certificate still be valid in 14 days?" The number is in seconds, and 14 days make 1,209,600 seconds.

On the server
sudo openssl x509 -checkend 1209600 -noout -in /etc/letsencrypt/live/exemple.fr/fullchain.pem
Expected output
Certificate will not expire

If the answer becomes Certificate will expire, the command fails with a non-zero exit code, which the script knows how to read. Certbot renews a certificate when a third of its life is left, so 30 days out of 90, and it tries twice a day. A certificate that still hasn't been renewed 14 days before its end has missed two weeks of attempts. Automatic renewal is broken, and there is still enough time to fix it. The five questions are asked: time to bring them together.

5Put it all in one script

A script is a text file holding commands that run one after the other. Ours, supervise.sh, repeats the five checks and keeps each problem in a list. It is 58 lines long and works in four stages: the thresholds and the recipient at the top, the five checks, the comparison with the last run, then sending the message. You don't have to read it all. The whole script is folded below for anyone who wants to check what they install, and these key lines are enough to find your way.

  • DEST="alice": the account that receives the messages.
  • MAX_DISQUE=${MAX_DISQUE:-85}: "take the value of MAX_DISQUE if it exists, otherwise 85". You change the threshold by editing that number.
  • probleme nom "message": adds a problem to the list, with a short name and a sentence to explain it.
  • awk -v c="$charge": awk is a small calculation program, and -v hands it a value, here the load, to compare with the threshold.
  • $((JOURS_CERT * 86400)): a calculation in seconds, because a day has 86,400 of them.
  • ETAT: the file where the script writes the list of problems from the last run.
  • exit 0: quit without a word when the list hasn't changed.
Read the script before installing it
Contents of supervise.sh
#!/bin/sh
# Surveille le serveur et ecrit a DEST seulement quand la situation change.
DEST="alice"
ETAT=/var/lib/supervise/etat
MAX_DISQUE=${MAX_DISQUE:-85}
MIN_MEMOIRE=${MIN_MEMOIRE:-10}
JOURS_CERT=${JOURS_CERT:-14}
CHARGE_PAR_COEUR=${CHARGE_PAR_COEUR:-2}

cles=""
details=""
probleme() {
    cles="$cles$1;"
    details="$details$2
"
}

# Charge : plus de CHARGE_PAR_COEUR fois le nombre de coeurs.
charge=$(cut -d ' ' -f 1 /proc/loadavg)
awk -v c="$charge" -v n="$(nproc)" -v f="$CHARGE_PAR_COEUR" 'BEGIN { exit !(c > f * n) }' &&
    probleme charge "Charge elevee : $charge pour $(nproc) coeurs"

# Memoire disponible, en pourcentage du total.
libre=$(free | awk '/^Mem:/ { print int($7 * 100 / $2) }')
[ "$libre" -lt "$MIN_MEMOIRE" ] && probleme memoire "Memoire disponible : $libre %"

# Disques pleins.
pleins=$(df -P -x tmpfs -x devtmpfs | awk -v max="$MAX_DISQUE" 'NR > 1 && int($5) >= max { print $6 "=" $5 }')
for p in $pleins; do
    probleme "disque-${p%=*}" "Disque plein : ${p%=*} a ${p#*=}"
done

# Services en echec.
for u in $(systemctl --failed --no-legend --plain | awk '{ print $1 }'); do
    probleme "service-$u" "Service en echec : $u"
done

# Certificats Let's Encrypt qui expirent bientot.
for c in /etc/letsencrypt/live/*/fullchain.pem; do
    [ -e "$c" ] || continue
    openssl x509 -checkend $((JOURS_CERT * 86400)) -noout -in "$c" > /dev/null ||
        probleme "cert-$c" "Certificat bientot expire : $c ($(openssl x509 -enddate -noout -in "$c"))"
done

# Alerte seulement si la liste des problemes a change depuis le dernier passage.
mkdir -p "$(dirname "$ETAT")"
[ "$cles" = "$(cat "$ETAT" 2>/dev/null)" ] && exit 0

if [ -n "$cles" ]; then
    sujet="ALERTE $(hostname)"
    corps=$details
else
    sujet="OK $(hostname) : retour a la normale"
    corps="Plus aucun probleme."
fi
printf '%s\n' "$corps" | mail -s "$sujet" "$DEST" || exit 1
printf '%s' "$cles" > "$ETAT"
logger -t supervise "$sujet"

Every check follows the same mold: if something is wrong, it calls probleme with a short name and a message. The name (memoire, disque-/, service-demo-panne.service) identifies the problem, and the message explains it. Four thresholds can be set at the top of the file: MAX_DISQUE=85 (percent), MIN_MEMOIRE=10 (percent), JOURS_CERT=14 and CHARGE_PAR_COEUR=2, the alert load per core: 2 gives 4 for a 2-core machine. Once the script is installed, open it with sudo nano /usr/local/bin/supervise.sh and change the number after :-. Nothing else needs touching. The script also needs the mail command, which the apt tool from Updates and packages with apt installs. Then download the script.

On the server
sudo apt install -y bsd-mailx
sudo curl -fsSo /usr/local/bin/supervise.sh https://anime-sanctuary.net/tutoriels/supervision-simple/files/supervise.sh
sudo chmod 755 /usr/local/bin/supervise.sh

The bsd-mailx package provides the mail command and a small mail server, the program that stores and sends messages. A setup screen may appear and ask which kind of mail setup to install. Choose the line local delivery only; not on a network: the messages stay on this server, which is what we need to start. If the screen looks different or doesn't appear, pick the option about local delivery only. To skip the screen, install with the command below: it picks local delivery for you, and the second command confirms it. In the script, DEST="alice" points to the mailbox of the user alice, on the server itself.

On the server
sudo DEBIAN_FRONTEND=noninteractive apt install -y bsd-mailx
grep configtype /etc/exim4/update-exim4.conf.conf
Expected output
dc_eximconfig_configtype='local'

Let's try the script: on a healthy server, it should say nothing.

But the demo-panne service from step 3 is still failed. Clear it first so we start from a healthy server.

On the server
sudo systemctl reset-failed demo-panne.service
sudo /usr/local/bin/supervise.sh
echo "code de retour : $?"
Expected output
code de retour : 0

Silence is the right answer: no problem, so no message. Code 0 means "finished without error". Now let's see what happens when something breaks.

6Alert only when something changes

The script writes at the first failure and when things return to normal, but never in between. It keeps the list of problems from the last run in /var/lib/supervise/etat and compares it with the new list. If they are identical, it stops without a word.

Six script runs, two messages The script runs every 5 minutes. At 10:00 and 10:05 all is well and it stays quiet. At 10:10 the disk is full: alert message. At 10:15 and 10:20 the disk is still full: silence. At 10:25 all is well again: recovery message. Time Situation The script 10:00all is wellstays quiet 10:05all is wellstays quiet 10:10disk fullALERT sent 10:15disk fullstays quiet 10:20disk fullstays quiet 10:25all is wellOK sent
Text version: over six runs, 5 minutes apart, the script sends only two messages, one when the problem appears (10:10) and one when it goes away (10:25). It stays quiet the other four times.

A script that wrote on every run during the failure would send three alerts (10:10, 10:15, 10:20) and then the recovery message: four in all. Ours sends only two, and the gap grows with the length of the outage. An alert repeated every five minutes ends up ignored. Let's replay this scenario with our demo service. Recreate the failure, then run the script twice.

On the server
sudo systemd-run --unit=demo-panne /bin/false
sudo /usr/local/bin/supervise.sh
sudo /usr/local/bin/supervise.sh
sudo grep -E '^(Subject|Service)' /var/mail/alice
Expected output
Running as unit: demo-panne.service; invocation ID: 1d7a5f0cb4e24a0d8a3b9f6e2c5d7a10
Subject: ALERTE mon-serveur
Service en echec : demo-panne.service

Delivery takes a second or two: if the last grep shows nothing, run it again. Two runs, one message: the first raised the alert, and the second saw the same list and stayed quiet. Your server's name will replace mon-serveur. Now let's fix the failure and run the script again.

On the server
sudo systemctl reset-failed demo-panne.service
sudo /usr/local/bin/supervise.sh
sudo grep -E '^(Subject|Plus)' /var/mail/alice
Expected output
Subject: ALERTE mon-serveur
Subject: OK mon-serveur : retour a la normale
Plus aucun probleme.

The second message arrived as soon as the list was empty again. Every message is also written to the journal, which gives you a history, as in Read your logs with journalctl.

On the server
sudo journalctl -t supervise --no-pager
Expected output
Nov 05 07:22:44 mon-serveur supervise[950]: ALERTE mon-serveur
Nov 05 07:22:45 mon-serveur supervise[977]: OK mon-serveur : retour a la normale

These messages arrive in alice's local mailbox on the server, the /var/mail/alice file that the grep commands above were reading. To read it more comfortably, log in as alice and type mail. The list of messages appears, 1 then Enter opens the first one, and q quits.

On the server, logged in as alice
mail
Expected output
Mail version 8.1.2 01/15/2001.  Type ? for help.
"/var/mail/alice": 1 message 1 new
>N  1 root@mon-serveur   Thu Nov 05 08:09   20/616   ALERTE mon-serveur

This mailbox stays on the server: the alert won't find you on its own. Until you forward it, you have to remember to read it.

Get the alert in your real email

For that, the server must be able to send mail to the outside. We haven't tried this part, so here is the lead without the details. The package from step 5 installed Exim, its mail program. The command sudo dpkg-reconfigure exim4-config reopens the setup screen. Choose mail sent by smarthost; received via SMTP or fetchmail, which means mail sent through a relay, then enter the sending server and the account that your email provider gives you. After that, a .forward file in alice's folder, containing your address, forwards everything that arrives.

The other way: replace the mail line in the script with a call to the notification service of your choice. The rest of the script doesn't change.

Both messages went out at the right time. All that's left is to let the server run this without you.

7Run it every 5 minutes

Cron runs the script for you, as with the backup in Your first backup with rsync. This script covers the whole server, not a single user, so we declare it in the /etc/cron.d/ folder, where each line names the user who runs the command, here root. The first five fields, */5 * * * *, mean "every 5 minutes".

On the server
echo '*/5 * * * * root /usr/local/bin/supervise.sh' | sudo tee /etc/cron.d/supervise
Expected output
*/5 * * * * root /usr/local/bin/supervise.sh

Cron rereads this folder on its own, with no restart. To check that it really runs the script, wait 5 minutes and look for its trace in the journal.

On the server
sudo journalctl -u cron --since "10 minutes ago" --no-pager | grep supervise
Expected output
Nov 05 07:19:01 mon-serveur CRON[597]: (root) CMD (/usr/local/bin/supervise.sh)
Nov 05 07:24:01 mon-serveur CRON[641]: (root) CMD (/usr/local/bin/supervise.sh)

One line per run: monitoring is working. Nothing more to do, except read the server's mailbox now and then, or forward it as in the folded section of step 6.

Use a systemd timer instead of cron

A timer is the systemd version of cron: two small files, one that describes the task and one that describes the schedule. Its advantage: every run leaves a trace in the journal. Create /etc/systemd/system/supervise.service and /etc/systemd/system/supervise.timer, then enable the timer.

On the server
sudo tee /etc/systemd/system/supervise.service <<'EOF'
[Unit]
Description=Verification de sante du serveur

[Service]
Type=oneshot
ExecStart=/usr/local/bin/supervise.sh
EOF
sudo tee /etc/systemd/system/supervise.timer <<'EOF'
[Unit]
Description=Lance supervise toutes les 5 minutes

[Timer]
OnCalendar=*:0/5

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now supervise.timer

Don't use both at the same time: delete /etc/cron.d/supervise if you switch to the timer. See the list with systemctl list-timers supervise.timer.

If it fails

  • mail: not found: the mail command is missing. Install bsd-mailx, as in step 5.
  • mkdir: cannot create directory '/var/lib/supervise': Permission denied: the script was run without sudo. It needs root rights to write its state file.
  • Could not open file or uri for loading certificate: the domain name in your openssl command is wrong. Run sudo ls /etc/letsencrypt/live/ again and use the name it shows.
  • The script says nothing: that's normal. No output means no change. To force a message, recreate the demo failure from step 3.

Here is the full message when mail is missing: the script stops without saving its state, so it will try again on the next run.

On the server
sudo /usr/local/bin/supervise.sh
Expected output
/usr/local/bin/supervise.sh: 55: mail: not found

Going further

Ready-made tools

When your fleet of servers grows, other tools do the same job with richer graphs and alerts. They need more memory and more settings than our 58-line script.

  • Monit: watches services and files, and can restart a service that went down.
  • Netdata: shows hundreds of live graphs on a web page.
  • Prometheus with node_exporter: keeps the history of measurements from several machines so you can compare them.

A single server doesn't need them to start. My advice: add one of these tools the day you wonder "how long has this been going on?" and the script can't answer.

What's next

Your server now writes to its mailbox when a disk fills up, a service goes down or a certificate gets old, and you can forward those messages to your email. Knowing a failure is coming won't save your files if the disk dies. The next article, Encrypted backups with restic, adds encrypted, versioned backups: the net that remains when the alert comes too late.

Sources (accessed November 4, 2026)

  • Commands: uptime(1), free(1), df(1), systemctl(1), systemd-run(1), openssl-x509(1ssl).
  • Scheduling: crontab(5), systemd.timer(5). Certificate renewal: Certbot documentation.
  • Tools: Monit, Netdata, node_exporter.
  • Commands: tested on Debian 13.7 (systemd 257, cron 3.0pl1, bsd-mailx 8.1.2, OpenSSL 3.5.7). Sample values for uptime, free, df and the machine name; test certificate made by hand; script replayed in a container without systemd, with a simulated systemctl, load threshold included. Not tested: the load threshold under real conditions, sending to an outside address and forwarding with .forward, the mail package's setup screen (the wording of the choice comes from the package), and the 5-minute interval (tested with one run per minute).