Anime-Sanctuary

When things break: the checklist

Server down: what do you check, and in what order? Here is the troubleshooting checklist to follow when "it stopped working", from the widest check to the most precise, without breaking anything more.

Course
Tutorial 35 of 35
Time
25 min to read
Tested with
Commands: tested on Debian 13.7 (systemd 257.13, Apache 2.4.69, restic 0.18.0)
What you will be able to do
  • Follow ten checks in order, from "does the machine answer?" to "what happened?".
  • Read the real error output of a full disk and a failed service.
  • Pick the first command according to the symptom.
  • Roll back without making the failure worse.

Before you start: Read your logs with journalctl, A firewall with ufw, Your first website with Apache.

Contents
  1. A funnel, not a maze
  2. Breathe and don't break anything more
  3. Is the server reachable?
  4. Is there space and memory left?
  5. Which services are down?
  6. What do the logs say?
  7. Do the network and firewall let traffic through?
  8. Does the website answer?
  9. Disk or memory exhausted
  10. Roll back
  11. Write down what happened
  12. The table: symptom, first command
  13. If it fails
  14. To finish
  15. Sources

Server down: what do you do first? Follow a troubleshooting checklist, a list of checks in a fixed order, from the widest to the most precise. Ten steps are enough. Each one rules out a family of failures, and the first one that answers "no" tells you where to dig. A failure isn't a defeat: it is the server asking for your attention.

A funnel, not a maze

We start with the widest questions, because they cost almost nothing and rule out a lot: is the machine even switched on? It would be a shame to dig through an Apache log if the network cable is unplugged. At each step the question narrows, down to a single command that points at the cause.

The diagnosis funnel Seven bars getting narrower from top to bottom: breathe, reachable, resources, services, logs, network, web. 1 Breathe, take notes 2 Reachable? 3 Resources 4 Services 5 Logs 6 Network 7 Web from the widest to the most precise
Text version: steps 1 to 7 form a funnel. You start with the widest, "breathe and take notes", and narrow down to the website. Steps 8 to 10 (exhaustion, rollback, write-up) come after.

To make this concrete, let's follow a simulated failure. On mon-serveur, the site exemple.fr is served by Apache, as in Your first website with Apache. You changed a setting, restarted Apache, and it answers with an error:

On the server
sudo systemctl restart apache2
Expected output
Job for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details.

The site is down. Before chaining commands, the first step asks for none.

1Breathe and don't break anything more

Before any command, note the time and what changed recently, and don't restart anything at random. A restart wipes the machine's memory, and with it some of the clues. It can also turn a small failure into a big one. Three notes are enough:

  • The time the failure started, or of your first alert.
  • What changed just before: an update, a setting you edited, a new user.
  • Who is affected: only you, or all the visitors.

Two commands find what changed. The first reads the package install log, /var/log/dpkg.log, and keeps the last three lines. The second sorts the settings in /etc from newest to oldest. Between the two, the vertical bar | is a pipe: it passes what the left command prints to the right one, here tail -n 3 or head -n 5, which keep only the last or first lines.

On the server
date
sudo grep -E ' (install|upgrade) ' /var/log/dpkg.log | tail -n 3
ls -lt /etc | head -n 5
Expected output
Thu Nov  5 08:01:14 UTC 2026
2026-11-05 08:01:09 install libncursesw6:amd64 <none> 6.5+20250216-2
2026-11-05 08:01:10 install nano:amd64 <none> 8.4-1+deb13u1
2026-11-05 08:01:10 install libgpm2:amd64 <none> 1.20.7-11+b2
total 236
drwxr-xr-x 2 root root    4096 Nov  5 08:01 apache2
-rw-r--r-- 1 root root    4399 Nov  5 08:01 ld.so.cache
drwxr-xr-x 1 root root    4096 Nov  5 08:01 alternatives
-rw-r--r-- 1 root root     172 Nov  5 08:01 hosts

Each line of the log starts with a date and a time, then the word install or upgrade and the package name. In ls -lt, the date comes just before the name. In our test, these dates don't point at anything useful: only the method matters. On your machine, compare them with the time of the failure: a setting dated ten minutes earlier is a suspect. Restarting at random is like shaking the TV and hoping it picks up the signal. What remains is finding out whether the machine still answers.

2Is the server reachable?

Two tests tell you whether the machine answers: ping for the network, then ssh for the front door. The ping command sends small messages and waits for their echo, like sonar. Run it from your computer, with -c 3 to stop after three messages.

On your computer
ping -c 3 203.0.113.10
Expected output
PING 203.0.113.10 (203.0.113.10) 56(84) bytes of data.

--- 203.0.113.10 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2049ms

Here, "0 received" means no echo came back. A healthy machine answers three times, with "0% packet loss". But some hosting providers block ping, so silence alone proves nothing. Try the connection itself, as in Your first SSH connection.

On your computer
ssh alice@203.0.113.10
Expected output
ssh: connect to host 203.0.113.10 port 22: Connection timed out

The message tells you where the failure is:

  • You are connected: the machine is alive, and everything else happens on it.
  • Connection timed out: nothing reaches the server. It is off, the network is cut, or a firewall swallows the messages.
  • Connection refused: the machine answers, but nobody is listening on port 22. The SSH service is stopped.

In the last two cases, you can no longer get in over SSH. Use your hosting provider's remote console instead: a virtual screen and keyboard, reachable from the provider's website. It works even when the server's network is down. If SSH answers, you are inside the machine: let's look at what it has in its belly.

3Is there space and memory left?

A full disk or exhausted memory is enough to bring down services that aren't broken at all. Four commands give you the picture in ten seconds:

  • Space: df -h lists the disks and how full they are. Look at the Use% column, and beware of values close to 100%.
  • Index cards: df -i counts inodes, the cards that describe each file. A disk can run out of cards while it still has space, for example with millions of tiny files.
  • Memory: free -h shows the RAM. Read the available column: it is what is really still free.
  • Load: uptime prints three numbers, the average load over 1, 5 and 15 minutes. Compare them with the number of processor cores, given by nproc. Far above it, the machine is overloaded.

Suppose your data lives in /donnees. Here is a full disk. This one is a small test space in memory, and yours will have a different name:

On the server
df -h /donnees
Expected output
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           2.0M  2.0M     0 100% /donnees

The Avail column is at zero and Use% at 100%: every write fails with the message No space left on device. To find what takes the space, du adds up the file sizes, and sort -h orders them from smallest to biggest.

On the server
sudo du -ah /donnees | sort -h | tail -n 3
Expected output
1.0M	/donnees/export.sql
1.0M	/donnees/gros.log
2.0M	/donnees

The culprits are at the bottom of the list, here two files of 1 MB. If df -i reports 100% in IUse%, look instead for the folder that holds millions of small files. Space and memory are fine? Then a program may simply be stopped.

4Which services are down?

The systemctl --failed command lists the services that systemd tried to start and couldn't. A service is a program that runs all the time, like Apache. The --plain and --no-legend options remove the column title line and the little dot that marks each service, to keep only the useful lines.

On the server
systemctl --failed --plain --no-legend
Expected output
apache2.service loaded failed failed The Apache HTTP Server

Each line is a service that went down. When all is well, the command prints nothing at all: silence is the good news. Now let's question the culprit, with sudo so we also see its last log lines.

On the server
sudo systemctl status apache2 --no-pager
Expected output
     Loaded: loaded (/usr/lib/systemd/system/apache2.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Thu 2026-11-05 07:48:59 UTC; 20ms ago
   Duration: 17.284s
 Invocation: 27ea0799a7d349bc8266e4b4f5d80903
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 153 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)
   Mem peak: 1.5M
        CPU: 15ms

Nov 05 07:48:59 mon-serveur systemd[1]: Starting apache2.service - The Apache HTTP Server...
Nov 05 07:48:59 mon-serveur apachectl[155]: AH00526: Syntax error on line 226 of /etc/apache2/apache2.conf:
Nov 05 07:48:59 mon-serveur apachectl[155]: Invalid command 'ServerNaem', perhaps misspelled or defined by a module not included in the server configuration
Nov 05 07:48:59 mon-serveur systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Nov 05 07:48:59 mon-serveur systemd[1]: apache2.service: Failed with result 'exit-code'.
Nov 05 07:48:59 mon-serveur systemd[1]: Failed to start apache2.service - The Apache HTTP Server.

The Active: failed line confirms the failure, and the dated lines at the bottom already give the cause: Invalid command 'ServerNaem', a typo for ServerName. Your dates and numbers will differ. Here the excerpt is enough, but that isn't always the case: you then need to read the whole journal.

5What do the logs say?

To read the logs of a failure, journalctl gives three angles, detailed in Read your logs with journalctl. The first keeps the errors since the machine last booted (-p err -b): it says what.

On the server
sudo journalctl -p err -b --no-pager
Expected output
Nov 05 07:48:59 mon-serveur systemd[1]: Failed to start apache2.service - The Apache HTTP Server.

The line announces the failure, without the reason. The second angle reads a single service over a short period (-u and --since): it says why. Here are Apache's last four lines.

On the server
sudo journalctl -u apache2 --since "10 minutes ago" --no-pager | tail -n 4
Expected output
Nov 05 07:51:04 mon-serveur apachectl[600]: Invalid command 'ServerNaem', perhaps misspelled or defined by a module not included in the server configuration
Nov 05 07:51:04 mon-serveur systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Nov 05 07:51:04 mon-serveur systemd[1]: apache2.service: Failed with result 'exit-code'.
Nov 05 07:51:04 mon-serveur systemd[1]: Failed to start apache2.service - The Apache HTTP Server.

The journal confirms the typo. The third angle, -b -1, reads the previous boot: useful when the machine crashed and then restarted by itself. These three commands point at the service. What remains is checking that the network lets it be reached.

6Do the network and firewall let traffic through?

Three commands answer: ss shows what is listening, ufw what the firewall allows, dig what the domain name answers. A reminder from The words: IP, DNS, domain, port, firewall: a port is a numbered door of the machine, and a program listens at it, waiting for visitors. First let's see who is listening.

On the server
sudo ss -tlnp
Expected output
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess

The options -t, -l, -n and -p ask for the ports where a program is listening (over TCP, the protocol of the web), with their number rather than their name, and the program involved. Only the header shows: nobody is listening on port 80, so Apache isn't there. When all is well, one line appears under the header for each listening program.

If a program is listening but visitors don't get through, look at the firewall. The ufw from A firewall with ufw must allow ports 80 and 443 for a website. Finally, dig asks the domain name for its address.

On the server
sudo ufw status
dig +short exemple.fr

The first command lists the rules: with no rule for 80 and 443, visitors see a timeout. The second should print 203.0.113.10, your server's address. Another address, or nothing, sends you back to the chapter on Domain name and DNS. Here, everything is in order except Apache: let's look at the site itself.

7Does the website answer?

The curl -I command asks your site for its response header, the first line it sends back, which carries a code. The code 200 means "all is well". Test from the server itself, with -sS to hide the progress bar while keeping the errors.

On the server
curl -sSI http://localhost
Expected output
curl: (7) Failed to connect to localhost port 80 after 0 ms: Could not connect to server

The error code (7) means nobody answers: consistent with the empty ss. Before restarting Apache, one habit saves a lot of trouble: check the syntax of its settings files with apachectl configtest.

On the server
sudo apachectl configtest
Expected output
AH00526: Syntax error on line 226 of /etc/apache2/apache2.conf:
Invalid command 'ServerNaem', perhaps misspelled or defined by a module not included in the server configuration

The test gives the file, the line and the mistake. Fix ServerNaem to ServerName with sudo nano /etc/apache2/apache2.conf. nano is a text editor in the terminal: move with the arrows, fix the word, then Ctrl + O and Enter to save, Ctrl + X to quit (details in The terminal for beginners). Then test again.

On the server
sudo apachectl configtest
sudo systemctl restart apache2
curl -sSI http://localhost | head -n 1
Expected output
Syntax OK
HTTP/1.1 200 OK

The site answers, and sudo ss -tlnp shows Apache's line on port 80 again. If the code isn't 200, read the end of the site's error log, which is not the systemd one:

On the server
sudo tail -n 20 /var/log/apache2/error.log

Codes that start with 5 signal an error on the server side, and those that start with 4 a refused request or a missing page. But the cause isn't always a setting: the disk or memory may be exhausted.

8Disk or memory exhausted

When df, free or uptime rang the alarm in step 3, the failure comes from a resource that ran out, not from a setting. Three situations stand out:

  • A slow disk, or one swamped with reads: everything drags without any service going down. Diagnose a saturated disk teaches you to see who is using it.
  • A failing disk: read errors, unreadable files. Software RAID and SMART checks shows how to watch its health.
  • Full memory: the kernel, the core of the system, then kills a program to save the machine. A trace of it stays in the journal.
On the server
sudo journalctl -k | grep -i "out of memory"

The command looks for that trace in the kernel messages (-k). If it prints nothing, memory isn't the cause. Otherwise it prints an "Out of memory: Killed process" line followed by the number and name of the program that was killed. To be warned before the failure rather than after, Simple monitoring sets up an alert. If the cause stays hidden or time is short, there is one last resort.

9Roll back

When the cause hides or time is short, put back the last version that worked. There are two cases, depending on what you lost:

  • A file you edited: keep a copy before every change, with sudo cp /etc/apache2/apache2.conf /etc/apache2/apache2.conf.bak. To go back, copy the .bak over it.
  • Lost or damaged data: get it back from a backup, like the ones in Encrypted backups with restic.

With restic, you first list the backups, which it calls snapshots. Here the repository is /sauvegardes: replace it with the path to yours, for example /home/alice/depot-restic if you followed that article. Before answering, restic asks for the repository password, the one chosen when you created it in Encrypted backups with restic: type it (nothing shows while you type) then Enter. Expect this question with every restic command in this step.

On the server
sudo restic -r /sauvegardes snapshots
Expected output
enter password for repository:
repository c1443eb0 opened (version 2, compression level auto)
ID        Time                 Host         Tags        Paths      Size
-----------------------------------------------------------------------
0b782240  2026-11-05 07:51:30  mon-serveur              /srv/site  17 B
-----------------------------------------------------------------------
1 snapshots

Each line is a point you can return to. Restore into a folder next to it, never over it: you check first, you replace afterwards.

On the server
sudo restic -r /sauvegardes restore latest --target /tmp/restauration --include /srv/site/index.html
sudo cat /tmp/restauration/srv/site/index.html
Expected output
enter password for repository:
repository c1443eb0 opened (version 2, compression level auto)
restoring snapshot 0b782240 of [/srv/site] at 2026-11-05 07:51:30.076496234 +0000 UTC by root@mon-serveur to /tmp/restauration
Summary: Restored 3 / 1 files/dirs (17 B / 17 B) in 0:00
<h1>Bonjour</h1>

The IDs and times will be your own. The content shows: you can copy the file back into place with sudo cp. One last step remains, which everyone skips and which you will be glad of later.

10Write down what happened

A short write-up, written while you still remember, keeps you from falling into the same failure again. It is called a post-mortem: a review written after the failure, which looks for the cause and not for someone to blame. Five lines are enough:

  • Start and end: the times you noted in step 1.
  • Symptom: what the visitors saw.
  • Cause: here, a typo in apache2.conf.
  • Fix: the corrected line, then restarting Apache.
  • What changes: here, running apachectl configtest before every restart.

The last line counts the most: a failure that improves nothing will happen again. To keep all this at hand, here is the full list of the ten steps, to print or open next to your terminal.

The table: symptom, first command

When the symptom is clear, this table saves you from going through the whole funnel again: it gives the first command, then where to look next.

SymptomFirst commandWhere to look next
Nothing answers (no ping, no SSH)the hosting provider's remote consolethe machine's state in the customer area; Software RAID and SMART checks if the disk is suspect
SSH: Connection refusedsudo systemctl status ssh through the consolesudo ufw status, then journalctl -u ssh
The server answers but everything dragsuptimefree -h, df -h, then Diagnose a saturated disk
No space left on devicedf -h and df -isudo du -ah /donnees | sort -h | tail
A service won't startsudo systemctl status namesudo journalctl -u name --since "10 minutes ago"
The site won't opencurl -sSI http://localhostsudo apachectl configtest, then error.log
The domain name leads nowheredig +short exemple.frDomain name and DNS, then sudo ufw status
A program vanished without a soundsudo journalctl -k | grep -i "out of memory"free -h, then Simple monitoring
It worked yesterday, not todayls -lt /etc | head -n 5the /var/log/dpkg.log log, then the backup (step 9)

If it fails

Two messages come back when you use the journal without the right permissions:

  • Hint: You are currently not seeing messages from other users and the system. sudo is missing in front of journalctl. Run the same command with it.
  • systemctl status without log lines. Same cause: without sudo, alice only sees the status information, not the dated lines at the bottom.

To finish

This is the last step of the course, and here you are with a server that can tell you what is happening to it, and someone to listen. It will break again one day, like every server in the world, and on that day you will have a list, a cup of tea and all the time you need. Fair winds to mon-serveur, and to you.