When things break: the checklist
Server down: what do you check, and in what order? Here is the troubleshooting checklist to follow when "it stopped working", from the widest check to the most precise, without breaking anything more.
- Course
- Tutorial 35 of 35
- Time
- 25 min to read
- Tested with
- Commands: tested on Debian 13.7 (systemd 257.13, Apache 2.4.69, restic 0.18.0)
What you will be able to do
- Follow ten checks in order, from "does the machine answer?" to "what happened?".
- Read the real error output of a full disk and a failed service.
- Pick the first command according to the symptom.
- Roll back without making the failure worse.
Before you start: Read your logs with journalctl, A firewall with ufw, Your first website with Apache.
Contents
- A funnel, not a maze
- Breathe and don't break anything more
- Is the server reachable?
- Is there space and memory left?
- Which services are down?
- What do the logs say?
- Do the network and firewall let traffic through?
- Does the website answer?
- Disk or memory exhausted
- Roll back
- Write down what happened
- The table: symptom, first command
- If it fails
- To finish
- Sources
Server down: what do you do first? Follow a troubleshooting checklist, a list of checks in a fixed order, from the widest to the most precise. Ten steps are enough. Each one rules out a family of failures, and the first one that answers "no" tells you where to dig. A failure isn't a defeat: it is the server asking for your attention.
A funnel, not a maze
We start with the widest questions, because they cost almost nothing and rule out a lot: is the machine even switched on? It would be a shame to dig through an Apache log if the network cable is unplugged. At each step the question narrows, down to a single command that points at the cause.
To make this concrete, let's follow a simulated failure. On mon-serveur, the site exemple.fr is served by Apache, as in Your first website with Apache. You changed a setting, restarted Apache, and it answers with an error:
sudo systemctl restart apache2Job for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details.The site is down. Before chaining commands, the first step asks for none.
1Breathe and don't break anything more
Before any command, note the time and what changed recently, and don't restart anything at random. A restart wipes the machine's memory, and with it some of the clues. It can also turn a small failure into a big one. Three notes are enough:
- The time the failure started, or of your first alert.
- What changed just before: an update, a setting you edited, a new user.
- Who is affected: only you, or all the visitors.
Two commands find what changed. The first reads the package install log, /var/log/dpkg.log, and keeps the last three lines. The second sorts the settings in /etc from newest to oldest. Between the two, the vertical bar | is a pipe: it passes what the left command prints to the right one, here tail -n 3 or head -n 5, which keep only the last or first lines.
date
sudo grep -E ' (install|upgrade) ' /var/log/dpkg.log | tail -n 3
ls -lt /etc | head -n 5Thu Nov 5 08:01:14 UTC 2026
2026-11-05 08:01:09 install libncursesw6:amd64 <none> 6.5+20250216-2
2026-11-05 08:01:10 install nano:amd64 <none> 8.4-1+deb13u1
2026-11-05 08:01:10 install libgpm2:amd64 <none> 1.20.7-11+b2
total 236
drwxr-xr-x 2 root root 4096 Nov 5 08:01 apache2
-rw-r--r-- 1 root root 4399 Nov 5 08:01 ld.so.cache
drwxr-xr-x 1 root root 4096 Nov 5 08:01 alternatives
-rw-r--r-- 1 root root 172 Nov 5 08:01 hostsEach line of the log starts with a date and a time, then the word install or upgrade and the package name. In ls -lt, the date comes just before the name. In our test, these dates don't point at anything useful: only the method matters. On your machine, compare them with the time of the failure: a setting dated ten minutes earlier is a suspect. Restarting at random is like shaking the TV and hoping it picks up the signal. What remains is finding out whether the machine still answers.
2Is the server reachable?
Two tests tell you whether the machine answers: ping for the network, then ssh for the front door. The ping command sends small messages and waits for their echo, like sonar. Run it from your computer, with -c 3 to stop after three messages.
ping -c 3 203.0.113.10PING 203.0.113.10 (203.0.113.10) 56(84) bytes of data.
--- 203.0.113.10 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2049msHere, "0 received" means no echo came back. A healthy machine answers three times, with "0% packet loss". But some hosting providers block ping, so silence alone proves nothing. Try the connection itself, as in Your first SSH connection.
ssh alice@203.0.113.10ssh: connect to host 203.0.113.10 port 22: Connection timed outThe message tells you where the failure is:
- You are connected: the machine is alive, and everything else happens on it.
Connection timed out: nothing reaches the server. It is off, the network is cut, or a firewall swallows the messages.Connection refused: the machine answers, but nobody is listening on port 22. The SSH service is stopped.
In the last two cases, you can no longer get in over SSH. Use your hosting provider's remote console instead: a virtual screen and keyboard, reachable from the provider's website. It works even when the server's network is down. If SSH answers, you are inside the machine: let's look at what it has in its belly.
3Is there space and memory left?
A full disk or exhausted memory is enough to bring down services that aren't broken at all. Four commands give you the picture in ten seconds:
- Space:
df -hlists the disks and how full they are. Look at theUse%column, and beware of values close to 100%. - Index cards:
df -icounts inodes, the cards that describe each file. A disk can run out of cards while it still has space, for example with millions of tiny files. - Memory:
free -hshows the RAM. Read theavailablecolumn: it is what is really still free. - Load:
uptimeprints three numbers, the average load over 1, 5 and 15 minutes. Compare them with the number of processor cores, given bynproc. Far above it, the machine is overloaded.
Suppose your data lives in /donnees. Here is a full disk. This one is a small test space in memory, and yours will have a different name:
df -h /donneesFilesystem Size Used Avail Use% Mounted on
tmpfs 2.0M 2.0M 0 100% /donneesThe Avail column is at zero and Use% at 100%: every write fails with the message No space left on device. To find what takes the space, du adds up the file sizes, and sort -h orders them from smallest to biggest.
sudo du -ah /donnees | sort -h | tail -n 31.0M /donnees/export.sql
1.0M /donnees/gros.log
2.0M /donneesThe culprits are at the bottom of the list, here two files of 1 MB. If df -i reports 100% in IUse%, look instead for the folder that holds millions of small files. Space and memory are fine? Then a program may simply be stopped.
4Which services are down?
The systemctl --failed command lists the services that systemd tried to start and couldn't. A service is a program that runs all the time, like Apache. The --plain and --no-legend options remove the column title line and the little dot that marks each service, to keep only the useful lines.
systemctl --failed --plain --no-legendapache2.service loaded failed failed The Apache HTTP ServerEach line is a service that went down. When all is well, the command prints nothing at all: silence is the good news. Now let's question the culprit, with sudo so we also see its last log lines.
sudo systemctl status apache2 --no-pager Loaded: loaded (/usr/lib/systemd/system/apache2.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Thu 2026-11-05 07:48:59 UTC; 20ms ago
Duration: 17.284s
Invocation: 27ea0799a7d349bc8266e4b4f5d80903
Docs: https://httpd.apache.org/docs/2.4/
Process: 153 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)
Mem peak: 1.5M
CPU: 15ms
Nov 05 07:48:59 mon-serveur systemd[1]: Starting apache2.service - The Apache HTTP Server...
Nov 05 07:48:59 mon-serveur apachectl[155]: AH00526: Syntax error on line 226 of /etc/apache2/apache2.conf:
Nov 05 07:48:59 mon-serveur apachectl[155]: Invalid command 'ServerNaem', perhaps misspelled or defined by a module not included in the server configuration
Nov 05 07:48:59 mon-serveur systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Nov 05 07:48:59 mon-serveur systemd[1]: apache2.service: Failed with result 'exit-code'.
Nov 05 07:48:59 mon-serveur systemd[1]: Failed to start apache2.service - The Apache HTTP Server.The Active: failed line confirms the failure, and the dated lines at the bottom already give the cause: Invalid command 'ServerNaem', a typo for ServerName. Your dates and numbers will differ. Here the excerpt is enough, but that isn't always the case: you then need to read the whole journal.
5What do the logs say?
To read the logs of a failure, journalctl gives three angles, detailed in Read your logs with journalctl. The first keeps the errors since the machine last booted (-p err -b): it says what.
sudo journalctl -p err -b --no-pagerNov 05 07:48:59 mon-serveur systemd[1]: Failed to start apache2.service - The Apache HTTP Server.The line announces the failure, without the reason. The second angle reads a single service over a short period (-u and --since): it says why. Here are Apache's last four lines.
sudo journalctl -u apache2 --since "10 minutes ago" --no-pager | tail -n 4Nov 05 07:51:04 mon-serveur apachectl[600]: Invalid command 'ServerNaem', perhaps misspelled or defined by a module not included in the server configuration
Nov 05 07:51:04 mon-serveur systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Nov 05 07:51:04 mon-serveur systemd[1]: apache2.service: Failed with result 'exit-code'.
Nov 05 07:51:04 mon-serveur systemd[1]: Failed to start apache2.service - The Apache HTTP Server.The journal confirms the typo. The third angle, -b -1, reads the previous boot: useful when the machine crashed and then restarted by itself. These three commands point at the service. What remains is checking that the network lets it be reached.
6Do the network and firewall let traffic through?
Three commands answer: ss shows what is listening, ufw what the firewall allows, dig what the domain name answers. A reminder from The words: IP, DNS, domain, port, firewall: a port is a numbered door of the machine, and a program listens at it, waiting for visitors. First let's see who is listening.
sudo ss -tlnpState Recv-Q Send-Q Local Address:Port Peer Address:PortProcessThe options -t, -l, -n and -p ask for the ports where a program is listening (over TCP, the protocol of the web), with their number rather than their name, and the program involved. Only the header shows: nobody is listening on port 80, so Apache isn't there. When all is well, one line appears under the header for each listening program.
If a program is listening but visitors don't get through, look at the firewall. The ufw from A firewall with ufw must allow ports 80 and 443 for a website. Finally, dig asks the domain name for its address.
sudo ufw status
dig +short exemple.frThe first command lists the rules: with no rule for 80 and 443, visitors see a timeout. The second should print 203.0.113.10, your server's address. Another address, or nothing, sends you back to the chapter on Domain name and DNS. Here, everything is in order except Apache: let's look at the site itself.
7Does the website answer?
The curl -I command asks your site for its response header, the first line it sends back, which carries a code. The code 200 means "all is well". Test from the server itself, with -sS to hide the progress bar while keeping the errors.
curl -sSI http://localhostcurl: (7) Failed to connect to localhost port 80 after 0 ms: Could not connect to serverThe error code (7) means nobody answers: consistent with the empty ss. Before restarting Apache, one habit saves a lot of trouble: check the syntax of its settings files with apachectl configtest.
sudo apachectl configtestAH00526: Syntax error on line 226 of /etc/apache2/apache2.conf:
Invalid command 'ServerNaem', perhaps misspelled or defined by a module not included in the server configurationThe test gives the file, the line and the mistake. Fix ServerNaem to ServerName with sudo nano /etc/apache2/apache2.conf. nano is a text editor in the terminal: move with the arrows, fix the word, then Ctrl + O and Enter to save, Ctrl + X to quit (details in The terminal for beginners). Then test again.
sudo apachectl configtest
sudo systemctl restart apache2
curl -sSI http://localhost | head -n 1Syntax OK
HTTP/1.1 200 OKThe site answers, and sudo ss -tlnp shows Apache's line on port 80 again. If the code isn't 200, read the end of the site's error log, which is not the systemd one:
sudo tail -n 20 /var/log/apache2/error.logCodes that start with 5 signal an error on the server side, and those that start with 4 a refused request or a missing page. But the cause isn't always a setting: the disk or memory may be exhausted.
8Disk or memory exhausted
When df, free or uptime rang the alarm in step 3, the failure comes from a resource that ran out, not from a setting. Three situations stand out:
- A slow disk, or one swamped with reads: everything drags without any service going down. Diagnose a saturated disk teaches you to see who is using it.
- A failing disk: read errors, unreadable files. Software RAID and SMART checks shows how to watch its health.
- Full memory: the kernel, the core of the system, then kills a program to save the machine. A trace of it stays in the journal.
sudo journalctl -k | grep -i "out of memory"The command looks for that trace in the kernel messages (-k). If it prints nothing, memory isn't the cause. Otherwise it prints an "Out of memory: Killed process" line followed by the number and name of the program that was killed. To be warned before the failure rather than after, Simple monitoring sets up an alert. If the cause stays hidden or time is short, there is one last resort.
9Roll back
When the cause hides or time is short, put back the last version that worked. There are two cases, depending on what you lost:
- A file you edited: keep a copy before every change, with
sudo cp /etc/apache2/apache2.conf /etc/apache2/apache2.conf.bak. To go back, copy the.bakover it. - Lost or damaged data: get it back from a backup, like the ones in Encrypted backups with restic.
With restic, you first list the backups, which it calls snapshots. Here the repository is /sauvegardes: replace it with the path to yours, for example /home/alice/depot-restic if you followed that article. Before answering, restic asks for the repository password, the one chosen when you created it in Encrypted backups with restic: type it (nothing shows while you type) then Enter. Expect this question with every restic command in this step.
sudo restic -r /sauvegardes snapshotsenter password for repository:
repository c1443eb0 opened (version 2, compression level auto)
ID Time Host Tags Paths Size
-----------------------------------------------------------------------
0b782240 2026-11-05 07:51:30 mon-serveur /srv/site 17 B
-----------------------------------------------------------------------
1 snapshotsEach line is a point you can return to. Restore into a folder next to it, never over it: you check first, you replace afterwards.
sudo restic -r /sauvegardes restore latest --target /tmp/restauration --include /srv/site/index.html
sudo cat /tmp/restauration/srv/site/index.htmlenter password for repository:
repository c1443eb0 opened (version 2, compression level auto)
restoring snapshot 0b782240 of [/srv/site] at 2026-11-05 07:51:30.076496234 +0000 UTC by root@mon-serveur to /tmp/restauration
Summary: Restored 3 / 1 files/dirs (17 B / 17 B) in 0:00
<h1>Bonjour</h1>The IDs and times will be your own. The content shows: you can copy the file back into place with sudo cp. One last step remains, which everyone skips and which you will be glad of later.
10Write down what happened
A short write-up, written while you still remember, keeps you from falling into the same failure again. It is called a post-mortem: a review written after the failure, which looks for the cause and not for someone to blame. Five lines are enough:
- Start and end: the times you noted in step 1.
- Symptom: what the visitors saw.
- Cause: here, a typo in
apache2.conf. - Fix: the corrected line, then restarting Apache.
- What changes: here, running
apachectl configtestbefore every restart.
The last line counts the most: a failure that improves nothing will happen again. To keep all this at hand, here is the full list of the ten steps, to print or open next to your terminal.
The table: symptom, first command
When the symptom is clear, this table saves you from going through the whole funnel again: it gives the first command, then where to look next.
| Symptom | First command | Where to look next |
|---|---|---|
| Nothing answers (no ping, no SSH) | the hosting provider's remote console | the machine's state in the customer area; Software RAID and SMART checks if the disk is suspect |
SSH: Connection refused | sudo systemctl status ssh through the console | sudo ufw status, then journalctl -u ssh |
| The server answers but everything drags | uptime | free -h, df -h, then Diagnose a saturated disk |
No space left on device | df -h and df -i | sudo du -ah /donnees | sort -h | tail |
| A service won't start | sudo systemctl status name | sudo journalctl -u name --since "10 minutes ago" |
| The site won't open | curl -sSI http://localhost | sudo apachectl configtest, then error.log |
| The domain name leads nowhere | dig +short exemple.fr | Domain name and DNS, then sudo ufw status |
| A program vanished without a sound | sudo journalctl -k | grep -i "out of memory" | free -h, then Simple monitoring |
| It worked yesterday, not today | ls -lt /etc | head -n 5 | the /var/log/dpkg.log log, then the backup (step 9) |
If it fails
Two messages come back when you use the journal without the right permissions:
Hint: You are currently not seeing messages from other users and the system.sudois missing in front ofjournalctl. Run the same command with it.systemctl statuswithout log lines. Same cause: withoutsudo, alice only sees the status information, not the dated lines at the bottom.
To finish
This is the last step of the course, and here you are with a server that can tell you what is happening to it, and someone to listen. It will break again one day, like every server in the world, and on that day you will have a list, a cup of tea and all the time you need. Fair winds to mon-serveur, and to you.