Harden a service with systemd
How do you harden a service with systemd? A few lines added to its file forbid it from touching what isn't its business. You add them one by one to a practice service, and you measure the gain.
- Course
- Tutorial 34 of 35
- Time
- 40 min to read
- Tested with
- Debian 13.7 (trixie), systemd 257
What you will be able to do
- Say what a service can do by default, and why that is a risk.
- Measure a service's exposure with systemd-analyze security.
- Add the locks ProtectSystem, ReadWritePaths, ProtectHome, PrivateTmp and NoNewPrivileges, then check their effect.
- Read the 226/NAMESPACE error and roll back without breaking the service.
Before you start: Users, sudo and file permissions, Read your logs with journalctl, Simple monitoring.
Contents
What a service can do by default
A service started by systemd can read and write almost the whole server, as long as nobody takes that right away. A service is a program that Debian starts and watches for you, as in Simple monitoring. Many run as root, the account with every right, covered in Users, sudo and file permissions.
The danger comes the day an attacker takes control of that program. They inherit all its rights, and a service that only needed one folder opens the whole server to them. Picture an employee who holds the master key to the building while they only open their own office: if it gets stolen, the whole building is open.
systemd can hand out an office key instead of the master key. In the service's file, you add lines, called directives. Each one takes away a right. Here are the five we add.
| Directive | What it changes for the service |
|---|---|
ProtectSystem=strict | Almost the whole disk becomes read-only |
ReadWritePaths= | Lists the folders where writing is still allowed |
ProtectHome=yes | Home folders become invisible |
PrivateTmp=yes | The service gets its own empty /tmp folder |
NoNewPrivileges=yes | A program started by the service can no longer gain rights |
To see these locks at work without risking your real server, we first build a practice service.
1Install a practice service
The practice service is called bonjour, and it tries four things before reporting which ones are allowed. It writes in its own folder, writes in /etc, looks at alice's home folder and reads a file placed in /tmp. A fifth marker prints the state of NoNewPrivileges.
Read the script before installing it
#!/bin/sh
# Service d'essai : il tente quatre gestes et dit lesquels sont permis.
essai() {
nom=$1
shift
if "$@" >/dev/null 2>&1; then
echo "permis : $nom"
else
echo "refuse : $nom"
fi
}
essai "ecrire dans /var/lib/bonjour" touch /var/lib/bonjour/passage
essai "ecrire dans /etc" touch /etc/bonjour-essai
essai "lire le dossier /home/alice" ls /home/alice
essai "lire /tmp/note-d-alice" cat /tmp/note-d-alice
grep NoNewPrivs /proc/self/status
rm -f /etc/bonjour-essai
Its service file is four lines long. Type=oneshot says the program runs once, then stops.
[Unit]
Description=Service d'essai bonjour
[Service]
Type=oneshot
ExecStart=/usr/local/bin/bonjour.sh
Download the two files, create the service's folder and drop the secret note in /tmp. Then ask systemd to reread its files with daemon-reload, and start the service.
U=https://anime-sanctuary.net/tutoriels/durcir-service-systemd/files
sudo curl -fsSo /usr/local/bin/bonjour.sh $U/bonjour.sh
sudo chmod 755 /usr/local/bin/bonjour.sh
sudo curl -fsSo /etc/systemd/system/bonjour.service $U/bonjour.service
sudo mkdir -p /var/lib/bonjour
echo "secret" > /tmp/note-d-alice
sudo systemctl daemon-reload
sudo systemctl start bonjourThe script wrote its report in the journal, the notebook where Debian records what each program says. The journalctl command reads it. Four options come with it.
-u bonjour: keep only this service.-I: keep only its last run.-o cat: show only the message, without the date or the machine name.--no-pager: show everything at once, with no page to scroll.
sudo journalctl -u bonjour -I --no-pager -o catStarting bonjour.service - Service d'essai bonjour...
permis : ecrire dans /var/lib/bonjour
permis : ecrire dans /etc
permis : lire le dossier /home/alice
permis : lire /tmp/note-d-alice
NoNewPrivs: 0
bonjour.service: Deactivated successfully.
Finished bonjour.service - Service d'essai bonjour.All four actions are allowed, even writing in /etc, and NoNewPrivs: 0 says the service can still gain rights. That is the starting point of most services. Next we give it a score, to measure the improvement afterwards.
2Measure its exposure
The systemd-analyze security command scores a service from 0 to 10: the higher the score, the less locked down it is. It compares its settings with a list of known locks, and each missing lock adds points. We keep only the final sentence with grep -o, which prints only the part starting at Overall.
systemd-analyze security bonjour.service --no-pager | grep -o "Overall.*"Overall exposure level for bonjour.service: 9.6 UNSAFE :-{The score is 9.6, and the word after it rates it: UNSAFE for the worst, then EXPOSED, MEDIUM and OK going down. Without the filter, the command prints one line per lock, and the crosses mark the missing ones. This score doesn't say the service is vulnerable: it only says it would be better off more locked in. Let's add the first lock.
3Make the system read-only
The ProtectSystem=strict directive makes the whole disk read-only from the service's point of view, except for a few technical folders. We don't edit the service file itself: an update could overwrite it. Instead we add a companion file next to it, a drop-in, stored in a folder called bonjour.service.d.
sudo mkdir -p /etc/systemd/system/bonjour.service.d
sudo tee /etc/systemd/system/bonjour.service.d/durcir.conf <<'CONF'
[Service]
ProtectSystem=strict
CONF
sudo systemctl daemon-reload
sudo systemctl start bonjour
sudo journalctl -u bonjour -I --no-pager -o cat[Service]
ProtectSystem=strict
Starting bonjour.service - Service d'essai bonjour...
refuse : ecrire dans /var/lib/bonjour
refuse : ecrire dans /etc
permis : lire le dossier /home/alice
permis : lire /tmp/note-d-alice
NoNewPrivs: 0
bonjour.service: Deactivated successfully.
Finished bonjour.service - Service d'essai bonjour.Writing in /etc is now refused, which was the goal. But the service can no longer write in its own folder either, which would make it unusable. The ReadWritePaths= directive reopens writing for the folders you name, and only those.
sudo tee /etc/systemd/system/bonjour.service.d/durcir.conf <<'CONF'
[Service]
ProtectSystem=strict
ReadWritePaths=/var/lib/bonjour
CONF
sudo systemctl daemon-reload
sudo systemctl start bonjour
sudo journalctl -u bonjour -I --no-pager -o cat
systemd-analyze security bonjour.service --no-pager | grep -o "Overall.*"[Service]
ProtectSystem=strict
ReadWritePaths=/var/lib/bonjour
Starting bonjour.service - Service d'essai bonjour...
permis : ecrire dans /var/lib/bonjour
refuse : ecrire dans /etc
permis : lire le dossier /home/alice
permis : lire /tmp/note-d-alice
NoNewPrivs: 0
bonjour.service: Deactivated successfully.
Finished bonjour.service - Service d'essai bonjour.
Overall exposure level for bonjour.service: 9.2 UNSAFE :-{The service writes at home and nowhere else, and the score goes from 9.6 to 9.2. It drops only a little, because this lock is just one of many possible ones. What remains is to close what the service can still see: home folders and /tmp.
4Hide /home, isolate /tmp, block extra rights
Three more lines hide the home folders, isolate /tmp and block extra rights. Here is the complete file: its first two directives are the ones from step 3.
[Service]
ProtectSystem=strict
ReadWritePaths=/var/lib/bonjour
ProtectHome=yes
PrivateTmp=yes
NoNewPrivileges=yes
ProtectHome=yes:/homebecomes empty for the service, and alice and her files disappear.PrivateTmp=yes: the service gets a fresh/tmp, emptied when it stops. It no longer sees anyone else's.NoNewPrivileges=yes: the service and the programs it starts can no longer gain rights, even by opening a privileged program.
U=https://anime-sanctuary.net/tutoriels/durcir-service-systemd/files
sudo curl -fsSo /etc/systemd/system/bonjour.service.d/durcir.conf $U/durcir.conf
sudo systemctl daemon-reload
sudo systemctl start bonjour
sudo journalctl -u bonjour -I --no-pager -o cat
systemd-analyze security bonjour.service --no-pager | grep -o "Overall.*"Starting bonjour.service - Service d'essai bonjour...
permis : ecrire dans /var/lib/bonjour
refuse : ecrire dans /etc
refuse : lire le dossier /home/alice
refuse : lire /tmp/note-d-alice
NoNewPrivs: 1
bonjour.service: Deactivated successfully.
Finished bonjour.service - Service d'essai bonjour.
Overall exposure level for bonjour.service: 8.7 EXPOSED :-(The three refusals are there and NoNewPrivs is 1: the locks hold. The score stays at 8.7, though, because the service still runs as root.
5Bring the score down
The score really drops when the service leaves the root account. Two lines change the service's identity. The others close what it doesn't use.
DynamicUser=yes: systemd creates a temporary account with no rights when the service starts, and deletes it when it stops.StateDirectory=bonjour: systemd creates/var/lib/bonjourand makes it writable for that account. Here it replacesReadWritePaths=.
The other lines, about twenty, forbid what our service doesn't need: touching the kernel (the core of Linux), the clock, devices (disks, USB sticks), or opening a network connection. No need to read them one by one: the file is in the block below.
Read the durcir-complet.conf file
[Service]
DynamicUser=yes
StateDirectory=bonjour
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
NoNewPrivileges=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
ProtectProc=invisible
ProcSubset=pid
RestrictSUIDSGID=yes
RestrictRealtime=yes
RestrictNamespaces=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
CapabilityBoundingSet=
SystemCallFilter=@system-service
SystemCallArchitectures=native
RestrictAddressFamilies=AF_UNIX
UMask=0077
U=https://anime-sanctuary.net/tutoriels/durcir-service-systemd/files
sudo curl -fsSo /etc/systemd/system/bonjour.service.d/durcir.conf $U/durcir-complet.conf
sudo systemctl daemon-reload
sudo systemctl start bonjour
sudo journalctl -u bonjour -I --no-pager -o cat
systemd-analyze security bonjour.service --no-pager | grep -o "Overall.*"Starting bonjour.service - Service d'essai bonjour...
permis : ecrire dans /var/lib/bonjour
refuse : ecrire dans /etc
refuse : lire le dossier /home/alice
refuse : lire /tmp/note-d-alice
NoNewPrivs: 1
bonjour.service: Deactivated successfully.
Finished bonjour.service - Service d'essai bonjour.
Overall exposure level for bonjour.service: 1.2 OK :-)The service does exactly the same job, and the score falls to 1.2, rated OK. Here is the road traveled.
| Step | Locks added | Score |
|---|---|---|
| 2 | None | 9.6 UNSAFE |
| 3 | ProtectSystem and ReadWritePaths | 9.2 UNSAFE |
| 4 | Plus ProtectHome, PrivateTmp, NoNewPrivileges | 8.7 EXPOSED |
| 5 | Temporary account and about twenty locks | 1.2 OK |
If a hardening step breaks your service, you need to know how to remove it quickly and cleanly.
6Roll back and clean up
To remove hardening, delete the drop-in folder and ask systemd to reread its files. The original service file never moved, so you only need to take away what was added.
sudo rm -r /etc/systemd/system/bonjour.service.d
sudo systemctl daemon-reload
sudo systemctl start bonjour
sudo journalctl -u bonjour -I --no-pager -o catStarting bonjour.service - Service d'essai bonjour...
permis : ecrire dans /var/lib/bonjour
permis : ecrire dans /etc
permis : lire le dossier /home/alice
permis : lire /tmp/note-d-alice
NoNewPrivs: 0
bonjour.service: Deactivated successfully.
Finished bonjour.service - Service d'essai bonjour.All four actions are allowed again: you are back at the start. Now delete the practice service and its files.
sudo rm /etc/systemd/system/bonjour.service /usr/local/bin/bonjour.sh /tmp/note-d-alice
sudo rm -rf /var/lib/bonjour /var/lib/private/bonjour
sudo systemctl daemon-reloadYour machine is back to its earlier state. For a real service, the method stays the same: one lock at a time, with the journal in front of you.
If it fails
status=226/NAMESPACE: a folder named inReadWritePaths=doesn't exist. Create it withsudo mkdir -p, or hand it over toStateDirectory=.Read-only file system: the service wants to write in a folder that is now closed. Add that folder toReadWritePaths=.- The score doesn't move: a directive is misspelled, and systemd ignores it without stopping the service. The
systemd-analyze verify bonjour.servicecommand flags it.
Here is what the journal prints for the first error, when the /var/lib/bonjour folder is missing.
sudo journalctl -u bonjour -I --no-pager -o catStarting bonjour.service - Service d'essai bonjour...
bonjour.service: Failed to set up mount namespacing: /var/lib/bonjour: No such file or directory
bonjour.service: Failed at step NAMESPACE spawning /usr/local/bin/bonjour.sh: No such file or directory
bonjour.service: Main process exited, code=exited, status=226/NAMESPACE
bonjour.service: Failed with result 'exit-code'.
Failed to start bonjour.service - Service d'essai bonjour.And here is how a typo is flagged, here ProtectSytem instead of ProtectSystem.
systemd-analyze verify bonjour.service/etc/systemd/system/bonjour.service.d/durcir.conf:2: Unknown key 'ProtectSytem' in section [Service], ignoring.Going further
Try a lock without writing a file
The systemd-run command starts a program as a temporary service, and -p gives it a directive. That way you test a lock in one line. The first write in /etc succeeds, the last one is blocked.
sudo systemd-run -q --wait --pipe touch /etc/essai
ls /etc/essai
sudo rm /etc/essai
sudo systemd-run -q --wait --pipe -p ProtectSystem=strict touch /etc/essai/etc/essai
touch: cannot touch '/etc/essai': Read-only file systemHarden a real service
Run systemd-analyze security without a service name: it scores all the running services. Pick one, show its detailed table with systemd-analyze security followed by its name, then add one lock at a time in a drop-in. After each lock, restart the service and read its journal. If an error appears, remove the last lock, as in step 6.
systemd-analyze security --no-pagerA service that writes its logs or its data outside /var/lib needs those folders in ReadWritePaths=. A service that talks to others through files in /tmp breaks with PrivateTmp=yes.
ProtectSystem=true and ProtectSystem=full
These are less strict versions. With true, only /usr and the boot folders become read-only. With full, /etc joins them. strict covers the whole disk, hence the need for ReadWritePaths=.
What's next
You can now lock a service in and measure the result. Your server is monitored, backed up and hardened. It can still go down, though. The next article, When things break: the checklist, gathers the checklist to follow on that day. It comes last because it reuses everything you have set up.