Day 16 - systemd Service Management

Systemd manages how services start, stop, and interact with the rest of the system. This lesson explains unit types, everyday commands, reading logs, writing and overriding unit files, dependency ordering, sandboxing options, and practical troubleshooting.

Prerequisites

• Day 1 through Day 15 completed
• A user with sudo on a system that uses systemd

Unit types and naming

A unit is a configuration file that describes something systemd can manage. Common types:

• .service long running or one shot processes
• .socket sockets that can trigger a service on demand
• .timer schedules that trigger a service (Day 15)
• .mount and .automount filesystem mounts
• .target a milestone or grouping of units
• .path path watchers that start a service on changes

Units are stored in multiple places. Later entries override earlier ones.

• /usr/lib/systemd/system/ or /lib/systemd/system/ packaged defaults
• /etc/systemd/system/ local admin customizations and drop ins
• ~/.config/systemd/user/ per user units

Show a unit file as systemd sees it:

systemctl cat ssh.service

List active units and installed unit files:

systemctl list-units --type=service
systemctl list-unit-files --type=service

Everyday management commands

# start, stop, restart, reload
sudo systemctl start nginx.service
sudo systemctl stop  nginx.service
sudo systemctl restart nginx.service
sudo systemctl reload nginx.service   # uses ExecReload if defined
 
# enable at boot, disable, and check state
sudo systemctl enable nginx.service
sudo systemctl disable nginx.service
systemctl is-enabled nginx.service
 
# status and logs
systemctl status nginx.service
journalctl -u nginx.service -e --no-pager
journalctl -u nginx.service --since "-2h"
 
# list dependencies
systemctl list-dependencies nginx.service

reload depends on the unit providing ExecReload=. If not present, restart may be required.

Targets (runlevel equivalents)

A target groups units to reach a state such as multi user or graphical. Common targets:

• multi-user.target non graphical multi user
• graphical.target desktops, depends on multi-user.target
• rescue.target single user maintenance mode
• default.target the target reached at boot (usually an alias to one of the above)

systemctl get-default
sudo systemctl set-default multi-user.target
# change state immediately (Caution: drops to console on desktops)
sudo systemctl isolate multi-user.target

Write a simple service

Create a managed service for a script at /usr/local/bin/hello.sh.

sudo install -m 0755 -d /usr/local/bin
echo -e '#!/usr/bin/env bash\necho "$(date +%F'"' '"'%T) hello"' | sudo tee /usr/local/bin/hello.sh >/dev/null
sudo chmod 0755 /usr/local/bin/hello.sh

Service unit:

# /etc/systemd/system/hello.service
[Unit]
Description=Hello logger
After=network.target
 
[Service]
Type=simple
ExecStart=/usr/local/bin/hello.sh
Restart=on-failure
RestartSec=5s
 
[Install]
WantedBy=multi-user.target

Enable and test:

sudo systemctl daemon-reload
sudo systemctl enable --now hello.service
systemctl status hello.service
journalctl -u hello.service -n 20 --no-pager

Type=simple is the default. Other types:

• oneshot for tasks that run then exit; pair with RemainAfterExit=yes
• forking for legacy daemons that background themselves
• notify when the service tells systemd it is ready (via libsystemd)
• exec starts the process directly and tracks it

Environment and configuration files

Pass variables inline or via a file.

[Service]
Environment=APP_MODE=prod LOG_LEVEL=info
EnvironmentFile=-/etc/default/hello

The dash before a path allows the unit to start even if the file is missing.

Runtime and persistent directories can be created automatically with safe permissions:

[Service]
User=www-data
Group=www-data
RuntimeDirectory=hello
StateDirectory=hello
CacheDirectory=hello
LogsDirectory=hello

Systemd creates these under /run, /var/lib, /var/cache, and /var/log with correct ownership.

Service ordering and dependencies

Control start order and requirements with these directives:

• Requires= hard dependency; failure stops this unit
• Wants= soft dependency; failure does not stop this unit
• Before= and After= order only; they do not imply dependency

Example:

[Unit]
Requires=network-online.target
After=network-online.target

To ensure the network is fully configured, use network-online.target provided by your network stack along with its corresponding wait service (for example, NetworkManager-wait-online.service).

Safer services with sandboxing

Apply built in hardening flags to reduce risk.

[Service]
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
MemoryDenyWriteExecute=true
RestrictSUIDSGID=true
CapabilityBoundingSet=~CAP_SYS_MODULE CAP_SYS_RAWIO
ReadWritePaths=/var/lib/hello

Not all services tolerate strict settings. Start with a few and expand after testing.

Overrides and drop ins

Avoid editing packaged unit files. Use drop in overrides so updates do not overwrite changes.

sudo systemctl edit nginx.service
# this opens an editor and saves to /etc/systemd/system/nginx.service.d/override.conf

An example override to add environment and restart policy:

[Service]
Environment=NGINX_WORKERS=4
Restart=on-failure
RestartSec=3s

Reload and check the merged unit:

sudo systemctl daemon-reload
systemctl cat nginx.service

Mask a unit to prevent any start:

sudo systemctl mask avahi-daemon.service
sudo systemctl unmask avahi-daemon.service

Socket and path activation (on demand)

A service can be started when a socket is hit or when a path changes.

Socket example:

# /etc/systemd/system/echo.socket
[Unit]
Description=Echo demo socket
 
[Socket]
ListenStream=127.0.0.1:9999
Accept=no
 
[Install]
WantedBy=sockets.target

# /etc/systemd/system/echo.service
[Unit]
Description=Echo demo service
 
[Service]
ExecStart=/usr/bin/socat - TCP4-LISTEN:9999,fork,reuseaddr

Enable the socket (it pulls in the service on demand):

sudo systemctl daemon-reload
sudo systemctl enable --now echo.socket
systemctl status echo.socket

Path example triggers on file changes:

# /etc/systemd/system/reindex.path
[Path]
PathChanged=/srv/data/index.trigger
 
# /etc/systemd/system/reindex.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/reindex.sh

Enable both .path and .service units.

Boot diagnostics and performance

Analyze boot timing and dependency issues.

systemd-analyze time
systemd-analyze blame | head
systemd-analyze critical-chain
 
# show why a unit is not running
systemctl status myapp.service
journalctl -u myapp.service -b --no-pager
systemctl list-dependencies --reverse myapp.service

Practical lab

1. Create hello.service as shown, enable it, confirm logs, and then harden it with NoNewPrivileges=true and PrivateTmp=true via an override.

2. Write a oneshot service rotate-logs.service that runs /usr/sbin/logrotate -f /etc/logrotate.conf. Add After=network.target and Type=oneshot. Hook it to a timer from Day 15 that runs daily at 03:00.

3. Add EnvironmentFile=/etc/default/hello and set LOG_LEVEL=debug. Print it from the script and confirm in the journal.

4. Use systemd-analyze blame and critical-chain to identify the slowest boot items. Record the top three.

5. Create a path activated unit that runs an indexer when /srv/data/index.trigger changes. Test by touching the file and watching the journal.

Troubleshooting

• Unit not found. Check the exact name and suffix. Run systemctl daemon-reload after adding files.
• Failed to start with exit status 203 or 200. The binary in ExecStart is missing or not executable. Verify path and mode.
• Service starts manually but not at boot. Confirm WantedBy= in [Install] is correct and that the unit is enabled. Inspect journalctl -b.
• Ordering issues. Add After= or Requires= appropriately, but avoid circular dependencies. Use systemctl list-dependencies.
• Permissions denied writing to /var or /run. Use StateDirectory= or RuntimeDirectory= to create managed directories with correct ownership, or loosen sandboxing flags cautiously.
• Environment file not read. Ensure EnvironmentFile= path exists and has KEY=value lines without quotes.

Next steps

Day 17 introduces firewall management. It compares uncomplicated firewall (ufw) and firewalld, shows how to open and list ports, add rich rules, make changes persistent, and validate with ss and nmap.