Monitoring and auto-restarting systemd services: Restart=, OnFailure=, watchdogs, and when that's not enough

Most "the service is down" tickets have a boring cause: the process crashed at 03:12 and nobody had set Restart=; or it was running but no longer answering; or it had simply been off since last week's reboot because nobody ran enable. systemd can catch all three, but not with the defaults, and not without you thinking about what "healthy" means for that service.

Step 1: know what isn't running right now

# Units in failed state (crash, exit ≠ 0, start limit reached)
systemctl --failed
# Services that are enabled but NOT running (e.g. didn't come up after a reboot)
systemctl list-unit-files --type=service --state=enabled --no-legend | awk '$1 !~ /@\.service$/ {print $1}' \
  | while read -r u; do
      # oneshots and units whose Condition isn't met (cloud-init on bare metal) are supposed to be inactive
      [ "$(systemctl show "$u" -p Type --value)" = oneshot ] && continue
      [ "$(systemctl show "$u" -p ConditionResult --value)" = no ] && continue
      # finished cleanly (Result=success, e.g. cloud-init-main) is not a failure; a crash gives exit-code/signal
      [ "$(systemctl show "$u" -p Result --value)" = success ] && ! systemctl is-active --quiet "$u" && continue
      systemctl is-active --quiet "$u" || echo "enabled-but-inactive: $u"
    done
# How often has a service restarted since boot?
systemctl show nginx -p NRestarts
# Why did it fail?
journalctl -u nginx -b -p warning --no-pager | tail -20

That second list is the surprising one: units that are "enabled" but inactive. Oneshots, template units and units whose Condition… isn't met (cloud-init on a physical machine) belong there; a web server doesn't: which is why the loop filters those three out.

Step 2: set Restart= properly (with a drop-in, not in the unit itself)

Never edit /lib/systemd/system/*.service: a package update overwrites it. A drop-in in /etc/systemd/system/<unit>.service.d/ survives everything.

sudo systemctl edit nginx

[Unit]
# Start limit: default 5 attempts in 10 s, then failed and NOTHING more. Too tight for
# a service waiting on a database. Wider window.
StartLimitIntervalSec=300
StartLimitBurst=10

[Service]
# on-failure: on crash or exit ≠ 0. always: also on clean exit 0 (for daemons that think they're "done").
Restart=on-failure
RestartSec=5s

Mind the section: StartLimitIntervalSec and StartLimitBurst belong in [Unit] (since systemd 230); older examples put them in [Service], which warns and is ignored on newer versions.

Then check what systemd actually uses:

sudo systemctl daemon-reload
systemctl show nginx -p Restart -p RestartUSec -p StartLimitBurst -p StartLimitIntervalUSec

For services with dependencies (an app that needs a database): After=postgresql.service in [Unit], and a connection retry in the app itself. Requires= sounds logical but makes your app stop along with the database when you restart it: usually not what you want.

Step 3: OnFailure=, an alert to your phone, built in

systemd can start another unit when a service fails. One generic notify unit for all services:

sudo tee /etc/systemd/system/notify-failed@.service >/dev/null <<'EOF'
[Unit]
Description=Push failure of %i to ntfy

[Service]
Type=oneshot
# %i = the name of the failed unit (via OnFailure=notify-failed@%n.service)
ExecStart=/bin/sh -c 'journalctl -u %i -n 15 --no-pager -o cat | curl -s -H "Title: FAILED %i on $(hostname -s)" -H "Priority: high" --data-binary @- https://ntfy.example.be/services >/dev/null'
EOF

# Attach to a service via drop-in
sudo systemctl edit nginx

[Unit]
OnFailure=notify-failed@%n.service

%n is the full unit name (nginx.service); it becomes the instance of the notify unit. Test with a throwaway unit:

sudo systemd-run --unit=crashtest -p OnFailure=notify-failed@crashtest.service /bin/false
# → push with the last 15 journal lines of crashtest

Note: OnFailure only fires when the unit really fails, i.e. after hitting the start limit, not on every individual crash that Restart= absorbs. That's exactly right: you don't want a push per restart, you do want one when restarting no longer helps.

Step 4: hanging services, the watchdog

Restart= reacts to a process that stops. A process stuck in a deadlock, all threads busy, doesn't stop. For services supporting systemd's sd_notify (nginx doesn't; many Go/Rust/Python daemons do, and anything talking via systemd-notify), there's WatchdogSec=:

[Service]
Type=notify
WatchdogSec=30s
# If the process doesn't send WATCHDOG=1 every 30 s, it gets SIGABRT and systemd restarts it
Restart=on-watchdog

If your service doesn't support that, build the watchdog outside the service: a timer that does a real request and restarts on failure.

sudo tee /etc/systemd/system/healthcheck-web.service >/dev/null <<'EOF'
[Unit]
Description=HTTP healthcheck for the web app; restart on failure
[Service]
Type=oneshot
ExecStart=/bin/sh -c 'curl -fsS -m 5 -o /dev/null http://127.0.0.1:8080/healthz || { echo "healthz failed, restarting"; systemctl restart webapp.service; }'
EOF
sudo tee /etc/systemd/system/healthcheck-web.timer >/dev/null <<'EOF'
[Unit]
Description=Run web healthcheck every minute
[Timer]
OnBootSec=2min
OnUnitActiveSec=1min
[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now healthcheck-web.timer

Two rules for such an external watchdog: the healthcheck must do something real (a query, not just "200 OK"), and the restart must be logged and counted, otherwise you mask a memory leak as "the occasional restart".

Step 5: the cron script that closes the gaps

What systemd itself doesn't report: enabled-but-inactive units, restart loops that stay just under the start limit, and services that didn't come back after a reboot.

sudo tee /usr/local/sbin/service-check.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
NTFY="https://ntfy.example.be/services"
HOST=$(hostname -s)
STATE=/var/lib/service-check.state
touch "$STATE"; MSG=()

# 1. Failed units
f=$(systemctl --failed --no-legend | awk '{print $1}' | paste -sd, -)
[ -n "$f" ] && MSG+=("failed: $f")

# 2. Enabled but not active (templates, oneshots and unmet-Condition units excluded)
while read -r u; do
  [ "$(systemctl show "$u" -p Type --value)" = oneshot ] && continue
  [ "$(systemctl show "$u" -p ConditionResult --value)" = no ] && continue
  systemctl is-active --quiet "$u" && continue
  # Finished cleanly (Result=success) = boot job like cloud-init-main, not a failure.
  # A daemon that unexpectedly stops with exit 0 you catch with Restart=always (step 2).
  [ "$(systemctl show "$u" -p Result --value)" = success ] && continue
  MSG+=("enabled-but-inactive: $u ($(systemctl show "$u" -p Result --value))")
done < <(systemctl list-unit-files --type=service --state=enabled --no-legend | awk '$1 !~ /@\.service$/ {print $1}')

# 3. Restart loops: NRestarts up by ≥ 3 since the previous check (10 min)
while read -r u; do
  n=$(systemctl show "$u" -p NRestarts --value); [ "${n:-0}" -eq 0 ] && continue
  prev=$(grep "^$u " "$STATE" | awk '{print $2}'); prev=${prev:-0}
  [ $((n - prev)) -ge 3 ] && MSG+=("restart-loop: $u ($((n - prev)) restarts in 10 min, total $n)")
  sed -i "/^$u /d" "$STATE"; echo "$u $n" >> "$STATE"
done < <(systemctl list-units --type=service --state=running --no-legend | awk '{print $1}')

[ ${#MSG[@]} -gt 0 ] && printf '%s\n' "${MSG[@]}" | curl -s -H "Title: services@$HOST" -H "Priority: high" --data-binary @- "$NTFY" >/dev/null
EOF
sudo chmod 0755 /usr/local/sbin/service-check.sh
echo '*/10 * * * * root /usr/local/sbin/service-check.sh' | sudo tee /etc/cron.d/service-check

Pitfalls

• Restart=always on a oneshot. A backup script with Restart=always runs again forever. Restart= is for daemons; for jobs you use timers.
• A failing ExecStartPre counts towards the start limit. A mkdir failing because a mount is missing burns your ten attempts in seconds. Prefix the command with - (ExecStartPre=-/bin/mkdir …) if failure is acceptable.
• systemctl restart in a healthcheck without rate limiting. If the database is gone, you restart an app every minute that won't work anyway, and fill the logs. Count the restarts and stop after three: then it's an alert, not a fix.
• KillMode= and children. A service spawning worker processes (gunicorn, php-fpm): with Restart=, old workers sometimes linger if KillMode=process. The default (control-group) is almost always right.
• Forgetting daemon-reload. A drop-in without systemctl daemon-reload is a text file. systemctl show tells you what's really active.

What you still don't have

• History. NRestarts resets on every boot. "How often has this service restarted in the last 30 days" needs a log you keep yourself.
• Fleet view. Thirty servers, thirty --failed lists, thirty ntfy topics.
• Context. The service that crashed at 03:12 and the apt upgrade at 03:10 are in two logs; the correlation is manual.

How monsys does it

The monsys agent reads the systemd state of every host: failed units, enabled-but-inactive, NRestarts, restart events with timestamps, and the exit status. Restart loops and down services become deduplicated alerts with the journal lines and the host metrics of that moment next to them; the SLA engine computes an availability percentage per service over 5-minute buckets, so "99.7% last month" is a fact rather than an estimate. Restarting remains an action a human approves: via a signed Emergency Action Token, with the output of systemctl restart in the audit log.

FAQ

Does systemd restart a service automatically after a crash?

Only if Restart= is set (on-failure or always); the default is no. And even then systemd stops after StartLimitBurst attempts within StartLimitIntervalSec and marks the unit failed.

How do I see why a service failed?

systemctl status <unit> shows the last lines; journalctl -u <unit> -b -p warning the full context since boot. systemctl show <unit> -p Result -p ExecMainStatus gives the exit code and the reason (exit-code, signal, start-limit-hit, watchdog).

Does the watchdog work with every service?

Only with services that use Type=notify and periodically send WATCHDOG=1 via sd_notify. For other services you build an external healthcheck with a timer, as in step 4.