Quick Linux systemd troubleshooting overview
When a systemd unit fails on Linux, rapid diagnosis matters. This guide focuses on pragmatic systemd troubleshooting techniques sysadmins use every day, centering on journalctl and systemctl to find root causes, plus practical fixes you can apply at runtime or persistently.
Start by gathering the unit status and recent logs, then iterate through common culprits: unit file errors, dependency or socket issues, cgroup and resource constraints, and environment or permission problems. The examples here assume shell access with sudo privileges.
Inspect failed unit status and logs
Begin with systemctl to see high level status and recent failures. Use systemctl status unit to get the exit code and a short log excerpt. Then run journalctl for the full log context that often contains the real error string to act on.
Common commands to run immediately include:
sudo systemctl status nginx.service
sudo journalctl -u nginx.service --no-pager --since "10 minutes ago"
sudo journalctl -xe --no-pagerCommon unit file issues to check
Unit file syntax errors or incorrect paths are frequent causes of failed units. Check ExecStart and other directives for typos, missing binaries, or wrong quoting. Also confirm the Service Type is appropriate for the process model.
Quick checklist:
- Paths and executable permissions for ExecStart and ExecStartPre
- Correct Type, Restart, and User settings
- Environment variables and EnvironmentFile presence
Dependency and socket problems
systemd units declare dependencies with Wants, Requires, After, and Before. A required dependency that fails can pull the whole chain down. Use systemctl list-dependencies unit to view the tree and identify upstream failures.
Socket-activated services can fail silently if the socket unit is inactive or its permissions prevent activation. Inspect socket units separately and check for SELinux or AppArmor denials if sockets are created but cannot bind.
Cgroups and resource limit troubleshooting
systemd uses cgroups to enforce resource limits. A service killed by OOM or by reaching CPU, memory, or IO quotas will show relevant messages in the journal. Look for messages mentioning cgroup, memory, OOM, or killed by signal.
To inspect runtime limits, check systemctl show unit and examine MemoryLimit, CPUQuota, and TasksMax. Temporarily increasing limits at runtime can validate whether quotas are the cause before making persistent changes.
Runtime versus persistent fixes and editing unit files
Decide if the fix needs to be applied at runtime only, or persisted. For runtime changes use systemctl edit –runtime unit or systemctl set-property. For persistent edits use systemctl edit unit which creates drop-in files under /etc/systemd/system.
Never edit files under /lib/systemd/system directly. Use drop-ins to override directives safely, for example to change ExecStart or increase MemoryLimit. After persistent edits reload the daemon with systemctl daemon-reload.
Safe recovery steps and rescue targets
If a critical unit fails during boot or prevents normal operation, use rescue and emergency targets to recover. Boot with systemd.unit=rescue.target to get a minimal environment, or use emergency.target for single-user root shell without mounting most filesystems.
Steps for safe recovery include isolating the failed unit with systemctl isolate, disabling automatic restart loops with systemctl mask unit, and booting into a rescue target to adjust configs without dependencies interfering.
Examples: fixing a failed service
Example diagnosis for a failing nginx.service: check status, examine logs for permission denied or address already in use, then confirm the binary path and user permissions. Apply a runtime restart once a fix is identified.
Commands you might run during the example:
sudo systemctl status nginx.service
sudo journalctl -u nginx.service --no-pager -n 200
sudo ss -lptn | grep :80
sudo systemctl edit nginx.service # create drop-in if you need to change ExecStart
sudo systemctl daemon-reload
sudo systemctl restart nginx.servicePreventative measures and automation
Reduce future incidents by adding monitoring and automated remediation. Use systemd watchdog settings, service health checks, and external monitoring that can execute remediation playbooks or restart units on failure. Keep unit files under version control to track changes.
Automation ideas:
- Add Restart=on-failure with a sensible RestartSec to avoid tight loops
- Use systemd drop-ins stored in config repository deployed via automation tools
- Log and alert on specific journal patterns using journalctl forwarded to your logging system
Conclusion and FAQs
Effective Linux systemd troubleshooting combines fast log inspection with methodical checks of unit files, dependencies, and resource constraints. Start with systemctl status and journalctl to capture the error context, then validate unit configuration, dependency trees, and cgroup limits. Use runtime edits to test fixes safely, then persist changes with drop-ins and daemon-reload. For production systems, prefer conservative Restart policies and implement monitoring that can detect repeated failures and surface root cause data. Keep unit files in version control to simplify rollbacks and audits. Practicing these steps reduces mean time to repair and prevents many common failures from escalating.
Q1: How do I view logs for a specific service?
A: Use journalctl -u unit –since “time” to view logs for that unit. Add –no-pager for full output.
Q2: Should I edit files under /lib/systemd/system?
A: No, create drop-ins with systemctl edit or place overrides in /etc/systemd/system to keep vendor files intact.
Q3: My service restarts in a loop after a config change, how to stop it?
A: Mask the unit with systemctl mask unit to prevent starts, then diagnose. You can also set RestartSec to add delay before restarts.
Q4: How to tell if cgroups killed my process?
A: Check the journal for OOM or cgroup messages and inspect MemoryLimit via systemctl show unit. Increase limits at runtime to validate.