Overview: journald log management for Linux servers
systemd journald collects structured logs on most modern Linux systems, and left unchecked the journal can consume significant disk space. This guide targets sysadmins and engineers who need practical steps to inspect journal size, enforce retention and rate limits, reclaim space, and forward logs for production environments.
We focus on commands and configuration changes that are safe for production, with options for temporary cleanup and permanent policy, while keeping the approach vendor neutral for distributions using systemd.
Inspect current journal size and file usage
Start by measuring how much space the journal uses and which files are largest, so you know whether growth is caused by a runaway service or a policy mismatch. Useful commands report disk usage, per boot footprint, and top entries by size or age.
Example checks include journal disk usage and listing journal directories, run them as root or via sudo to see complete results.
sudo journalctl --disk-usage
sudo du -sh /var/log/journal /run/log/journal
sudo journalctl --list-boots
sudo journalctl --since "1 day ago" | wc -l
Configure retention and size limits
journald respects settings in /etc/systemd/journald.conf, where you can set effective limits to control retained data. Key settings let you bound total size, limit per database file, and choose to keep logs only in volatile memory if that fits your risk profile.
Edit the config and restart systemd journald to apply changes. Common options to consider include keeping a fixed maximum size and preventing infinite growth on root partitions.
- System level keys to set: SystemMaxUse, SystemKeepFree, SystemMaxFileSize, SystemMaxFiles, RuntimeMaxUse
- Behavior keys: Compress, ForwardToSyslog, MaxRetentionSec
sudo vim /etc/systemd/journald.conf
# example
SystemMaxUse=500M
SystemKeepFree=100M
Compress=yes
Set rate limits and runtime controls
Rate limiting prevents a noisy service from filling the journal in minutes. journald offers RateLimitInterval and RateLimitBurst to throttle logging volume from individual clients. Use these to protect disk and downstream log collectors.
For runtime adjustments use systemctl to set temporary limits or tune logging at the service unit level so a misbehaving unit cannot overwhelm the system journal. Audit unit log levels and reduce debug chatter where appropriate.
Vacuum and compress journal files
When you need to reclaim space immediately, use journalctl vacuum operations. Vacuuming by size or by time reduces retained journals safely, without altering live logging. Compressing old journals also reduces footprint if Compress is not enabled.
Use these commands to reclaim space, and verify free space after each operation. Run vacuum steps during low load windows if your system has heavy I O.
sudo journalctl --vacuum-size=200M
sudo journalctl --vacuum-time=7d
sudo journalctl --verify
Forwarding and centralized logging options
For production systems, long term retention and deeper analysis belong in a centralized log store. Forwarding journald to syslog or a collector reduces local retention needs and provides better search and alerting capability.
Typical forwarding strategies include relaying to rsyslog or syslog ng, using the systemd journal gateway, or shipping to a collector such as Fluentd, Logstash, or an agent that sends to hosted platforms. Match retention policies between the host and the central store to avoid duplication.
- Forwarding methods: ForwardToSyslog, TCP or TLS agents, journal gateway API
- Collector targets: Elastic stack, Graylog, Splunk, hosted SIEMs
Rotate and integrate journald with logrotate
journald keeps its own files and does not rely on traditional logrotate, but integrating with logrotate may be useful for legacy logs. For journald itself, prefer vacuuming and the config options over external rotation attempts.
If you have services that write rotated text logs, continue managing those with logrotate and ensure rotation scripts do not conflict with journald forwarding or permissions. Keep an eye on file descriptors and journal ownership after any rotation operations.
Monitoring and alerting for journal growth
Set up monitoring on journal directory sizes and inode use to detect runaway growth early. Alerts should trigger on total journal size crossing thresholds or when SystemKeepFree is breached, so operations teams can respond before services fail.
Implement simple checks via Nagios, Prometheus node exporter, or custom scripts. Monitor related signals such as sudden increase in log rate, new units generating many errors, or kernel messages that might indicate systemic issues.
FAQs
Below are common operational questions about journald log management with concise answers for day to day administration.
- Q: Will vacuuming delete current logs? A: Vacuum operations remove archived journals and will not truncate the live, in memory journal. They only affect retained files on disk.
- Q: Can I keep logs only in memory? A: Yes, set RuntimeMaxUse and disable persistent storage by removing /var/log/journal or setting Storage=volatile in the config, this keeps data off disk but increases post reboot data loss risk.
- Q: How to identify which service creates most logs? A: Use journalctl with the unit filter and count entries per unit, for example list entries per service and sort by frequency to find noisy units.
- Q: Should I compress journals? A: Enable Compress in journald.conf to reduce disk usage automatically. For already large archives use journalctl –vacuum-size after enabling compression to reclaim space.
Conclusion and recommended checklist
Managing journald growth is a combination of inspection, policy, and automation. Start by measuring current usage and identifying noisy services, then apply conservative retention and rate limits in /etc/systemd/journald.conf. Use vacuuming as a safe cleanup mechanism, and enable compression to reduce long term footprint. For production systems, forward logs to a central collector where retention and indexing are easier to manage.
Operationally, adopt a short checklist to keep journal growth predictable: monitor journal size in your observability stack, set SystemMaxUse and SystemKeepFree so the system remains stable, apply RateLimit settings to noisy units, and script periodic vacuum tasks if you cannot rely on a central collector immediately. Finally, document changes and test them on staging before rolling them into production so you avoid accidental loss of critical logs during incidents.