Keep systemd journal logs useful without filling the VPS

journalctl is often the fastest route from “service is down” to the actual error. It becomes much less useful when logs vanish on reboot or consume the last free gigabyte.

Make logs persistent and bounded

The drop-in directory does not exist on a default install, so create it first:

sudo mkdir -p /etc/systemd/journald.conf.d

Then create /etc/systemd/journald.conf.d/retention.conf as root, for example with sudo nano /etc/systemd/journald.conf.d/retention.conf:

[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=2G
SystemMaxFileSize=100M
MaxRetentionSec=30day
Compress=yes
RateLimitIntervalSec=30s
RateLimitBurst=10000

SystemMaxUse caps journal storage. SystemKeepFree makes journald yield when the filesystem gets tight. Choose values appropriate to the disk; a 10 GB root volume should use smaller limits.

Before you restart journald, check what is already on disk with journalctl --disk-usage. If the existing journal is larger than 1G or holds entries older than 30 days, the restart prunes the excess immediately, and there is no undo. Export anything you still need first, for example journalctl --since '2026-01-01' > journal-archive.log, or raise the limits above before applying.

One more caveat for older distributions: on systemd versions older than 246 (Ubuntu 20.04 ships 245), restarting journald disconnects the stdout and stderr streams of services that are already running, and their output is silently lost until each service is restarted. Check systemctl --version; on an affected system, plan to restart those services afterwards or apply the change at the next maintenance reboot. Ubuntu 24.04 ships systemd 255 and is not affected.

Apply and inspect:

sudo systemctl restart systemd-journald
journalctl --disk-usage

Query by unit and time

journalctl -u nginx --since '30 minutes ago'
journalctl -u myapp -p warning..alert --since today
journalctl -b -1
journalctl --since '2026-06-10 14:00' --until '2026-06-10 14:15'

-b -1 shows the previous boot, which is valuable after a crash or reboot. Follow live output with journalctl -fu myapp.

Use structured fields where possible:

journalctl _SYSTEMD_UNIT=ssh.service _PID=1234 -o json-pretty

Vacuum an oversized journal

If the journal is already too large, vacuuming trims it. Be clear about what that means before you run it: vacuum permanently deletes the oldest archived entries, and there is no undo. If any of that history still matters, export it first, for example journalctl -u myapp --since '2026-05-01' > myapp-archive.log, or confirm it has already been forwarded off the host.

sudo journalctl --rotate
sudo journalctl --vacuum-size=750M

Do not delete files from /var/log/journal while journald is running. Vacuuming preserves journal structure.

Handle noisy services at the source

Rate limits protect the host, but they also mean messages may be dropped. Fix loops that log on every failed retry, reduce debug logging in production, and use application-level sampling for repeated events.

Keep critical evidence elsewhere

Local logs disappear with the VPS. Forward authentication, firewall, application error, and audit events to another host or a managed log service. Use authenticated, encrypted transport and set retention based on operational and legal needs.

Do not ship secrets. Scrub tokens, passwords, session cookies, and full request bodies before they enter any log system.

The rest of this section builds that pipeline with systemd's own tooling: systemd-journal-remote receives on a second host, systemd-journal-upload ships from the VPS, and mutual TLS keeps the transport encrypted and authenticated. The examples use logs.example.com for the receiver and 203.0.113.10 for the VPS; substitute your own names and addresses.

Set up the receiver

On the log host, install the receiver:

sudo apt update
sudo apt install -y systemd-journal-remote

Both ends authenticate with certificates from a small private CA. Generate everything in a working directory on the receiver. The server certificate's name must match the hostname the VPS will connect to:

mkdir ~/journal-certs && cd ~/journal-certs

openssl req -x509 -newkey rsa:4096 -nodes -days 3650 \
  -keyout ca-key.pem -out ca-cert.pem -subj '/CN=journal-ca'

openssl req -newkey rsa:4096 -nodes \
  -keyout server-key.pem -out server.csr -subj '/CN=logs.example.com'
openssl x509 -req -in server.csr -CA ca-cert.pem -CAkey ca-key.pem \
  -CAcreateserial -days 825 -out server-cert.pem \
  -extfile <(printf 'subjectAltName=DNS:logs.example.com')

openssl req -newkey rsa:4096 -nodes \
  -keyout client-key.pem -out client.csr -subj '/CN=vps1.example.com'
openssl x509 -req -in client.csr -CA ca-cert.pem -CAkey ca-key.pem \
  -CAcreateserial -days 825 -out client-cert.pem

ca-key.pem is the trust anchor for the whole pipeline. Anyone who copies it can mint client certificates your receiver will accept, so it stays on the receiver with tight permissions and never goes in the bundle you ship to the VPS.

Install the server pieces where the receiver can read them:

sudo mkdir -p /etc/journal-remote/certs
sudo cp ca-cert.pem server-cert.pem server-key.pem /etc/journal-remote/certs/
sudo chown systemd-journal-remote /etc/journal-remote/certs/server-key.pem
sudo chmod 600 /etc/journal-remote/certs/server-key.pem

Point the receiver at them in /etc/systemd/journal-remote.conf:

[Remote]
ServerKeyFile=/etc/journal-remote/certs/server-key.pem
ServerCertificateFile=/etc/journal-remote/certs/server-cert.pem
TrustedCertificateFile=/etc/journal-remote/certs/ca-cert.pem

Enabling the socket opens TCP 19532 on the receiver. TrustedCertificateFile means only clients presenting a certificate signed by your CA can write, but still restrict the port to your own machines at the firewall before you open it; a reachable log receiver is an inviting place to flood disks. An allow rule only restricts anything if the firewall is actually enforcing a default-deny policy, so confirm sudo ufw status reports Status: active first (and if you are enabling ufw now, add your SSH allow rule before ufw enable or you cut off your own session). Then:

sudo ufw allow from 203.0.113.10 to any port 19532 proto tcp
sudo systemctl enable --now systemd-journal-remote.socket

Received entries land in /var/log/journal/remote/, one journal file per sending host. The SystemMaxUse cap you configured on the VPS does not apply here on the receiver, so this directory grows without bound unless you prune it. Schedule the pruning; a root cron entry works (sudo crontab -e on the receiver):

0 3 * * * journalctl -D /var/log/journal/remote --vacuum-time=90d

Note that the vacuum permanently deletes entries older than 90 days; pick a horizon that matches how far back your incident reviews actually reach.

Ship from the VPS

Copy the client credentials from the receiver's working directory to the VPS, over SSH only:

scp ca-cert.pem client-cert.pem client-key.pem 203.0.113.10:

On the VPS, install the same package (it also provides the uploader) and move the credentials where the upload service can read them:

sudo apt update
sudo apt install -y systemd-journal-remote
sudo mkdir -p /etc/journal-upload
sudo mv ~/ca-cert.pem ~/client-cert.pem ~/client-key.pem /etc/journal-upload/
sudo chown systemd-journal-upload /etc/journal-upload/client-key.pem
sudo chmod 600 /etc/journal-upload/client-key.pem

Edit /etc/systemd/journal-upload.conf:

[Upload]
URL=https://logs.example.com:19532
ServerKeyFile=/etc/journal-upload/client-key.pem
ServerCertificateFile=/etc/journal-upload/client-cert.pem
TrustedCertificateFile=/etc/journal-upload/ca-cert.pem

The URL hostname must match the name in the server certificate, so make sure the VPS resolves logs.example.com to your receiver, via DNS or an /etc/hosts entry. Then start the uploader:

sudo systemctl enable --now systemd-journal-upload.service
systemctl status systemd-journal-upload --no-pager

Verify end to end

On the VPS, write a marker into the journal:

logger -t upload-test "shipping check $(hostname) $(date --iso-8601=seconds)"

On the receiver, read it back from the remote directory:

sudo journalctl -D /var/log/journal/remote -t upload-test -n 5

If nothing arrives, check systemctl status systemd-journal-remote on the receiver and journalctl -u systemd-journal-upload on the VPS. Certificate name mismatches and a blocked port 19532 are the usual causes.

Beyond this smoke test, rehearse the real workflow: generate a known application error, find it by unit and timestamp, then confirm it appears on the receiver. Logging is an incident tool, so exercise it before an incident.

systemd-journal-upload reads the journal with a cursor and saves its position in /var/lib/systemd/journal-upload/state. Test a network outage too: block the receiver briefly, generate events, then unblock it. Local logs retain the events until the uploader reconnects, and it resumes from the saved cursor with nothing lost. Keep a short local retention even when remote logging exists.

Give services useful identities

Prefer one systemd unit per application component so logs can be filtered cleanly. Use SyslogIdentifier= when a process writes generic output, and include a release or instance identifier in application messages:

journalctl -u myapp.service --since '10 minutes ago'

If a service logs JSON, keep it valid JSON on one line and query it with journalctl -o json. Do not mix unbounded stack traces and health checks at the same severity.

Check disk pressure from logs

Include journal usage in the same capacity dashboard as application data:

journalctl --disk-usage
du -sh /var/log/* 2>/dev/null | sort -h | tail

If a service produces a burst, fix the retry or debug setting before increasing retention. Increasing the cap makes the symptom last longer and can still consume the filesystem needed for databases and package updates.