Ship logs off the VPS with Loki before you need them

Logs stored only on the server describe every incident except the one where you lose the server. They are also unsearchable across machines, which stops mattering at exactly one host and starts mattering at two.

Keeping journald useful solves the local half: caps, rotation, and a disk that does not fill. This is the other half, getting the logs somewhere else while they are still worth reading.

Loki is a reasonable choice for a small deployment because it indexes labels rather than log contents. That keeps it cheap to run on modest hardware, and it puts all the design pressure on one decision: which labels you use.

The one rule that decides whether this works

Labels are for identifying streams. They are not for storing values.

Loki creates a separate stream for every unique combination of label values. A label with unbounded values, a request ID, a user ID, a full path, a trace ID, produces an unbounded number of streams, and that will bring down a small Loki server faster than any volume of log text.

Everything you filter on occasionally, rather than identify by, belongs in the log line and is searched at query time. Loki is designed for that, and grep-style filtering over a narrow stream is fast.

1. Deploy Loki on a separate VPS

Put the log server somewhere other than the machines it collects from. A log server that dies with the application it was watching has failed at its one job. A separate edge is better still, because it survives a location-level problem.

A minimal Compose file:

services:
  loki:
    image: grafana/loki:3.0.0
    command: -config.file=/etc/loki/config.yaml
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml:ro
      - loki-data:/loki
    ports:
      - "127.0.0.1:3100:3100"
    restart: unless-stopped

volumes:
  loki-data:

Note the bind to 127.0.0.1. Loki has no authentication of its own in this configuration, so it must not listen on a public address. Traffic from other hosts arrives over a private path, covered below.

A configuration that fits a small server:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 720h
  ingestion_rate_mb: 8
  ingestion_burst_size_mb: 16
  max_label_names_per_series: 15

compactor:
  working_directory: /loki/compactor
  delete_request_store: filesystem
  retention_enabled: true

Two settings there are doing real work. retention_period with retention_enabled is what actually deletes old data; without the compactor setting, retention is configured and never applied, and the disk fills anyway. The ingestion rate limits protect the server from one misbehaving host in a loop.

2. Do not expose it to the internet

Loki as configured has no authentication. Collectors should reach it over a private path, not over a public port.

WireGuard between the hosts is the straightforward answer, and you likely have it already: see the WireGuard guide if not. Point collectors at the tunnel address, and keep the public firewall closed:

sudo nft add rule inet filter input ip saddr 10.10.0.0/24 tcp dport 3100 accept

If you put Grafana or Loki behind a reverse proxy instead, put authentication on the proxy. An open Loki endpoint is both a data leak and an easy way for someone to fill your disk.

3. Collect journald with Alloy

Alloy is Grafana's collector and reads journald directly, which means you get every service on the box without configuring each one.

services:
  alloy:
    image: grafana/alloy:latest
    command:
      - run
      - /etc/alloy/config.alloy
    volumes:
      - ./config.alloy:/etc/alloy/config.alloy:ro
      - /var/log/journal:/var/log/journal:ro
      - /etc/machine-id:/etc/machine-id:ro
    restart: unless-stopped

The collector config, reading the journal and forwarding to Loki:

loki.source.journal "journal" {
  max_age       = "12h"
  relabel_rules = loki.relabel.journal.rules
  forward_to    = [loki.write.central.receiver]
  labels        = {
    job  = "systemd-journal",
    host = constants.hostname,
  }
}

loki.relabel "journal" {
  forward_to = []

  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label  = "unit"
  }

  rule {
    source_labels = ["__journal_priority_keyword"]
    target_label  = "level"
  }
}

loki.write "central" {
  endpoint {
    url = "http://10.10.0.1:3100/loki/api/v1/push"
  }
}

Only job, host, unit, and level become labels. Everything else stays in the line. That is four bounded dimensions, which a small Loki handles comfortably.

The read-only mounts of /var/log/journal and /etc/machine-id are both required; without the machine ID, the journal reader cannot resolve the local journal.

4. Query it

Every query starts by selecting a stream, then filters the text:

{host="web-01", unit="nginx.service"} |= "500"

{job="systemd-journal", level="err"}

Extract fields at query time rather than making them labels:

{unit="my-app.service"} | json | duration > 500

Count errors per service over time, which is the shape most alerts want:

sum by (unit) (rate({level="err"}[5m]))

Find the machine that suddenly went quiet, which is often the more useful signal:

sum by (host) (count_over_time({job="systemd-journal"}[10m]))

A host whose line count drops to zero has either stopped logging or stopped existing, and both are worth knowing.

5. Alert on patterns, not on lines

An alert per error line is a pager that everyone learns to ignore. Alert on rates and on absence.

groups:
  - name: logs
    rules:
      - alert: ErrorRateHigh
        expr: sum by (host, unit) (rate({level="err"}[5m])) > 1
        for: 10m
        annotations:
          summary: "Sustained errors on {{ $labels.unit }} at {{ $labels.host }}"

      - alert: HostStoppedLogging
        expr: sum by (host) (count_over_time({job="systemd-journal"}[15m])) == 0
        for: 15m
        annotations:
          summary: "No logs from {{ $labels.host }} in 15 minutes"

The second one catches the failure mode people forget: the collector dying, the tunnel dropping, or the host going away entirely. Without it, silence looks like health.

6. Size the disk before it sizes you

Log volume is the thing that surprises people. Measure rather than estimate:

journalctl --disk-usage
journalctl --since '24 hours ago' | wc -c

That second number, times your retention in days, times the number of hosts, is a rough floor for storage before compression. Loki compresses well, often several times over, but plan against the uncompressed figure and you will not be caught out.

Watch the log server itself:

df -h /var/lib/docker/volumes
du -sh /loki/chunks

If retention and volume do not fit the disk, the honest options are shorter retention, fewer hosts, or a plan with more storage. Do not solve it by turning off retention enforcement, which converts a capacity decision into an outage at an unpredictable date.

Diagnosing the pipeline

Checklist

• Loki runs on a different VPS from the hosts it collects.
• It does not listen on a public address, and collectors reach it privately.
• Labels are bounded: host, job, unit, level, and little else.
• Retention is set and the compactor is enabled to enforce it.
• Ingestion rate limits are configured.
• An alert fires when a host stops logging, not only when it errors.
• Disk usage measured against real volume, with headroom.
• You have queried the logs of a host you deliberately stopped, and found them.

The test of a logging system is not whether it collects logs. It is whether, an hour after a server is gone, you can still read what it said in its last minute.