Monitor a VPS with Prometheus Node Exporter

CPU graphs are reassuring, but the metrics that prevent incidents are usually disk space, memory pressure, filesystem errors, and sustained load. Prometheus Node Exporter exposes those Linux signals in a format Prometheus can scrape.

Install Node Exporter

One thing to know before you start the service: the exporter has no authentication, it listens on every interface, and anyone who can reach port 9100 can read your system metrics. Until you complete the firewall section below, that means the whole internet, so plan to do the next section immediately after this one.

On distributions that package it:

sudo apt update
sudo apt install -y prometheus-node-exporter
sudo systemctl enable --now prometheus-node-exporter

Confirm it listens on port 9100:

systemctl status prometheus-node-exporter
ss -lntp | grep 9100
curl -s http://127.0.0.1:9100/metrics | head

The metrics endpoint contains system data but no authentication. Do not expose it to the entire internet.

Restrict the metrics port

If Prometheus runs at 203.0.113.40, that should be the only source allowed to reach port 9100. An allow rule on its own restricts nothing: unless ufw is enabled with a default-deny incoming policy, the port stays open to the whole internet.

Enabling ufw with a default-deny policy drops every inbound connection you have not explicitly allowed, and that includes SSH. Add the OpenSSH rule before you enable the firewall, keep your current session open, and test a fresh SSH login from a second terminal after enabling. If SSH listens on a nonstandard port, allow that port instead of the OpenSSH profile. Locking yourself out is a real risk; the GetVPS console is the back door, but it's awkward.

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw allow from 203.0.113.40 to any port 9100 proto tcp
sudo ufw enable
sudo ufw status verbose

Better still, scrape over WireGuard or a private network and bind the exporter to that address so it never listens on a public interface. On Ubuntu the packaged exporter reads its flags from /etc/default/prometheus-node-exporter:

sudo sed -i 's|^ARGS=.*|ARGS="--web.listen-address=10.20.0.11:9100"|' /etc/default/prometheus-node-exporter
sudo systemctl restart prometheus-node-exporter

After the restart the exporter answers on http://10.20.0.11:9100/metrics only. Either way, verify from a machine that is not the Prometheus server that port 9100 is closed to everyone else. With the VPS at 198.51.100.20:

curl -sS --max-time 5 http://198.51.100.20:9100/metrics

The request should time out or be refused. If metrics come back, the restriction is not in place yet.

Run Prometheus as a service

You need a Prometheus server to scrape the exporter. If you already run one, skip to the next section. Ubuntu packages Prometheus (sudo apt install prometheus), but the release archive gives you a current version and a unit file you control, so that is the path shown here.

Create a dedicated user and keep configuration separate from the data directory:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
sudo install -d -o prometheus -g prometheus /var/lib/prometheus
sudo install -d -o root -g prometheus -m 750 /etc/prometheus

Download the release archive and check its integrity:

PROM_VERSION=3.5.0
cd /tmp
curl -LO https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
curl -LO https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/sha256sums.txt
sha256sum --check --ignore-missing sha256sums.txt

The check must print OK for the archive. If it prints anything else, stop, delete the file, and download it again; do not install binaries from an archive that fails the check. Once it passes, install the binaries and the starter configuration:

PROM_VERSION=3.5.0
cd /tmp
tar -xzf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
sudo install prometheus-${PROM_VERSION}.linux-amd64/prometheus /usr/local/bin/prometheus
sudo install prometheus-${PROM_VERSION}.linux-amd64/promtool /usr/local/bin/promtool
sudo install -o root -g prometheus -m 640 prometheus-${PROM_VERSION}.linux-amd64/prometheus.yml /etc/prometheus/prometheus.yml

Create the systemd unit:

sudo tee /etc/systemd/system/prometheus.service > /dev/null <<'EOF'
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --web.listen-address=127.0.0.1:9090
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/prometheus

[Install]
WantedBy=multi-user.target
EOF

Reload systemd, start the service, and confirm it is ready:

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
systemctl status prometheus
curl -s http://127.0.0.1:9090/-/ready

The readiness endpoint should answer Prometheus Server is Ready. The unit binds Prometheus to 127.0.0.1, which keeps it private; expose a dashboard through an authenticated reverse proxy rather than opening port 9090.

Add the scrape target

On the Prometheus server, add a job to the scrape_configs list in /etc/prometheus/prometheus.yml (the starter file already has one for Prometheus itself):

scrape_configs:
  - job_name: linux-vps
    scrape_interval: 30s
    static_configs:
      - targets:
          - 10.20.0.11:9100
        labels:
          environment: production
          region: fremont

Validate the file with promtool, then reload the service:

sudo promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

If you run Prometheus with --web.enable-lifecycle, curl -X POST http://127.0.0.1:9090/-/reload does the same job. Check the Targets page before building dashboards; a green target confirms collection. With Prometheus bound to 127.0.0.1, reach the UI at http://127.0.0.1:9090/targets through an SSH tunnel.

Four alerts worth having

Start with conditions that require action:

groups:
  - name: vps-basics
    rules:
      - alert: FilesystemNearlyFull
        expr: |
          100 * (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
          / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) > 85
        for: 15m
        labels: { severity: warning }
        annotations:
          summary: "Filesystem above 85% on {{ $labels.instance }}"

      - alert: HostOutOfMemory
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1
        for: 10m
        labels: { severity: warning }

      - alert: SustainedHighLoad
        expr: node_load15 / count without(cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1.5
        for: 20m
        labels: { severity: warning }

      - alert: NodeExporterDown
        expr: up{job="linux-vps"} == 0
        for: 5m
        labels: { severity: critical }

Rules do nothing until Prometheus loads them. Save the YAML above as /etc/prometheus/rules.yml, reference it from /etc/prometheus/prometheus.yml at the top level:

rule_files:
  - /etc/prometheus/rules.yml

Then validate both files and reload:

sudo promtool check rules /etc/prometheus/rules.yml
sudo promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

Firing alerts now show on the Alerts page of the Prometheus UI. To have them delivered anywhere (email, Slack, PagerDuty), you additionally need Alertmanager, configured via alerting: in prometheus.yml; the Prometheus docs cover that setup, and until it exists the alerts are visible but silent.

Adjust thresholds to the workload. A build runner can legitimately sit at high CPU; a mostly idle DNS server cannot.

Build a useful dashboard

The quickest start needs nothing new: open the Prometheus UI through an SSH tunnel (ssh -L 9090:127.0.0.1:9090 you@prometheus-host, then http://127.0.0.1:9090/graph) and try a query like 100 - avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100 for CPU busy percentage. For persistent dashboards, install Grafana and point its Prometheus data source at http://127.0.0.1:9090; its default Node Exporter dashboards cover most of what follows.

Graph CPU busy percentage, available memory, load per vCPU, filesystem usage, disk latency, network throughput, and TCP connection count. Label every chart with units and use the same time range.

Dashboards answer “what changed?” Alerts answer “do we need to act?” You need both, but alerts should stay few and actionable. A page that fires every day will be ignored on the day it matters.

Watch the TSDB's disk usage and adjust the retention flag if 30 days does not fit the disk. Back up alert rules and scrape configuration. Test one alert end to end, including the notification route and the person who owns the response.