Automatic Linux security updates without surprise reboots

Manual patching works until the person responsible is busy for three weeks. Automatic security updates close that gap, but automatic reboots need a clear maintenance policy.

Enable security updates

On Ubuntu or Debian:

sudo apt update
sudo apt install -y unattended-upgrades apt-listchanges
sudo dpkg-reconfigure -plow unattended-upgrades

Review /etc/apt/apt.conf.d/50unattended-upgrades. Keep allowed origins focused on the distribution's security repositories. Do not automatically pull every third-party repository unless you trust its release process and have tested upgrades.

Set a predictable schedule

APT's timers add random delay to spread load. Inspect them:

systemctl list-timers 'apt-*'
systemctl cat apt-daily-upgrade.timer

If you need a strict window, add an override rather than editing the packaged unit:

sudo systemctl edit apt-daily-upgrade.timer

[Timer]
OnCalendar=
OnCalendar=*-*-* 04:15:00
RandomizedDelaySec=15m

Saving the editor applies the override: systemctl edit reloads systemd for you. If you changed the file any other way, run sudo systemctl daemon-reload. Then confirm the next run:

systemctl list-timers apt-daily-upgrade.timer

Protect service availability

Before relying on reboot automation:

• Enable required services with systemd.
• Use restart policies for containers.
• Confirm filesystems mount without prompts.
• Ensure encrypted disks have a remote-unlock plan.
• Test one real reboot from the provider console.

An application launched from a forgotten terminal is not reboot-safe. The test reboot deserves a real rehearsal: on a staging VPS with the same image, trigger a dry run, install a harmless update, and reboot during the chosen window. Confirm that SSH, firewall, database, reverse proxy, workers, timers, and monitoring all return. Keep a snapshot before planned distribution upgrades; a release upgrade is a separate change with its own rehearsal.

Decide how reboots happen

Kernel and core-library updates may create /var/run/reboot-required. Check for the marker and the packages behind it:

test -f /var/run/reboot-required && cat /var/run/reboot-required.pkgs

For a single stateless web VPS, an automatic reboot during a quiet window can be reasonable. It is also the riskiest setting in this article: the server reboots itself with no one watching. Only enable it after you have completed the checklist in the previous section, including the test reboot from the provider console. If a service does not start on boot, a filesystem prompts during mount, or an encrypted disk has no working remote unlock, an unattended reboot leaves the machine down or locks you out until you reach the provider console. On database or clustered nodes, leave automatic reboot off: alert on the marker file instead, drain traffic, confirm replicas, and reboot one node at a time.

If the checklist is done and the rehearsal came back clean, create /etc/apt/apt.conf.d/52unattended-upgrades-local with:

Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "04:45";

A separate file keeps these settings out of 50unattended-upgrades, which a package upgrade can overwrite. Confirm APT picked them up:

apt-config dump | grep Automatic-Reboot

Monitor the updater

sudo unattended-upgrade --dry-run --debug
sudo journalctl -u unattended-upgrades --since yesterday
sudo tail -n 100 /var/log/unattended-upgrades/unattended-upgrades.log

These commands tell you what the updater did on a single host. For the signals worth alerting on, see "Make update state observable" below. After a reboot, verify application health externally rather than assuming “system is running” means “service is available.”

Check holds and repositories

Once automation is enabled, verify that no held package or repository problem is silently blocking updates:

apt-mark showhold
grep -RhE '^(deb |URIs:)' /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null
sudo apt-get update

The URIs: pattern covers deb822 .sources files, the default format on current Ubuntu. Watch the apt-get update output for EXPKEYSIG or NO_PUBKEY errors: an expired or missing signing key makes APT skip that repository, and unattended-upgrades will not tell you. A held kernel or expired repository key can make a server appear patched while updates are stuck. Treat the unattended-upgrades log as an operational signal.

Make update state observable

Expose three simple signals to monitoring: the timestamp of the last unattended-upgrades run, the presence of /var/run/reboot-required, and failed package units. Each is a one-line check:

stat -c %y /var/lib/apt/periodic/unattended-upgrades-stamp
test -f /var/run/reboot-required && cat /var/run/reboot-required.pkgs
systemctl --failed

A small cron job can push these into whatever your monitoring already scrapes, for example a node-exporter textfile:

#!/bin/sh
# /etc/cron.hourly/apt-state
dir=/var/lib/node_exporter/textfile_collector
mkdir -p "$dir"
{
  printf 'apt_reboot_required %d\n' "$(test -f /var/run/reboot-required && echo 1 || echo 0)"
  printf 'apt_last_unattended_run_seconds %d\n' "$(stat -c %Y /var/lib/apt/periodic/unattended-upgrades-stamp 2>/dev/null || echo 0)"
} > "$dir/apt-state.prom.tmp"
mv "$dir/apt-state.prom.tmp" "$dir/apt-state.prom"

Alert on failed package units, on a reboot-required marker that persists past the next maintenance window, and on any server that is online but has not installed security updates for 30 days: that should page or create a ticket.

When an update fails, preserve the log before retrying. Do not blind-run the repair on a production host: apt-get -f install resolves broken dependencies by installing or removing packages, and on a bad day the package it proposes to remove is your database or reverse proxy. Read the list of changes at the interactive prompt and only confirm if nothing critical is being removed.

sudo cp -a /var/log/unattended-upgrades /var/backups/unattended-upgrades-$(date +%F)
sudo dpkg --audit
sudo apt-get -f install

Automation does not eliminate maintenance. It turns routine security patches into a controlled default and leaves unusual upgrades for deliberate review.