Anycast a service across two VPS edges

Anycast means announcing the same prefix from more than one location and letting the internet's own path selection decide which one a given user reaches. There is no load balancer, no health-check service in the middle, and no DNS trickery. The routing table is the load balancer.

That also describes the failure mode. If a node stops serving but keeps announcing, BGP will keep sending it traffic, because BGP has no idea whether your application is healthy. Every design decision below exists to close that gap.

This assumes you already have a working single-location session. If you do not, start with the BGP session walkthrough, and make sure the prefix is authorised first via RPKI and IRR.

Addresses below are documentation examples: AS64500, 203.0.113.0/24, 2001:db8:100::/48. Replace them with your own resources.

When anycast is the right tool

Anycast suits stateless services where any node can answer any request:

• authoritative DNS
• a static or heavily cached HTTP endpoint
• NTP
• a health or status endpoint
• TURN and other UDP services that renegotiate cheaply

It suits stateful services badly. A TCP connection is pinned to whichever node the route pointed at when it opened. If the path changes mid-connection, and paths do change, the connection lands on a node with no knowledge of it and resets. Long-lived uploads, websockets, database protocols, and anything with a session in local memory will produce sporadic resets that are miserable to debug.

The honest test: if you cannot lose a node mid-request and have the client simply retry, do not anycast the service. Use per-location names, or a proxy tier in front of a stateful backend.

1. One prefix, one origin, two locations

You announce the same prefix from both edges, from the same ASN. That means the RPKI side needs nothing extra: a single ROA authorising AS64500 to originate 203.0.113.0/24 covers both announcements, because both originate from the same ASN.

Two nodes is the right number to start with. Pick locations that are genuinely far apart, because two edges in the same metro give you resilience without meaningful latency benefit. London and Ashburn, or Frankfurt and Fremont, are useful pairs. Both nodes must be deployed and serving before you announce from either.

2. Provision a session at each edge

Each VPS gets its own session with its own neighbour addresses. Register the ASN once in VPS Control; verification goes to the contact address published for the ASN. After that you attach a session to each VPS and authorise the same prefix on both.

Record the values separately, because mixing up which source address belongs to which node is the most common cause of a session that will not leave Active:

The announcement is identical on both. Everything else differs.

3. Put the service address on loopback

The anycast address lives on lo, not on the public interface. The interface address is what BGP peers with; the loopback address is what the world reaches.

Do this identically on both nodes:

sudo ip addr add 203.0.113.10/32 dev lo
sudo ip -6 addr add 2001:db8:100::10/128 dev lo

Make it persist. On a systemd-networkd host, create /etc/systemd/network/10-anycast.network:

[Match]
Name=lo

[Network]
Address=203.0.113.10/32
Address=2001:db8:100::10/128

Then bind your service to it, and confirm it answers locally on both nodes before any route exists:

curl --resolve example.test:443:203.0.113.10 https://example.test/healthz

If that fails on either node, fix it now. Announcing a prefix to a node that cannot serve it is precisely the outage you are trying to avoid.

4. Announce from both, then verify they differ

Bring up the second session only after the first is established and serving. Then confirm that the two are actually being selected by different parts of the internet, which is the only evidence anycast is working at all.

Looking glasses and public route collectors will show the prefix arriving via two distinct paths. From your own side, the check is simpler: ask each node how much traffic it is answering.

A quick way to see which node answered, without guessing, is to have the service report its own location:

curl -s https://example.test/healthz
{"node":"lhr-01","status":"ok"}

Query from several networks and you should see different nodes answer. If every probe lands on one node, the other is either not announcing, announcing with a longer path, or being filtered upstream. Check show route export on the quiet node first.

5. Withdraw automatically when the service fails

This is the part that makes anycast safe, and the part most guides omit.

The rule: BGP should announce the prefix only while the local service is actually answering. If the service dies, the route must be withdrawn so traffic moves to the other node.

In BIRD 2, express this by exporting the anycast route from a static protocol that you enable and disable based on health, rather than announcing unconditionally.

Define the route separately from the session:

protocol static anycast_v4 {
    ipv4;
    route 203.0.113.0/24 blackhole;
}

Then drive it from a health check. A minimal script that runs from a timer:

#!/bin/bash
set -uo pipefail

if curl -fsS --max-time 2 http://127.0.0.1:8080/healthz >/dev/null; then
    birdc enable anycast_v4 >/dev/null
else
    birdc disable anycast_v4 >/dev/null
fi

And a systemd timer to run it every few seconds:

[Unit]
Description=Anycast health gate

[Timer]
OnBootSec=30s
OnUnitActiveSec=5s

[Install]
WantedBy=timers.target

Three details matter more than the specific implementation:

• Check the service, not the machine. A node that responds to ping while its application is dead is the exact case anycast needs to handle.
• Fail closed. If the health check itself cannot run, withdraw. A node that cannot evaluate its own health should not be attracting traffic.
• Require a couple of consecutive failures before withdrawing, and a couple of successes before re-announcing. Flapping a route on every transient timeout is worse than either state, because each change ripples outward and disturbs paths globally.

6. Test the failure you are designing for

Do not ship this without pulling the plug on purpose, during a window you chose.

sudo systemctl stop my-service

Within a few seconds the health gate should disable the protocol, the route should be withdrawn, and requests should be answered by the surviving node. Watch it from outside, not from the node:

watch -n1 'curl -s --max-time 2 https://example.test/healthz'

You are looking for the reported node name to change, with few or no failed requests in between. Then start the service again and confirm the route returns.

If clients see a burst of errors rather than a clean shift, the usual causes are a health interval that is too slow, a check that tests the wrong thing, or a client with a long connection that had to time out first. The last one is a reminder of the stateless rule.

Diagnosing anycast

Checklist

• The service is genuinely stateless, or state lives outside the node.
• Both nodes serve correctly before either announces.
• One ROA covers the prefix; both nodes originate from the same ASN.
• The anycast address is on lo and survives reboot on both nodes.
• The route is gated on a real service health check, not on machine liveness.
• The gate fails closed and requires consecutive results before changing state.
• You have withdrawn a node on purpose and watched traffic move.
• You know how to disable both protocols and withdraw the prefix entirely.

Anycast is one of the few pieces of infrastructure that is genuinely simple once the routing is correct, and genuinely dangerous when nobody has tested the failure. Announce nothing you cannot serve, and withdraw the moment you cannot serve it.