Diagnose latency and packet loss without misreading the output

Most network reports are wrong, and they are wrong in the same way: someone runs a traceroute, sees red in the middle, and concludes that hop is broken. Almost always it is not.

Understanding why takes about five minutes and saves entire afternoons.

The one thing to understand first

traceroute and mtr work by sending packets with a low TTL and collecting the ICMP messages routers send back when the TTL expires. That gives you a list of hops on the way out.

Two consequences follow, and nearly every misdiagnosis comes from ignoring them.

Generating those ICMP replies is the router's lowest priority. Forwarding packets is done in hardware. Replying to an expired TTL is done by the control plane, which is comparatively slow and is usually rate-limited on purpose. A busy core router will deprioritise or drop your probes while forwarding real traffic through the same interface at line rate, without loss and without delay.

So a middle hop showing 40 percent loss, while every hop after it shows zero, is not losing your traffic. It is declining to answer you. Real loss propagates: if a hop genuinely drops packets, every hop beyond it shows loss too, because the actual traffic never arrived.

The path you see is the forward path only. Return traffic can and frequently does take a different route, through different networks. When a hop reports high latency, that number is the round trip for that probe, including a return path you cannot see from this side. An asymmetric route with congestion on the way back looks identical to a slow hop on the way out.

This is why a single traceroute from one side is not evidence. You need both directions.

Read mtr correctly

mtr is traceroute run continuously, which is what you want, because a single traceroute is a snapshot of a moment.

mtr -rwzbc 200 203.0.113.10

Twenty packets is noise. Two hundred is a sample.

Reading the report, in order:

1. Look at the final hop first. That is your destination and the only loss figure that describes the user's experience. If the last hop shows zero loss, there is no loss, whatever the middle looks like.
2. Loss that starts at a hop and continues to the end is real. Loss that appears at one hop and disappears afterwards is ICMP deprioritisation.
3. Compare Best and Wrst. A large gap with a normal average is jitter or queueing. A high Best means the floor itself is high, which is distance or a genuinely congested path, not a transient.
4. Watch for a latency step that persists. If latency jumps 80ms at one hop and stays there, the path crossed an ocean or took an unexpected detour. A jump that recovers at the next hop is that one router being slow to reply.

Sanity-check the floor against distance

Light in fibre travels roughly 200 km per millisecond, and real paths are not straight. A useful rule: about 1ms of round trip per 100 km, plus the equipment in between.

That gives you a floor to compare against:

If you measure close to the floor, the network is doing its job and no amount of complaining will improve it. Physics is not a support issue. If you measure well above the floor, something is worth investigating.

This is also the argument for picking a location near your users rather than near you. Choosing an edge is a latency decision more than anything else.

Test both directions

The single most useful thing you can do, and the step most reports skip.

From your machine to the VPS:

mtr -rwzbc 200 203.0.113.10

Then from the VPS back to your address, at the same time:

mtr -rwzbc 200 198.51.100.20

If your own address is behind NAT and not reachable, use a stable public target on your side of the problem, such as your ISP's own gateway or a well-connected host in the same city, and say clearly in the report which you used.

Run both simultaneously. A problem that appears in one direction only tells you and the network team exactly where to look, and rules out half the possible causes immediately.

Distinguish latency from throughput

They are different problems and the fix for one does nothing for the other.

For a quick reachability and round-trip measure:

ping -c 100 203.0.113.10 | tail -3

That gives min, average, max, and standard deviation. High standard deviation with a low average is jitter, which hurts real-time traffic and barely affects a file transfer.

For actual throughput, measure a transfer rather than guessing from latency:

iperf3 -s

iperf3 -c 203.0.113.10 -t 30 -P 4

Four parallel streams matter. A single TCP stream over a long path is limited by window size and loss recovery long before it is limited by the link, so a single-stream test on a transcontinental path measures TCP behaviour, not capacity.

If parallel streams reach the expected rate and a single stream does not, the link is fine and you are looking at TCP tuning or distance, not a network fault.

When ICMP is blocked entirely

Some networks drop ICMP outright, producing a traceroute full of asterisks that says nothing. Switch to TCP probes against a port that is actually open:

sudo mtr -rwzbc 200 -T -P 443 203.0.113.10

sudo traceroute -T -p 443 203.0.113.10

This also tests something closer to the truth, since it follows the same path and treatment as real traffic to that service rather than a protocol many networks handle differently.

To check whether a specific port is reachable at all, without inferring it from a traceroute:

nc -vz 203.0.113.10 443

Check the machine before blaming the network

Some of what looks like network trouble is the server being unable to answer promptly.

uptime
vmstat 1 5
ss -s

In vmstat, the st column is CPU steal, time your virtual CPU was ready to run but waiting. Sustained steal shows up to users as latency that no traceroute will explain. High run-queue length or heavy swap-in does the same thing. If the machine is short on CPU or memory for its workload, resize the plan to fit the workload rather than trying to tune around it.

Check whether the interface itself is dropping anything:

ip -s link show

Non-zero and increasing dropped or errors counters point at the host or its configuration, not at a transit provider three hops away.

Write a report someone can act on

A network team can act on evidence and can do nothing with an impression. Include:

• Source address and destination address, both specific.
• mtr output in both directions, 200 cycles, with -z so ASNs are visible.
• Timestamps in UTC, and whether the problem is constant or intermittent.
• What "bad" means in numbers: latency, loss percentage, or throughput, measured.
• Whether it affects one protocol only. Test IPv4 and IPv6 separately, because they frequently take different paths.

mtr -rwzbc 200 -4 203.0.113.10
mtr -rwzbc 200 -6 2001:db8:100::10

That last pair catches a whole class of problems that a v4-only test hides completely.

Reading the common patterns

Checklist

• The final hop is the number you judge by.
• At least 200 cycles before drawing a conclusion.
• Both directions measured, at the same time.
• IPv4 and IPv6 tested separately.
• Measured floor compared against distance before calling it slow.
• Throughput tested with parallel streams, not inferred from ping.
• Host checked for CPU steal and interface errors.
• Report contains addresses, timestamps, and numbers.

Most reported network problems are either physics, a busy router declining to answer, or a saturated host. Being able to tell which within five minutes is most of the skill.