mtr Command Guide: Diagnosing Packet Loss and Latency

When a network path misbehaves - intermittent packet loss, latency spikes, or throughput that never matches the circuit - a single traceroute is often not enough: it sends a few probes per hop and exits, so it misses intermittent problems entirely. mtr (My Traceroute) combines traceroute and ping: it discovers the path and then keeps probing every hop continuously, accumulating hundreds of samples per hop. This guide explains how to run mtr, how to read its columns, and how to avoid the classic false alarm of mistaking ICMP rate-limiting for real loss.

How mtr Works

Like traceroute, mtr manipulates the TTL of probe packets so each router along the path sends back an ICMP Time Exceeded message; the difference is that mtr keeps sending probes to every hop simultaneously and updates live statistics. Most implementations default to ICMP probes, but TCP and UDP modes are available when ICMP is filtered.

$ mtr 8.8.8.8
$ mtr -n 8.8.8.8        # numeric, no DNS lookups
$ mtr --report 8.8.8.8  # send 10 rounds, print one report, exit
$ mtr --tcp -P 443 example.com
$ mtr --udp -P 53 1.1.1.1

Because every hop is probed at the same time rather than one after another, mtr produces a picture of the whole path's behaviour over the same interval — which is exactly what you need when a problem is intermittent. A conventional traceroute finishes in a second or two and reports whatever the network was doing in that second; mtr runs until you stop it and reports the distribution of what it saw.

Reading the Output Columns

Interactive mtr output shows one row per hop with the following columns:

• Loss%: percentage of probes that received no reply at this hop.
• Snt: probes sent to this hop so far - the more samples, the more trustworthy the loss figure.
• Last: round-trip time of the most recent probe.
• Avg: average RTT - the primary latency indicator per hop.
• Best / Wrst: fastest and slowest RTT seen; a large gap means jitter or congestion events.
• StDev: RTT standard deviation; a high value signals an unstable, jittery hop.

Three of these columns deserve more attention than they usually get. Snt is the sample size, and a loss percentage computed from ten probes is nearly worthless — leave mtr running for at least a minute, or use --report-cycles 100, before you quote a number to anyone. Best is the cleanest estimator of true propagation delay for a hop, because it reflects the packets that experienced no queueing; comparing Best at hop 1 with Best at the destination gives you the physical floor for the path. StDev is the column people ignore and then spend a week chasing: a hop with a modest average but a huge standard deviation is one where traffic is being buffered and released in bursts, which shows up to users as stutter and video calls that degrade but never fail outright.

The One Rule That Prevents False Alarms

If loss appears at an intermediate hop (say 30-70% at hop 4) but every subsequent hop - including the final destination - shows 0% loss, that router is almost certainly rate-limiting or refusing the ICMP probes mtr uses, while still forwarding your real traffic normally. Many routers cap ICMP Time Exceeded replies to protect their control plane. The rule: loss at an intermediate hop only matters if the loss carries through to the final destination. Conversely, loss that begins at one hop and continues on every hop after it indicates a real problem at or beyond that point.

The reasoning behind the rule is worth internalising: the router generating the Time Exceeded message is a different code path from the router forwarding your packet. Control-plane policing affects the former and not the latter, so a router can drop 80% of the diagnostic replies it owes you while forwarding 100% of your data. Because downstream hops are measured independently, they reveal the truth — if the destination reports zero loss, the packets got there, and the intermediate figure was an artifact of the measurement method.

Patterns and What They Mean

• Loss starts at hop 1 (your gateway) and continues: the problem is inside your own site - Wi-Fi, cable, or your router. Test from a wired connection.
• Hops 1-2 clean, loss starts at hop 3-4 and continues to the end: the issue sits at your ISP's edge - exactly the report your provider needs.
• Avg latency jumps sharply at one hop and stays high afterward: that hop is congested or is a long-distance link; the added delay carries downstream.
• High StDev or a wide Best-Wrst gap at one hop: jitter or congestion events on that segment.
• Loss at the final hop only, with clean intermediate hops: usually the destination itself rate-limiting ICMP. Confirm with a TCP-based probe to a listening port before filing a ticket.
• Latency that climbs steadily across all hops: not a single bad hop, but a saturated uplink somewhere near the destination — or a path that is being shaped by a policy on your own router.

Choosing the Right Probe Type

ICMP is the default because it is simple and unauthenticated, but it is also the first thing firewalls drop and the first thing routers deprioritise. When the network you are testing filters it, switch to the protocol your application actually uses — the results are more representative anyway:

mtr --tcp -P 443 example.com      # TLS/HTTPS paths, passes most firewalls
mtr --tcp -P 80 example.com       # plain HTTP
mtr --udp -P 53 1.1.1.1           # DNS, useful when only UDP is allowed
mtr -4 example.com                # force IPv4
mtr -6 example.com                # force IPv6
mtr -a 203.0.113.10 example.com   # bind to a specific source address
mtr -s 1200 example.com           # probe size, to test MTU behaviour

Comparing an ICMP run with a TCP run on the same destination is one of the most productive five minutes in network troubleshooting. If both show end-to-end loss, the loss is real. If ICMP shows loss at the destination but TCP to port 443 is clean, the destination is policing ICMP and your users are fine. If TCP is worse than ICMP, something is shaping or inspecting that specific protocol — often a stateful firewall or a load balancer in front of the service.

Running mtr against a specific source address is the technique to remember when a host has several uplinks or several addresses in different subnets, since the path chosen by the router depends on what you send from. Use ip route get 8.8.8.8 first to see which source address the kernel would pick, then pin it explicitly to confirm.

Installing mtr and Permission Requirements

apt install mtr-tiny          # Debian/Ubuntu, minimal build
dnf install mtr               # RHEL/Fedora family
brew install mtr              # macOS via Homebrew

Most implementations use a setuid helper or raw sockets, so mtr usually needs root or a specific capability. On hardened systems where running as root is undesirable, grant the capability directly:

setcap cap_net_raw+ep /usr/bin/mtr
getcap /usr/bin/mtr

A related caveat on Linux: reverse DNS lookups for every hop can take several seconds each, so the first run appears to hang on the first hop. Use -n to suppress name resolution when you care about speed, and only enable it when hop identity actually matters.

Running mtr in Report Mode for Evidence

Interactive mode is for investigation; report mode is for documentation. A report is stable, copy-pasteable and can be attached to a ticket or emailed to an ISP, which is exactly what an escalation needs.

mtr --report --report-cycles 100 8.8.8.8
mtr --report-wide --report-cycles 200 example.com
mtr --csv --report 8.8.8.8 > mtr_8.8.8.8_$(date +%F).csv
mtr --json --report example.com > mtr.json

Run the report at the same time as, and from the same machine as, whatever the users are complaining about. A report from a healthy network proves nothing about a broken one. Two reports taken a few minutes apart, or from two sites, are far more persuasive than one, because they show whether the pattern is persistent and whether it is location-specific. Where the site publishes path quality to an internal monitoring system, the pattern of a lossy uplink is visible in IP SLA probes and tracked objects as well as in mtr.

Caveats: ECMP and Asymmetry

Internet paths frequently use Equal Cost Multi-Path (ECMP) routing: different probes can take different parallel paths to the same destination, so the per-hop statistics mix multiple routes. Repeat the test several times and, if possible, run the test in the reverse direction - a lossy path in one direction is not necessarily lossy in the other.

Asymmetry is the second trap. Routing is decided per direction, so the path from A to B and the path from B to A can share almost no hops. When a user reports that "the file server is slow", the delay may be entirely in the return path — the one carrying TCP acknowledgements — which mtr from the client cannot see. Run mtr from both ends, and if only one end is accessible, reason about the direction your probes actually travelled.

Validating the Result

Because mtr manipulates TTLs, it can produce false positives. Validate an end-to-end finding with plain tools: ping for continuous end-to-end loss/latency, and hping3 to test the same TCP or ICMP type and port you used in the mtr run:

$ ping -i 0.2 -c 100 8.8.8.8
$ hping3 -S -p 443 -c 100 example.com

If mtr shows end-to-end loss but a long ping run is clean, re-check the mtr parameters (ICMP vs TCP) before escalating.

Three more corroborating checks are cheap and often decisive:

• Large-packet test — ping -M do -s 1472 8.8.8.8 verifies that a full-MTU packet gets through. Loss that appears only near the MTU points at an MTU or PMTU discovery problem rather than congestion; see MTU, MSS and PMTUD troubleshooting.
• Interface counters — CRC errors, discards and interface resets on the local NIC or switch port are the counterpart to a clean-looking path. A quick ip -s link before and after a test run tells you whether your own hardware is dropping frames.
• TCP-level analysis — if the path looks clean but transfers stall, the loss may be happening at the application layer. Retransmissions and out-of-order segments in a capture distinguish link loss from endpoint problems; the method is covered in Wireshark TCP retransmission analysis.

When you want to establish whether an application is loss-sensitive before blaming the network, tc netem lets you inject a known amount of latency, loss and jitter on a lab host so you can compare the user-visible impact against the mtr numbers.

A Worked Example

Suppose users at one branch report slow file transfers. The first report looks like this:

$ mtr --report --report-cycles 100 --no-dns 10.10.10.10
Start: 2024-05-02T09:14:11+0000
HOST: branch-pc01            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.20.1          0.0%   100    0.6   0.7   0.4   3.2   0.3
  2.|-- 100.64.0.1            0.0%   100    8.1   9.0   7.9  22.4   2.1
  3.|-- 100.64.0.9           12.0%   100   10.2  11.4   9.8  88.0  12.7
  4.|-- 203.0.113.1           0.0%   100   11.0  11.6  10.0  94.2  13.1
  5.|-- 198.51.100.7          1.0%   100   12.4  13.0  11.2 120.4  15.8
  6.|-- 10.10.10.10           1.0%   100   12.9  13.4  11.5 131.0  16.2

Reading it properly: hop 1 is clean, so the local LAN is fine. Hop 3 shows 12% loss, but hops 4, 5 and 6 show 0-1%, and the destination itself reports 1% — so the 12% at hop 3 is that router limiting its own ICMP replies, not the path dropping traffic. However, every hop from 3 onward shows a large Wrst and a StDev above 12 ms, and the destination's worst case of 131 ms is 10× its best case. That combination — low average loss, high jitter, latency spikes that appear at the ISP edge — is buffering somewhere upstream, and the evidence to send your provider is hops 2 through 6 with their Wrst and StDev columns highlighted. A single 1% end-to-end loss figure would have been dismissed; the jitter makes it a real complaint.

Common Mistakes

• Quoting intermediate-hop loss as a problem. Check the destination first, always.
• Running for ten seconds. Small sample sizes produce noise; use --report-cycles of 100 or more.
• Testing from the wrong place. A clean result from a datacentre says nothing about a branch on a different circuit.
• Assuming one direction. Paths are asymmetric; test the return direction too.
• Forgetting ECMP. Load-balanced paths blend statistics — repeat the run.
• Ignoring the parent network. If the loss begins at your own gateway, no amount of ISP escalation will fix it.

Quick Reference

mtr --report --report-cycles 100 --no-dns TARGET     # the go-to diagnostic report
mtr --tcp -P 443 TARGET                              # when ICMP is filtered
mtr -w --csv TARGET > report.csv                     # evidence for a ticket
mtr -4 -s 1472 TARGET                                # test full-size packets

Related reading: tcpdump filters for troubleshooting and Linux network troubleshooting with ss, netstat and tcpdump.

Original article: AWS re:Post: Diagnosing packet loss and latency with MTR