Wireshark TCP: Retransmission vs Out-of-Order vs Loss - 夜莺博客

Wireshark TCP: Retransmission vs Out-of-Order vs Loss

A capture full of red “TCP Retransmission” lines is the most over-interpreted artefact in network troubleshooting. Those labels are Wireshark’s interpretation of a single capture, written from one point in the path. Packet loss, reordering and capture drops look similar on screen and require completely different fixes. Here is how to tell them apart using sequence numbers, ACK progress and SACK blocks - and how to write the finding up so it survives review.

Sequence numbers are the source of truth

TCP is a byte stream. Every data byte has a sequence number, and every analysis question reduces to the relationship between sequence number, segment length, the ACK number coming back, SACK blocks and timestamps.

Sender -> Receiver: Seq 1000 Len 1000
Sender -> Receiver: Seq 3000 Len 1000
Sender -> Receiver: Seq 2000 Len 1000

If the segment starting at 2000 arrives soon after the one at 3000, this is reordering: TCP carries on without retransmission, and the fix is path-level (ECMP hashing, link aggregation, multi-path tunnels), not a server tuning issue. If the sender emits a second copy of 2000 after duplicate ACKs or an RTO, you observed a retransmission - and the original may still have been lost, delayed, or merely invisible to your capture point.

What the Wireshark flags actually mean

  • tcp.analysis.retransmission - bytes were observed from the sender with a sequence range seen before. It does not prove loss.
  • tcp.analysis.fast_retransmission - triggered by duplicate ACKs or a large jump, before the RTO expires. Normal recovery, not necessarily a problem.
  • tcp.analysis.out_of_order - a later segment arrived before an earlier one, within the out-of-order RTT threshold.
  • tcp.analysis.duplicate_ack - the receiver repeats the same ACK; strong evidence of a gap in what the receiver has.
  • tcp.analysis.spurious_retransmission - the original data was already acknowledged; typically a delayed original segment.
  • tcp.analysis.lost_segment / “previous segment not captured” - frequently means your capture missed a packet, not the network.
tcp.analysis.flags
tcp.analysis.retransmission || tcp.analysis.fast_retransmission
tcp.flags.syn == 1 && tcp.analysis.retransmission

Why Wireshark has to guess

Wireshark reconstructs a TCP conversation from frames it saw, in the order it saw them. It has no idea whether a missing sequence range was dropped by a switch, delayed by a queue, reordered by a hash, or simply not offered to the capture interface. Every one of the flags above is therefore a hypothesis about the local observation, not a fact about the network. The only facts in the capture are: which frames arrived at this sensor, in which order, with which timestamps, and what the sender and receiver said to each other.

This is why "the capture proves packet loss" is a claim you should never make from one point. From a single sensor you can prove that a gap existed in what you saw; you cannot prove where the bytes went. The classic trap is the server-side capture on a hardworking host: the NIC, the kernel and the capture stack all compete for CPU, and under load the capture drops frames while the TCP stream itself is perfectly healthy.

Loss, reordering and capture drop: the decision tree

  1. Did the missing range ever appear, in any order? Yes, and it appeared later with a lower sequence number than a following segment = reordering. It never appeared at this sensor = loss at this sensor or upstream of it, or a capture drop.
  2. Did the sender send it twice? A second copy of the same range after duplicate ACKs or an RTO = retransmission. If the first copy was never seen, the gap is upstream of your sensor.
  3. Did the receiver SACK the later range? SACK blocks prove the receiver holds bytes above the gap. That localises the gap to the segment between sender and receiver, not to a capture artefact of a later segment.
  4. Is the gap one segment or many? A single missing segment with later data acknowledged is a loss event. A cliff of missing sequence numbers with no ACK progress often means the capture simply stopped taking packets for a while.
  5. Does the timing make sense? An RTO is roughly a round trip plus back-off; a fast retransmit follows three duplicate ACKs. If the "retransmission" is microseconds after the original, suspect an offload or mirroring artefact rather than a real timeout.

Fast retransmit, RTO and what the ACK stream says

Three duplicate ACKs followed by a retransmission is fast retransmit: the receiver saw later segments, so the missing one is a real gap in the stream, and the sender repaired it without waiting. A retransmission that follows a silence of one RTO or more is a timeout-driven retransmission, which usually means the receiver saw nothing at all above the gap - either the whole window was lost or the path stalled. The distinction matters: fast retransmit implies isolated loss with a healthy ACK path; RTO implies the ACK path itself was stalled, and points at congestion, a flapping link, or a queue collapse.

tcp.analysis.duplicate_ack
tcp.analysis.duplicate_ack_num == 3
tcp.analysis.fast_retransmission
tcp.time_delta > 0.2 && tcp.analysis.retransmission

Use tcp.time_delta and tcp.time_relative after enabling conversation timestamps, so the deltas are per-conversation rather than per-capture. Then look at the time-sequence graph (Statistics, then Flow Graph, or tcptrace): a staircase that recovers one step back is normal reordering repair; a flat line with a vertical cliff is a retransmission after loss.

Do not confuse buffer problems with loss

Zero-window and window-full events look nothing like retransmissions once you know the shape. A receiver advertising Win=0 is telling the sender to stop; a subsequent window update reopens the stream. You will see long gaps in the time-sequence graph with no red frames at all, and the sender's ACKs simply stop advancing. Fixes are about application read speed, socket buffer sizes and receiver CPU - not about the network path.

tcp.analysis.zero_window
tcp.analysis.window_full
tcp.window_size_value == 0

Options that change the analysis: SACK, timestamps and window scaling

Two options decide how much evidence you actually have. SACK-permitted, negotiated in the SYN and SYN-ACK, tells you the receiver can report discontinuous block ranges; without it, duplicate ACKs are your only clue that a gap exists, and reordering is much harder to distinguish from loss. TCP timestamps let Wireshark measure RTT per acknowledgement and expose tcp.analysis.ack_rtt, which converts a vague "looks slow" into a number you can trend. Window scaling, negotiated once at handshake, determines whether a zero-window event is a real buffer exhaustion or a measurement mistake.

tcp.options.sack_perm
tcp.options.timestamp
tcp.analysis.ack_rtt > 0.05
tcp.analysis.ack_rtt > 0.05 && tcp.len > 0

If a conversation was established before your capture started, Wireshark still labels the missing handshake context and timestamps may be absent - say so in the report rather than assuming the option was not offered.

Offload, mirroring and the artefacts they create

Large Send Offload (TSO/LSO) shows the capture a huge "sent" segment that the wire never carried; the NIC fragmented it after the capture point. Checksum offload shows bad checksums that are in fact computed later. Receive-side coalescing (LRO) can hide segments entirely. On a SPAN or mirror port, oversubscription drops frames under burst, and a busy mirror produces exactly the lost-segment pattern of a lossy network. Before you blame the path, confirm the capture's own health: the capture summary window reports drops, and on Linux tcpdump -i eth0 -w file.pcap prints "packets dropped by kernel" when the ring buffer overruns.

Five conclusions that are almost always wrong

  • "There is 5% packet loss." You measured 5% of the frames at one sensor. Loss toward the server and loss toward the client are different numbers and both are usually needed.
  • "The server is retransmitting, so the network is dropping." The server may be retransmitting because the client's ACKs were lost, because the server's own transmit queue overflowed, or because the capture point sits downstream of a load balancer rewriting the stream.
  • "Wireshark says out-of-order, so a link is faulty." Reordering along a multi-path fabric with per-packet hashing is normal and TCP repairs it silently.
  • "Lost segment means loss." It usually means the capture missed a frame, especially at high rates or on a mirror port.
  • "Fast retransmissions are the problem." Fast retransmit is the recovery mechanism working as designed; the loss that triggered it is the finding.

The analysis order that produces defensible conclusions

  1. Isolate the TCP conversation and confirm where the capture was taken and in which direction.
  2. Check capture health: dropped-packet counters, ring buffer overruns, CPU pressure, and whether offload reshaped segment sizes.
  3. Start at the first unexpected ACK, not the first red frame. Write down the receiver’s next-expected byte.
  4. Read the SACK blocks: they prove the receiver got later data while missing a lower range.
  5. Decide whether the missing range arrived late (reordering) or only as a sender copy (retransmission, or a capture artefact).
  6. If loss location matters, capture at a second point and compare - the difference between two observers bounds where the packet disappeared.

Backlog and buffer problems present differently: look for zero-window events and a flat time-sequence graph rather than retransmissions. A good report reads “segment range X arrived 40 ms after later range Z and before any sender copy” or “sender retransmitted X after three duplicate ACKs; original X was not observed at this sensor”. Both are reviewable; “lots of retransmits” is not.

Two-point capture: bounding where the packet died

The fastest way to turn a hypothesis into a finding is to capture on both sides of the suspected hop. An embedded capture on the switch (EPC on IOS-XE, a monitor session on NX-OS, tcpdump on Aruba CX) gives you a second observer without disturbing the server. If the sender emits sequence range X once and the far sensor sees it, the packet was not lost. If the near sensor sees X and the far one does not, the loss is between them. Repeat, halving the interval, and you have localised the drop to a link or a queue rather than an entire path.

! IOS-XE embedded packet capture around a suspected hop
monitor capture CAP interface GigabitEthernet0/1 both
monitor capture CAP match ipv4 protocol tcp any any
monitor capture CAP start
! reproduce, then export:
monitor capture CAP export flash:cap.pcap

A sensor that sees the retransmission but never the original copy is the single most useful observation in packet forensics; it tells you the loss happened upstream of that sensor, and the fix belongs there.

Writing it up

State the capture points, the exact sequence ranges, the ACK and SACK evidence, the timestamps, and the conclusion. Separate observation from inference explicitly. Note the capture's own drop counters, because a reviewer will ask. Where a fix was applied, re-capture and show the same ranges now completing without a sender copy - that before/after pair is what closes a ticket.

Related reading: capture filters versus display filters, tcpdump command examples, diagnosing packet loss with mtr and IOS-XE embedded packet capture.

原文链接:https://hannes-software.com/pcap-surgery/blogs/tcp-out-of-order-vs-retransmission-pcap/