Packet Order Correction in SD-WAN

Overview - Packet Order Correction

Packet delivery across an IP network is not guaranteed to occur in the same sequence in which the packets were transmitted. Variations in queuing, congestion, routing, and path latency can cause some packets to overtake others in transit. This becomes particularly relevant in SD-WAN environments where traffic may traverse multiple WAN paths with different performance characteristics. Although the packets ultimately reach their destination, their arrival sequence can affect how receiving protocols process the traffic.

Out-of-order delivery is particularly significant for TCP, which uses sequence numbers to identify missing data and ensure that information is presented correctly to the application. When later packets arrive before an earlier packet, TCP may initially interpret the gap as packet loss, generating duplicate acknowledgements and potentially triggering unnecessary retransmission and congestion-control behaviour. Packet Order Correction (POC) addresses this problem by temporarily buffering out-of-sequence packets and restoring the correct sequence before the traffic is forwarded, reducing the performance impact of packet reordering.

Why Packet Order Matters

Most application traffic is divided into a sequence of packets before being transmitted across the network. Each packet may experience slightly different conditions in transit, but protocols such as TCP expect the receiving system to reconstruct the original byte stream in the correct order.

TCP uses sequence numbers to identify the position of each segment within that stream. If a receiver obtains a later segment before an earlier one, it identifies a gap in the sequence. The later segment may be retained temporarily, but the receiver continues to acknowledge the last contiguous sequence of data it has successfully received.

Why Packets Arrive Out of Order

Packet reordering occurs when packets belonging to the same traffic flow experience different amounts of delay while travelling across the network. Although packets may be transmitted in sequence, variations in queuing, routing, congestion, and path characteristics can cause later packets to arrive before earlier ones.

Variable Network Delay

Even when packets follow the same logical path, they may encounter different levels of queuing at routers and other intermediate devices. A packet delayed in a congested queue can therefore be overtaken by packets transmitted shortly afterwards.

Multiple WAN Paths

Packet reordering becomes more likely when traffic is distributed across multiple WAN links. Different paths may have different propagation delays, routing distances, queuing behaviour, and service-provider characteristics.

For example, if one packet is transmitted over a path with a 20 ms delay and the next packet is transmitted over a path with a 50 ms delay, subsequent packets placed on the faster path may arrive before the earlier packet travelling on the slower path. The packets have not been lost, but they are no longer presented to the receiver in the sequence in which they were transmitted.

Path Changes and Load Balancing

Routing changes, dynamic path selection, and load-balancing mechanisms can also introduce packet reordering. When traffic is moved between paths, packets already in transit on the previous path may arrive after packets sent later on the new path.

Jitter

Jitter is the variation in packet delay over time. A path may have an average latency that appears acceptable while individual packets experience significantly different delays. As this variation increases, so does the probability that packets will arrive out of sequence.

Why This Matters in SD-WAN

SD-WAN platforms are specifically designed to make use of multiple available WAN links. This improves resiliency and allows traffic to be distributed according to policy, but it also means that packets within a flow may encounter different path conditions. Packet Order Correction is therefore particularly valuable where link bonding or multipath forwarding is used, because it allows the receiving appliance to compensate for differences in path delay before forwarding the traffic onward.

Impact on TCP Performance

TCP is designed to deliver a reliable, ordered byte stream. When packets arrive out of sequence, the receiver identifies a gap in the expected sequence and continues to acknowledge the last contiguous block of data that has been received successfully. This can cause the sender to interpret delayed packets as evidence of packet loss.

Duplicate Acknowledgements

If later packets continue to arrive while an earlier packet is still missing, the receiver may generate repeated acknowledgements for the same sequence position. These duplicate acknowledgements indicate that data beyond the gap has arrived, but that the expected packet has not yet been received.

Unnecessary Retransmission

When sufficient duplicate acknowledgements are received, TCP can trigger a fast retransmission of the apparently missing data. If the original packet was only delayed and subsequently arrives, the retransmission was unnecessary and additional bandwidth has been consumed without improving delivery.

Congestion-Control Response

TCP commonly treats packet loss as an indication of network congestion. A retransmission event can therefore cause the sender to reduce its congestion window and lower its transmission rate. The connection must then gradually increase its sending rate again, even though the underlying problem may have been packet reordering rather than actual congestion or packet loss.

Greater Impact on Higher-Latency Links

The performance penalty becomes more significant as round-trip latency increases. Recovery from an unnecessary retransmission or congestion-window reduction takes longer on a high-latency WAN, because each acknowledgement and control response requires more time to traverse the network.

As a result, repeated packet reordering can reduce effective throughput even where the underlying WAN links have sufficient available bandwidth and relatively low actual packet loss.

Preventing Reordering from Reaching TCP

Packet Order Correction addresses this problem below the TCP layer. By temporarily buffering out-of-sequence packets and restoring the expected sequence before forwarding the traffic, the SD-WAN system can prevent transient path-delay differences from being interpreted by TCP as packet loss.

How Packet Order Correction Works

Packet Order Correction operates by temporarily buffering packets that arrive out of sequence and waiting for the missing sequence numbers to arrive. Once the required packets are present, the receiver restores the original packet order and forwards the traffic without exposing the reordering event to higher-layer protocols.

The key engineering variable is the amount of time the system is prepared to wait. If the reorder window is too short, a delayed packet may be treated as lost before it has an opportunity to arrive. If the window is too long, the correction process itself can introduce unnecessary latency.

Reorder Buffer

When an expected packet is missing but later packets have already arrived, those later packets are placed into a reorder buffer. The receiver holds them until either the missing packet arrives or the configured wait period expires.

For example, if packets are transmitted as:

1 → 2 → 3 → 4 → 5

and arrive as:

1 → 3 → 4 → 2 → 5

packets 3 and 4 can be held temporarily while the receiver waits for packet 2. When packet 2 arrives, the buffered packets can be released in the correct sequence:

1 → 2 → 3 → 4 → 5

Reorder Wait Time

The reorder wait time determines how long the system waits for a missing packet before assuming that it has been lost. This setting must balance two competing requirements: allowing sufficient time for delayed packets to arrive while avoiding excessive additional latency.

Dynamic Resequencing

More advanced implementations can adjust the reorder wait time dynamically as network conditions change. If packets are frequently arriving late or out of sequence, the system can increase the wait period to accommodate greater variation in path delay. When conditions improve, the wait period can be reduced to minimize buffering latency.

This is particularly useful in multipath WAN environments, where different links may have significantly different latency and jitter characteristics. Rather than relying on a single fixed timeout, dynamic resequencing allows Packet Order Correction to respond to the behaviour of the underlying network.

When the Wait Time Expires

If the missing packet does not arrive within the reorder window, it is treated as lost and the remaining buffered packets are released. At that point, other recovery mechanisms, including Forward Error Correction or higher-layer retransmission, may be used to recover the missing data.

Packet Order Correction in Multipath SD-WAN

Packet Order Correction becomes particularly important when an SD-WAN platform uses multiple WAN links simultaneously. Each underlay can have different latency, jitter, congestion, and routing characteristics, so packets belonging to the same flow may arrive at the far-end appliance in a different order from the one in which they were transmitted.

Consider two WAN paths with different one-way delays:

Path A: 20 ms
Path B: 50 ms

If packet 1 is sent across Path B and packet 2 is sent immediately afterwards across Path A, packet 2 may arrive first even though it was transmitted later. The same behaviour can occur repeatedly when traffic is distributed across paths with changing performance characteristics.

Path Delay Differential

The difference in latency between available paths is one of the primary causes of reordering in a bonded WAN. The greater the delay differential, the larger the resequencing window may need to be to allow slower-path packets to arrive before later packets are released.

Changing Link Conditions

The challenge is not limited to links with permanently different latency. Internet and wireless connections can experience rapid changes in queuing delay and jitter, which means the relative performance of two paths can vary over time. A path that is normally faster may temporarily become slower because of congestion or other network conditions.

A packet-order-correction mechanism therefore needs to account not only for the average latency of each path, but also for the variation in delay that occurs while traffic is in transit.

Maintaining a Single Ordered Flow

At the receiving SD-WAN appliance, packets from the available paths are brought back together into a single traffic flow. Packet Order Correction uses sequence information to identify packets that have arrived early, hold them where necessary, and release the traffic in the correct order.

This allows the SD-WAN system to take advantage of multiple WAN links while reducing the risk that path-delay differences will be exposed directly to TCP and the application.

Interaction Between Packet Order Correction and Forward Error Correction

Packet Order Correction and Forward Error Correction address different types of WAN impairment. Packet Order Correction deals with packets that have arrived, but have arrived in the wrong sequence. FEC deals with packets that have not arrived at all and may need to be reconstructed from redundant data.

The distinction is important because a delayed packet should not be treated as lost too quickly. Packet Order Correction gives an out-of-sequence packet a short period in which to arrive and be restored to its proper position. If that wait period expires and the packet is still missing, the system can then treat the packet as lost.

Late Versus Lost

From the receiver's perspective, a missing sequence number initially creates an ambiguity: the packet may have been dropped somewhere in the network, or it may simply be taking longer to arrive than the packets that followed it. Packet Order Correction helps resolve that ambiguity by allowing delayed packets to arrive before loss recovery is invoked.

Complementary Recovery Mechanisms

Once a packet is determined to be genuinely lost, FEC may be able to reconstruct the missing data without requiring retransmission from the original sender. In this way, the two mechanisms complement each other: Packet Order Correction handles delay-related reordering, while FEC handles actual packet loss.

Used together, these mechanisms can prevent both transient reordering and recoverable packet loss from being exposed unnecessarily to higher-layer protocols such as TCP, helping maintain more consistent application performance across impaired WAN links.

Summary

Packet Order Correction addresses a common WAN impairment in which packets arrive at their destination in a different sequence from the one in which they were sent. This can occur because of queuing, jitter, route changes, or differences in latency between multiple WAN paths.

By temporarily buffering out-of-sequence packets and restoring the correct order before forwarding the traffic, Packet Order Correction can prevent transient delay variations from unnecessarily affecting higher-layer protocols such as TCP. The key is to balance sufficient wait time for delayed packets against the additional latency introduced by buffering.

In multipath SD-WAN environments, this capability becomes especially important because packets may routinely traverse links with different and changing performance characteristics. When combined with Forward Error Correction, Packet Order Correction helps distinguish between packets that are merely late and packets that are genuinely lost, allowing each impairment to be handled by the appropriate recovery mechanism.