[Cake] A few puzzling Cake results

[Jonathan Morton]

>> I'm saying that there's a tradeoff between intra-flow induced latency and packet loss, and I've chosen 4 MTUs as the operating point.
> 
> Is there a reason for picking 4 MTUs vs 2 MTUs vs 2 packets, etc?

To be more precise, I'm using a sojourn time equivalent to 4 MTU-sized packets per bulk flow at line rate, as a modifier to existing AQM behaviour.

The worst case for packet loss within the AQM occurs when the inherent latency of the links is very low but the available bandwidth per flow is also low.  This is easy to replicate using a test box flanked by GigE links to endpoint hosts; GigE has sub-millisecond inherent delays.  In this case, the entire BDP of each flow exists within the queue.

A general recommendation exists for TCP to use a minimum of 4 packets in flight, in order to keep the ack-clock running smoothly in the face of packet losses which might otherwise trigger an RTO (retransmit timeout).  This allows one packet to be lost and detected by the triple-repetition ACK method, without SACK.

It isn't necessary for these packets to all carry an MSS payload; theoretically a TCP could reduce the payload per packet to maintain four packets in flight with a congestion window below 4x MSS.  I'm not aware of any TCP which actually bothers to do that, though I might have missed recent developments in Linux TCP.

It's also possible for a TCP to pace its output so that fewer than 4 packets are physically in flight at a time, but still functionally have a congestion window that's significantly larger.  BBR could be said to fall into that category under some conditions.  TSQ might also produce this behaviour under some conditions.

The vast majority of widely deployed TCPs, however, are unable to operate efficiently at less than 4x MSS congestion windows.  Additionally, actual use of ECN remains deplorably low.  That's the reason for choosing 4 MTUs per bulk flow.

Originally, Cake had a similar AQM tweak but imposing a flat minimum of 1.5 MTUs, irrespective of flow count.  This mechanism is what was adapted into the present scheme.

 - Jonathan Morton