[Bloat] Packet reordering and RACK (was The "Some Congestion Experienced" ECN codepoint)

[David Lang]

On Sun, 17 Mar 2019, Sebastian Moeller wrote:

>> All I’m trying to say is that this is bad engineering, apparently perpetuated 
>> by bad transport layer implementations.
>
> 	??? Now I am confuzed, to me engineering is all about balancing 
> trade-offs, and this is all about how to evaluate different dimensions. I 
> believe we agree that a network should not re-order packets artificially 
> without some justification and we also agree that some level of re-ordering 
> might be un-avoidable, we basically haggle over how much is acceptable. I also 
> believe, and correct me if I am wrong, that we agree that with TCP endpoints 
> prefer less over more en-passage re-ordering. So why is it bad for a transport 
> layer to aim for pleasing its users (in my book the transport network only 
> exists to allow end-to-end communication)?

what I am seeing is that you seem to be claiming that the network MUST NOT allow 
packets to pass each other on the network.

from the network layer I am hearing that ensuring that packets always arrive 
in-order has a cost, in buffer space, in latency, and in processing overhead. It 
forces packet processing to be single threaded at some point to check if the 
packets are in order rather than just letting the device forward all packets as 
fast as it can.

In the past, this has not been very significant, but as you get the ability to 
send (and/or receive) packets over multiple links (be they physical wire links, 
or more probably, multiple parallel channels over multi-mode fiber or RF), I can 
easily see it becoming more important to try and avoid these bottlenecks.

> I note that the benefit of reordering is all in the transport network, while 
> the costs are all carried by the endpoints. With this kind of skewed 
> incentives, what outcome do you expect.

well, if the transport routers end up slowing down all traffic because they run 
out of cpu to process packets, that will hurt the endpoint very significantly.

See what happens to router performance when you have to get the CPUs involved in 
packet processing as opposed to letting it happen on the ASICs. Look what 
happens when you hook a WNDR3800 home router to a 100Mb link and try to run 
cake (even before you saturate the network links). The endpoint very much 
notices.

the bottleneck link is not always the final hop with the lowest bitrate (it 
frequently is, but not always)

David Lang