[Bloat] updating the theory of buffer sizing

[Jonathan Morton]

> On 10 Oct, 2021, at 8:48 pm, Dave Taht <dave.taht at gmail.com> wrote:
> 
> This latest from Nick & co, was quite good:
> 
> https://arxiv.org/pdf/2109.11693.pdf

Skip the false modesty - I think this is very important work, actually.  I would expect it to get cited a heck of a lot in future work, both in academia and in the IETF.

In terms of its content, it confirms, contextualises, and formalises various things that I already understood at an intuitive level.  The mathematics involved is simple and accessible (unlike some papers I've read recently), and the practical explanations of the observed behaviours are clear and to the point.  I particularly appreciate the way they were able to parameterise certain characteristics on a continuum, rather than all-or-nothing, as that captures the complex characteristics of real traffic much better.

The observations about synchronisation of congestion responses are also very helpful.  When synchronised, the aggregate behaviour of many flows is similar to that of a much smaller number, perhaps even a single flow.  When desynchronised, the well-known statistical multiplexing effects apply.  They also clearly explain why the "hard threshold" type of ECN marking is undesirable - because it provokes synchronisation in a way that tail-drop does not (and this is also firmly related to a point we discussed last week).

Notably, they started seeing the effects of burstiness, on a small and theoretically "smooth" network, on timescales of approximately a seventh of a millisecond (20 packets, 9000 byte MTU, 10Gbps).  They were unable to reduce buffer sizes below that level without throughput dropping well below their theoretical predictions, which had held true down to that point.  This has implications for setting AQM targets and tolerances in even near-ideal network environments.  But they did also note that BBR showed much less sensitivity to this effect, as it uses pacing.  In any case, it confirms that the first role of a buffer is to absorb bursts without excessive loss.

 - Jonathan Morton