[Bloat] Buffer bloat at the sender [was: About Stochastic Fair Blue (SFB)]

[Jim Gettys]

On 02/04/2011 12:33 PM, Juliusz Chroboczek wrote:
>> Juliusz,  have you thought about the host case at all?
>
>> One of the places we're getting insane buffering is in the operating
>> systems themselves (e.g. the experiment I did with a 100Mbps
>> switch).
>
> Yes.  You have three to four layers of buffering:
>
>    (1) the device driver's buffer;
>    (2) the packet scheduler's buffer;
>    (3) TCP's buffer;
>    (4) the application's buffer.

There are a few more than the four you identify:
	5) sometimes device drivers have some private buffers, independent of 
the device itself; for example the Marvell driver for the wireless 
module used on OLPC has such an internal buffer; dwmw2 used it to 
simplify his locking problems with USB.
	6) the device themselves may have buffers.  Again, on the OLPC wireless 
module (which does a form of mesh routing internally, without needing 
host support), there are 4 packet buffers out in the device (which is 
itself an ARM processor with hundreds of kilobytes of code).

I also hypothesise that there could be busses that support multiple 
outstanding transactions, that could add yet more buffering. I have no 
idea if this hypothesis is true.

>
> It will come as no surprise to the readers of this list that (1) and (2)
> are usually too large.  For example, (1) the ath9k driver has a buffer
> of 200 packets; and (2) the default scheduler queue is 1000 packets (!).
>
>> My intuition is that we have to do AQM in hosts, not just routers.
>
> Hmm... I would argue that the sending host is somewhat easier than the
> intermediate router.  In the sender, the driver/packet scheduler can
> apply backpressure to the transport layer, to cause it to slow down
> without the need for the lengthy feedback loop that dropping/delaying
> a packet in an intermediate router has to rely on [1].

Yup.  I was noting, however, with 1-4 above, we today have a major 
problem here in practice.

And I don't know if there is any way to signal back pressure to UDP 
based protocols; I've never worked on one.
>
> Unfortunately, at least under Linux, most drivers do not apply
> backpressure correctly.  Markus Kittenberger has recently determined [2]
> that among b43-legacy, ath5k, ath9k and madwifi, only the former two do
> the right thing.

Yes, the host side can and should apply backpressure correctly, and it 
may be an easier case.  It sounds like we have work to that area to do.

>
> --Juliusz
>
> [1] Now why did we give up on source quench again?
> [2] http://article.gmane.org/gmane.network.olsr.user/4264
>