[Bloat] using tcp_notsent_lowat in various apps?

[Jonathan Morton]

> On 19 Jun, 2015, at 05:47, Juliusz Chroboczek <jch at pps.univ-paris-diderot.fr> wrote:
> 
>> I am curious if anyone has tried this new socket option in appropriate apps,
> 
> I'm probably confused, but I don't see how this is different from setting SO_SNDBUF.  I realise that's lower in the stack, but it should have a similar effect, shouldn't it?

What I understand of it is:

Reducing SO_SNDBUF causes send() to block until all of the data can be accommodated in the smaller buffer.  But select() will return the socket as soon as there is *any* space in that buffer to stuff data into.

TCP_NOTSENT_LOWAT causes select() to not return the socket until the data in the buffer falls below the mark, which may (and should) be a mere fraction of the total buffer size.

It’s a subtle difference, but worth noting.  The two options effectively apply to completely different system calls.

You could use both in the same program, but generally SO_SNDBUF would be set to a higher value than the low water mark.  This allows a complete chunk of data to be stuffed into the buffer, and the application can then spend more time waiting in select() - where it is in a better position to make control decisions which are likely to be latency sensitive, and it can service other sockets which might be draining or filling at a different rate.

 - Jonathan Morton