[Bloat] SO_SNDBUF and SO_RCVBUF

[Eric Dumazet]

On Thu, 2015-04-23 at 21:40 -0700, Dave Taht wrote:
> and of course, after writing the previous email, I go reading the
> original commit for this option. Yea, that is a huge increase in
> context switches...
> 
> https://lwn.net/Articles/560082/
> 
> ... but totally worth it for many apps that can do something else
> while their connection congests, and totally awesome for tcp vpns,
> x11, screen sharers, etc....

It all depends on how many bytes are pushed by the application per
sendmsg()

To keep the amount of unsent bytes low, the application should not issue
a large write, but it still can if it needs to for whatever reason.

netperf -t TCP_STREAM" uses a default size of 16384 bytes per sendmsg.

So obviously, if a wakeup is needed per sendmsg(), number of context
switches is exactly bandwidth_in_bytes_per_second / 16384

Normally, without this TCP_NOTSENT_LOWAT option, number of wakeups is
more like bandwidth_in_bytes_per_second / SO_SNDBUF, because kernel
wakes up the blocked task when output buffers size occupancy reached 50%