[Bloat] using tcp_notsent_lowat in various apps?

[Bloat] using tcp_notsent_lowat in various apps?

[Dave Taht]

In light of apple's: https://developer.apple.com/videos/wwdc/2015/?id=719

I am curious if anyone has tried this new socket option in appropriate apps,
(web browsers, screen sharers like tightvnc, X11, etc)? Would it be
helpful in openssh/dropbear?

What other sorts of apps? It looks like using it in chrome got stuck
on battery life analsysis:
https://code.google.com/p/chromium/issues/detail?id=310927

It does not appear to be a define in my fairly recently version of
gcc, and looking at the kernel for linux it looks like "25" is correct
there. it is defined to be 0x201 in OSX/ios:
http://fxr.watson.org/fxr/source/bsd/netinet/tcp.h?v=xnu-2050.18.24#L217

 (and now on universally in dev builds of ios)


I just tossed off a quick patch for rsync, not that I have a clue as
to whether it would make any difference there.

diff --git a/socket.c b/socket.c
index 3f5786b..bbb2461 100644
--- a/socket.c
+++ b/socket.c
@@ -406,6 +406,8 @@ static int *open_socket_in(int type, int port,
const char *bind_addr,
                           int af_hint)
 {
        int one = 1;
+       const int lowat = 16 * 1024;
+       int rc;
        int s, *socks, maxs, i, ecnt;
        struct addrinfo hints, *all_ai, *resp;
        char portbuf[10], **errmsgs;
@@ -451,6 +453,12 @@ static int *open_socket_in(int type, int port,
const char *bind_addr,

                setsockopt(s, SOL_SOCKET, SO_REUSEADDR,
                           (char *)&one, sizeof one);
+#ifndef TCP_NOTSENT_LOWAT
+#define TCP_NOTSENT_LOWAT 25
+#endif
+               rc = setsockopt(s, IPPROTO_TCP, TCP_NOTSENT_LOWAT,
+                          (char *)&lowat, sizeof lowat);
+               if(rc != 0) { perror("lowatfailed"); }
                if (sockopts)
                        set_socket_options(s, sockopts);
                else



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] using tcp_notsent_lowat in various apps?

[Steinar H. Gunderson]

On Tue, Jun 16, 2015 at 09:11:08AM -0700, Dave Taht wrote:
> I just tossed off a quick patch for rsync, not that I have a clue as
> to whether it would make any difference there.

For bulk applications (like rsync), how would this make sense at all?
I thought the entire point of this option was if you knew what data to send
now, but that you might want to change your mind later if it takes some time
to send it. The latter doesn't apply to rsync.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: [Bloat] using tcp_notsent_lowat in various apps?

[Dave Taht]

On Tue, Jun 16, 2015 at 9:18 AM, Steinar H. Gunderson
<sgunderson@bigfoot.com> wrote:
> On Tue, Jun 16, 2015 at 09:11:08AM -0700, Dave Taht wrote:
>> I just tossed off a quick patch for rsync, not that I have a clue as
>> to whether it would make any difference there.
>
> For bulk applications (like rsync), how would this make sense at all?
> I thought the entire point of this option was if you knew what data to send
> now, but that you might want to change your mind later if it takes some time
> to send it. The latter doesn't apply to rsync.

That was mostly there for a quick coding example (I have also had out
of tree patches
for classification and tcp cc selection in there, and was fiddling
with cdg, so it was seconds
more to toss off that patch. rsync does also mix command and control
in the data flow).
What I'd wanted to do was measure the cpu impact of the
additional context switches along the lines of the original posting on
this option

https://lwn.net/Articles/560082/

(In the case of rsync and scp I have been known to abort it part of
the way through, too.)

The larger question was about anyone trying vnc and similar questions
in chrome and other web
browsers, and servers. ex:

https://insouciant.org/tech/prioritization-only-works-when-theres-pending-data-to-prioritize/

> /* Steinar */
> --
> Homepage: http://www.sesse.net/
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] using tcp_notsent_lowat in various apps?

[Jonathan Morton]

> On 16 Jun, 2015, at 19:18, Steinar H. Gunderson <sgunderson@bigfoot.com> wrote:
> 
> On Tue, Jun 16, 2015 at 09:11:08AM -0700, Dave Taht wrote:
>> I just tossed off a quick patch for rsync, not that I have a clue as
>> to whether it would make any difference there.
> 
> For bulk applications (like rsync), how would this make sense at all?
> I thought the entire point of this option was if you knew what data to send
> now, but that you might want to change your mind later if it takes some time
> to send it. The latter doesn't apply to rsync.

Actually, it does.  Rsync is designed to be used to update an existing set of files, so the protocol interleaves control and data information asynchronously.

More generally, I think it’s worth setting LOWAT on *any* application that uses select() or poll() with a readable and writable socket population simultaneously.

- Jonathan Morton

Re: [Bloat] using tcp_notsent_lowat in various apps?

[Dave Taht]

On Tue, Jun 16, 2015 at 9:33 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 16 Jun, 2015, at 19:18, Steinar H. Gunderson <sgunderson@bigfoot.com> wrote:
>>
>> On Tue, Jun 16, 2015 at 09:11:08AM -0700, Dave Taht wrote:
>>> I just tossed off a quick patch for rsync, not that I have a clue as
>>> to whether it would make any difference there.
>>
>> For bulk applications (like rsync), how would this make sense at all?
>> I thought the entire point of this option was if you knew what data to send
>> now, but that you might want to change your mind later if it takes some time
>> to send it. The latter doesn't apply to rsync.
>
> Actually, it does.  Rsync is designed to be used to update an existing set of files, so the protocol interleaves control and data information asynchronously.
>
> More generally, I think it’s worth setting LOWAT on *any* application that uses select() or poll() with a readable and writable socket population simultaneously.

Take samba as another potential example. I commonly see this
increasing the SO_SNDBUF to a given value, but I am not sure if this
is the right thing anymore. As samba is commonly used for filesharing
(and things that take locks and do database-y stuff), improving
interactivity might be a big win.

Seeing the 50%! decrease in kernel memory on the original tests of
TCP_SENT_LOWAT is very exciting in the context of those running samba
on tiny tiny nas devices common in my world.

And seeing apple enable it universally points to perhaps exploring the
effects of just enabling it universally in linux (or in certain kinds
of linux-based devices) at a "reasonable" value, for whatever value of
reasonable exists.

> - Jonathan Morton
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] using tcp_notsent_lowat in various apps?

[Eric Dumazet]

On Tue, 2015-06-16 at 10:22 -0700, Dave Taht wrote:

> Take samba as another potential example. I commonly see this
> increasing the SO_SNDBUF to a given value, but I am not sure if this
> is the right thing anymore. As samba is commonly used for filesharing
> (and things that take locks and do database-y stuff), improving
> interactivity might be a big win.
> 
> Seeing the 50%! decrease in kernel memory on the original tests of
> TCP_SENT_LOWAT is very exciting in the context of those running samba
> on tiny tiny nas devices common in my world.
> 
> And seeing apple enable it universally points to perhaps exploring the
> effects of just enabling it universally in linux (or in certain kinds
> of linux-based devices) at a "reasonable" value, for whatever value of
> reasonable exists.

Another use case for TCP_SENT_LOWAT is the cancel phase.

If for one reason, application decides to abort the transfer, by doing a
close() or shutdown(), it is better to have at most few KB of data still
queued in socket, that kernel tries to deliver no matter what.

I've seen stupid applications having timeouts on their sockets. When
they believe a flow is stuck, they close it and open a new one. And you
get many flows competing on the same bottleneck.

Re: [Bloat] using tcp_notsent_lowat in various apps?

[Juliusz Chroboczek]

> I am curious if anyone has tried this new socket option in appropriate apps,

I'm probably confused, but I don't see how this is different from setting 
SO_SNDBUF.  I realise that's lower in the stack, but it should have 
a similar effect, shouldn't it?

-- Juliusz

Re: [Bloat] using tcp_notsent_lowat in various apps?

[Jonathan Morton]

> On 19 Jun, 2015, at 05:47, Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> wrote:
> 
>> I am curious if anyone has tried this new socket option in appropriate apps,
> 
> I'm probably confused, but I don't see how this is different from setting SO_SNDBUF.  I realise that's lower in the stack, but it should have a similar effect, shouldn't it?

What I understand of it is:

Reducing SO_SNDBUF causes send() to block until all of the data can be accommodated in the smaller buffer.  But select() will return the socket as soon as there is *any* space in that buffer to stuff data into.

TCP_NOTSENT_LOWAT causes select() to not return the socket until the data in the buffer falls below the mark, which may (and should) be a mere fraction of the total buffer size.

It’s a subtle difference, but worth noting.  The two options effectively apply to completely different system calls.

You could use both in the same program, but generally SO_SNDBUF would be set to a higher value than the low water mark.  This allows a complete chunk of data to be stuffed into the buffer, and the application can then spend more time waiting in select() - where it is in a better position to make control decisions which are likely to be latency sensitive, and it can service other sockets which might be draining or filling at a different rate.

 - Jonathan Morton

Re: [Bloat] using tcp_notsent_lowat in various apps?

[Eric Dumazet]

On Fri, 2015-06-19 at 07:07 +0300, Jonathan Morton wrote:
> > On 19 Jun, 2015, at 05:47, Juliusz Chroboczek
> <jch@pps.univ-paris-diderot.fr> wrote:
> > 
> >> I am curious if anyone has tried this new socket option in
> appropriate apps,
> > 
> > I'm probably confused, but I don't see how this is different from
> setting SO_SNDBUF.  I realise that's lower in the stack, but it should
> have a similar effect, shouldn't it?
> 
> What I understand of it is:
> 
> Reducing SO_SNDBUF causes send() to block until all of the data can be
> accommodated in the smaller buffer.  But select() will return the
> socket as soon as there is *any* space in that buffer to stuff data
> into.
> 
> TCP_NOTSENT_LOWAT causes select() to not return the socket until the
> data in the buffer falls below the mark, which may (and should) be a
> mere fraction of the total buffer size.
> 
> It’s a subtle difference, but worth noting.  The two options
> effectively apply to completely different system calls.
> 
> You could use both in the same program, but generally SO_SNDBUF would
> be set to a higher value than the low water mark.  This allows a
> complete chunk of data to be stuffed into the buffer, and the
> application can then spend more time waiting in select() - where it is
> in a better position to make control decisions which are likely to be
> latency sensitive, and it can service other sockets which might be
> draining or filling at a different rate.

SO_SNDBUF needs to be large enough to accommodate with losses/repairs.

If flow has no losses, SNDBUF needs to be at least BDP :
 ( cwnd * MSS / rtt)

If a packet can be lost once, then SNDBUF needs to be :
2 * (cwnd * MSS / rtt)

If a packet can be lost twice, then we need
3 * (cwnd * MSS / rtt)

... etc ...

But really TCP write queue is logically split into 2 different logical
parts :

[1] Already sent data, waiting for ACK. This one can be arbitrary big,
depending on network conditions.

[2] Not sent data.

1) Part is hard to size, because it depends on losses, which cannot be
predicted.

2) Part is easy to size, if we have some reasonable ways to schedule
the application to provide additional data (write()/send()) when empty.


SO_SNDBUF sizes the overall TCP write queue ([1] + [2])

While NOTSENT_LOWAT is able to restrict (2) only, avoiding filling write
queue when/if no drops are actually seen.

Re: [Bloat] using tcp_notsent_lowat in various apps?

[Juliusz Chroboczek]

[Removed bloat-devel from the CC]

> [1] Already sent data [...]
> [2] Not sent data.

[...]

> While NOTSENT_LOWAT is able to restrict (2) only, avoiding filling write 
> queue when/if no drops are actually seen.

Thanks for the explanation, Eric.

Do you have any concrete examples where this has been found to yield 
better results than what the kernel can do on its own?  I'm probably 
missing something, but my intuition is that conversational protocols 
should be naturally self-pacing, while for streaming protocols the right 
amount of buffering depends on the kernel latency, and hence should be 
better done by the kernel itself.

-- Juliusz

Re: [Bloat] using tcp_notsent_lowat in various apps?

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 73 bytes --]

Cue link to the recently posted Apple presentation...

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 112 bytes --]

Re: [Bloat] using tcp_notsent_lowat in various apps?

[Eric Dumazet]

On Sun, 2015-06-21 at 17:04 +0200, Juliusz Chroboczek wrote:

> Do you have any concrete examples where this has been found to yield 
> better results than what the kernel can do on its own?  I'm probably 
> missing something, but my intuition is that conversational protocols 
> should be naturally self-pacing, while for streaming protocols the right 
> amount of buffering depends on the kernel latency, and hence should be 
> better done by the kernel itself.

Unless you have thousands of tcp flows maybe, where (unswapable) kernel
memory becomes a precious resource.

Of course, if you have a few tcp sockets, just queuing 10 Mbytes per
socket will be your best shot, in terms of cpu overhead (one schedule
every 5 MBytes or so)

Re: [Bloat] using tcp_notsent_lowat in various apps?

[Juliusz Chroboczek]

>> amount of buffering depends on the kernel latency, and hence should be 
>> better done by the kernel itself.

> Unless you have thousands of tcp flows maybe, where (unswapable) kernel 
> memory becomes a precious resource.

Thanks, that makes sense.

-- Juliusz