using tcp_notsent_lowat in various apps?

using tcp_notsent_lowat in various apps?

[Dave Taht]

In light of apple's: https://developer.apple.com/videos/wwdc/2015/?id=719

I am curious if anyone has tried this new socket option in appropriate apps,
(web browsers, screen sharers like tightvnc, X11, etc)? Would it be
helpful in openssh/dropbear?

What other sorts of apps? It looks like using it in chrome got stuck
on battery life analsysis:
https://code.google.com/p/chromium/issues/detail?id=310927

It does not appear to be a define in my fairly recently version of
gcc, and looking at the kernel for linux it looks like "25" is correct
there. it is defined to be 0x201 in OSX/ios:
http://fxr.watson.org/fxr/source/bsd/netinet/tcp.h?v=xnu-2050.18.24#L217

 (and now on universally in dev builds of ios)


I just tossed off a quick patch for rsync, not that I have a clue as
to whether it would make any difference there.

diff --git a/socket.c b/socket.c
index 3f5786b..bbb2461 100644
--- a/socket.c
+++ b/socket.c
@@ -406,6 +406,8 @@ static int *open_socket_in(int type, int port,
const char *bind_addr,
                           int af_hint)
 {
        int one = 1;
+       const int lowat = 16 * 1024;
+       int rc;
        int s, *socks, maxs, i, ecnt;
        struct addrinfo hints, *all_ai, *resp;
        char portbuf[10], **errmsgs;
@@ -451,6 +453,12 @@ static int *open_socket_in(int type, int port,
const char *bind_addr,

                setsockopt(s, SOL_SOCKET, SO_REUSEADDR,
                           (char *)&one, sizeof one);
+#ifndef TCP_NOTSENT_LOWAT
+#define TCP_NOTSENT_LOWAT 25
+#endif
+               rc = setsockopt(s, IPPROTO_TCP, TCP_NOTSENT_LOWAT,
+                          (char *)&lowat, sizeof lowat);
+               if(rc != 0) { perror("lowatfailed"); }
                if (sockopts)
                        set_socket_options(s, sockopts);
                else



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: using tcp_notsent_lowat in various apps?

[Juliusz Chroboczek]

> I am curious if anyone has tried this new socket option in appropriate apps,

I'm probably confused, but I don't see how this is different from setting 
SO_SNDBUF.  I realise that's lower in the stack, but it should have 
a similar effect, shouldn't it?

-- Juliusz

Re: using tcp_notsent_lowat in various apps?

[Jonathan Morton]

> On 19 Jun, 2015, at 05:47, Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> wrote:
> 
>> I am curious if anyone has tried this new socket option in appropriate apps,
> 
> I'm probably confused, but I don't see how this is different from setting SO_SNDBUF.  I realise that's lower in the stack, but it should have a similar effect, shouldn't it?

What I understand of it is:

Reducing SO_SNDBUF causes send() to block until all of the data can be accommodated in the smaller buffer.  But select() will return the socket as soon as there is *any* space in that buffer to stuff data into.

TCP_NOTSENT_LOWAT causes select() to not return the socket until the data in the buffer falls below the mark, which may (and should) be a mere fraction of the total buffer size.

It’s a subtle difference, but worth noting.  The two options effectively apply to completely different system calls.

You could use both in the same program, but generally SO_SNDBUF would be set to a higher value than the low water mark.  This allows a complete chunk of data to be stuffed into the buffer, and the application can then spend more time waiting in select() - where it is in a better position to make control decisions which are likely to be latency sensitive, and it can service other sockets which might be draining or filling at a different rate.

 - Jonathan Morton

Re: [Bloat] using tcp_notsent_lowat in various apps?

[Eric Dumazet]

On Fri, 2015-06-19 at 07:07 +0300, Jonathan Morton wrote:
> > On 19 Jun, 2015, at 05:47, Juliusz Chroboczek
> <jch@pps.univ-paris-diderot.fr> wrote:
> > 
> >> I am curious if anyone has tried this new socket option in
> appropriate apps,
> > 
> > I'm probably confused, but I don't see how this is different from
> setting SO_SNDBUF.  I realise that's lower in the stack, but it should
> have a similar effect, shouldn't it?
> 
> What I understand of it is:
> 
> Reducing SO_SNDBUF causes send() to block until all of the data can be
> accommodated in the smaller buffer.  But select() will return the
> socket as soon as there is *any* space in that buffer to stuff data
> into.
> 
> TCP_NOTSENT_LOWAT causes select() to not return the socket until the
> data in the buffer falls below the mark, which may (and should) be a
> mere fraction of the total buffer size.
> 
> It’s a subtle difference, but worth noting.  The two options
> effectively apply to completely different system calls.
> 
> You could use both in the same program, but generally SO_SNDBUF would
> be set to a higher value than the low water mark.  This allows a
> complete chunk of data to be stuffed into the buffer, and the
> application can then spend more time waiting in select() - where it is
> in a better position to make control decisions which are likely to be
> latency sensitive, and it can service other sockets which might be
> draining or filling at a different rate.

SO_SNDBUF needs to be large enough to accommodate with losses/repairs.

If flow has no losses, SNDBUF needs to be at least BDP :
 ( cwnd * MSS / rtt)

If a packet can be lost once, then SNDBUF needs to be :
2 * (cwnd * MSS / rtt)

If a packet can be lost twice, then we need
3 * (cwnd * MSS / rtt)

... etc ...

But really TCP write queue is logically split into 2 different logical
parts :

[1] Already sent data, waiting for ACK. This one can be arbitrary big,
depending on network conditions.

[2] Not sent data.

1) Part is hard to size, because it depends on losses, which cannot be
predicted.

2) Part is easy to size, if we have some reasonable ways to schedule
the application to provide additional data (write()/send()) when empty.


SO_SNDBUF sizes the overall TCP write queue ([1] + [2])

While NOTSENT_LOWAT is able to restrict (2) only, avoiding filling write
queue when/if no drops are actually seen.