[Cake] DSCP ramblings

[Stephen Hemminger]

On Wed, 22 Apr 2020 09:20:29 -0700
Dave Taht <dave.taht at gmail.com> wrote:

> and because of your I'm off building collectd because those graphs
> look so good. :)
> 
> https://forum.openwrt.org/t/sqm-reporting/59960/24
> 
> I have long just used snmpd, and collectd looks interesting. I fear
> it's too heavyweight, particularly shelling out to a script....
> 
> On Wed, Apr 22, 2020 at 9:15 AM Dave Taht <dave.taht at gmail.com> wrote:
> >
> > On Wed, Apr 22, 2020 at 8:58 AM Kevin Darbyshire-Bryant
> > <kevin at darbyshire-bryant.me.uk> wrote:  
> > >
> > > During these strange times of lockdown I’ve been trying to keep myself occupied/entertained/sane(???) by ‘fiddling with stuff’ and improving my coding.  This started with an idea of learning Python which was great until the on-line bit of it ran out and someone posted an idea on the Openwrt forum about graphing Cake stats.
> > >
> > > That had nothing to do with Python and involved (new to me) technologies such as ‘collectd’, ‘JSON’, a bit of javascript and my usual level of cobbling something together in ‘ash’…. So that course was well spent :-)
> > >
> > > Anyway, data was collected and graphs produced in a very small household.  What’s immediately apparent from those graphs and cake in ‘diffserv4’ mode is that very, very few applications are using DSCP at all.  Most things are to port 443.
> > >
> > > I was also a little surprised to see that my DNS over foo proxies such as stubby & https-dns-proxy don’t use DSCP coding.  It surprised me even more to see RFC recommendations that DNS be treated as ‘Best Effort’.  Now in the days of udp only and no dnssec (with fallback to tcp) this may be good enough, but I wonder if this is realistic these days?
> > >
> > > So putting aside the discussion of what codepoint should be used, I then wondered how hard it would be to actually set a dscp in these applications.  And this is where I had another surprise.  For example https-dns-proxy uses libcurl.  libcurl has no standard ‘in-library’ method for setting a socket’s dscp.  I cobbled a workaround in the application https://github.com/aarond10/https_dns_proxy/pull/83 - it works.
> > >
> > > Next I attacked stubby, which uses getdns.  getdns doesn’t even have a callback or parameters passing so you can set a dscp on the socket from a client application, pure ‘hack the library’ stuff.
> > >
> > > To be blunt and on a small sample of 2 libraries/applications, it seems that DSCP is completely ignored.  Applications signalling ’this is/isnt latency sensitive/bulk’ isn’t going to happen if it isn’t easy to do.
> > >
> > > Apple should be marking facetime calls as being ‘video conference’ or whatever.  BBC iplayer Radio apps should be marking ‘audio streaming’. But every f*ing thing is CS0 port 443.  And I’m wondering how much of this is because library support is simply missing.  Maybe gaming apps are better? (I don’t game)
> > >
> > > Right, I’m off for a lie down.  Sorry for the rant.  
> >
> > Welcome to my explorations... in 2011. Diffserv is rather underused, isn't it?
> >
> > I took a survey of every (500+) gaming console at a convention. nearly
> > zero diffserv usage and it was all over the map, and I think, mostly,
> > from osx.
> >
> > windows requires admin privs to set the tos bits at all
> > webrtc has an api to set the bits, but it doesn't work on windows.
> >
> > ssh will set the imm bit for interactive, I forget what it sets for bulk
> > bgp sets cs6. so does babel. Arguably both usages are wrong.
> > some windows stuff sets cs1 for things like ping
> > I got the mosh folk to use AF42 as a (worldwide) test, for nearly a
> > year. they had one user with a problem and they turned it off. It was
> > funny, keith thought I was making an expert recommendation rather than
> > a test and just copy pasted my code into the tree and shipped it.
> >
> > linux implements a strict priority queue in pfifo_fast. You can dos it
> > if you hit it by setting the bits.
> > irtt and netperf let you set the bits. iperf also.
> >
> > I produced a patch for rsync in particular (since I use it heavily)
> >
> > sqm at least used to mark dns and ntp as some elivated prio, but I
> > forget which and for all I know the cake qos system doesn't implement
> > those filters.
> >
> > A few multi-queue ethernet devices actually do interpret the bits.
> > Undocumented as to which one..
> >
> > and lets not get started on ecn.
> >  
> > >
> > >
> > > Hack for getdns/stubby
> > >
> > > diff --git a/src/stub.c b/src/stub.c
> > > index 2547d10f..7e47aba5 100644
> > > --- a/src/stub.c
> > > +++ b/src/stub.c
> > > @@ -52,6 +52,7 @@
> > >  #include "platform.h"
> > >  #include "general.h"
> > >  #include "pubkey-pinning.h"
> > > +#include <netinet/ip.h>
> > >
> > >  /* WSA TODO:
> > >   * STUB_TCP_RETRY added to deal with edge triggered event loops (versus
> > > @@ -381,6 +382,9 @@ tcp_connect(getdns_upstream *upstream, getdns_transport_list_t transport)
> > >  # else
> > >         static const int  enable = 1;
> > >  # endif
> > > +#endif
> > > +#if defined(IP_TOS)
> > > +       int dscp = IPTOS_CLASS_CS4;
> > >  #endif
> > >         int fd = -1;
> > >
> > > @@ -390,6 +394,12 @@ tcp_connect(getdns_upstream *upstream, getdns_transport_list_t transport)
> > >                    __FUNC__, (void*)upstream);
> > >         if ((fd = socket(upstream->addr.ss_family, SOCK_STREAM, IPPROTO_TCP)) == -1)
> > >                 return -1;
> > > +#if defined(IP_TOS)
> > > +       if (upstream->addr.ss_family == AF_INET6)
> > > +               (void)setsockopt(fd, IPPROTO_IPV6, IP_TOS, &dscp, sizeof(dscp));
> > > +       else if (upstream->addr.ss_family == AF_INET)
> > > +               (void)setsockopt(fd, IPPROTO_IP, IP_TOS, &dscp, sizeof(dscp));
> > > +#endif
> > >
> > >
> > > Cheers,
> > >
> > > Kevin D-B
> > >
> > > gpg: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A
> > >

In my experience, except for a small number of cases (RDMA etc) Diffserv is a
complete waste of time. There is no global ordering, there is no guarantee against
starvation and any sane ISP strips the bits off or ignores them.

Diffserv is even an issue at scale in the cloud. What does DSCP mean exactly on
outer headers, who gets to decide for which service. And what about inner headers
and propogating inner to outer. Its a mess.