From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd44.google.com (mail-io1-xd44.google.com [IPv6:2607:f8b0:4864:20::d44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 95D1A3B29E for ; Wed, 22 Apr 2020 12:58:26 -0400 (EDT) Received: by mail-io1-xd44.google.com with SMTP id i3so3114627ioo.13 for ; Wed, 22 Apr 2020 09:58:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=BYbT7gw0HcU2D6d157nyuyGeT1rWuTfT6fFXHjLLvy4=; b=AstVIzDaHqn5B4GA2N3pzTyKIozakqLkxNhKPugECasl505jHl6o2xNG8zU58g3fYw Ns4BA43ZSY9P2JqAfqS2oflQCnF+oQQP4Bxz16pjMKa8QBDVlgILXzvACUF4uK3lasJt BhvgllvagOnTLNDs1SWrO4yOp4dSnzEJg9ZvswsVDSQWnllpynAPMefr+q77xLPzpOxD pN563/c4TwohAVXVe7VykP4VAThao83hySrCYNQ3LATk5Xgv5jsZS/F5nDiWUACcrH8W QcN0k3dlGVaLVraieYArpNgEZVzMrAA7L/HKv7sBsg8lmuBnNetjLP09D4lvBHa+mXZv zrGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=BYbT7gw0HcU2D6d157nyuyGeT1rWuTfT6fFXHjLLvy4=; b=gLgRu5YBacT1YfgItGKcDcnI8O4MvJIxAgsQssCqZB76Qe56FCQ7oqTTNx9DJ3hWk5 Qu4SIRVq2JLlD70106FgOIUXfY6Mh4tStrXWNteuVCDmYhh+NczAC2A5mzxTzr7N2IOB IWgeEIM+OHAKe1HDMC5ij1FDAkmPgl+HsWYMi8amHWm8PtY3HQ8m2wdjkzgMhHpjXZEo 8f/kcfyKe6/ING7Pg4g7YrjD7Y3OCse5uXOtvJGPYGdEwPJtJPJopKCifjkya2E+6HGu Uaz4Avf66ikldTGnXe4k61rYD5FMTXwRNjL+SPVt/Dlc03WypyUGax7TmcnhPJOvVF3j 7eJQ== X-Gm-Message-State: AGi0PuY7akniNhGQQc3hzWNL1Tm26pHBeG9lNkyCd8eU5M7x8Z0w2vJx 3NAmbEppzZB33DjZgN6I7YwKQVI3Znby/k4jA80Lu5FWk20= X-Google-Smtp-Source: APiQypLkCJ1jurnH9xcJjgszx4zVIXZbKqcywZFia2YVufXxjL0HZLn4EeJ+v0+4jMK9G93/6XlyRPucsihl9FBbKuU= X-Received: by 2002:a6b:b8d6:: with SMTP id i205mr27489317iof.123.1587574705895; Wed, 22 Apr 2020 09:58:25 -0700 (PDT) MIME-Version: 1.0 References: <20200422094428.2ee29433@hermes.lan> In-Reply-To: <20200422094428.2ee29433@hermes.lan> From: Dave Taht Date: Wed, 22 Apr 2020 09:58:14 -0700 Message-ID: To: Stephen Hemminger Cc: Kevin Darbyshire-Bryant , Cake List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Cake] DSCP ramblings X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Apr 2020 16:58:26 -0000 On Wed, Apr 22, 2020 at 9:44 AM Stephen Hemminger wrote: > > On Wed, 22 Apr 2020 09:20:29 -0700 > Dave Taht wrote: > > > and because of your I'm off building collectd because those graphs > > look so good. :) > > > > https://forum.openwrt.org/t/sqm-reporting/59960/24 > > > > I have long just used snmpd, and collectd looks interesting. I fear > > it's too heavyweight, particularly shelling out to a script.... > > > > On Wed, Apr 22, 2020 at 9:15 AM Dave Taht wrote: > > > > > > On Wed, Apr 22, 2020 at 8:58 AM Kevin Darbyshire-Bryant > > > wrote: > > > > > > > > During these strange times of lockdown I=E2=80=99ve been trying to = keep myself occupied/entertained/sane(???) by =E2=80=98fiddling with stuff= =E2=80=99 and improving my coding. This started with an idea of learning P= ython which was great until the on-line bit of it ran out and someone poste= d an idea on the Openwrt forum about graphing Cake stats. > > > > > > > > That had nothing to do with Python and involved (new to me) technol= ogies such as =E2=80=98collectd=E2=80=99, =E2=80=98JSON=E2=80=99, a bit of = javascript and my usual level of cobbling something together in =E2=80=98as= h=E2=80=99=E2=80=A6. So that course was well spent :-) > > > > > > > > Anyway, data was collected and graphs produced in a very small hous= ehold. What=E2=80=99s immediately apparent from those graphs and cake in = =E2=80=98diffserv4=E2=80=99 mode is that very, very few applications are us= ing DSCP at all. Most things are to port 443. > > > > > > > > I was also a little surprised to see that my DNS over foo proxies s= uch as stubby & https-dns-proxy don=E2=80=99t use DSCP coding. It surprise= d me even more to see RFC recommendations that DNS be treated as =E2=80=98B= est Effort=E2=80=99. Now in the days of udp only and no dnssec (with fallb= ack to tcp) this may be good enough, but I wonder if this is realistic thes= e days? > > > > > > > > So putting aside the discussion of what codepoint should be used, I= then wondered how hard it would be to actually set a dscp in these applica= tions. And this is where I had another surprise. For example https-dns-pr= oxy uses libcurl. libcurl has no standard =E2=80=98in-library=E2=80=99 met= hod for setting a socket=E2=80=99s dscp. I cobbled a workaround in the app= lication https://github.com/aarond10/https_dns_proxy/pull/83 - it works. > > > > > > > > Next I attacked stubby, which uses getdns. getdns doesn=E2=80=99t = even have a callback or parameters passing so you can set a dscp on the soc= ket from a client application, pure =E2=80=98hack the library=E2=80=99 stuf= f. > > > > > > > > To be blunt and on a small sample of 2 libraries/applications, it s= eems that DSCP is completely ignored. Applications signalling =E2=80=99thi= s is/isnt latency sensitive/bulk=E2=80=99 isn=E2=80=99t going to happen if = it isn=E2=80=99t easy to do. > > > > > > > > Apple should be marking facetime calls as being =E2=80=98video conf= erence=E2=80=99 or whatever. BBC iplayer Radio apps should be marking =E2= =80=98audio streaming=E2=80=99. But every f*ing thing is CS0 port 443. And= I=E2=80=99m wondering how much of this is because library support is simpl= y missing. Maybe gaming apps are better? (I don=E2=80=99t game) > > > > > > > > Right, I=E2=80=99m off for a lie down. Sorry for the rant. > > > > > > Welcome to my explorations... in 2011. Diffserv is rather underused, = isn't it? > > > > > > I took a survey of every (500+) gaming console at a convention. nearl= y > > > zero diffserv usage and it was all over the map, and I think, mostly, > > > from osx. > > > > > > windows requires admin privs to set the tos bits at all > > > webrtc has an api to set the bits, but it doesn't work on windows. > > > > > > ssh will set the imm bit for interactive, I forget what it sets for b= ulk > > > bgp sets cs6. so does babel. Arguably both usages are wrong. > > > some windows stuff sets cs1 for things like ping > > > I got the mosh folk to use AF42 as a (worldwide) test, for nearly a > > > year. they had one user with a problem and they turned it off. It was > > > funny, keith thought I was making an expert recommendation rather tha= n > > > a test and just copy pasted my code into the tree and shipped it. > > > > > > linux implements a strict priority queue in pfifo_fast. You can dos i= t > > > if you hit it by setting the bits. > > > irtt and netperf let you set the bits. iperf also. > > > > > > I produced a patch for rsync in particular (since I use it heavily) > > > > > > sqm at least used to mark dns and ntp as some elivated prio, but I > > > forget which and for all I know the cake qos system doesn't implement > > > those filters. > > > > > > A few multi-queue ethernet devices actually do interpret the bits. > > > Undocumented as to which one.. > > > > > > and lets not get started on ecn. > > > > > > > > > > > > > > > Hack for getdns/stubby > > > > > > > > diff --git a/src/stub.c b/src/stub.c > > > > index 2547d10f..7e47aba5 100644 > > > > --- a/src/stub.c > > > > +++ b/src/stub.c > > > > @@ -52,6 +52,7 @@ > > > > #include "platform.h" > > > > #include "general.h" > > > > #include "pubkey-pinning.h" > > > > +#include > > > > > > > > /* WSA TODO: > > > > * STUB_TCP_RETRY added to deal with edge triggered event loops (v= ersus > > > > @@ -381,6 +382,9 @@ tcp_connect(getdns_upstream *upstream, getdns_t= ransport_list_t transport) > > > > # else > > > > static const int enable =3D 1; > > > > # endif > > > > +#endif > > > > +#if defined(IP_TOS) > > > > + int dscp =3D IPTOS_CLASS_CS4; > > > > #endif > > > > int fd =3D -1; > > > > > > > > @@ -390,6 +394,12 @@ tcp_connect(getdns_upstream *upstream, getdns_= transport_list_t transport) > > > > __FUNC__, (void*)upstream); > > > > if ((fd =3D socket(upstream->addr.ss_family, SOCK_STREAM, I= PPROTO_TCP)) =3D=3D -1) > > > > return -1; > > > > +#if defined(IP_TOS) > > > > + if (upstream->addr.ss_family =3D=3D AF_INET6) > > > > + (void)setsockopt(fd, IPPROTO_IPV6, IP_TOS, &dscp, s= izeof(dscp)); > > > > + else if (upstream->addr.ss_family =3D=3D AF_INET) > > > > + (void)setsockopt(fd, IPPROTO_IP, IP_TOS, &dscp, siz= eof(dscp)); > > > > +#endif > > > > > > > > > > > > Cheers, > > > > > > > > Kevin D-B > > > > > > > > gpg: 012C ACB2 28C6 C53E 9775 9123 B3A2 389B 9DE2 334A > > > > > > In my experience, except for a small number of cases (RDMA etc) Diffserv = is a > complete waste of time. There is no global ordering, there is no guarante= e against > starvation and any sane ISP strips the bits off or ignores them. I am under the impression that comcast at least, uses diffserv internally, and also has all kinds of fun mapping from MPLS back and forth. they also annoyingly, on some CMTSes, remark most traffic to CS1, which is why I was so insistent on getting the wash option into cake. diffserv is totally OK on your internal network, but most of the time, you really, really don't care. really time sensitive traffic gets tossed onto a priority vlan.... given the track record of diffserv, the idea of handing ECT(1) to the same folk as an identifier does not seem like a good idea, either. the first thing cake got when it hit the real world was proper tc support so you could identify and prioritize packets without twiddling the diffserv field. That said, I do rather support the ability to mark you own traffic within your domain, and in particular, I like the idea of a least effort or background marking for traffic you don't care about much. transmission does support qos markings... for tcp traffic.ledbat, no. Bunch of one line patches to a lot of tools and then where do we get? > Diffserv is even an issue at scale in the cloud. What does DSCP mean exac= tly on > outer headers, who gets to decide for which service. And what about inner= headers > and propogating inner to outer. Its a mess. yep. we tried, by following the proposed standards for webrtc and so on, to come up with a sane set of interpretations. If you think diffserv was a messy idea... se= e: https://datatracker.ietf.org/doc/draft-white-tsvwg-nqb/ and for that matter the recent thread on a new hop by hop options header from china telecom on netdev titled "net: ipv6: support Application-aware IPv6 Network (APN6)" inmates, asylum, running amuck. > -- Make Music, Not War Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-435-0729