From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id F11D03B29E for ; Wed, 22 Apr 2020 12:44:37 -0400 (EDT) Received: by mail-pg1-x544.google.com with SMTP id d17so1388923pgo.0 for ; Wed, 22 Apr 2020 09:44:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sak98CX+j5NFCvExvgok4ZLZSmIwyajflLUSBGnORZo=; b=FMGX8DIkjoJnwJ8+BnzwXao4kuaxe3XzolpxXlE4KlVvbZemWLkRdGMz9vY5b4psTD WRXyff+Lt3sB0mssqKsmPNNEt6p2I1k1xf+yD8bsa+pv25vc3Aw97uT7wjikit/EYVIm 2+1deqiGZYKRRdUGvYrAfliYXvvtUapbcGWxhGpe4wwvKgzDAN3ePkAGA0ibZxEH491a +J+RcSziEnKlXe7RIP10THmBoJnrAHp81GAJlfGXXVNK73RJOLrb/5SGT5qWacsa2LNs sxLYHaOsbeFbqXH2yJtyMIBOAeoVtC1ItKF8sH1HhVIfGB2g1qW6L7uodkR7+0Fe629F 7uDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sak98CX+j5NFCvExvgok4ZLZSmIwyajflLUSBGnORZo=; b=nc37qOvuLNm03/T/ggWrmqm718PYfVl9FKLXQUht7405b8jUrt4xOLZZxJl/hdKkU3 VlujHgxMvDuBxXbBh4x+dI9DNaHvf1wAoRwmefQ4UFNAHF/VO7AmaazGl0YYxNNV6OER x0v4+0div9cfWFKlwn+yRmah25pFA54yX+J5neDxKa/j6bYdgz8ODr7OEomtt0UU0JFt k8vsud4pVAGtzQWrpRIoe8TxAa7ILia5iv9ZTSi7Xfu5cxl2pqgPV5mncb3KrmmjrhXS aSMQXJ5UFMi0IGFQZ9izq31aSwlKvovyAi63OSWW2ymfFqQUBues7gUVOxn/Sn68wE+v j5dw== X-Gm-Message-State: AGi0PuYp+2mQen/AYSWcOSA9wWR5lE/XvFW+A2Kc5rioaSVkr5X55Hfe Snj88EHb32waeKh0krPHxaHRACRVZ7Y= X-Google-Smtp-Source: APiQypIVvMv/NUXcZvUByul0zKaQgNU2Rv2CN5OlGw/E4vTXpfi75S8Z4mXbW+SDJElx2z33ehKUVg== X-Received: by 2002:a63:c604:: with SMTP id w4mr7271097pgg.270.1587573876991; Wed, 22 Apr 2020 09:44:36 -0700 (PDT) Received: from hermes.lan (204-195-22-127.wavecable.com. [204.195.22.127]) by smtp.gmail.com with ESMTPSA id d20sm5928452pjs.12.2020.04.22.09.44.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Apr 2020 09:44:36 -0700 (PDT) Date: Wed, 22 Apr 2020 09:44:28 -0700 From: Stephen Hemminger To: Dave Taht Cc: Kevin Darbyshire-Bryant , Cake List Message-ID: <20200422094428.2ee29433@hermes.lan> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Cake] DSCP ramblings X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Apr 2020 16:44:38 -0000 On Wed, 22 Apr 2020 09:20:29 -0700 Dave Taht wrote: > and because of your I'm off building collectd because those graphs > look so good. :) >=20 > https://forum.openwrt.org/t/sqm-reporting/59960/24 >=20 > I have long just used snmpd, and collectd looks interesting. I fear > it's too heavyweight, particularly shelling out to a script.... >=20 > On Wed, Apr 22, 2020 at 9:15 AM Dave Taht wrote: > > > > On Wed, Apr 22, 2020 at 8:58 AM Kevin Darbyshire-Bryant > > wrote: =20 > > > > > > During these strange times of lockdown I=E2=80=99ve been trying to ke= ep myself occupied/entertained/sane(???) by =E2=80=98fiddling with stuff=E2= =80=99 and improving my coding. This started with an idea of learning Pyth= on which was great until the on-line bit of it ran out and someone posted a= n idea on the Openwrt forum about graphing Cake stats. > > > > > > That had nothing to do with Python and involved (new to me) technolog= ies such as =E2=80=98collectd=E2=80=99, =E2=80=98JSON=E2=80=99, a bit of ja= vascript and my usual level of cobbling something together in =E2=80=98ash= =E2=80=99=E2=80=A6. So that course was well spent :-) > > > > > > Anyway, data was collected and graphs produced in a very small househ= old. What=E2=80=99s immediately apparent from those graphs and cake in =E2= =80=98diffserv4=E2=80=99 mode is that very, very few applications are using= DSCP at all. Most things are to port 443. > > > > > > I was also a little surprised to see that my DNS over foo proxies suc= h as stubby & https-dns-proxy don=E2=80=99t use DSCP coding. It surprised = me even more to see RFC recommendations that DNS be treated as =E2=80=98Bes= t Effort=E2=80=99. Now in the days of udp only and no dnssec (with fallbac= k to tcp) this may be good enough, but I wonder if this is realistic these = days? > > > > > > So putting aside the discussion of what codepoint should be used, I t= hen wondered how hard it would be to actually set a dscp in these applicati= ons. And this is where I had another surprise. For example https-dns-prox= y uses libcurl. libcurl has no standard =E2=80=98in-library=E2=80=99 metho= d for setting a socket=E2=80=99s dscp. I cobbled a workaround in the appli= cation https://github.com/aarond10/https_dns_proxy/pull/83 - it works. > > > > > > Next I attacked stubby, which uses getdns. getdns doesn=E2=80=99t ev= en have a callback or parameters passing so you can set a dscp on the socke= t from a client application, pure =E2=80=98hack the library=E2=80=99 stuff. > > > > > > To be blunt and on a small sample of 2 libraries/applications, it see= ms that DSCP is completely ignored. Applications signalling =E2=80=99this = is/isnt latency sensitive/bulk=E2=80=99 isn=E2=80=99t going to happen if it= isn=E2=80=99t easy to do. > > > > > > Apple should be marking facetime calls as being =E2=80=98video confer= ence=E2=80=99 or whatever. BBC iplayer Radio apps should be marking =E2=80= =98audio streaming=E2=80=99. But every f*ing thing is CS0 port 443. And I= =E2=80=99m wondering how much of this is because library support is simply = missing. Maybe gaming apps are better? (I don=E2=80=99t game) > > > > > > Right, I=E2=80=99m off for a lie down. Sorry for the rant. =20 > > > > Welcome to my explorations... in 2011. Diffserv is rather underused, is= n't it? > > > > I took a survey of every (500+) gaming console at a convention. nearly > > zero diffserv usage and it was all over the map, and I think, mostly, > > from osx. > > > > windows requires admin privs to set the tos bits at all > > webrtc has an api to set the bits, but it doesn't work on windows. > > > > ssh will set the imm bit for interactive, I forget what it sets for bulk > > bgp sets cs6. so does babel. Arguably both usages are wrong. > > some windows stuff sets cs1 for things like ping > > I got the mosh folk to use AF42 as a (worldwide) test, for nearly a > > year. they had one user with a problem and they turned it off. It was > > funny, keith thought I was making an expert recommendation rather than > > a test and just copy pasted my code into the tree and shipped it. > > > > linux implements a strict priority queue in pfifo_fast. You can dos it > > if you hit it by setting the bits. > > irtt and netperf let you set the bits. iperf also. > > > > I produced a patch for rsync in particular (since I use it heavily) > > > > sqm at least used to mark dns and ntp as some elivated prio, but I > > forget which and for all I know the cake qos system doesn't implement > > those filters. > > > > A few multi-queue ethernet devices actually do interpret the bits. > > Undocumented as to which one.. > > > > and lets not get started on ecn. > > =20 > > > > > > > > > Hack for getdns/stubby > > > > > > diff --git a/src/stub.c b/src/stub.c > > > index 2547d10f..7e47aba5 100644 > > > --- a/src/stub.c > > > +++ b/src/stub.c > > > @@ -52,6 +52,7 @@ > > > #include "platform.h" > > > #include "general.h" > > > #include "pubkey-pinning.h" > > > +#include > > > > > > /* WSA TODO: > > > * STUB_TCP_RETRY added to deal with edge triggered event loops (ver= sus > > > @@ -381,6 +382,9 @@ tcp_connect(getdns_upstream *upstream, getdns_tra= nsport_list_t transport) > > > # else > > > static const int enable =3D 1; > > > # endif > > > +#endif > > > +#if defined(IP_TOS) > > > + int dscp =3D IPTOS_CLASS_CS4; > > > #endif > > > int fd =3D -1; > > > > > > @@ -390,6 +394,12 @@ tcp_connect(getdns_upstream *upstream, getdns_tr= ansport_list_t transport) > > > __FUNC__, (void*)upstream); > > > if ((fd =3D socket(upstream->addr.ss_family, SOCK_STREAM, IPP= ROTO_TCP)) =3D=3D -1) > > > return -1; > > > +#if defined(IP_TOS) > > > + if (upstream->addr.ss_family =3D=3D AF_INET6) > > > + (void)setsockopt(fd, IPPROTO_IPV6, IP_TOS, &dscp, siz= eof(dscp)); > > > + else if (upstream->addr.ss_family =3D=3D AF_INET) > > > + (void)setsockopt(fd, IPPROTO_IP, IP_TOS, &dscp, sizeo= f(dscp)); > > > +#endif > > > > > > > > > Cheers, > > > > > > Kevin D-B > > > > > > gpg: 012C ACB2 28C6 C53E 9775 9123 B3A2 389B 9DE2 334A > > > In my experience, except for a small number of cases (RDMA etc) Diffserv is= a complete waste of time. There is no global ordering, there is no guarantee = against starvation and any sane ISP strips the bits off or ignores them. Diffserv is even an issue at scale in the cloud. What does DSCP mean exactl= y on outer headers, who gets to decide for which service. And what about inner h= eaders and propogating inner to outer. Its a mess.