From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id BDF583B29E for ; Fri, 2 Aug 2019 08:05:17 -0400 (EDT) Received: by mail-io1-xd2e.google.com with SMTP id i10so38287139iol.13 for ; Fri, 02 Aug 2019 05:05:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=uSL4TGlvuu4pKXyLhZOhuwiD3IgXXnsXPdrn5o2a5N8=; b=kOYGilnjtR5yWw+zdIctZoG0wdotYtW200UaZglETVVoyDBfPwC5lr2zcpDFMRoORM t4mJmWrP5U46YAlqC7auym6qrDGCwfTsCdKTKBggRbVMcCCqr1ObDeJMDWVQZJscMyHu 8KlPJcl847yN7BAufL5zBLwv0Y4Yr/GeOnJOV81V7C9ckHTwyB2E2fbnBCM4JojRpURe hZA7kTHYIc6gzSEPXTpeeP6sR8jv1oF89LlRPmaZ2bwGaHx4dNngokUB8lP9GKD1lBVL zvuj503ard8J1ZSRE/R+ilhJUn9xAkMVAOQBuIQUNh270apaFd+Lsx7oRgMtruktsvzF uhsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=uSL4TGlvuu4pKXyLhZOhuwiD3IgXXnsXPdrn5o2a5N8=; b=Ias2GUT5FVu4tz0GdAc29XugecO1+uzEjaKV8v+mG7/aHy0cvxWBQR7SznSFgncpeO 639R02PObuIe1Ygj5RR8mYQEGn9Bxy1F7VPdr/Chfvz93JtihUC03lA8pCZE6wBv3EKy mWPVQv4eAexRZRGHkjgADWurNRADFuKauK8Q/icRzDgIah+fZlLXfx/mLpidjVeUIkKx PBwDtUrQk8Yr+3oi4sy/deibFr0jylKR6YN6GKMOqfe64yy/ZYjn1lc8WASwAPuezLc0 ax9iBCQJVICsXQ3o2GZuyBtUZJbOAp3RTq126GlsH6A1oXws1ISaVy3pm3rTV5Higq7s pv1w== X-Gm-Message-State: APjAAAVInJiJxAZ8smKd3eC4rcBzNAYUaaswZ46JUWqU8TeE6Si5h/5k UpWI5nNzcmV/bUIuk2miXlv+sM4nu/N4kuZHFRE= X-Google-Smtp-Source: APXvYqy9TMUMgMZE7E1PHijIhKB201vaKmndpJX0y4Ca/agrKHG6jmXYQNdA7HEgEAIaVJsNFB2JqVQKZN/yRHNccYE= X-Received: by 2002:a6b:790a:: with SMTP id i10mr24038762iop.150.1564747516991; Fri, 02 Aug 2019 05:05:16 -0700 (PDT) MIME-Version: 1.0 References: <24f7b15a-129f-ca44-60e0-32c7d23eadf4@bobbriscoe.net> <0FD0931A-3035-4F65-BC91-E0267DF6537B@gmail.com> In-Reply-To: From: Dave Taht Date: Fri, 2 Aug 2019 05:05:04 -0700 Message-ID: To: Jonathan Morton Cc: Ruediger.Geib@telekom.de, tcpm IETF list , ECN-Sane , tsvwg IETF list Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Ecn-sane] [tsvwg] ECN CE that was ECT(0) incorrectly classified as L4S X-BeenThere: ecn-sane@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of explicit congestion notification's impact on the Internet List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2019 12:05:17 -0000 On Fri, Aug 2, 2019 at 4:10 AM Dave Taht wrote: > > On Fri, Aug 2, 2019 at 2:47 AM Jonathan Morton wr= ote: > > > > > On 2 Aug, 2019, at 9:29 am, wrote: > > > > > > Hi Jonathan, > > > > > > could you provide a real world example of links which are consecutive= ly narrower than sender access links? > > > > > > I could figure out a small campus network which has a bottleneck at t= he Internet access and a second one connecting the terminal equipment. But = in a small campus network, the individual terminal could very well have a h= igher LAN access bandwidth, than the campus - Internet connection (and then= there's only one bottleneck again). > > > > > > There may be a tradeoff between simplicity and general applicability.= Awareness of that tradeoff is important. To me, simplicity is the design a= im. > > > > A progressive narrowing of effective link capacity is very common in co= nsumer Internet access. Theoretically you can set up a chain of almost unl= imited length of consecutively narrowing bottlenecks, such that a line-rate= burst injected at the wide end will experience queuing at every intermedia= te node. In practice you can expect typically three or more potentially na= rrowing points: > > 0: Container and vm users are frequently using htb + something to keep > their bandwidths under control. > > 0.5: Cloudy providers use "something" to also rate limit traffic. > Policers and shapers, I assume. Stuff in the cloud thus far is looking quite jittery; sub-ms marking thresholds do not look feasible. I have no idea what sorts of software jitter and burstyness exist in NFV and ddpk based implementatons. > > > 1: Core to Access network peering. Most responsible ISPs regularly upg= rade these links to minimise congestion, subject to cost effectiveness. So= me consumer ISPs however are less responsible, and permit regular congestio= n here, often on a daily and/or weekly cycle according to demand. Even the= responsible ones may experience short-term congestion here due to exceptio= nal events. Even if the originating server's NIC is slower than the peerin= g link, queuing may occur here if the link is congested overall due to stat= istical multiplexing. > > > > 2: Access to Backhaul provisioning shaper. Many ISPs have a provisioni= ng shaper to handle "poverty tariffs" with deliberately limited capacity. = It may also be used to impose a sanity check on inbound traffic bursts, to = limit backhaul network traffic to that actually deliverable to the customer= (especially when the backhaul network is itself subcontracted on a gigabyt= es-carried basis, as is common in the UK). In the ADSL context it's often = called a BRAS. > > > > 3: Backhaul to head-end device. Generally the backhaul network is size= d to support many head-end devices, each of which serves some number of con= sumer last-mile links. I'm being deliberately generic here; it could be a = CMTS on a cable network, a DSLAM in a phone exchange, a cell tower, or a lo= ng-range wifi hub. In many cases the head-end device shares several subscr= ibers' lines over a common last-mile medium, so there is still some statist= ical multiplexing. In the particular case of a cell tower, the subscriber = usually gets less link capacity (due to propagation conditions) than his ta= riff limit. There are also bursts here from the vpn crypto engine. > > 4: CPE bufferbloat mitigation shaper. This is *post-last-mile* ingress= shaping, with AQM and FQ, to reduce the effects of the still-ubiquitous du= mb FIFOs on the above bottlenecks, especially the head-end device. The IQr= outer is a ready-made CPE device which does this. I would say "inbound shaper" to be clear. And its far, far wider than just that commercial product, Nearly everyone using this stuff shapes both in and outbound - be it untangle, netduma, asus "adaptive qos", edgerouter's or eero's sqm implemention, all of openwrt, dd-wrt, and deratives, bsd-based pfsense and opfsense, preseem does a bump in the wire for WISPs, streamboost, I think the list of off the shelf home router qos systems that shape inbound is well above 80-90%, and users have been trained to turn it on in both directions and off only when they run out of cpu. We see new product sales driven by the cpu cost of having to shape inbound, too. Policers have become quite useless in the past decade. > > > > 5: LAN, wifi, or powerline link. Most people now have GigE, but 100bas= e-TX is still in use in cheaper CPE and some low-end I'd so love to see powerline gain fq and AQM. I see it used a lot in busy apt buildings to drag stuff from room to room. It can be *awful* >computers, and this will be a further bottleneck in cases where the last m= ile is faster. For example, the Raspberry Pi has only just upgraded to Gig= E in its latest versions, and used 100base-TX before. Wifi is also a likel= y bottleneck, especially if the mobile station is on the opposite side of t= he house from the base CPE, and is additionally a "bursty MAC" with heavy a= ggregation. Some CPE now runs airtime-fairness and FQ-AQM on wifi to help = manage that. "some" includes most of qca's ath9k or ath10k products. (40% of the AP market?) Prominently known fq-codel for wifi users are ~3m chromebook users and ~3m google wifi, ( http://flent-newark.bufferbloat.net/~d/Airtime%20based%20queue%20limit%20fo= r%20FQ_CoDel%20in%20wireless%20interface.pdf ) and nearly everyone else in the 802.11s meshy market... meraki is a known sfq + codel user... starting in 2014... A large number of wifi APs and p2p links oft have a faster wifi speed than their 100base-tx link, and thus use the ethernet (with either a short fifo or fq_codel) to smooth the bursts out. I can point to a few ubnt products that I know a bit too much about. There's also some 802.11ad and ay.... there is a quite remarkable amount of 100base-tx gear still being deployed. If it helps any to those here doing simulation, long ago I put a "slot" and statistical distribution of busty mac delay feature into netem. I can say now that that was used to help guide the development of google "stadia", and certainly "slotting" is a MAJOR tool that I'd plug into the SCE work to better emulate the real orld behaviors of bursty macs. google has collected WAY more examples characterizing real world micro(bursts) than I could ever deal with, and I hope they publish that data someday for others to use. > > > > I think the above collection is not at all exotic. Not all of these wi= ll apply in every case, or at all times in any given case, but it is certai= nly possible for all five to apply consecutively. For bursty. > > - Jonathan Morton > > _______________________________________________ > > Ecn-sane mailing list > > Ecn-sane@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/ecn-sane > > > > -- > > Dave T=C3=A4ht > CTO, TekLibre, LLC > http://www.teklibre.com > Tel: 1-831-205-9740 --=20 Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740