From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt1-x82c.google.com (mail-qt1-x82c.google.com [IPv6:2607:f8b0:4864:20::82c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 8B3033B2A4; Mon, 11 Mar 2019 12:34:28 -0400 (EDT) Received: by mail-qt1-x82c.google.com with SMTP id b16so4670504qtt.2; Mon, 11 Mar 2019 09:34:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=FbKyjbLDzZtw+M/PBfKXFblO5MD+GCcBqkJY3IQInP0=; b=YBG1QZsiEKs8QbsuwNTyyY4Lufy3fvmQM6nUEuJNjx7xgGWUBeLgq2diToMRwt/CoE 7/hwt1+ncdjwm2Y1PHDqaU2RByqMPng2aGWumR5VVy5w/CRoYDDAna3TCcHbMcz2d78l nshkQPK8Bg3WFLO7fJE53Qwg1uSJESJpHNIexsPyw+qI7+Y3IRWQopHdRmxTxtHVfqg+ qzTzf9CTVrsaFQ+NaUtW/09Oo2RyW3twyPXndR5AG93x7V+u9JrVoiqgyhWf9RVbQNf1 kMhtMHLup6yCxOK25yj14RtflmJn5QGYPAatlWw+KAiuQHZEHaTv6pFXa9gqzZrjx7nS PK2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=FbKyjbLDzZtw+M/PBfKXFblO5MD+GCcBqkJY3IQInP0=; b=E5BDbUTe8J/tE86xk3LvVqM2hQpIcoT+dFpo7n7hKzCensRzsOphR7NBX+K2dsx0Es bnhq4PVnrB80oa9RrB/EHyriVqb48Lvk7Ojd4RcKPbzxmaTPLXXNLW6gHWTewbWFAOSj qH8G+gSJhQYJPOSds+KLW62aWuCiR5NuSZfFKmMAYjvAU7sk9M+P75b4g/Ttgpu3yUX9 h36Py9aHBULHrYdDlyex8C8qnorwAhLZ3sI8fUsqETAVN5KxzWWlKt0j02JmQBVGQciA 9ZeAXV2IQGhCgJPRpCU0R/braG//ySjScSh8kocjqyLCzGg0k9q9925TjD7r/5dydFgS JUFw== X-Gm-Message-State: APjAAAWQvYDWpbJjf9ZSd/MNm/5r22SRALUskNWBBy10p8JNd7C2iJqt 4rDh6P6Tmoa2iftxc87tc7PyEVbtvCBVhEL9Pvg= X-Google-Smtp-Source: APXvYqweYdpeaTnLcBmIOrBwCCGj9UstCMbwwyQ8UxSVOF1rSIQtviAzWPXf5qh7YqxME2BhvF5kE//cIR+F7ZtpErc= X-Received: by 2002:ac8:3f46:: with SMTP id w6mr27087764qtk.175.1552322067926; Mon, 11 Mar 2019 09:34:27 -0700 (PDT) MIME-Version: 1.0 References: <550C0248-1704-49DA-ABDC-49A91E0AC6F3@akamai.com> <1EE25778-8571-4506-A334-38C544470ACE@gmail.com> <2594F939-829A-443C-B1FE-2193F870E48B@gmail.com> <00133e20-6639-c6db-a06c-57856bf78167@bobbriscoe.net> In-Reply-To: <00133e20-6639-c6db-a06c-57856bf78167@bobbriscoe.net> From: Dave Taht Date: Mon, 11 Mar 2019 09:34:16 -0700 Message-ID: To: Bob Briscoe Cc: Jonathan Morton , bloat , ecn-sane@lists.bufferbloat.net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] [Codel] The "Some Congestion Experienced" ECN codepoint - a new internet draft - X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 16:34:28 -0000 On Mon, Mar 11, 2019 at 7:14 AM Bob Briscoe wrote= : > > Dave, > > L4S is far from dead. It's merely been working differently from how you'r= e used to. Those working on an L4S AQM (at least those in the cable industr= y) had to have a private WG for the last ~18 months, but now we're allowed = to publish and talk openly again. Similarly, there's work under the covers = on an L4S AQM in switch hardware. After traffic essentially vanished from the various ietf mailing lists, and specs kept dropping, I assumed there was work being done somewhere. I'm very glad that it's now more open We announced the ecn-sane project and it's goals last august. If you and/or you group cannot participate under those house rules please suggest changes. And/or resurrect an appropriate list within the ietf. https://www.bufferbloat.net/projects/ecn-sane/wiki/ >And I see external signs of work under covers on DSL access equipment (cov= ers that I am not under any longer). It has certainly been my hope that the DSL folk would at least wake up and implement BQL in their lowest level firmware for about 8 years now. Free.fr basically did that in 2012 when they shipped fq_codel on their revolution series of modems. > > Nonetheless, I think you will see updated Linux code for an L4S DualQ Cou= pled AQM built against the mainline tree appear on netdev list today. I am beyond delighted to finally have a chance to evaluate this. Have you run any flent related tests through it yet? Regrettably since this code posting is so close to netdevconf it's going to be very difficult for me to do a comprehensive evaluation in time to form an objective opinion. I'm busy on something else. > =3D=3DIn summary=3D=3D > > The problem that the SCE draft identifies with TCP's sharp multiplicative= decrease is also the primary problem that L4S identified. Yes, we have long been in agreement that some congestion signal should be an earlier notification than drop. > Like L4S, SCE requires changes to network, sender and receiver (see comme= nt later about the rcv-window approach hinted at in the SCE draft). But SCE= is just starting on its journey. Having to change end systems and network = together is really tough and takes many years. Not really, the AQM portion of SCE could roll out very quickly across all of the linux and freebsd universe, if consensus is achieved that it's a good idea. > > It seems you're trying to do the same thing as L4S, but by slightly diffe= rent means. Before splitting the people involved in this into two factions,= can you say what you didn't like about the L4S approach in the first place= ? We've been very careful to specify L4S broadly enough so that it can enco= mpass many different approaches within it. I've read the 100+ of spec now multiple times over the years (and all your work on ecn in general), and I hope, that once we get a bit of time, we can do a detailed comparison of the two approaches. But, honestly, based on the total inactivity on the tcp prague mailing list that it had died, until recently. > > The only thing stated against L4S I can find is that it's taking a long t= ime. Starting an identically difficult approach now is going to restart the= clock, and take a lot lot longer. SCE and the modifications to the relevant already IETF approved AQMs are extremely straightforward, backward compatible approaches to extending RFC3168 for all existing transports. It's not an identically difficult approach at all. Aside from some more detailed analysis of transport effects and the inevitable debate in netdevconf and ietf, rolling out SCE, at least in openwrt and linux, could be happen by about.... june. Particularly with the now more readily available source code to compare for the two approaches, independent experts should be able to leap in and provide feedback. > =3D=3D2 output values vs. 2 input values.=3D=3D > > We considered schemes where the AQM can use a second marking as a lower s= trength /output/ (like VCP, my own QV and now SCE). But we worked out that = there were a wider range of advantages and much more significant performanc= e improvements from the sender using a second marking to distinguish how it= will behave (i.e. a second /input/ to the classifier in front of the AQM). > > Don't get me wrong. It's useful that you guys are putting the work in on = SCE. Then everyone can compare the two approaches (again), as a check on wh= ether that decision was correct. That's important, cos ECT(1) is the last u= seful codepoint in the IP header. See: "Notification of Less Severe Congest= ion than CE" at https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#= appendix-C.2 where we've written: Yep. I am glad that both of our cards are on the table. > > Before assigning ECT(1) as an identifer for L4S, we must carefully > consider whether it might be better to hold ECT(1) in reserve for > future standardisation of rapid flow acceleration, which is an > important and enduring problem [RFC6077]. > > > =3D=3DFQ-only vs. FQ or DualQ=3D=3D > > One of the problems with the 2 outputs approach (SCE etc.) is that it is = only possible with per-flow queuing. I doubt you'll get the last useful cod= epoint in the IP header for just that. It's sort-of obvious that, if you tr= y to implement SCE in a FIFO, you can only have one queue length for all th= e flows. Then legacy TCP flows that don't understand SCE would push the que= ue deeper to the CE threshold, ruining it for the flows that support SCE. > > We worked out that the 2 inputs approach (L4S) is more generic - ie. it c= an be used with FQ or DualQ (multiple or just 2 queues). > > For instance, you can modify fq_CoDel for senders that uses ECT(1) to ind= icate that they support a small multiplicative decrease (L4S senders). You = only need the following: Include the last bit of the ECN field with the flo= w ID when you do the classification for sfq. Then in the queues with ECN=3D= =3DX1, you instantiate a shallow threshold ECN AQM. This could be CoDel wit= h a shallow 'target', but you also want it to respond immediately (zero 'in= terval'), so even a simple step at about 1ms will work, but a random RED-li= ke ramp on the /instantaneous/ queue is much better. > > =3D=3DRe-purposing the Receive Window?=3D=3D=3D > > Receiver congestion control using the receive window may seem like a usef= ul stop-gap, but it runs counter to how nearly all today's transport protoc= ols are intended to work (except, I know of a LEDBAT-like example from Micr= osoft Research). So you will have your work cut out proving that it is stab= le and that the two ends don't fight, etc. if you think L4S is taking years= , you will find that takes longer. There is current research on this that I= can point you to, if you want. I kind of wish we'd cut that from the draft. My proposal was considerably different, and a bit longer in the prior paragraph. > That's why we chose an approach that had a pre-existing widely deployed e= xistence proof (DCTCP) to start from. I have never had a good number for DCTCP's actual deployment. In the cloud services I use, it's all (paced) cubic and BBR nowadays. > > IETF groups like rmcat explicitly decided early on to require the approac= h where the receiver is a dumb reflector, then new sender congestion contro= l algorithms can be deployed unilaterally. The argument was that the feedba= ck function can be thought of as a sub-layer below the congestion control f= unction. The ongoing addition of accurate ECN feedback to TCP and to QUIC a= lso take the dumb reflector approach. And RTCP already does it that way. I do think there is more work to do here. > =3D=3DECN feedback problems=3D=3D=3D > > Over the last decades, we've made sure that the ECN feedback schemes for = TCP, QUIC, RTP (but not SCTP yet) can all feed back ECT(1) as well as CE, i= n case a scheme like SCE came along. > > However, the solution in the TCP case [draft-ietf-tcpm-accurate-ecn] is s= till problematic for SCE if you're impatient. The base scheme overloads 3 b= its in the TCP header, which it uses to feed back CE only. To feed back ECT= (1) we had to add a TCP option. That's not going to get through middleboxes= for many years. The TCP option is also optional to implement. Two of the m= ain TCP developers are currently saying they will probably not implement it= , at least not initially. One thing I've not understood about L4S is why didn't you just pick a new NNN codepoint for it and not fiddled with the ECT(1) at all. > =3D=3DTunnels and lower layers=3D=3D > > Over the years I've maintained a fairly lonely activity to make sure that= the ECN propagation behaviour of tunnels and layer 2 protocols will treat = ECT(1) as either a stronger output signal (as in SCE) or as an alternative = input signal to an AQM (as in L4S). Theoretically, this allows either the S= CE or the L4S approach. > > HOWEVER, you would probably not be surprised at how many people read the = spec [RFC6040], and say "Ah, no router alters ECT(0) to ECT(1) today, so I'= m not going to implement that unnecessary extra line of code in my tunnel d= ecap." For the record I have made sure your efforts to do ecn as right as possible made it into new things - notably wireguard "does it right", now - and mosh has a wonderful response to ecn that we put in there years ago. I am grateful for your work on clarifying these things. > > =3D=3DWider benefit: Relaxing link ordering=3D=3D > > By overloading the ECT(1) marking to mean "the sender uses time for loss = detection" a link can relax the reordering requirement on ECT(1) packets to= day. You can do that with L4S, cos the sender is selecting the marking. You= can't do that when the AQM is selecting the marking (as with SCE). > > If transport protocols detect loss in time units without tying it to any = marking (as in RACK on its own), a link cannot use this to relax the orderi= ng requirement until it is sure that all the legacy non-RACK transports hav= e decayed out of the network. That would be measured in decades. There's an awful lot here worthy of discussion that I cannot respond to today, and I've got this other, open, mailing list for it. You are welcome to join. > HTH > > > > Bob > > On 11/03/2019 10:11, Dave Taht wrote: > > Everybody, calm down. I put this out merely to get comment before we > submitted the first of several drafts. That draft is now submitted and > we've asked for a talk slot in the tsvwg for it. I cc'd the world to > get quick initial feedback, and I want to shut this overbroad > conversation down and move it to just the ecn-sane mailing list. > > The l4s mailing list is dead, and the debates on the AQM mailing list and= here, > unhelpful - for decades. So, back in august I started a new working > group here, under house rules that I thought would be more productive, > and asked that people that wanted to debate ecn more sanely, join. few > did. > > And jon and I have been working for months (and largely not on the > list) to try and create a compromise proposal of which y'all just saw > the first output. There's more in the bufferbloat-rfcs repo. > > The rules for joining the ecn-sane list are simple - take the time to > step back and write a write a short position paper, and join (or > create) a team. You needn't do that immediately. If you disagree with > the rules of operation of the ecn-sane working group, submit a pull > request or file a bug on the web site. where we can discuss it. > > Ironically our ssl cert just expired and I don't remember how to fix it. > > Please join the ecn-sane mailing list for discussing this stuff and > stop cc-ing the whole bufferbloat.net world on it, please. > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > > > -- > ________________________________________________________________ > Bob Briscoe http://bobbriscoe.net/ --=20 Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740