From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt1-x82e.google.com (mail-qt1-x82e.google.com [IPv6:2607:f8b0:4864:20::82e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id D51F63BA8E for ; Mon, 18 Mar 2019 23:19:09 -0400 (EDT) Received: by mail-qt1-x82e.google.com with SMTP id v20so20553750qtv.12 for ; Mon, 18 Mar 2019 20:19:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=jVZZc16a0KXQ3EjKcsKkChR+fv+aXCChjqU19gyBQBs=; b=b+OPSsHAazgGOKUK3USk3bbhdx8kooEfedIBBvokeUDVGipLtS47YAwtkLW3frsSx1 7BCmsZsAgayRAictGYOwQwf9UHRGsoiyNqP2WA39hRhJqM8r1VPNyyirbUu6NqX/E86n d0Otz3llXmspH5IkC7OFJGbs6HgD5Fq985Y+Wn89KuIjqwri1H7Il6SQh5L9Buhs5Hhp fTZjEByUqjPRVaaQdIbQSuyc/1Wt6a/XDekNhIwrIp3Eg2c2xvTlg8ExxQrvi1HBIhGc SLc4G/3JXckahP8ZC0KQoSe7Ux3Czpw1mIzjZNaL0QHW9lJwmhI49qrYxvjaTAUeKs31 /ilw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=jVZZc16a0KXQ3EjKcsKkChR+fv+aXCChjqU19gyBQBs=; b=dy0q8IlzcgblgcTVRE7BvKYlTyMmETSJMsqGPUkiFKXU49e57eRiHUYb0NSL8ZBeNL n1terNzAwJNi/BsGfsDJ0B3MPr/MDB8yaR0IjZmcuG10KNNh2YqUfd4Q145xiyLpc0N1 jWL4YqMrf95lvGGQ3B7i0zNuUZRE5IuadeHcIDBJ9l0w/hOiTF0tCbJmpGA3jRtuSZBH 02X0K2G3wI68XhF/iyGJRhP1LJS5l+IKZWw7DGu039DgDrWlcXLoKI6WVdI7XMyYdKA8 H2y1ZzbHpQiYHaDemxGjnrmmg1Rt1pWZL2wbzY2P6RwoEg+POWzfgC7bL2qJmvMd2XUe Mp2g== X-Gm-Message-State: APjAAAVV4bNbD8jY/2uFoZfTkgwASLop3gxQNmRc6w6wsqDzBHKfRiK5 yaBTxs3jLJ8QF5QfnNxRRZ5FkjIznC7Uje5NYAk= X-Google-Smtp-Source: APXvYqwLlc7hq4yBjveR8xjifY2Wpha+C75JIpEiZ3ZvVLaa6gkVt6Wdt7h2a117LW09xA5SMCZw2RsoqeKzgcoXmlA= X-Received: by 2002:ac8:2207:: with SMTP id o7mr180866qto.376.1552965549121; Mon, 18 Mar 2019 20:19:09 -0700 (PDT) MIME-Version: 1.0 References: <1E80578D-A589-4CA0-9015-B03B63042355@gmx.de> <27FA673A-2C4C-4652-943F-33FAA1CF1E83@gmx.de> <1552669283.555112988@apps.rackspace.com> <7029DA80-8B83-4775-8261-A4ADD2CF34C7@akamai.com> <1552846034.909628287@apps.rackspace.com> <5458c216-07b9-5b06-a381-326de49b53e0@bobbriscoe.net> In-Reply-To: <5458c216-07b9-5b06-a381-326de49b53e0@bobbriscoe.net> From: Dave Taht Date: Tue, 19 Mar 2019 04:18:56 +0100 Message-ID: To: Bob Briscoe Cc: "David P. Reed" , Vint Cerf , tsvwg IETF list , bloat Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Mar 2019 03:19:10 -0000 On Tue, Mar 19, 2019 at 2:07 AM Bob Briscoe wrote: > > David, > > On 17/03/2019 18:07, David P. Reed wrote: > > Vint - > > > > BBR is the end-to-end control logic that adjusts the source rate to match= the share of the bolttleneck link it should use. > > > > It depends on getting reliable current congestion information via packet = drops and/or ECN. > > > > So the proposal by these guys (not the cable guys) is an attempt to impro= ve the quality of the congestion signal inserted by the router with the bot= tleneck outbound link. > > What do you mean 'not the cable guys'? > This thread was reasonably civil until this intervention. I think the inventor of udp, and the e2e argument has good reason to blow his top. I also think vint asked a reasonable question - has this been tested vs bbr? > > > THe cable guys are trying to get a "private" field in the IP header for t= heir own use. > > > There is nothing private about this codepoint, and there never has been. = Here's some data points: I hope https://github.com/dtaht/bufferbloat-rfcs/blob/master/sce/draft-mort= on-taht-tsvwg-sce.txt gets comprehensively evaluated in comparison to the more private use of ect_1 you are talking about, > * The IP header codepoint in question (ECT(1) in the ECN field) was propo= sed for use as an alternative ECN behaviour in July 2105 in the IETF AQM WG= and the IETF's transport area WG (which handles all ECN matters). I called for the closure of that wg because it had become an endless bikeshed. Since the aqm chairs refused to apply long honored traditions of the ietf - like *running code*. I ended up forming a new wg https://www.bufferbloat.net/projects/ecn-sane/wiki/rules/ I like my rules a lot better than what ended up going on here. I do hope more folk join my wg. > * A year later there followed a packed IETF BoF on the subject (after 2 o= pen Bar BoFs). >From which many, including myself, ran screaming. I actually quit the ietf last prague, and went off to implement real code in real products. Shipping. I lost track of the numbers after they cracked 10m. > * Long discussion ensued on the merits of different IP header field combi= nations, on both these IETF lists, involving people active on this list (bl= oat), including Dave Taht, who is acknowledged for his contributions in the= IETF draft. I have called repeatedly that my name be removed from the l4s related drafts, in private, and in public. I in no way endorsed this effort, except as a (somewhat dubious) experiment. > * That was when it was decided that ECT(1) was most appropriate. Love the passive voice there. Everybody else had left the room. > * The logic of the decision is written up in an appendix of draft-ietf-ec= n-l4s-id. Yea, read that. Lousy logic. My notes from that period said - "prob won't work in a container/vm/network namespace. design an experiment to test it when running code arrives." Running code never did. None of this stuff has been subjected to rigor we put the aqms through in the aqm group. There's no public ns2, or ns3 models, no public implementations at all... and y'all are proposing to write tcp prague from scratch at a hackathon this weekend? come on. > * David Black, one of the co-chairs of the IETF's transport area WG and c= o-author of both the original ECN and Diffserv RFCs, wrote RFC8311 to lay o= ut the process for reclaiming and reusing the necessary codepoints. yea, that was good. used that for wireguard and now SCE. > * This work and the process of freeing up codepoints has been very visibl= e at every IETF ever since, with multiple drafts to fix other aspects of th= e protocols working their way through the IETF process in multiple WGs (tsv= wg, tcpm, trill, etc). amazing levels of funding for that bit. Why on earth couldn't you hire a decent hacker and get a version of tcpprague.c done 3 years ago? You're Doing it at a hackathon next week? You know how much testing even a simple change to tcp or and aqm goes through at google or bufferbloat.net? Even with running code and widescale live testing? I've been trying to analyze one tiny change to fq_codel on real networks for close to a year now.... seems to work, so that's part of the now SCE enabled repo here: https://github.com/dtaht/fq_codel_fast > * L4S has also been mentioned in IETF liaisons with the IEEE and 3GPP. > > Some history: > * I had been researching the idea since 2012. > * In fact my first presentation on it was scheduled directly after Van Ja= cobson's first presentation of CoDel at the IETF in July 2012. VJ overran b= y nearly 20mins leaving just 3 mins for my presentation. > * Mirja Kuehlewind and I did early groundwork in 2013 and published a pap= er > * Then I (working for BT) brought the work into the EU RITE project (Redu= cing Internet Transport Latency) with Simula, Alcatel-Lucent, etc. > * By 2015 the two main L4S proponents were Koen De Schepper from Alcatel = Lucent and myself (I had just switched from BT to Simula), along with Olga = Bondarenko (now Albisser), a PhD student at Simula who now works for Micros= oft (on something else) and is still doing her PhD part-time with Simula > o By that time, Al-Lu and Simula had a cool prototype running. > o This was demonstrated publicly for the first time in the IETF AQM WG = over DC+core+backhaul+DSL+home networks. with a contrived benchmark. > https://riteproject.eu/dctth/#1511dispatchwg > * In May 2016, L4S was demonstrated at MultiMediaSystems'16 with /every/ = packet from all the following simultaneous applications achieving ~1ms queu= ing delay or less over a 40Mb/s Internet access link (7ms base RTT): not measuring any normal traffic. on a contrived, proprietary, "bigbuckbunny" benchmark I have blasted for its misrepresentation of how basic internet protocols work multiple times and suggested instead that the enormous suite of tests we developed for pie/codel/fq_codel at least be run. Your group has refused to do so. Our flent and irtt benchmarks are GPL3, developed in public, with a bog-std output format and terabytes of data published.They are licensed that way so that they cannot be gamed. Now that the code is finally starting to land everyone can go and run some real benchmarks and form their own opinions. > o cloud-rendered remote presence in a racing car within a VR headset > o the interactive cloud-rendered video already linked above > o an online gaming benchmark > o HAS video streaming > o a number of bulk file downloads > o a heavy synthetic load of web browsing Not one of these being independently reproducible or repeatable. No source code, no means to verify. > > L4S has never been access-technology-specific. Indeed the cable industry = has been funding my work at the IETF to help encourage a wider L4S ecosyste= m. There is nothing private to the cable industry in this: This works on wifi? Got code? > * Al-Lu used DSL as a use-case, but L4S was relevant to all the access te= chnologies they supplied. > * BT was obviously interested in DSL, > * but BT's initial motivating use-case was to incrementally deploy the lo= w queuing delay of DCTCP over BT's data centre interconnect networks. Just in terms of deployment fq_codel, in free.fr *alone* is well above a million units (as of 2015), and persistently in the top rankings of http://www.dslreports.com/speedtest/results/bufferbloat?up=3D1 You guys are at ZERO. Still. > * In Jul 2016 the open-source Linux repo for the Coupled AQM was publishe= d, with a fully working version to be used and abused. No open source developer can get near that code legally. https://fsfe.org/activities/os/why-frand-is-bad-for-free-software.en.html It's not open source with a frand patent. OSI statement here: https://opensource.org/node/616 Period. I'd asked repeatedly that patent be lifted. crickets. I asked nokia's contact for the ipr. crickets. > Now at: https://github.com/L4STeam/sch_dualpi2_upstream I am delighted you finally got some code possibly worth reviewing by expert= s. > * Of course, DCTCP was already open-sourced in Linux and FreeBSD, as well= as available in Windows Beating that patent took forever. I'm glad microsoft joined OIN. > * In Jul 2016, the main IETF BoF on L4S was held: > o Ingemar Johansson from Ericsson was one of the proponents, focused on= using L4S in LTE > o along with Kevin Smith from Vodafone and > o Praveen Balasubramanian from Microsoft (who maintains the Windows TCP= stack, including DCTCP). > o Ingemar has since written an open-source L4S variant of the SCReAM co= ngestion controller for real-time media: https://github.com/EricssonResearc= h/scream/ I have to note that google congestion control (gcc) won in the marketplace and I have no idea why anyone tries with scream anymore. I liked some bits of nada. > o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented the necessary= feedback in TCP https://github.com/mirjak/linux-accecn it does help to submit code to netdev for mainline consideration? this repo has not had a commit since 2017. > * In summer 2017 CableLabs started work on Low Latency DOCSIS, and hired = me later in the year to help develop and specify it, along with support for= L4S > o In Jan 2019 the Low Latency DOCSIS spec was published and is now bein= g implemented. Thought it was just another limited experiment til last month. *really*. Not once in all our communications in the past 2 years did I have the slightest clue what was up. Formed my own wg, under rules true to the ietf's original spirit ( https://www.bufferbloat.net/projects/ecn-sane/wiki/rules/ ) - made progress, submitted a *goooood * draft, and then, wow, all kinds of stuff started coming to light that nobody knew about. > o You can find the primary companies involved at the end of the White P= aper. https://cablela.bs/low-latency-docsis-technology-overview-february-20= 19 > o Operators: > Liberty Global > Charter > Rogers > Comcast > Shaw > Cox Communications I think david's characterization of the "cable industry" being behind this is accurate. > o Equipment vendors > ARRIS > Huawei > Broadcom > Intel > Casa > Nokia > Cisco > Videotron > * Nicolas Kuhn of CNES has been assessing the use of L4S for satellite. > * Magnus Westerlund of Ericsson with a team of others has written the nec= essary ECN feedback into QUIC where is the code? > * L4S hardware is also being implemented for hi-speed switches at the mom= ent > (the developer wants to announce it themselves, so I have been asked = not to identify them). groovy. > > There's a lot more stuff been going on, but I've tried to pick out highli= ghts. > > All this is good healthy development of much lower latency for Internet t= echnology. I look forward to benching it against normal traffic finally, esp against bbr, and in a realistic scenario with fq_codeled home routers (and sch_cake), and on wifi. IMHO there's no L4S deployment plan that makes sense vs the existing 10-100m+ deployment of rfc3168 compliant fq_codel, but, now that l4s/dualpi/tcppraguecode exists to test - I started drafting some experiments to finally take a look at it. Maybe I'll get a chance to talk about those. > > > I find it extremely disappointing that some people on this list are takin= g such a negative attitude to the major development in their own field that= they seem not to have noticed since it first hit the limelight in 2015. > > L4S has been open-sourced since 2016 so that everyone can develop it and = make it better... > > If I was in this position, having overlooked something important for mult= iple years, I would certainly not condemn it while I was trying to understa= nd what it was and how it worked. Can I suggest everyone takes a step back,= and suspends judgement until they have understood the potential, the goals= and the depth of what they have missed. People who know me, know that I am= very careful with Internet architecture, and particularly with balancing p= ublic policy against commercial issues. Please presume respect unless prove= n otherwise. > > Best Regards > > > > Bob > > PS. Oh and BBR would be welcome to use the ECT(1) codepoint to get into t= he L4S queue. But only if it can keep latency down below around 1ms. Curren= tly those ~15-25ms delay spikes would not pass muster. Indeed, its delay is= not consistently low enough between the spikes either. > > > > > > > > -----Original Message----- > From: "Vint Cerf" > Sent: Saturday, March 16, 2019 5:57pm > To: "Holland, Jake" > Cc: "Mikael Abrahamsson" , "David P. Reed" , "ecn-sane@lists.bufferbloat.net" , "bloat" > Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation a= nd experimentation of TCP Prague/L4S hackaton at IETF104 > > where does BBR fit into all this? > v > > On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake wrote= : >> >> On 2019-03-15, 11:37, "Mikael Abrahamsson" wrote: >> L4S has a much better possibility of actually getting deployment int= o the >> wider Internet packet-moving equipment than anything being talked ab= out >> here. Same with PIE as opposed to FQ_CODEL. I know it's might not be= as >> good, but it fits better into actual silicon and it's being proposed= by >> people who actually have better channels into the people setting har= d >> requirements. >> >> I suggest you consider joining them instead of opposing them. >> >> >> Hi Mikael, >> >> I agree it makes sense that fq_anything has issues when you're talking >> about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE >> makes better sense there. >> >> But fq_x makes great sense and provides real value for the uplink in a >> home, small office, coffee shop, etc. (if you run the final rate limit >> on the home side of the access link.) I'm thinking maybe there's a >> disconnect here driven by the different use cases for where AQMs can go. >> >> The thing is, each of these is the most likely congestion point at >> different times, and it's worthwhile for each of them to be able to >> AQM (and mark packets) under congestion. >> >> One of the several things that bothers me with L4S is that I've seen >> precious little concern over interfering with the ability for another >> different AQM in-path to mark packets, and because it changes the >> semantics of CE, you can't have both working at the same time unless >> they both do L4S. >> >> SCE needs a lot of details filled in, but it's so much cleaner that it >> seems to me there's reasonably obvious answers to all (or almost all) of >> those detail questions, and because the semantics are so much cleaner, >> it's much easier to tell it's non-harmful. >> >> >> >> But as you also said so well in another thread, this is important. ("Th= e >> last unicorn", IIRC.) How much does it matter if there's a feature that >> has value today, but only until RACK is widely deployed? If you were >> convinced RACK would roll out everywhere within 3 years and SCE would >> produce better results than L4S over the following 15 years, would that >> change your mind? >> >> It would for me, and that's why I'd like to see SCE explored before >> making a call. I think at its core, it provides the same thing L4S does >> (a high-fidelity explicit congestion signal for the sender), but with >> much cleaner semantics that can be incrementally added to congestion >> controls that people are already using. >> >> Granted, it still remains to be seen whether SCE in practice can match >> the results of L4S, and L4S was here first. But it seems to me L4S come= s >> with some problems that have not yet been examined, and that are nicely >> dodged by a SCE-based approach. >> >> If L4S really is as good as they seem to think, I could imagine getting >> behind it, but I don't think that's proven yet. I'm not certain, but >> all the comparative analyses I remember seeing have been from more or >> less the same team, and I'm not convinced they don't have some >> misaligned incentives of their own. >> >> I understand a lot of work has gone into L4S, but this move to jump it >> from interesting experiment to de-facto standard without a more critical >> review that digs deeper into some of the potential deployment problems >> has me concerned. >> >> If it really does turn out to be good enough to be permanent, I'm not >> opposed to it, but I'm just not convinced that it's non-harmful, and my >> default position is that the cleaner solution is going to be better in >> the long run, if they can do the same job. >> >> It's not that I want it to be a fight, but I do want to end up with the >> best solution we can get. We only have the one internet. >> >> Just my 2c. >> >> -Jake >> >> >> _______________________________________________ >> Ecn-sane mailing list >> Ecn-sane@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/ecn-sane > > > -- > New postal address: > Google > 1875 Explorer Street, 10th Floor > Reston, VA 20190 > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > > > -- > ________________________________________________________________ > Bob Briscoe http://bobbriscoe.net/ > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat --=20 Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740