From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 468F33CB55; Sun, 5 Jul 2020 13:29:57 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1593970188; bh=wRPdWjFKjJG5P5OUZbLWcwYLCsItfUpovczG91wDpqY=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=GXqrdSpQBPcx99PZCKCHsFgaBdL5VeFLjvwTomUEZjgJAUtojM0PZHCy7IvpVhaQS svWa+NuhEVPeaSaitNSEAfVbamNAyIvJ//of+7OgHZcTHykhUJdZMy7bWPtTypnrPh jatIHdEsxf9AEknKWVlO9qtxS2OB5p88aBVnlfRI= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [192.168.42.229] ([77.0.164.216]) by mail.gmx.com (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1N4Qwg-1ksW5a0gkQ-011Oyj; Sun, 05 Jul 2020 19:29:48 +0200 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.14\)) From: Sebastian Moeller In-Reply-To: Date: Sun, 5 Jul 2020 19:29:47 +0200 Cc: Daniel Sterling , Make-Wifi-fast , Carlo Augusto Grazia , bloat , Jamshid Mahdavi Content-Transfer-Encoding: quoted-printable Message-Id: <5DE69639-F8E7-4DAC-9A20-C978EFF2D41E@gmx.de> References: <5405F10B-F446-4B74-8894-33232145EB2E@gmx.de> <9A7FBFF1-7F10-41DD-B5C3-5A45254CEB54@gmx.de> To: Matt Mathis X-Mailer: Apple Mail (2.3445.104.14) X-Provags-ID: V03:K1:NMdbZ5KWavvZpsBFVGZMhrA+7VoY7ApVctfdze7nJngwqH5ztao z6q6nlmSPP6VhSIZnlI8axM8dxwwo0iUaAsCR39drWnnnVMibIDBeU5hAnWOjs75Kf3b9In uZIsjTYwNw2KvNgTlFLoFU2ZWVZFDmuEKDuTwJYcC9+ln34O/mKZSDSmY5fSZwRSZTY1t1I WS6WzzOCeunZe9maF5pJQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:amzVXRI0DUk=:nJx1wTQPT431eW2usOIH7e Zie67PK0RQs4MNmUKOZkaihex0CrEyn0pLxcLli+ePF50K2mIMmtuL+XoJ5eExf/HVVlWp8wT EQ+shDr/VozzyeLIiatZ1zo/yhyTjEbBnuoGLqrB98g/JzLT41nKGqE/tVLbIeDqhRDBeDyY8 hk7KFWPEqHgxA/ZoiKSRri8d2iM0mydtP95iowkiAWw5IZKG6Q1hUYPCs6Ya7HQcFuvjyrA94 3yI3raSdmxW22sGdEQYQtvyAegUxtpfkQADTWCjyFlEhmJ0VZ6UOxYK6S1RstyJcbBNLFA2fL m/ewgXn9RlZitNDlcIncAjNFSkVK4Oy1RG83FAA+BL/1NORZY+9LWvvUJwGpnRpeJuAxtII5F vxj2WAPTvMOEU25vO4mM9lJVPjBIpR1Ra5fVXO+9ixswAqmAFwEJ2KZpj4361AzxWONfFwoxR 2gK8lCqfKVsNO8/kuDyaq8Fn0LoiPK6K/nwGEXR+tKMzpBnBmAtf4gySaFJldMcBaSDTzUfr6 oZCg7l87RYidJOt4LZDvA3qDc9OSwe/kMbr86xQ+QvF8HWjFq9K2QORbkFQv5P/RjydnxH934 qEZRSTX+uTyEiuleATdASFIm4ym0z4x3tzeMv48aW6/ZRyCi6iSQp+K1/ZSKoAyO4Cy24tcC4 bMzxVS9PBfAO879x5hdpuOwg3kcLrZ3LqcjkyiNFyqvWsGNbsf/P9pBeY7hH8halmPRzVSe6k ZzTUmxjjSJ1Ukg5ozdTi2Ob8OnNC1IQEGZCO+Y+1BQRNXszMJIi/+JuCAj4VxGuUnDQ7w1R4e EpUb35uVtKZLvwKFlDOQJ2nK4jOh545vFFH40dbH4mdElPtsTZmDPkLNdYNhFPBWntVTOZL51 A/LyggmsPsotIelusi5vy2wyhIqrmKGOdg7qlGKdaEL12GMDYwIpz51lAFuT+4eI+jquXUhhS hMO3iM/ozQSFI4zxn299HJKlzyN7oDXWEUsd+TwKUmwlCf8ZRKhiJHxYe4DPPTge4jbB5VsOe 4yvstbE6R+tVVMCjE2V+XMgw6lJ9HJdyJRmJvsj8Ihx/F6J5rEIyd01HypCE8w40Pv0JkGBpt DJQzwz8tcTtEZxQjHIlVAUDEYM53jbk5ROdpiHal8MBi7vDjmO7w1kNb+4jzZuKmpXvCvfsLi QTRyLtEmT4OmxFIVEuKqNKHGuaWO2E/+l0X6x/95nRQ85bYEag8S7silQqk0HvKytwleJe6qn p7n38byCQkdUVCupB Subject: Re: [Bloat] the future belongs to pacing X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jul 2020 17:29:57 -0000 Hi Matt, > On Jul 5, 2020, at 19:07, Matt Mathis wrote: >=20 > The consensus in the standards community is that 3168 ECN is not so = useful - too late to protect small queues, too much signal (gain) to use = it to hint at future congestion. =20 I follow the discussion in the tsw working group and believe I = have a good overview of the state of the discussion. I also have = gathered some experience in the bufferbloat effort to be able to realize = that the L4S proposal is mostly based on wishful thinking than on solid = engineering. But yes, the time seems ripe for 1/p-type congestion = signaling, but how to do this seems an open question. > The point of non-3168 ECN is to permit earlier gentle signalling. I = am not following the ECN conversation, but as stated at recent IETFs, = the ECN code in BBRv2 is really a placeholder, and when the ECN = community comes to consensus on a standard, I would expect BBR to do the = standard. I respectfully argue that this is the wrong way around, first = implement the current RFC standard aka rfc3168 and only if there is a = new standard switch over to that. ATM BBRv seems to bank on the L4S = proposals to sail through the IETF completely ignoring the lack of = critical testing the L4S design has been treated to. >=20 > Tor has its own special challenge with traffic management. =20 Sorry, TOR was intended to expand to Top Of Rack, which I = assumed to be a common way to call the devices that house "the most = expensive silicon in the data center, the switch buffer memory". I = apologize for not being clear. > Easy solutions leak information, secure solutions are very hard. = Remember to be useful, the ECN bits need to be in the clear. All good points, thanks, but more applicable to the onion router = and to top of rack switches... Best Regards Sebastian >=20 > Thanks, > --MM-- > The best way to predict the future is to create it. - Alan Kay >=20 > We must not tolerate intolerance; > however our response must be carefully measured:=20 > too strong would be hypocritical and risks spiraling out = of control; > too weak risks being mistaken for tacit approval. >=20 >=20 > On Sun, Jul 5, 2020 at 5:01 AM Sebastian Moeller = wrote: > Hi Matt, >=20 >=20 >=20 > > On Jul 5, 2020, at 08:10, Matt Mathis wrote: > >=20 > > I strongly suggest that people (re)read VJ88 - I do every couple of = years, and still discover things that I overlooked on previous readings. >=20 > I promise to read it. And before I give the wrong impression = and for what it is worth*, I consider BBR (even v1) an interesting and = important evolutionary step and agree that "pacing" is a gentler = approach then bursting a full CWN into a link. >=20 >=20 > >=20 > > All of the negative comments about BBR and loss, ECN marks, >=20 > As far as I can tell, BBRv2 aims for a decidedly non-rfc3168 = response to CE-marks. This IMHO is not a clear cut case of meaningfully = addressing my ECN comment. In the light of efficiently using TOR? switch = buffers efficiently, that kind of response might be defensible but it = does not really address my remark about it being unfortunate that BBR = ignores both immediate signals of congestion, (sparse) packet drops AND = explicit CE marks, the proposed (dctcp-like) CE-response seems rather = weak compared to the naive expectation of halving/80%-ing of the sending = rate, no? BBRv2 as I understand it will happily run roughshod over any = true rfc3168 AQM on the path, I do not have the numbers, but I am not = fully convinced that typically the most significant throttling on a CDN = to end-user path happens still inside the CDN's data center...=20 >=20 >=20 > > or unfairness to cubic were correct for BBRv1 but have been = addressed in BBRv2. >=20 > I am not sure that unfairness was brought up as an issue in = this thread. >=20 >=20 > >=20 > > My paper has a synopsis of BBR, which is intended to get people = started. See the references in the paper for more info: >=20 > I will have a look at these as well... Thanks >=20 > Best Regards > Sebastian >=20 > *) Being from outside the field, probably not much... >=20 > >=20 > > [12] Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas = Yeganeh, and Van Jacobson. 2016. BBR: Congestion-Based Congestion = Control. Queue 14, 5, Pages 50 (October 2016). DOI: = https://doi.org/10.1145/3012426.3022184 > > [13] Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas = Yeganeh, and Van Jacobson. 2017. BBR: Congestion-Based Congestion = Control. Commun. ACM 60, 2 (January 2017), 58-66. DOI: = https://doi.org/10.1145/3009824 > > [22] google/bbr. 2019. GitHub repository, retrieved = https://github.com/google/bbr > >=20 > > Key definitions: self clocked: data is triggered by ACKs. All = screwy packet and ACK scheduling in the network is reflected back into = the network on the next RTT. > >=20 > > Paced: data is transmitted on a timer, independent of ACK arrivals = (as long as the ACKs take less than twice the measured minRTT). Thus in = bulk transport there is little or no correlation between data = transmissions and events elsewhere in the network.=20 > >=20 > > Clarification about my earlier WiFi comment: The BBRv1 WiFi fix = missed 4.19 LTS, so bad results are "expected" for many distros. If you = want to do useful experiments, you must read = https://groups.google.com/g/bbr-dev/ and start from BBRv2 in [22]. > >=20 > > Thanks, > > --MM-- > > The best way to predict the future is to create it. - Alan Kay > >=20 > > We must not tolerate intolerance; > > however our response must be carefully measured:=20 > > too strong would be hypocritical and risks spiraling out = of control; > > too weak risks being mistaken for tacit approval. > >=20 > >=20 > > On Sat, Jul 4, 2020 at 11:29 AM Sebastian Moeller = wrote: > >=20 > >=20 > > > On Jul 4, 2020, at 19:52, Daniel Sterling = wrote: > > >=20 > > > On Sat, Jul 4, 2020 at 1:29 PM Matt Mathis via Bloat > > > wrote: > > > "pacing is inevitable, because it saves large content providers = money > > > (more efficient use of the most expensive silicon in the data = center, > > > the switch buffer memory), however to use pacing we walk away from = 30 > > > years of experience with TCP self clock" > > >=20 > > > at the risk of asking w/o doing any research, > > >=20 > > > could someone explain this to a lay person or point to a doc = talking > > > about this more? > > >=20 > > > What does BBR do that's different from other algorithms? > >=20 > > Well, it does not believe the network (blindly), that is = currently it ignores both ECN marks and (sparse) drops as signs of = congestion, instead it uses its own rate estimates to set its send rate = and cyclically will re-assess its rate estimate. Sufficiently severe = drops will be honored. IMHO a somewhat risky approach, that works = reasonably well, as often sparse drops are not real signs of congestion = but just random drops of say a wifi link (that said, these drops on wifi = typically also cause painful latency spikes as wifi often takes heroic = measures in attempting retransmitting for several 100s of milliseconds). > >=20 > >=20 > > > Why does it > > > break the clock? > >=20 > > One can argue that there is no real clock to break. TCP = gates the release on new packets on the reception of ACK signals from = the receiver, this is only a clock, if one does not really care for the = equi-temporal period property of a real clock. But for better or worse = that is the term that is used. IMHO (and I really am calling this from = way out in the left-field) gating would be a better term, but changing = the nomenclature probably is not an option at this point. > >=20 > > > Before BBR, was the clock the only way TCP did CC? > >=20 > > No, TCP also interpreted a drop (or rather 3 duplicated = ACKs) as signal of congestion and hit the brakes, by halving the = congestion window (the amount of data that could be in flight = unacknowledged, which roughly correlates with the send rate, if averaged = over long enough time windows). BBR explicitly does not do this unless = it really is convinced that someone dropped multiple packets = purposefully to signal congestion. > > In practice it works rather well, in theory it could do with = at least an rfc3168 compliant response to ECN marks (which an AQM uses = to explicitly signal congestion, unlike a drop an ECN mark is really = unambiguous, some hop on the way "told" the flow slow down). > >=20 > >=20 > > >=20 > > > Also, > > >=20 > > > I have UBNT "Amplifi" HD wifi units in my house. (HD units only; = none > > > of the "mesh" units. Just HD units connected either via wifi or > > > wired.) Empirically, I've found that in order to reduce latency, I > > > need to set cake to about 1/4 of the total possible wifi speed; > > > otherwise if a large download comes down from my internet link, = that > > > flow causes latency. > > >=20 > > > That is, if I'm using 5ghz at 20mhz channel width, I need to set > > > cake's bandwidth argument to 40mbits to prevent video streams / > > > downloads from impacting latency for any other stream. This is w/o = any > > > categorization at all; no packet marking based on port or anything > > > else; cake set to "best effort". > > >=20 > > > Anything higher and when a large amount of data comes thru, = something > > > (assumedly the buffer in the Amplifi HD units) causes 100s of > > > milliseconds of latency. > > >=20 > > > Can anyone speak to how BBR would react to this? My ISP is full > > > gigabit; but cake is going to drop a lot of packets as it = throttles > > > that down to 40mbit before it sends the packets to the wifi AP. > > >=20 > > > Thanks, > > > Dan > > > _______________________________________________ > > > Bloat mailing list > > > Bloat@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/bloat > >=20 >=20