From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-il1-x12f.google.com (mail-il1-x12f.google.com [IPv6:2607:f8b0:4864:20::12f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id B73773CB52 for ; Sun, 5 Jul 2020 13:07:54 -0400 (EDT) Received: by mail-il1-x12f.google.com with SMTP id a11so22776300ilk.0 for ; Sun, 05 Jul 2020 10:07:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=7qLXDJX2dwzgEr17toMqpTZP0hpRUVQyuUdfM+9XrvI=; b=vwXveNSSnhP3Edyr9RyZ+Nu6Qye6RyqcdO4g2EiqloRQA2mywemSUZUVmSY0sb5YJD FDQfX0iLsl+nfnyeqY3E57tFd97veODcPyBf1Mqq7plEq3Gto4blQGk5jc/gxZ7H01nl 6QZbb/owSzg6gT6U2/5XMw+Z0UQiEf4XpKUGyZWbCUqsdhDpV2lApV5OCbnXV/WXLbFh I+UV1elgmzm9MRDksykowvanuVErtIqdbDZ2ICOdJSSV8h6jDtLAjVGiuBbtM1WA+erX Fl5kBJD6fiLNVAvd/qXc/Ba1xAaVxIg4G2Q66vMe4ZjZ6qFfgEbg2MaaSBF8TOyeB+tN mbXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=7qLXDJX2dwzgEr17toMqpTZP0hpRUVQyuUdfM+9XrvI=; b=mX1iliUujawW624CFODsVGbvoEsVjMQCH+HJHiuwSISvTz50P9dd+EAdGkkRhQqUGQ 7oBTGaPTVQWkSBqoxHMAPd0pQKKfHxWimBppclVoJs+UgbsbmZxQFHH/vHe9HeTVLGOZ ttcwZwUKydQ7np/KM39iFYlqx+PKUZoTasRpWhVB/996v3dBmxWDQFtdHh0Og2sTqB8X ZE/FXkHtSY6kR0/inKuR9c3RlJ1ShI5ruZNWq7e2W5zabrDRtnonAaL5WCqrsuzh5CGx g4B/vXj76a3Y9oIn46heME+CgA8OD7ZTWY+DsFRehi+x6Vtm6GxllT6cBefvpmjUABvY PEdw== X-Gm-Message-State: AOAM531C7WV4tdfDRuTYF5oE3GZqhOIAapNm9EsHye8OZRwHHy22J8iS 5JI8vdA4lu5y5eOzP0jOSsuteH1BngvZ9rBlcOoD6g== X-Google-Smtp-Source: ABdhPJyWUqsas5baDR+Ls0notbPGXOYqdi8LNiS055fBdOGSgoztgoNh8GdT9sF/Pqx2BiZfo6pQqtXoifsNKOiJDt8= X-Received: by 2002:a92:c10b:: with SMTP id p11mr8682261ile.60.1593968873919; Sun, 05 Jul 2020 10:07:53 -0700 (PDT) MIME-Version: 1.0 References: <5405F10B-F446-4B74-8894-33232145EB2E@gmx.de> <9A7FBFF1-7F10-41DD-B5C3-5A45254CEB54@gmx.de> In-Reply-To: <9A7FBFF1-7F10-41DD-B5C3-5A45254CEB54@gmx.de> From: Matt Mathis Date: Sun, 5 Jul 2020 10:07:41 -0700 Message-ID: Subject: Re: [Bloat] the future belongs to pacing To: Sebastian Moeller Cc: Daniel Sterling , Make-Wifi-fast , Carlo Augusto Grazia , bloat , Jamshid Mahdavi Content-Type: multipart/alternative; boundary="00000000000083e51705a9b4ccb6" X-Mailman-Approved-At: Mon, 06 Jul 2020 06:38:11 -0400 X-List-Received-Date: Sun, 05 Jul 2020 17:07:54 -0000 --00000000000083e51705a9b4ccb6 Content-Type: text/plain; charset="UTF-8" The consensus in the standards community is that 3168 ECN is not so useful - too late to protect small queues, too much signal (gain) to use it to hint at future congestion. The point of non-3168 ECN is to permit earlier gentle signalling. I am not following the ECN conversation, but as stated at recent IETFs, the ECN code in BBRv2 is really a placeholder, and when the ECN community comes to consensus on a standard, I would expect BBR to do the standard. Tor has its own special challenge with traffic management. Easy solutions leak information, secure solutions are very hard. Remember to be useful, the ECN bits need to be in the clear. Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay We must not tolerate intolerance; however our response must be carefully measured: too strong would be hypocritical and risks spiraling out of control; too weak risks being mistaken for tacit approval. On Sun, Jul 5, 2020 at 5:01 AM Sebastian Moeller wrote: > Hi Matt, > > > > > On Jul 5, 2020, at 08:10, Matt Mathis wrote: > > > > I strongly suggest that people (re)read VJ88 - I do every couple of > years, and still discover things that I overlooked on previous readings. > > I promise to read it. And before I give the wrong impression and > for what it is worth*, I consider BBR (even v1) an interesting and > important evolutionary step and agree that "pacing" is a gentler approach > then bursting a full CWN into a link. > > > > > > All of the negative comments about BBR and loss, ECN marks, > > As far as I can tell, BBRv2 aims for a decidedly non-rfc3168 > response to CE-marks. This IMHO is not a clear cut case of meaningfully > addressing my ECN comment. In the light of efficiently using TOR? switch > buffers efficiently, that kind of response might be defensible but it does > not really address my remark about it being unfortunate that BBR ignores > both immediate signals of congestion, (sparse) packet drops AND explicit CE > marks, the proposed (dctcp-like) CE-response seems rather weak compared to > the naive expectation of halving/80%-ing of the sending rate, no? BBRv2 as > I understand it will happily run roughshod over any true rfc3168 AQM on the > path, I do not have the numbers, but I am not fully convinced that > typically the most significant throttling on a CDN to end-user path happens > still inside the CDN's data center... > > > > or unfairness to cubic were correct for BBRv1 but have been addressed in > BBRv2. > > I am not sure that unfairness was brought up as an issue in this > thread. > > > > > > My paper has a synopsis of BBR, which is intended to get people > started. See the references in the paper for more info: > > I will have a look at these as well... Thanks > > Best Regards > Sebastian > > *) Being from outside the field, probably not much... > > > > > [12] Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas > Yeganeh, and Van Jacobson. 2016. BBR: Congestion-Based Congestion Control. > Queue 14, 5, Pages 50 (October 2016). DOI: > https://doi.org/10.1145/3012426.3022184 > > [13] Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas > Yeganeh, and Van Jacobson. 2017. BBR: Congestion-Based Congestion Control. > Commun. ACM 60, 2 (January 2017), 58-66. DOI: > https://doi.org/10.1145/3009824 > > [22] google/bbr. 2019. GitHub repository, retrieved > https://github.com/google/bbr > > > > Key definitions: self clocked: data is triggered by ACKs. All screwy > packet and ACK scheduling in the network is reflected back into the network > on the next RTT. > > > > Paced: data is transmitted on a timer, independent of ACK arrivals (as > long as the ACKs take less than twice the measured minRTT). Thus in bulk > transport there is little or no correlation between data transmissions and > events elsewhere in the network. > > > > Clarification about my earlier WiFi comment: The BBRv1 WiFi fix missed > 4.19 LTS, so bad results are "expected" for many distros. If you want to > do useful experiments, you must read https://groups.google.com/g/bbr-dev/ > and start from BBRv2 in [22]. > > > > Thanks, > > --MM-- > > The best way to predict the future is to create it. - Alan Kay > > > > We must not tolerate intolerance; > > however our response must be carefully measured: > > too strong would be hypocritical and risks spiraling out of > control; > > too weak risks being mistaken for tacit approval. > > > > > > On Sat, Jul 4, 2020 at 11:29 AM Sebastian Moeller > wrote: > > > > > > > On Jul 4, 2020, at 19:52, Daniel Sterling > wrote: > > > > > > On Sat, Jul 4, 2020 at 1:29 PM Matt Mathis via Bloat > > > wrote: > > > "pacing is inevitable, because it saves large content providers money > > > (more efficient use of the most expensive silicon in the data center, > > > the switch buffer memory), however to use pacing we walk away from 30 > > > years of experience with TCP self clock" > > > > > > at the risk of asking w/o doing any research, > > > > > > could someone explain this to a lay person or point to a doc talking > > > about this more? > > > > > > What does BBR do that's different from other algorithms? > > > > Well, it does not believe the network (blindly), that is > currently it ignores both ECN marks and (sparse) drops as signs of > congestion, instead it uses its own rate estimates to set its send rate and > cyclically will re-assess its rate estimate. Sufficiently severe drops will > be honored. IMHO a somewhat risky approach, that works reasonably well, as > often sparse drops are not real signs of congestion but just random drops > of say a wifi link (that said, these drops on wifi typically also cause > painful latency spikes as wifi often takes heroic measures in attempting > retransmitting for several 100s of milliseconds). > > > > > > > Why does it > > > break the clock? > > > > One can argue that there is no real clock to break. TCP gates > the release on new packets on the reception of ACK signals from the > receiver, this is only a clock, if one does not really care for the > equi-temporal period property of a real clock. But for better or worse that > is the term that is used. IMHO (and I really am calling this from way out > in the left-field) gating would be a better term, but changing the > nomenclature probably is not an option at this point. > > > > > Before BBR, was the clock the only way TCP did CC? > > > > No, TCP also interpreted a drop (or rather 3 duplicated ACKs) as > signal of congestion and hit the brakes, by halving the congestion window > (the amount of data that could be in flight unacknowledged, which roughly > correlates with the send rate, if averaged over long enough time windows). > BBR explicitly does not do this unless it really is convinced that someone > dropped multiple packets purposefully to signal congestion. > > In practice it works rather well, in theory it could do with at > least an rfc3168 compliant response to ECN marks (which an AQM uses to > explicitly signal congestion, unlike a drop an ECN mark is really > unambiguous, some hop on the way "told" the flow slow down). > > > > > > > > > > Also, > > > > > > I have UBNT "Amplifi" HD wifi units in my house. (HD units only; none > > > of the "mesh" units. Just HD units connected either via wifi or > > > wired.) Empirically, I've found that in order to reduce latency, I > > > need to set cake to about 1/4 of the total possible wifi speed; > > > otherwise if a large download comes down from my internet link, that > > > flow causes latency. > > > > > > That is, if I'm using 5ghz at 20mhz channel width, I need to set > > > cake's bandwidth argument to 40mbits to prevent video streams / > > > downloads from impacting latency for any other stream. This is w/o any > > > categorization at all; no packet marking based on port or anything > > > else; cake set to "best effort". > > > > > > Anything higher and when a large amount of data comes thru, something > > > (assumedly the buffer in the Amplifi HD units) causes 100s of > > > milliseconds of latency. > > > > > > Can anyone speak to how BBR would react to this? My ISP is full > > > gigabit; but cake is going to drop a lot of packets as it throttles > > > that down to 40mbit before it sends the packets to the wifi AP. > > > > > > Thanks, > > > Dan > > > _______________________________________________ > > > Bloat mailing list > > > Bloat@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/bloat > > > > --00000000000083e51705a9b4ccb6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
The consensus in the standards community is that 3168 ECN = is not so useful - too late to protect small queues,=C2=A0too much signal (= gain) to use it to=C2=A0hint at future congestion.=C2=A0 =C2=A0 The point o= f non-3168 ECN is to permit earlier gentle signalling.=C2=A0 =C2=A0I am not= following the ECN conversation, but as stated at recent IETFs, the ECN cod= e in BBRv2 is really a placeholder, and when the ECN community comes to con= sensus on a standard, I would expect BBR to=C2=A0do the standard.

<= /div>
Tor has its own special challenge with=C2=A0traffic management.= =C2=A0 =C2=A0Easy solutions leak information, secure solutions are very har= d.=C2=A0 =C2=A0Remember to be useful, the ECN bits=C2=A0need to be in the c= lear.

Thanks,
--MM--
The best way to predict the future is = to create it. =C2=A0- Alan Kay

We must not tolerate intolerance;
=C2=A0 =C2=A0 =C2=A0 =C2=A0however our response must be = carefully measured:=C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 too strong would be hypocritical and risks spiraling out of control;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too weak risks being mist= aken for tacit approval.


On Sun, Jul 5, 2020 at 5:01 AM Sebastian Moeller <moeller0@gmx.de> wrote= :
Hi Matt,



> On Jul 5, 2020, at 08:10, Matt Mathis <mattmathis@google.com> wrote:
>
> I strongly suggest that people (re)read VJ88 - I do every couple of ye= ars, and still discover things that I overlooked on previous readings.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 I promise to read it. And before I give the wro= ng impression and for what it is worth*, I consider BBR (even v1) an intere= sting and important evolutionary step and agree that "pacing" is = a gentler approach then bursting a full CWN into a link.


>
> All of the negative comments about BBR and loss, ECN marks,

=C2=A0 =C2=A0 =C2=A0 =C2=A0 As far as I can tell, BBRv2 aims for a decidedl= y non-rfc3168 response to CE-marks. This IMHO is not a clear cut case of me= aningfully addressing my ECN comment. In the light of efficiently using TOR= ? switch buffers efficiently, that kind of response might be defensible but= it does not really address my remark about it being unfortunate that BBR i= gnores both immediate signals of congestion, (sparse) packet drops AND expl= icit CE marks, the proposed (dctcp-like) CE-response seems rather weak comp= ared to the naive expectation of halving/80%-ing of the sending rate, no? B= BRv2 as I understand it will happily run roughshod over any true rfc3168 AQ= M on the path, I do not have the numbers, but I am not fully convinced that= typically the most significant throttling on a CDN to end-user path happen= s still inside the CDN's data center...


> or unfairness to cubic were correct for BBRv1 but have been addressed = in BBRv2.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 I am not sure that unfairness was brought up as= an issue in this thread.


>
> My paper has a synopsis of BBR, which is intended to get people starte= d.=C2=A0 =C2=A0See the references in the paper for more info:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 I will have a look at these as well... Thanks
Best Regards
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Sebastian

*) Being from outside the field, probably not much...

>
> [12] Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yega= neh, and Van Jacobson. 2016. BBR: Congestion-Based Congestion Control. Queu= e 14, 5, Pages 50 (October 2016). DOI: https://doi.org/10.114= 5/3012426.3022184
> [13] Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yega= neh, and Van Jacobson. 2017. BBR: Congestion-Based Congestion Control. Comm= un. ACM 60, 2 (January 2017), 58-66. DOI: https://doi.org/10.1145/300= 9824
> [22] google/bbr. 2019. GitHub repository, retrieved https://github= .com/google/bbr
>
> Key definitions: self clocked: data is triggered by ACKs.=C2=A0 All sc= rewy packet and ACK scheduling in the network is reflected back into the ne= twork on the next RTT.
>
> Paced: data is transmitted on a timer, independent of ACK arrivals (as= long as the ACKs take less than twice the measured minRTT).=C2=A0 Thus in = bulk transport there is little or no correlation between data transmissions= and events elsewhere in the network.
>
> Clarification about my earlier WiFi comment:=C2=A0 The BBRv1 WiFi fix = missed 4.19 LTS, so bad results are "expected" for many distros.= =C2=A0 If you want to do useful experiments, you must read https= ://groups.google.com/g/bbr-dev/ and start from BBRv2 in [22].
>
> Thanks,
> --MM--
> The best way to predict the future is to create it.=C2=A0 - Alan Kay >
> We must not tolerate intolerance;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 however our response must be carefully meas= ured:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0too strong would be hyp= ocritical and risks spiraling out of control;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0too weak risks being mi= staken for tacit approval.
>
>
> On Sat, Jul 4, 2020 at 11:29 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>
>
> > On Jul 4, 2020, at 19:52, Daniel Sterling <sterling.daniel@gmail.com&g= t; wrote:
> >
> > On Sat, Jul 4, 2020 at 1:29 PM Matt Mathis via Bloat
> > <bloat@lists.bufferbloat.net> wrote:
> > "pacing is inevitable, because it saves large content provid= ers money
> > (more efficient use of the most expensive silicon in the data cen= ter,
> > the switch buffer memory), however to use pacing we walk away fro= m 30
> > years of experience with TCP self clock"
> >
> > at the risk of asking w/o doing any research,
> >
> > could someone explain this to a lay person or point to a doc talk= ing
> > about this more?
> >
> > What does BBR do that's different from other algorithms?
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Well, it does not believe the network= (blindly), that is currently it ignores both ECN marks and (sparse) drops = as signs of congestion, instead it uses its own rate estimates to set its s= end rate and cyclically will re-assess its rate estimate. Sufficiently seve= re drops will be honored. IMHO a somewhat risky approach, that works reason= ably well, as often sparse drops are not real signs of congestion but just = random drops of say a wifi link (that said, these drops on wifi typically a= lso cause painful latency spikes as wifi often takes heroic measures in att= empting retransmitting for several 100s of milliseconds).
>
>
> > Why does it
> > break the clock?
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0One can argue that there is no real c= lock to break. TCP gates the release on new packets on the reception of ACK= signals from the receiver, this is only a clock, if one does not really ca= re for the equi-temporal period property of a real clock. But for better or= worse that is the term that is used. IMHO (and I really am calling this fr= om way out in the left-field) gating would be a better term, but changing t= he nomenclature probably is not an option at this point.
>
> > Before BBR, was the clock the only way TCP did CC?
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0No, TCP also interpreted a drop (or r= ather 3 duplicated ACKs) as signal of congestion and hit the brakes, by hal= ving the congestion window (the amount of data that could be in flight unac= knowledged, which roughly correlates with the send rate, if averaged over l= ong enough time windows). BBR explicitly does not do this unless it really = is convinced that someone dropped multiple packets purposefully to signal c= ongestion.
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0In practice it works rather well, in = theory it could do with at least an rfc3168 compliant response to ECN marks= (which an AQM uses to explicitly signal congestion, unlike a drop an ECN m= ark is really unambiguous, some hop on the way "told" the flow sl= ow down).
>
>
> >
> > Also,
> >
> > I have UBNT "Amplifi" HD wifi units in my house. (HD un= its only; none
> > of the "mesh" units. Just HD units connected either via= wifi or
> > wired.) Empirically, I've found that in order to reduce laten= cy, I
> > need to set cake to about 1/4 of the total possible wifi speed; > > otherwise if a large download comes down from my internet link, t= hat
> > flow causes latency.
> >
> > That is, if I'm using 5ghz at 20mhz channel width, I need to = set
> > cake's bandwidth argument to 40mbits to prevent video streams= /
> > downloads from impacting latency for any other stream. This is w/= o any
> > categorization at all; no packet marking based on port or anythin= g
> > else; cake set to "best effort".
> >
> > Anything higher and when a large amount of data comes thru, somet= hing
> > (assumedly the buffer in the Amplifi HD units) causes 100s of
> > milliseconds of latency.
> >
> > Can anyone speak to how BBR would react to this? My ISP is full > > gigabit; but cake is going to drop a lot of packets as it throttl= es
> > that down to 40mbit before it sends the packets to the wifi AP. > >
> > Thanks,
> > Dan
> > _______________________________________________
> > Bloat mailing list
> > = Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>

--00000000000083e51705a9b4ccb6--