From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id A72A93CB35 for ; Sun, 5 Jul 2020 02:10:49 -0400 (EDT) Received: by mail-io1-xd2e.google.com with SMTP id a12so36232142ion.13 for ; Sat, 04 Jul 2020 23:10:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=prGLPrIZvZOiVnN8aRkjO6eYthDMP0w4VgUhkeLpA/0=; b=f9Uix+BulIgJtBlAT9ctAItVjmerTHqo1NswYW7DTN5KRWvyjhOpkDiXzMixV6+P5S pgZF5Na4vvOHtiD9Z6y/JtXAwEbe/jU/Gu8iXUul3+kzxC2nbdYvax/JOV9+Ze71rwPD +jVuOOx9MC8M1HH8krjGFBSmKgo+PNA+aqEOfN6XZOBUysLQCx82XvZ6TnVuATa7J82v dDBxpgR7JCRrUAlbjvFsFZZ2oP5j09f5F15LwcGgowH3pUhpHGaMXKS3x4BPkxEu1G8m ToDOAjWNSocn49RWIuqvoeJXarhNlyJbMKqrpTdTlJFcTFVSEa2z4yujzPVJ2769lvk7 UYmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=prGLPrIZvZOiVnN8aRkjO6eYthDMP0w4VgUhkeLpA/0=; b=aJKOCrWJ7mmtgpIrHF/KRYx224TD2R3FUVntNpbkC6HG4vL9Za1qCtrMD4ybOwHKwP 9Z8CFP65KvY9/gsnoSFQeUFrmy9Z0AWLP7ka5J2Ezs37QeVDua0GnlcBc8Q/ch/GcTV8 xPJRdrH+MQ/Q5ErTftofkffzQ6adb6ia4nmWG722jvwVJSp0kQiu58gvIVrJ75h1q20b rWpFiKOzmbRy3UxGFNLjwqE2YUWOZeqDnrl92Ewmkl/vWkCptp9TY1L1vm3DoNli0nTw IzJs5QXUDGZg/b+2lJztomnIqEg34iOxKHLwgQ65RkU0F77leKrqVV7xrUyXU0lJ/eOj w9Aw== X-Gm-Message-State: AOAM531lCA0gvrGD6ilCpQqY2u/2kauJv73Iutb9DZ8i7xm8UB4cdNZP knEygFeMENuBh+G2ON9AxuiTpHhyJ6+mfJfftizZmA== X-Google-Smtp-Source: ABdhPJxR2q1DxQFrA+jak+SJAPfKoHGFZZk4gmu4fxCaJwHHlAGsrIC5la5vEf9EZwZP6nqchRga17xQxAKimcG/6QQ= X-Received: by 2002:a02:a008:: with SMTP id a8mr44836761jah.68.1593929448861; Sat, 04 Jul 2020 23:10:48 -0700 (PDT) MIME-Version: 1.0 References: <5405F10B-F446-4B74-8894-33232145EB2E@gmx.de> In-Reply-To: <5405F10B-F446-4B74-8894-33232145EB2E@gmx.de> From: Matt Mathis Date: Sat, 4 Jul 2020 23:10:36 -0700 Message-ID: Subject: Re: [Bloat] the future belongs to pacing To: Sebastian Moeller Cc: Daniel Sterling , Make-Wifi-fast , Carlo Augusto Grazia , bloat , Jamshid Mahdavi Content-Type: multipart/alternative; boundary="0000000000009939c405a9ab9e38" X-List-Received-Date: Sun, 05 Jul 2020 06:10:49 -0000 --0000000000009939c405a9ab9e38 Content-Type: text/plain; charset="UTF-8" I strongly suggest that people (re)read VJ88 - I do every couple of years, and still discover things that I overlooked on previous readings. All of the negative comments about BBR and loss, ECN marks, or unfairness to cubic were correct for BBRv1 but have been addressed in BBRv2. My paper has a synopsis of BBR, which is intended to get people started. See the references in the paper for more info: [12] Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson. 2016. BBR: Congestion-Based Congestion Control. Queue 14, 5, Pages 50 (October 2016). DOI: https://doi.org/10.1145/3012426.3022184 [13] Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson. 2017. BBR: Congestion-Based Congestion Control. Commun. ACM 60, 2 (January 2017), 58-66. DOI: https://doi.org/10.1145/3009824 [22] google/bbr. 2019. GitHub repository, retrieved https://github.com/google/bbr Key definitions: self clocked: data is triggered by ACKs. All screwy packet and ACK scheduling in the network is reflected back into the network on the next RTT. Paced: data is transmitted on a timer, independent of ACK arrivals (as long as the ACKs take less than twice the measured minRTT). Thus in bulk transport there is little or no correlation between data transmissions and events elsewhere in the network. Clarification about my earlier WiFi comment: The BBRv1 WiFi fix missed 4.19 LTS, so bad results are "expected" for many distros. If you want to do useful experiments, you must read https://groups.google.com/g/bbr-dev/ and start from BBRv2 in [22]. Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay We must not tolerate intolerance; however our response must be carefully measured: too strong would be hypocritical and risks spiraling out of control; too weak risks being mistaken for tacit approval. On Sat, Jul 4, 2020 at 11:29 AM Sebastian Moeller wrote: > > > > On Jul 4, 2020, at 19:52, Daniel Sterling > wrote: > > > > On Sat, Jul 4, 2020 at 1:29 PM Matt Mathis via Bloat > > wrote: > > "pacing is inevitable, because it saves large content providers money > > (more efficient use of the most expensive silicon in the data center, > > the switch buffer memory), however to use pacing we walk away from 30 > > years of experience with TCP self clock" > > > > at the risk of asking w/o doing any research, > > > > could someone explain this to a lay person or point to a doc talking > > about this more? > > > > What does BBR do that's different from other algorithms? > > Well, it does not believe the network (blindly), that is currently > it ignores both ECN marks and (sparse) drops as signs of congestion, > instead it uses its own rate estimates to set its send rate and cyclically > will re-assess its rate estimate. Sufficiently severe drops will be > honored. IMHO a somewhat risky approach, that works reasonably well, as > often sparse drops are not real signs of congestion but just random drops > of say a wifi link (that said, these drops on wifi typically also cause > painful latency spikes as wifi often takes heroic measures in attempting > retransmitting for several 100s of milliseconds). > > > > Why does it > > break the clock? > > One can argue that there is no real clock to break. TCP gates the > release on new packets on the reception of ACK signals from the receiver, > this is only a clock, if one does not really care for the equi-temporal > period property of a real clock. But for better or worse that is the term > that is used. IMHO (and I really am calling this from way out in the > left-field) gating would be a better term, but changing the nomenclature > probably is not an option at this point. > > > Before BBR, was the clock the only way TCP did CC? > > No, TCP also interpreted a drop (or rather 3 duplicated ACKs) as > signal of congestion and hit the brakes, by halving the congestion window > (the amount of data that could be in flight unacknowledged, which roughly > correlates with the send rate, if averaged over long enough time windows). > BBR explicitly does not do this unless it really is convinced that someone > dropped multiple packets purposefully to signal congestion. > In practice it works rather well, in theory it could do with at > least an rfc3168 compliant response to ECN marks (which an AQM uses to > explicitly signal congestion, unlike a drop an ECN mark is really > unambiguous, some hop on the way "told" the flow slow down). > > > > > > Also, > > > > I have UBNT "Amplifi" HD wifi units in my house. (HD units only; none > > of the "mesh" units. Just HD units connected either via wifi or > > wired.) Empirically, I've found that in order to reduce latency, I > > need to set cake to about 1/4 of the total possible wifi speed; > > otherwise if a large download comes down from my internet link, that > > flow causes latency. > > > > That is, if I'm using 5ghz at 20mhz channel width, I need to set > > cake's bandwidth argument to 40mbits to prevent video streams / > > downloads from impacting latency for any other stream. This is w/o any > > categorization at all; no packet marking based on port or anything > > else; cake set to "best effort". > > > > Anything higher and when a large amount of data comes thru, something > > (assumedly the buffer in the Amplifi HD units) causes 100s of > > milliseconds of latency. > > > > Can anyone speak to how BBR would react to this? My ISP is full > > gigabit; but cake is going to drop a lot of packets as it throttles > > that down to 40mbit before it sends the packets to the wifi AP. > > > > Thanks, > > Dan > > _______________________________________________ > > Bloat mailing list > > Bloat@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat > > --0000000000009939c405a9ab9e38 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I strongly suggest that people (re)read VJ88 - I do every = couple of years, and still discover things that I overlooked on previous re= adings.

All of the negative comments=C2=A0about BBR and loss, = ECN marks, or unfairness to cubic were correct for BBRv1 but have been addr= essed in BBRv2.

My paper has a synopsis of BBR, wh= ich is intended to get people=C2=A0started.=C2=A0 =C2=A0See the references= =C2=A0in the paper for more info:

[12] Neal Cardwe= ll, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson= . 2016. BBR: Congestion-Based Congestion Control. Queue 14, 5, Pages 50 (Oc= tober 2016). DOI: https= ://doi.org/10.1145/3012426.3022184
[13] Neal Cardwell, Yuchung Cheng= , C. Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson. 2017. BBR: Cong= estion-Based Congestion Control. Commun. ACM 60, 2 (January 2017), 58-66. D= OI: https://doi.org/10.1145/300= 9824
[22] google/bbr. 2019. GitHub repository, retrieved https://github.com/google/bbr

Key definitions: self clocked: data is triggered by A= CKs.=C2=A0 All screwy packet and ACK scheduling in the network is reflected= back into the network on the next RTT.

Paced: dat= a is transmitted on a timer, independent of ACK arrivals (as long as the AC= Ks take less than twice the measured minRTT).=C2=A0 Thus in bulk transport = there is little or no correlation between data transmissions and events els= ewhere in the network.=C2=A0

Clarification about m= y earlier WiFi comment:=C2=A0 The BBRv1 WiFi fix missed 4.19 LTS, so bad re= sults are "expected" for many distros.=C2=A0 If you want to do us= eful experiments, you must read=C2=A0https://groups.google.com/g/bbr-dev/=C2=A0and start from BBR= v2 in [22].

Thanks,
--MM--
The best way to predic= t the future is to create it. =C2=A0- Alan Kay

We must not tolerate = intolerance;
=C2=A0 =C2=A0 =C2=A0 =C2=A0however our r= esponse must be carefully measured:=C2=A0
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 too strong would be hypocritical and risks spiraling o= ut of control;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too weak= risks being mistaken for tacit approval.


On Sat, Jul 4, 2020 at 11:29 AM Sebastian Moe= ller <moeller0@gmx.de> wrote:<= br>


> On Jul 4, 2020, at 19:52, Daniel Sterling <sterling.daniel@gmail.com> wr= ote:
>
> On Sat, Jul 4, 2020 at 1:29 PM Matt Mathis via Bloat
> <b= loat@lists.bufferbloat.net> wrote:
> "pacing is inevitable, because it saves large content providers m= oney
> (more efficient use of the most expensive silicon in the data center,<= br> > the switch buffer memory), however to use pacing we walk away from 30<= br> > years of experience with TCP self clock"
>
> at the risk of asking w/o doing any research,
>
> could someone explain this to a lay person or point to a doc talking > about this more?
>
> What does BBR do that's different from other algorithms?

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Well, it does not believe the network (blindly)= , that is currently it ignores both ECN marks and (sparse) drops as signs o= f congestion, instead it uses its own rate estimates to set its send rate a= nd cyclically will re-assess its rate estimate. Sufficiently severe drops w= ill be honored. IMHO a somewhat risky approach, that works reasonably well,= as often sparse drops are not real signs of congestion but just random dro= ps of say a wifi link (that said, these drops on wifi typically also cause = painful latency spikes as wifi often takes heroic measures in attempting re= transmitting for several 100s of milliseconds).


> Why does it
> break the clock?

=C2=A0 =C2=A0 =C2=A0 =C2=A0 One can argue that there is no real clock to br= eak. TCP gates the release on new packets on the reception of ACK signals f= rom the receiver, this is only a clock, if one does not really care for the= equi-temporal period property of a real clock. But for better or worse tha= t is the term that is used. IMHO (and I really am calling this from way out= in the left-field) gating would be a better term, but changing the nomencl= ature probably is not an option at this point.

> Before BBR, was the clock the only way TCP did CC?

=C2=A0 =C2=A0 =C2=A0 =C2=A0 No, TCP also interpreted a drop (or rather 3 du= plicated ACKs) as signal of congestion and hit the brakes, by halving the c= ongestion window (the amount of data that could be in flight unacknowledged= , which roughly correlates with the send rate, if averaged over long enough= time windows). BBR explicitly does not do this unless it really is convinc= ed that someone dropped multiple packets purposefully to signal congestion.=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 In practice it works rather well, in theory it = could do with at least an rfc3168 compliant response to ECN marks (which an= AQM uses to explicitly signal congestion, unlike a drop an ECN mark is rea= lly unambiguous, some hop on the way "told" the flow slow down).<= br>

>
> Also,
>
> I have UBNT "Amplifi" HD wifi units in my house. (HD units o= nly; none
> of the "mesh" units. Just HD units connected either via wifi= or
> wired.) Empirically, I've found that in order to reduce latency, I=
> need to set cake to about 1/4 of the total possible wifi speed;
> otherwise if a large download comes down from my internet link, that > flow causes latency.
>
> That is, if I'm using 5ghz at 20mhz channel width, I need to set > cake's bandwidth argument to 40mbits to prevent video streams / > downloads from impacting latency for any other stream. This is w/o any=
> categorization at all; no packet marking based on port or anything
> else; cake set to "best effort".
>
> Anything higher and when a large amount of data comes thru, something<= br> > (assumedly the buffer in the Amplifi HD units) causes 100s of
> milliseconds of latency.
>
> Can anyone speak to how BBR would react to this? My ISP is full
> gigabit; but cake is going to drop a lot of packets as it throttles > that down to 40mbit before it sends the packets to the wifi AP.
>
> Thanks,
> Dan
> _______________________________________________
> Bloat mailing list
> Bloat= @lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
--0000000000009939c405a9ab9e38--