From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ua1-x929.google.com (mail-ua1-x929.google.com [IPv6:2607:f8b0:4864:20::929]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 14FD63CB37 for ; Mon, 3 Apr 2023 09:41:25 -0400 (EDT) Received: by mail-ua1-x929.google.com with SMTP id n17so20854189uaj.10 for ; Mon, 03 Apr 2023 06:41:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680529285; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=HRsn8H3d14ZJQ6Hxa2tGXITQKaPPSKPCFSFUr0bYu7c=; b=cRPpJutLTUzSsK6dU8YlJX/QqRSjdl01VwlFFFWZx0dx9Y3Dsum0wBmnFKLzi/16KL lgMHkEQQG/6X/AzjZGy/Kl/CZ4qTXeZbf93VGvHhdJscvfKnsbRbzxywmDXM6EdileA0 pUpnOn42Sml9VvnjTXS5W6jT7TU7F2SrnteVhn3AJLH6hkNZV3GXQN6zHlLBU9H8Q67I DmnaGixdWm3JSxPY0l0g6+l+lrXJvMPWmHg6chZFaVnUD8i+bjqztojNI0dCl0TwbRaH nCk6ZdDRRUP8KNlApZUUyGPD5ve89SE6uMZXGfygyytfIv0JEdD+bEVnW6GvYxr6N50D gEnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680529285; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=HRsn8H3d14ZJQ6Hxa2tGXITQKaPPSKPCFSFUr0bYu7c=; b=tJVTlhE6BvWuAhkxSb9fRboxKZwbqvhaCPjLGJtCnNgg/LldCYupxXkctxZlgd651C Hk9a39DfViO+86U7Sva/Tw1bhl53RXihUsHLTuRCvqFM06XkIvMnp0qRhc+VYAUXeDEG v3mhMhf9zGFD1ZYmgUNZ8eL3kizHBH1lgXTp468RU1MX8ZrkfK4qSEIGuYA3lnlXPivq DY//AeNWLNLut0QmupS1bh6P3MLu0W+wKYrPfbNLxozCUwt+r2MC0JBgnOU4OQ5bhBAN TEm5m9jJ6DbUwiDXUZL3ImktKYUpQRZg7agEmD8DRnrg9rXsmPpESRxP1+VVYPvgBwfM sXaw== X-Gm-Message-State: AAQBX9dWxc8SGE/6QfRZv4RLjiXv73Kcjve4CRI3y6iO9GDgip+ZvURD P7kIelkMZYzw8T0DEO9gF0Gp/fPC8fstXxN4A+zjaA== X-Google-Smtp-Source: AKy350Z52hHoXsudpB8gsLC0lIFZ/i+04Vow9so8zo0eGJAtE9ntIf8Is6DuSVVz6z9x3Pbwl9T96JnnyarIyVRMza8= X-Received: by 2002:a05:6130:a6:b0:68b:9eed:1c7c with SMTP id x38-20020a05613000a600b0068b9eed1c7cmr27483231uaf.0.1680529285126; Mon, 03 Apr 2023 06:41:25 -0700 (PDT) MIME-Version: 1.0 References: <0C2095E9-B9A4-42D3-B86A-852A60508D2C@gmx.de> In-Reply-To: <0C2095E9-B9A4-42D3-B86A-852A60508D2C@gmx.de> From: Neal Cardwell Date: Mon, 3 Apr 2023 09:41:08 -0400 Message-ID: To: Sebastian Moeller Cc: Ayush Mishra , BBR Development , ayush@comp.nus.edu.sg, bloat Content-Type: multipart/alternative; boundary="00000000000013700d05f86eb744" Subject: Re: [Bloat] [bbr-dev] Re: Are we heading towards a BBR-dominant Internet? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Apr 2023 13:41:26 -0000 --00000000000013700d05f86eb744 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Apr 3, 2023 at 2:25=E2=80=AFAM Sebastian Moeller = wrote: > Hi Neil, > > > thanks for your response. To make it clear I appreciate this discussion > and I do in no way want to imply the BBRs are doing anything untoward her= e > this is about understanding the principles better. > > > > On Apr 2, 2023, at 16:02, Neal Cardwell wrote: > > > > > > > > On Sun, Apr 2, 2023 at 8:14=E2=80=AFAM Sebastian Moeller > wrote: > > Hi Ayush, > > > > > On Mar 28, 2023, at 11:36, Ayush Mishra via Bloat < > bloat@lists.bufferbloat.net> wrote: > > > > > > Hey Neal, > > > > > > I was revisiting this thread before presenting this paper in iccrg > tomorrow - and I was particularly intrigued by one of the motivations you > mentioned for BBR: > > > > > > "BBR is not trying to maintain a higher throughput than CUBIC in thes= e > kinds of scenarios with steady-state bulk flows. BBR is trying to be robu= st > to the kinds of random packet loss that happen in the real world when the= re > are flows dynamically entering/leaving a bottleneck." > > > > But isn't "when there are flows dynamically entering" actually a bona > fide reason for the already established flows to scale back a bit, to giv= e > the new-commers some room to establish themselves? > > > > Yes, I agree that "when there are flows dynamically entering" is > actually a bona fide reason for the already established flows to scale ba= ck > to give the newcomers some room to establish themselves. I'm not arguing > against scaling back to give the newcomers some room to establish > themselves. I'm arguing against the specific way that Reno and CUBIC beha= ve > to try to accomplish that. :-) > > [SM] Fair enough. There likely is room for improvements > > > > > > BBRv1 essentially tried to deal with this problem by doing away with > packet loss as a congestion signal and having an entirely different > philosophy to congestion control. However, if we set aside the issue of > buffer bloat, I would imagine packet loss is a bad congestion signal in > this situation because most loss-based congestion control algorithms use = it > as a binary signal with a binary response (back-off or no back-off). In > other words, I feel the blame must be placed on not just the congestion > signal, but also on how most algorithms respond to this congestion signal= . > > > > Fair enough, but even if we assume a capacity based loss we > really do not know: > > a) did the immediate traffic simply exceed the bottleneck's queue > (assuming a fixed egress capacity/rate) > > b) did the immediate traffic simply exceed the bottleneck's egress > capacity (think variable rate link that just dropped in rate, while traff= ic > rate was constant) > > > > In case a) we might be OK with doing a gentle reduction (and take a bit > to do so) in case b) we probably should be doing a less gentle reduction > and preferably ASAP. > > > > Agreed. And that's the approach that BBRv2 takes; it would behave > differently in the two cases. In case (a) it would essentially notice tha= t > packets are being dropped and yet the delivery rate remains high, so woul= d > infer that in-flight is too high but the estimated bandwidth seems OK, so > it would immediately reduce the cwnd slightly but maintain the pacing rat= e. > > [SM] Showing my confusion here: will reducing the cwnd not result > in a reduced pacing rate at one point? Or are we talking about the > immediate response here and not the (slightly) longer term average? > Good question. In BBR (v1 and v2) the cwnd and pacing rate are largely independent. In the BBR model the pacing rate is not computed using (pacing_rate =3D k * cwnd / srtt), as is the practice for most CC modules i= n Linux TCP, for example (including CUBIC and Reno, if those are used with pacing from the fq qdisc). So I do mean that the cwnd would not, by itself result in a reduced pacing rate. The cwnd might indirectly result in a reduced pacing rate if the lower cwnd causes a lower delivery rate. > > In case (b) it would notice that the loss rate is high and delivery rat= e > has reduced substantially, so would immediately and substantially reduce > both the cwnd and pacing rate. > > [SM] But to notice a high loss rate, will we not have to wait and > withhold our response (a bit) longer, or are we talking about DupACKs > showing more than one segment missing? (Both can be fine and used in > conjunction, I just wonder what you had in mind here) > The BBRv2 code makes the decision about the amount to reduce cwnd and pacing rate at the end of the first round trip of loss recovery (and at each round trip boundary beyond that if the recovery lasts multiple rounds). So the response is indeed slightly delayed. > > > > > > > > On a per-packet basis, packet loss is a binary signal. But over a > window, the loss percentage and distribution, for example, can be a rich > signal. There is probably scope for differentiating between different kin= ds > of packet losses > > > > Sure, as long as a veridical congestion detection is still > timely enough not to make case b) above worse... > > > > Agreed. > > > > > (and deciding how to react to them) when packet loss is coupled with > the most recent delay measurement too. > > > > Hmm, say we get a "all is fine" delay probe at time X, at X+1 > the capacity drops to 50% and we incur a drop, will the most recent delay > data actually be informative for the near future? > > > > Usually it takes an ACK (a dupack or ACK carrying a SACK block) ACKing > data that transited the network path *after* the loss to infer the loss > (consistent with the RACK philosophy), and that ACK will usually provide = a > delay sample. So when there is loss usually there will be a delay signal > that is at least as fresh as the loss signal, providing a hint about the > state of the bottleneck queue after the loss. So even with loss I'd imagi= ne > that using that most recent delay data should usually be informative abou= t > the near future. > > [SM] Thanks, I think I was confusing the timing of the bandwidth > probing steps with the latency measurements, thanks for clearing that up, > and sorry... Yes, I agree that delay measurement is as good as it gets, a= nd > yes typically we should be able to extrapolate a bit into the future... > Thanks! best regards, neal > Many Thanks & Best Regards > Sebastian > > > > > > best regards, > > neal > > > > > > Regards > > Sebastian > > > > > > > Now that BBRv2 reacts to packet loss, are you making any of these > considerations too? > > > > > > This is not something I plan to present in iccrg tomorrow, just > something I was curious about :) > > > > > > Warmest regards, > > > Ayush > > > > > > On Fri, Aug 26, 2022 at 9:36=E2=80=AFPM 'Neal Cardwell' via BBR Devel= opment < > bbr-dev@googlegroups.com> wrote: > > > Yes, I agree the assumptions are key here. One key aspect of this > paper is that it focuses on the steady-state behavior of bulk flows. > > > > > > Once you allow for short flows (like web pages, RPCs, etc) to > dynamically enter and leave a bottleneck, the considerations become > different. As is well-known, Reno/CUBIC will starve themselves if new flo= ws > enter and cause loss too frequently. For CUBIC, for a somewhat typical 30= ms > broadband path with a flow fair share of 25 Mbit/sec, if new flows enter > and cause loss more frequently than roughly every 2 seconds then CUBIC wi= ll > not be able to utilize its fair share. For a high-speed WAN path, with > 100ms RTT and fair share of 10 Gbit/sec, if new flows enter and cause lo= ss > more frequently than roughly every 40 seconds then CUBIC will not be able > to utilize its fair share. Basically, loss-based CC can starve itself in > some very typical kinds of dynamic scenarios that happen in the real worl= d. > > > > > > BBR is not trying to maintain a higher throughput than CUBIC in these > kinds of scenarios with steady-state bulk flows. BBR is trying to be robu= st > to the kinds of random packet loss that happen in the real world when the= re > are flows dynamically entering/leaving a bottleneck. > > > > > > cheers, > > > neal > > > > > > > > > > > > > > > On Thu, Aug 25, 2022 at 8:01 PM Dave Taht via Bloat < > bloat@lists.bufferbloat.net> wrote: > > > I rather enjoyed this one. I can't help but wonder what would happen > > > if we plugged some different assumptions into their model. > > > > > > https://www.comp.nus.edu.sg/~bleong/publications/imc2022-nash.pdf > > > > > > -- > > > FQ World Domination pending: > https://blog.cerowrt.org/post/state_of_fq_codel/ > > > Dave T=C3=A4ht CEO, TekLibre, LLC > > > _______________________________________________ > > > Bloat mailing list > > > Bloat@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/bloat > > > > > > -- > > > You received this message because you are subscribed to the Google > Groups "BBR Development" group. > > > To unsubscribe from this group and stop receiving emails from it, sen= d > an email to bbr-dev+unsubscribe@googlegroups.com. > > > To view this discussion on the web visit > https://groups.google.com/d/msgid/bbr-dev/CADVnQykKbnxpNcpuZATug_4VLhV1%3= DaoTTQE2263o8HF9ye_TQg%40mail.gmail.com > . > > > _______________________________________________ > > > Bloat mailing list > > > Bloat@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/bloat > > > > --00000000000013700d05f86eb744 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Apr 3, 2023 at 2:25=E2=80=AFA= M Sebastian Moeller <moeller0@gmx.de<= /a>> wrote:
H= i Neil,


thanks for your response. To make it clear I appreciate this discussion and= I do in no way want to imply the BBRs are doing anything untoward here thi= s is about understanding the principles better.


> On Apr 2, 2023, at 16:02, Neal Cardwell <ncardwell@google.com> wrote:
>
>
>
> On Sun, Apr 2, 2023 at 8:14=E2=80=AFAM Sebastian Moeller <moeller0@gmx.de> wrote= :
> Hi Ayush,
>
> > On Mar 28, 2023, at 11:36, Ayush Mishra via Bloat <bloat@lists.bufferblo= at.net> wrote:
> >
> > Hey Neal,
> >
> > I was revisiting this thread before presenting this paper in iccr= g tomorrow - and I was particularly intrigued by one of the motivations you= mentioned for BBR:
> >
> > "BBR is not trying to maintain a higher throughput than CUBI= C in these kinds of scenarios with steady-state bulk flows. BBR is trying t= o be robust to the kinds of random packet loss that happen in the real worl= d when there are flows dynamically entering/leaving a bottleneck."
>
> But isn't "when there are flows dynamically entering" ac= tually a bona fide reason for the already established flows to scale back a= bit, to give the new-commers some room to establish themselves?
>
> Yes, I agree that "when there are flows dynamically entering"= ; is actually a bona fide reason for the already established flows to scale= back to give the newcomers some room to establish themselves. I'm not = arguing against scaling back to give the newcomers some room to establish t= hemselves. I'm arguing against the specific way that Reno and CUBIC beh= ave to try to accomplish that. :-)

=C2=A0 =C2=A0 =C2=A0 =C2=A0 [SM] Fair enough. There likely is room for impr= ovements

>=C2=A0
> > BBRv1 essentially tried to deal with this problem by doing away w= ith packet loss as a congestion signal and having an entirely different phi= losophy to congestion control. However, if we set aside the issue of buffer= bloat, I would imagine packet loss is a bad congestion signal in this situ= ation because most loss-based congestion control algorithms use it as a bin= ary signal with a binary response (back-off or no back-off). In other words= , I feel the blame must be placed on not just the congestion signal, but al= so on how most algorithms respond to this congestion signal.
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Fair enough, but even if we assume a = capacity based loss we really do not know:
> a) did the immediate traffic simply exceed the bottleneck's queue = (assuming a fixed egress capacity/rate)
> b) did the immediate traffic simply exceed the bottleneck's egress= capacity (think variable rate link that just dropped in rate, while traffi= c rate was constant)
>
> In case a) we might be OK with doing a gentle reduction (and take a bi= t to do so) in case b) we probably should be doing a less gentle reduction = and preferably ASAP.
>
> Agreed. And that's the approach that BBRv2 takes; it would behave = differently in the two cases. In case (a) it would essentially notice that = packets are being dropped and yet the delivery rate remains high, so would = infer that in-flight is too high but the estimated bandwidth seems OK, so i= t would immediately reduce the cwnd slightly but maintain the pacing rate.<= br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 [SM] Showing my confusion here: will reducing t= he cwnd not result in a reduced pacing rate at one point? Or are we talking= about the immediate response here and not the (slightly) longer term avera= ge?

Good question. In BBR (v1 and v2) t= he cwnd and pacing rate are largely independent. In the BBR model the pacin= g rate is not computed using (pacing_rate =3D k * cwnd / srtt), as is the p= ractice for most CC modules in Linux TCP, for example (including CUBIC and = Reno, if those are used with pacing from the fq qdisc). So I do mean that t= he cwnd would not, by itself result in a reduced pacing rate. The cwnd migh= t indirectly result in a reduced pacing rate if the lower cwnd causes a low= er delivery rate.
=C2=A0
> In case (b) it would notice that the loss rate is high and delivery ra= te has reduced substantially, so would immediately and substantially reduce= both the cwnd and pacing rate.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 [SM] But to notice a high loss rate, will we no= t have to wait and withhold our response (a bit) longer, or are we talking = about DupACKs showing more than one segment missing? (Both can be fine and = used in conjunction, I just wonder what you had in mind here)

The BBRv2 code makes the decision about the amount t= o reduce cwnd and pacing rate at the end of the first round trip of loss re= covery (and at each round trip boundary beyond that if the recovery lasts m= ultiple rounds). So the response is indeed slightly delayed.
=C2= =A0
>=C2=A0
> >
> > On a per-packet basis, packet loss is a binary signal. But over a= window, the loss percentage and distribution, for example, can be a rich s= ignal. There is probably scope for differentiating between different kinds = of packet losses
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Sure, as long as a veridical congesti= on detection is still timely enough not to make case b) above worse...
>
> Agreed.
>=C2=A0
> > (and deciding how to react to them) when packet loss is coupled w= ith the most recent delay measurement too.
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Hmm, say we get a "all is fine&q= uot; delay probe at time X, at X+1 the capacity drops to 50% and we incur a= drop, will the most recent delay data actually be informative for the near= future?
>
> Usually it takes an ACK (a dupack or ACK carrying a SACK block) ACKing= data that transited the network path *after* the loss to infer the loss (c= onsistent with the RACK philosophy), and that ACK will usually provide a de= lay sample. So when there is loss usually there will be a delay signal that= is at least as fresh as the loss signal, providing a hint about the state = of the bottleneck queue after the loss. So even with loss I'd imagine t= hat using that most recent delay data should usually be informative about t= he near future.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 [SM] Thanks, I think I was confusing the timing= of the bandwidth probing steps with the latency measurements, thanks for c= learing that up, and sorry... Yes, I agree that delay measurement is as goo= d as it gets, and yes typically we should be able to extrapolate a bit into= the future...

Thanks!

best regards,
neal

=C2=A0
=
Many Thanks & Best Regards
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Sebastian


>
> best regards,
> neal
>
>=C2=A0
> Regards
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Sebastian
>
>
> > Now that BBRv2 reacts to packet loss, are you making any of these= considerations too?
> >
> > This is not something I plan to present in iccrg tomorrow, just s= omething I was curious about :)
> >
> > Warmest regards,
> > Ayush
> >
> > On Fri, Aug 26, 2022 at 9:36=E2=80=AFPM 'Neal Cardwell' v= ia BBR Development <bbr-dev@googlegroups.com> wrote:
> > Yes, I agree the assumptions are key here. One key aspect of this= paper is that it focuses on the steady-state behavior of bulk flows.
> >
> > Once you allow for short flows (like web pages, RPCs, etc) to dyn= amically enter and leave a bottleneck, the considerations become different.= As is well-known, Reno/CUBIC will starve themselves if new flows enter and= cause loss too frequently. For CUBIC, for a somewhat typical 30ms broadban= d path with a flow fair share of 25 Mbit/sec, if new flows enter and cause = loss more frequently than roughly every 2 seconds then CUBIC will not be ab= le to utilize its fair share. For a high-speed WAN path, with 100ms RTT and= fair share of 10 Gbit/sec,=C2=A0 if new flows enter and cause loss more fr= equently than roughly every 40 seconds then CUBIC will not be able to utili= ze its fair share. Basically, loss-based CC can starve itself in some very = typical kinds of dynamic scenarios that happen in the real world.
> >
> > BBR is not trying to maintain a higher throughput than CUBIC in t= hese kinds of scenarios with steady-state bulk flows. BBR is trying to be r= obust to the kinds of random packet loss that happen in the real world when= there are flows dynamically entering/leaving a bottleneck.
> >
> > cheers,
> > neal
> >
> >
> >
> >
> > On Thu, Aug 25, 2022 at 8:01 PM Dave Taht via Bloat <bloat@lists.buffe= rbloat.net> wrote:
> > I rather enjoyed this one. I can't help but wonder what would= happen
> > if we plugged some different assumptions into their model.
> >
> > https://www.comp.nus.edu.= sg/~bleong/publications/imc2022-nash.pdf
> >
> > --
> > FQ World Domination pending: https://blog.= cerowrt.org/post/state_of_fq_codel/
> > Dave T=C3=A4ht CEO, TekLibre, LLC
> > _______________________________________________
> > Bloat mailing list
> > = Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >
> > --
> > You received this message because you are subscribed to the Googl= e Groups "BBR Development" group.
> > To unsubscribe from this group and stop receiving emails from it,= send an email to bbr-dev+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit https://grou= ps.google.com/d/msgid/bbr-dev/CADVnQykKbnxpNcpuZATug_4VLhV1%3DaoTTQE2263o8H= F9ye_TQg%40mail.gmail.com.
> > _______________________________________________
> > Bloat mailing list
> > = Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>

--00000000000013700d05f86eb744--