From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ua1-x929.google.com (mail-ua1-x929.google.com [IPv6:2607:f8b0:4864:20::929]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 27AAE3B2A4 for ; Sun, 2 Apr 2023 10:03:01 -0400 (EDT) Received: by mail-ua1-x929.google.com with SMTP id 89so19173610uao.0 for ; Sun, 02 Apr 2023 07:03:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680444180; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=cMmOr1l6gXQY24NfzzGoE2CqgXmEYGQ6hfWDOjjPZwM=; b=NWQh7wuMxo0oTJxiwchk2ayuUyz6Ac2CkAgUVMNSKsYtO6kvAa7lac3bFoNSV3f6rx kGDpU1PXiJyuXKsEepMpThEc10pYVmbGz5j3OrJ1JXRsxw0/gZZlGZ9OU4pADWFD9dk9 vl4lLpWx7YAhpY1jZo0ThokxEqF/YO3fq1Txdz6yFmgAqbIng3pzqIvBij3gT/2SY1sx apeC7Eypzv3r0Ipw3RqutMF6XTLKcCmFZuxnyTwT5mcSGeXiD2WUFJBuVBY5OcU27LlM 0LYY1+xU+7nCOsI969wJrgiqfmcgEO8xPytO8SbxDeqNvd0+VBTksIMTDGm6S2Y5zRwC f9wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680444180; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=cMmOr1l6gXQY24NfzzGoE2CqgXmEYGQ6hfWDOjjPZwM=; b=ZCeHIkVxQru668nKQ/6I7BJhKA/KV8moDsALuvmtfqPVRpvsME4Ma1wBY9t5kUGi+p IX5mfsZdm7/6QSt6di+hFpJ9qyEuft3sMYXQRde5pZPYHMuDmUV/3JNnzfw+SAEZnIEg A83orsQHOX0fDIK9BH856wLheSWfepYJX41m1PkedE4LDyesWe2HpzerD7k4AnCQzRxQ 3YAHcFal/yRWwQgxT48Swx1gp4AlIX/ggTAFHSuD4xVRBm63gAcqUPSGScQkMoDJm2VB itEAEwZ8/t8LNSfTRfY7afkfUyyhd+dBcbNze9Ft0Z5AQiYMIedlp+20MacSyGdtUn3Q xhxg== X-Gm-Message-State: AAQBX9fBrTCulwvgBRF+HGOLFVV0Dlt6HLnaXHi5A5AxbZJzxNpQL+O/ hvgXyXY7K0CrvipFS+6aXkXvy9e2EXiyjaPWbl+cHg== X-Google-Smtp-Source: AKy350YISIJGvJW/fvkFvANbZhSsUGm05SMa1KvmSIi9UBxCetYFn46BEuSUGvsbg1xu5pqIP8iNEzsg0oOUuvNw5Hw= X-Received: by 2002:a05:6130:2a2:b0:764:584b:9fbb with SMTP id q34-20020a05613002a200b00764584b9fbbmr24003470uac.0.1680444180290; Sun, 02 Apr 2023 07:03:00 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Neal Cardwell Date: Sun, 2 Apr 2023 10:02:42 -0400 Message-ID: To: Sebastian Moeller Cc: Ayush Mishra , BBR Development , ayush@comp.nus.edu.sg, bloat Content-Type: multipart/alternative; boundary="0000000000006ee8a005f85ae6ea" Subject: Re: [Bloat] [bbr-dev] Re: Are we heading towards a BBR-dominant Internet? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Apr 2023 14:03:01 -0000 --0000000000006ee8a005f85ae6ea Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Apr 2, 2023 at 8:14=E2=80=AFAM Sebastian Moeller = wrote: > Hi Ayush, > > > On Mar 28, 2023, at 11:36, Ayush Mishra via Bloat < > bloat@lists.bufferbloat.net> wrote: > > > > Hey Neal, > > > > I was revisiting this thread before presenting this paper in iccrg > tomorrow - and I was particularly intrigued by one of the motivations you > mentioned for BBR: > > > > "BBR is not trying to maintain a higher throughput than CUBIC in these > kinds of scenarios with steady-state bulk flows. BBR is trying to be robu= st > to the kinds of random packet loss that happen in the real world when the= re > are flows dynamically entering/leaving a bottleneck." > > But isn't "when there are flows dynamically entering" actually a bona fid= e > reason for the already established flows to scale back a bit, to give the > new-commers some room to establish themselves? > Yes, I agree that "when there are flows dynamically entering" is actually a bona fide reason for the already established flows to scale back to give the newcomers some room to establish themselves. I'm not arguing against scaling back to give the newcomers some room to establish themselves. I'm arguing against the specific way that Reno and CUBIC behave to try to accomplish that. :-) > > BBRv1 essentially tried to deal with this problem by doing away with > packet loss as a congestion signal and having an entirely different > philosophy to congestion control. However, if we set aside the issue of > buffer bloat, I would imagine packet loss is a bad congestion signal in > this situation because most loss-based congestion control algorithms use = it > as a binary signal with a binary response (back-off or no back-off). In > other words, I feel the blame must be placed on not just the congestion > signal, but also on how most algorithms respond to this congestion signal= . > > Fair enough, but even if we assume a capacity based loss we reall= y > do not know: > a) did the immediate traffic simply exceed the bottleneck's queue > (assuming a fixed egress capacity/rate) > b) did the immediate traffic simply exceed the bottleneck's egress > capacity (think variable rate link that just dropped in rate, while traff= ic > rate was constant) > > In case a) we might be OK with doing a gentle reduction (and take a bit t= o > do so) in case b) we probably should be doing a less gentle reduction and > preferably ASAP. > Agreed. And that's the approach that BBRv2 takes; it would behave differently in the two cases. In case (a) it would essentially notice that packets are being dropped and yet the delivery rate remains high, so would infer that in-flight is too high but the estimated bandwidth seems OK, so it would immediately reduce the cwnd slightly but maintain the pacing rate. In case (b) it would notice that the loss rate is high and delivery rate has reduced substantially, so would immediately and substantially reduce both the cwnd and pacing rate. > > > > On a per-packet basis, packet loss is a binary signal. But over a > window, the loss percentage and distribution, for example, can be a rich > signal. There is probably scope for differentiating between different kin= ds > of packet losses > > Sure, as long as a veridical congestion detection is still timely > enough not to make case b) above worse... > Agreed. > > (and deciding how to react to them) when packet loss is coupled with th= e > most recent delay measurement too. > > Hmm, say we get a "all is fine" delay probe at time X, at X+1 the > capacity drops to 50% and we incur a drop, will the most recent delay dat= a > actually be informative for the near future? > Usually it takes an ACK (a dupack or ACK carrying a SACK block) ACKing data that transited the network path *after* the loss to infer the loss (consistent with the RACK philosophy), and that ACK will usually provide a delay sample. So when there is loss usually there will be a delay signal that is at least as fresh as the loss signal, providing a hint about the state of the bottleneck queue after the loss. So even with loss I'd imagine that using that most recent delay data should usually be informative about the near future. best regards, neal > Regards > Sebastian > > > > Now that BBRv2 reacts to packet loss, are you making any of these > considerations too? > > > > This is not something I plan to present in iccrg tomorrow, just > something I was curious about :) > > > > Warmest regards, > > Ayush > > > > On Fri, Aug 26, 2022 at 9:36=E2=80=AFPM 'Neal Cardwell' via BBR Develop= ment < > bbr-dev@googlegroups.com> wrote: > > Yes, I agree the assumptions are key here. One key aspect of this paper > is that it focuses on the steady-state behavior of bulk flows. > > > > Once you allow for short flows (like web pages, RPCs, etc) to > dynamically enter and leave a bottleneck, the considerations become > different. As is well-known, Reno/CUBIC will starve themselves if new flo= ws > enter and cause loss too frequently. For CUBIC, for a somewhat typical 30= ms > broadband path with a flow fair share of 25 Mbit/sec, if new flows enter > and cause loss more frequently than roughly every 2 seconds then CUBIC wi= ll > not be able to utilize its fair share. For a high-speed WAN path, with > 100ms RTT and fair share of 10 Gbit/sec, if new flows enter and cause lo= ss > more frequently than roughly every 40 seconds then CUBIC will not be able > to utilize its fair share. Basically, loss-based CC can starve itself in > some very typical kinds of dynamic scenarios that happen in the real worl= d. > > > > BBR is not trying to maintain a higher throughput than CUBIC in these > kinds of scenarios with steady-state bulk flows. BBR is trying to be robu= st > to the kinds of random packet loss that happen in the real world when the= re > are flows dynamically entering/leaving a bottleneck. > > > > cheers, > > neal > > > > > > > > > > On Thu, Aug 25, 2022 at 8:01 PM Dave Taht via Bloat < > bloat@lists.bufferbloat.net> wrote: > > I rather enjoyed this one. I can't help but wonder what would happen > > if we plugged some different assumptions into their model. > > > > https://www.comp.nus.edu.sg/~bleong/publications/imc2022-nash.pdf > > > > -- > > FQ World Domination pending: > https://blog.cerowrt.org/post/state_of_fq_codel/ > > Dave T=C3=A4ht CEO, TekLibre, LLC > > _______________________________________________ > > Bloat mailing list > > Bloat@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat > > > > -- > > You received this message because you are subscribed to the Google > Groups "BBR Development" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to bbr-dev+unsubscribe@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/bbr-dev/CADVnQykKbnxpNcpuZATug_4VLhV1%3= DaoTTQE2263o8HF9ye_TQg%40mail.gmail.com > . > > _______________________________________________ > > Bloat mailing list > > Bloat@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat > > --0000000000006ee8a005f85ae6ea Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sun, Apr 2, 2023 at 8:14=E2=80=AFA= M Sebastian Moeller <moeller0@gmx.de<= /a>> wrote:
H= i Ayush,

> On Mar 28, 2023, at 11:36, Ayush Mishra via Bloat <bloat@lists.bufferbloat.ne= t> wrote:
>
> Hey Neal,
>
> I was revisiting this thread before presenting this paper in iccrg tom= orrow - and I was particularly intrigued by one of the motivations you ment= ioned for BBR:
>
> "BBR is not trying to maintain a higher throughput than CUBIC in = these kinds of scenarios with steady-state bulk flows. BBR is trying to be = robust to the kinds of random packet loss that happen in the real world whe= n there are flows dynamically entering/leaving a bottleneck."

But isn't "when there are flows dynamically entering" actuall= y a bona fide reason for the already established flows to scale back a bit,= to give the new-commers some room to establish themselves?

Yes, I agree that "when there are flows dynamical= ly entering" is actually a bona fide reason for the already establishe= d flows to scale back to give the newcomers some room to establish themselv= es. I'm not arguing against scaling back to give the newcomers some roo= m to establish themselves. I'm arguing against the specific way that Re= no and CUBIC behave to try to accomplish that. :-)

=C2=A0
> BBRv1 essentially tried to deal with this problem by doing away with p= acket loss as a congestion signal and having an entirely different philosop= hy to congestion control. However, if we set aside the issue of buffer bloa= t, I would imagine packet loss is a bad congestion signal in this situation= because most loss-based congestion control algorithms use it as a binary s= ignal with a binary response (back-off or no back-off). In other words, I f= eel the blame must be placed on not just the congestion signal, but also on= how most algorithms respond to this congestion signal.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Fair enough, but even if we assume a capacity b= ased loss we really do not know:
a) did the immediate traffic simply exceed the bottleneck's queue (assu= ming a fixed egress capacity/rate)
b) did the immediate traffic simply exceed the bottleneck's egress capa= city (think variable rate link that just dropped in rate, while traffic rat= e was constant)

In case a) we might be OK with doing a gentle reduction (and take a bit to = do so) in case b) we probably should be doing a less gentle reduction and p= referably ASAP.

Agreed. And that's = the approach that BBRv2 takes; it would behave differently in the two cases= . In case (a) it would essentially notice that packets are being dropped an= d yet the delivery rate remains high, so would infer that in-flight is too = high but the estimated bandwidth seems OK, so it would immediately reduce t= he cwnd slightly but maintain the pacing rate. In case (b) it would notice = that the loss rate is high and delivery rate has reduced substantially, so = would immediately and substantially reduce both the cwnd and pacing rate.
=C2=A0
>
> On a per-packet basis, packet loss is a binary signal. But over a wind= ow, the loss percentage and distribution, for example, can be a rich signal= . There is probably scope for differentiating between different kinds of pa= cket losses

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Sure, as long as a veridical congestion detecti= on is still timely enough not to make case b) above worse...

Agreed.
=C2=A0
> (and deciding how to react to them) when packet loss is coupled with t= he most recent delay measurement too.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Hmm, say we get a "all is fine" delay= probe at time X, at X+1 the capacity drops to 50% and we incur a drop, wil= l the most recent delay data actually be informative for the near future?

Usually it takes an ACK (a dupack or ACK= carrying a SACK block) ACKing data that transited the network path *after*= the loss to infer the loss (consistent with the RACK philosophy), and that ACK will usually pr= ovide a delay sample. So when there is loss usually there will be a delay s= ignal that is at least as fresh as the loss signal,=C2=A0providing a hint a= bout the state of the bottleneck queue after the loss. So even with loss I&= #39;d imagine that using that most recent delay data should usually be info= rmative about the near future.

best regards,
=
neal

=C2=A0
Regards
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Sebastian


> Now that BBRv2 reacts to packet loss, are you making any of these cons= iderations too?
>
> This is not something I plan to present in iccrg tomorrow, just someth= ing I was curious about :)
>
> Warmest regards,
> Ayush
>
> On Fri, Aug 26, 2022 at 9:36=E2=80=AFPM 'Neal Cardwell' via BB= R Development <bbr-dev@googlegroups.com> wrote:
> Yes, I agree the assumptions are key here. One key aspect of this pape= r is that it focuses on the steady-state behavior of bulk flows.
>
> Once you allow for short flows (like web pages, RPCs, etc) to dynamica= lly enter and leave a bottleneck, the considerations become different. As i= s well-known, Reno/CUBIC will starve themselves if new flows enter and caus= e loss too frequently. For CUBIC, for a somewhat typical 30ms broadband pat= h with a flow fair share of 25 Mbit/sec, if new flows enter and cause loss = more frequently than roughly every 2 seconds then CUBIC will not be able to= utilize its fair share. For a high-speed WAN path, with 100ms RTT and fair= share of 10 Gbit/sec,=C2=A0 if new flows enter and cause loss more frequen= tly than roughly every 40 seconds then CUBIC will not be able to utilize it= s fair share. Basically, loss-based CC can starve itself in some very typic= al kinds of dynamic scenarios that happen in the real world.
>
> BBR is not trying to maintain a higher throughput than CUBIC in these = kinds of scenarios with steady-state bulk flows. BBR is trying to be robust= to the kinds of random packet loss that happen in the real world when ther= e are flows dynamically entering/leaving a bottleneck.
>
> cheers,
> neal
>
>
>
>
> On Thu, Aug 25, 2022 at 8:01 PM Dave Taht via Bloat <bloat@lists.bufferbloat.= net> wrote:
> I rather enjoyed this one. I can't help but wonder what would happ= en
> if we plugged some different assumptions into their model.
>
> https://www.comp.nus.edu.sg/~b= leong/publications/imc2022-nash.pdf
>
> --
> FQ World Domination pending: https://blog.cerow= rt.org/post/state_of_fq_codel/
> Dave T=C3=A4ht CEO, TekLibre, LLC
> _______________________________________________
> Bloat mailing list
> Bloat= @lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat >
> --
> You received this message because you are subscribed to the Google Gro= ups "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send= an email to bbr-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.go= ogle.com/d/msgid/bbr-dev/CADVnQykKbnxpNcpuZATug_4VLhV1%3DaoTTQE2263o8HF9ye_= TQg%40mail.gmail.com.
> _______________________________________________
> Bloat mailing list
> Bloat= @lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
--0000000000006ee8a005f85ae6ea--