[Bloat] geoff huston's take on BBR

[Bloat] geoff huston's take on BBR

[Dave Taht]

https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf

-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Bloat] geoff huston's take on BBR

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf

"More research needed". Naturally ;)

(But yeah, good points overall)

-Toke

Re: [Bloat] geoff huston's take on BBR

[Matthias Tafelmeier]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 696 bytes --]

On 06/12/2018 02:42 AM, Toke Høiland-Jørgensen wrote:
>> https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf
> "More research needed". Naturally ;)
>
> (But yeah, good points overall)

Interesting. Potentially, all affectuated. After having applied the BBR
2.0, we might are back to Cubic? :D

Moreover, if it tends to be unstable on larger scale - what is Google
doing then? Thought they've got a more or less homogeneous BBR driven
TCP flow ecosystem - at least internally!? Was all propaganda? When
speculating, might working for them since of centrally handled flow
steering approaches - "imposing inter-flow fairness".

-- 

Besten Gruß

Matthias Tafelmeier


[-- Attachment #1.1.1.2: Type: text/html, Size: 1436 bytes --]

[-- Attachment #1.1.2: 0xE0164F5B8ADF343B.asc --]
[-- Type: application/pgp-keys, Size: 5250 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 538 bytes --]

Re: [Bloat] geoff huston's take on BBR

[Bless, Roland (TM)]

Hi,

Am 12.06.2018 um 07:09 schrieb Matthias Tafelmeier:
> On 06/12/2018 02:42 AM, Toke Høiland-Jørgensen wrote:
>>> https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf
>> "More research needed". Naturally ;)
>>
>> (But yeah, good points overall)
> 
> Interesting. Potentially, all affectuated. After having applied the BBR
> 2.0, we might are back to Cubic? :D

I don't understand what you're saying. I think Geoff tested BBR v1.0.
Explanations for the experienced behavior can be found in our paper
http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf, esp. section
3. Geoff's findings in the wild nicely confirm our results that were
performed in more controlled lab settings. Important is though, that
you always test with multiple concurrent BBR flows...

> Moreover, if it tends to be unstable on larger scale - what is Google
> doing then? Thought they've got a more or less homogeneous BBR driven
> TCP flow ecosystem - at least internally!? Was all propaganda? When
> speculating, might working for them since of centrally handled flow
> steering approaches - "imposing inter-flow fairness".

There are certain situations where BBR might work well:
1) you only have a single flow at the bottleneck, might be the case in
their B4 scenario
2) The senders a application limited (e.g., YouTube)
3) The bottleneck buffer is much larger than a BDP
   (then BDP will limit the queue size between 1 and 1.5 BDP)
However, BBR has no explicit fairness mechanism, so sometimes
one will see quite unfair shares for longer periods,
even if there are only BBR flows present at then bottleneck.

Regards
 Roland

Re: [Bloat] geoff huston's take on BBR

[Dave Taht]

On Tue, Jun 12, 2018 at 12:36 AM, Bless, Roland (TM)
<roland.bless@kit.edu> wrote:
> Hi,
>
> Am 12.06.2018 um 07:09 schrieb Matthias Tafelmeier:
>> On 06/12/2018 02:42 AM, Toke Høiland-Jørgensen wrote:
>>>> https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf
>>> "More research needed". Naturally ;)
>>>
>>> (But yeah, good points overall)
>>
>> Interesting. Potentially, all affectuated. After having applied the BBR
>> 2.0, we might are back to Cubic? :D
>
> I don't understand what you're saying. I think Geoff tested BBR v1.0.
> Explanations for the experienced behavior can be found in our paper
> http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf, esp. section
> 3. Geoff's findings in the wild nicely confirm our results that were
> performed in more controlled lab settings. Important is though, that
> you always test with multiple concurrent BBR flows...

we always do that, 'round here, with flent. Glad more folk are doing it. :)

>> Moreover, if it tends to be unstable on larger scale - what is Google
>> doing then? Thought they've got a more or less homogeneous BBR driven
>> TCP flow ecosystem - at least internally!? Was all propaganda? When
>> speculating, might working for them since of centrally handled flow
>> steering approaches - "imposing inter-flow fairness".
>
> There are certain situations where BBR might work well:
> 1) you only have a single flow at the bottleneck, might be the case in
> their B4 scenario
> 2) The senders a application limited (e.g., YouTube)

I think the application limited scenario is the primary one. once
typical links are fully capable of video streaming
4k video a lot of the demand for better congestion control will drop.

> 3) The bottleneck buffer is much larger than a BDP
>    (then BDP will limit the queue size between 1 and 1.5 BDP)

Sadly we still see 2sec queues on cmtses, in particular.

> However, BBR has no explicit fairness mechanism, so sometimes
> one will see quite unfair shares for longer periods,
> even if there are only BBR flows present at then bottleneck.

except with fq at the bottleneck.

>
> Regards
>  Roland
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Bloat] geoff huston's take on BBR

[Dave Taht]

On Tue, Jun 12, 2018 at 4:40 AM, Dave Taht <dave.taht@gmail.com> wrote:
> On Tue, Jun 12, 2018 at 12:36 AM, Bless, Roland (TM)
> <roland.bless@kit.edu> wrote:
>> Hi,
>>
>> Am 12.06.2018 um 07:09 schrieb Matthias Tafelmeier:
>>> On 06/12/2018 02:42 AM, Toke Høiland-Jørgensen wrote:
>>>>> https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf
>>>> "More research needed". Naturally ;)
>>>>
>>>> (But yeah, good points overall)
>>>
>>> Interesting. Potentially, all affectuated. After having applied the BBR
>>> 2.0, we might are back to Cubic? :D
>>
>> I don't understand what you're saying. I think Geoff tested BBR v1.0.
>> Explanations for the experienced behavior can be found in our paper
>> http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf, esp. section
>> 3. Geoff's findings in the wild nicely confirm our results that were
>> performed in more controlled lab settings. Important is though, that
>> you always test with multiple concurrent BBR flows...
>
> we always do that, 'round here, with flent. Glad more folk are doing it. :)

OK, I read your paper. (It's 4am, can't sleep)

My principal observation remains the same: e2e fairness is nearly
hopeless on short
timescales.

FQ solves most of it, and even if if the rfc8290 aqm component is
targetted at cubics
response curve (and bbr treats it as noise), the resulting delay curve
in the fq'd
environment seemed to get BBR on the right track on it's first probe
(except when many flows were
started concurrently), but I didn't delve deep into it at the time.

I would love it if you could redo your tests with cake managing the
bottleneck link.
(we've got a bug in the thing right now at 40+gige, but...)

https://arxiv.org/abs/1804.07617

>>> Moreover, if it tends to be unstable on larger scale - what is Google
>>> doing then? Thought they've got a more or less homogeneous BBR driven
>>> TCP flow ecosystem - at least internally!? Was all propaganda? When
>>> speculating, might working for them since of centrally handled flow
>>> steering approaches - "imposing inter-flow fairness".
>>
>> There are certain situations where BBR might work well:
>> 1) you only have a single flow at the bottleneck, might be the case in
>> their B4 scenario
>> 2) The senders a application limited (e.g., YouTube)
>
> I think the application limited scenario is the primary one. once
> typical links are fully capable of video streaming
> 4k video a lot of the demand for better congestion control will drop.
>
>> 3) The bottleneck buffer is much larger than a BDP
>>    (then BDP will limit the queue size between 1 and 1.5 BDP)
>
> Sadly we still see 2sec queues on cmtses, in particular.
>
>> However, BBR has no explicit fairness mechanism, so sometimes
>> one will see quite unfair shares for longer periods,
>> even if there are only BBR flows present at then bottleneck.
>
> except with fq at the bottleneck.
>
>>
>> Regards
>>  Roland
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Bloat] geoff huston's take on BBR

[Matthias Tafelmeier]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 1840 bytes --]


>> Interesting. Potentially, all affectuated. After having applied the BBR
>> 2.0, we might are back to Cubic? :D
> I don't understand what you're saying. I think Geoff tested BBR v1.0.
> Explanations for the experienced behavior can be found in our paper
> http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf, esp. section
> 3. Geoff's findings in the wild nicely confirm our results that were
> performed in more controlled lab settings. Important is though, that
> you always test with multiple concurrent BBR flows...
To put this straight - I meant that all the efferescing outlines as to
BBR were potentially overly hasty, perceive it as a mere utterance. For
BBR2.0 I referred to the slide by Geoff listing the cued improvements
from 1.0 -> 2.0 - insinuating thereby ruling out thinkable 'vantage
aspects' of BBR (excuse my cynicism - early morning ranting!). Good.
Thanks for sharing your work.
>> Moreover, if it tends to be unstable on larger scale - what is Google
>> doing then? Thought they've got a more or less homogeneous BBR driven
>> TCP flow ecosystem - at least internally!? Was all propaganda? When
>> speculating, might working for them since of centrally handled flow
>> steering approaches - "imposing inter-flow fairness".
> There are certain situations where BBR might work well:
> 1) you only have a single flow at the bottleneck, might be the case in
> their B4 scenario
> 2) The senders a application limited (e.g., YouTube)
> 3) The bottleneck buffer is much larger than a BDP
>    (then BDP will limit the queue size between 1 and 1.5 BDP)
> However, BBR has no explicit fairness mechanism, so sometimes
> one will see quite unfair shares for longer periods,
> even if there are only BBR flows present at then bottleneck.

ACK

-- 
Besten Gruß

Matthias Tafelmeier


[-- Attachment #1.1.1.2: Type: text/html, Size: 2741 bytes --]

[-- Attachment #1.1.2: 0xE0164F5B8ADF343B.asc --]
[-- Type: application/pgp-keys, Size: 5250 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 538 bytes --]

Re: [Bloat] geoff huston's take on BBR

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 629 bytes --]



> On 11 Jun 2018, at 22:27, Dave Taht <dave.taht@gmail.com> wrote:
> 
> https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf

Fascinating!

"	• BBR changes all those assumptions, and could potentially push many networks into sustained instability

	• –  We cannot use the conventional network control mechanisms to regulate BBR flows

• Selective packet drop just won’t create back pressure on the flow”

And I keep on seeing questions on whether BBR understands ECN - if not…. well I think we see the results.

Cheers,

Kevin D-B

012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Bloat] geoff huston's take on BBR

[Dave Taht]

On Mon, Jun 11, 2018 at 10:58 PM, Kevin Darbyshire-Bryant
<kevin@darbyshire-bryant.me.uk> wrote:
>
>
>> On 11 Jun 2018, at 22:27, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf
>
> Fascinating!
>
> "       • BBR changes all those assumptions, and could potentially push many networks into sustained instability
>
>         • –  We cannot use the conventional network control mechanisms to regulate BBR flows
>
> • Selective packet drop just won’t create back pressure on the flow”
>
> And I keep on seeing questions on whether BBR understands ECN - if not…. well I think we see the results.

I think geoff goofed in his testing of BBR, starting all flows at the
same time, thus syncing up their probing periods. Real traffic is not
correlated this way.
(I made the same mistake on my initial bbr testing)

I do agree that bbr treats aqm drops as "noise", not backpressure. And
bbr scares me.

I look forward very much to bbr one day soon doing some sort of sane,
conservative, response to ecn marks.

PS having fq on the link makes cubic and bbr cohabitate just fine.
fq_codel vs bbr behavior was reasonable, though bbr lost a lot more
packets before finding a decent state.
> Cheers,
>
> Kevin D-B
>
> 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Bloat] geoff huston's take on BBR

[Geoff Huston]

> On 12 Jun 2018, at 4:55 pm, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Mon, Jun 11, 2018 at 10:58 PM, Kevin Darbyshire-Bryant
> <kevin@darbyshire-bryant.me.uk> wrote:
>> 
>> 
>>> On 11 Jun 2018, at 22:27, Dave Taht <dave.taht@gmail.com> wrote:
>>> 
>>> https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf
>> 
>> Fascinating!
>> 
>> "       • BBR changes all those assumptions, and could potentially push many networks into sustained instability
>> 
>>        • –  We cannot use the conventional network control mechanisms to regulate BBR flows
>> 
>> • Selective packet drop just won’t create back pressure on the flow”
>> 
>> And I keep on seeing questions on whether BBR understands ECN - if not…. well I think we see the results.
> 
> I think geoff goofed in his testing of BBR, starting all flows at the
> same time, thus syncing up their probing periods. Real traffic is not
> correlated this way.
> (I made the same mistake on my initial bbr testing)

no - I started the flows at 10, 20 and 30 seconds after the initial flow started.


> 
> I do agree that bbr treats aqm drops as "noise", not backpressure. And
> bbr scares me.
> 
> I look forward very much to bbr one day soon doing some sort of sane,
> conservative, response to ecn marks.
> 


I’m not sure that I understand this comment.

Part of the pressure going on here is the issue of whether the endpoints can and should trust the signals and.or manipulation that they get from the network infrastructure. BBR is using a different form of feedback to control its send rate. Essentially BBR is taking a delay variance measurement 1 / 8 of the time to adjust its internal model of the end-to-end delay bandwidth product (every 8th RTT). ECN provides a constant information flow, and this certainly matches the requirements of loss-based TCP, where every ACK contributes to the TCP flow dynamic, but it does not seem to me to be a good match to BBR’s requirements. 

The idea with BBR is to drive the network path such at the internal routers are sitting just at the initial onset of queuing. In theory ECN will not trigger at the onset of queuing, but will trigger later in the cycle of queue buildup.



> PS having fq on the link makes cubic and bbr cohabitate just fine.
> fq_codel vs bbr behavior was reasonable, though bbr lost a lot more
> packets before finding a decent state.

I guess that "It Depends" - my long delay experiments in the presentation referenced above showed cubic being completely crowded out by BBR.


Geoff

Re: [Bloat] geoff huston's take on BBR

[Dave Taht]

On Tue, Jun 12, 2018 at 12:49 AM, Geoff Huston <gih@apnic.net> wrote:
>
>
>> On 12 Jun 2018, at 4:55 pm, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> On Mon, Jun 11, 2018 at 10:58 PM, Kevin Darbyshire-Bryant
>> <kevin@darbyshire-bryant.me.uk> wrote:
>>>
>>>
>>>> On 11 Jun 2018, at 22:27, Dave Taht <dave.taht@gmail.com> wrote:
>>>>
>>>> https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf
>>>
>>> Fascinating!
>>>
>>> "       • BBR changes all those assumptions, and could potentially push many networks into sustained instability
>>>
>>>        • –  We cannot use the conventional network control mechanisms to regulate BBR flows
>>>
>>> • Selective packet drop just won’t create back pressure on the flow”
>>>
>>> And I keep on seeing questions on whether BBR understands ECN - if not…. well I think we see the results.
>>
>> I think geoff goofed in his testing of BBR, starting all flows at the
>> same time, thus syncing up their probing periods. Real traffic is not
>> correlated this way.
>> (I made the same mistake on my initial bbr testing)
>
> no - I started the flows at 10, 20 and 30 seconds after the initial flow started.

k.

>
>>
>> I do agree that bbr treats aqm drops as "noise", not backpressure. And
>> bbr scares me.
>>
>> I look forward very much to bbr one day soon doing some sort of sane,
>> conservative, response to ecn marks.
>>
>
>
> I’m not sure that I understand this comment.
>
> Part of the pressure going on here is the issue of whether the endpoints can and should trust the signals and.or manipulation that they get from the network infrastructure. BBR is using a different form of feedback to control its send rate. Essentially BBR is taking a delay variance measurement 1 / 8 of the time to adjust its internal model of the end-to-end delay bandwidth product (every 8th RTT). ECN provides a constant information flow, and this certainly matches the requirements of loss-based TCP, where every ACK contributes to the TCP flow dynamic, but it does not seem to me to be a good match to BBR’s requirements.

ECN CE is explicit. Someone on the path is saying "slow down". If BBR
merely responded like cubic to an ECN mark, it would be a win.

This of course doesn't handle the malicious middlebox or sender
problem, but neither does anything else.

I was pleased to see recently from
https://www.ietf.org/proceedings/98/slides/slides-98-maprg-tcp-ecn-experience-with-enabling-ecn-on-the-internet-padma-bhooma-00.pdf
that france was showing 6% CE markings on uplnks. Since free.fr
deployed fq_codel in 2011, I figure that's almost entirely from their
deployment. I was boggled at the argentine result (30%???), although I
know openwrt and derivatives with sqm is very popular there, that
can't explain all of that.

> The idea with BBR is to drive the network path such at the internal routers are sitting just at the initial onset of queuing. In theory ECN will not trigger at the onset of queuing, but will trigger later in the cycle of queue buildup.

a few ms of queue is not a problem.... we (in the fq_codel world) are
seeing 5ms queues and full throughput on ethernet, 15-20ms
on wifi. (and BBR and cubic share almost equally, all the time, at
least at the range of rates we've tested)

Wifi: https://arxiv.org/pdf/1703.00064.pdf
Cake shaper: https://arxiv.org/abs/1804.07617

There's a paper with the BBR vs cubic vs fq_codel result around here somewhere.

I am painfully aware that upgrading edge routers and other bottleneck
devices to have smart queue management is a long game...

... but I long ago gave up on pure end to end approaches.

>
>
>> PS having fq on the link makes cubic and bbr cohabitate just fine.
>> fq_codel vs bbr behavior was reasonable, though bbr lost a lot more
>> packets before finding a decent state.
>
> I guess that "It Depends" - my long delay experiments in the presentation referenced above showed cubic being completely crowded out by BBR.

Please give sqm or cake a shot.

>
> Geoff
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Bloat] geoff huston's take on BBR

[Bless, Roland (TM)]

Hi Geoff,

see inline.

Am 12.06.2018 um 09:49 schrieb Geoff Huston:
>> On 12 Jun 2018, at 4:55 pm, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> On Mon, Jun 11, 2018 at 10:58 PM, Kevin Darbyshire-Bryant
>> <kevin@darbyshire-bryant.me.uk> wrote:
>>>
>>>
>>>> On 11 Jun 2018, at 22:27, Dave Taht <dave.taht@gmail.com> wrote:
>>>>
>>>> https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf
>>>
>>> Fascinating!
>>>
>>> "       • BBR changes all those assumptions, and could potentially push many networks into sustained instability
>>>
>>>        • –  We cannot use the conventional network control mechanisms to regulate BBR flows
>>>
>>> • Selective packet drop just won’t create back pressure on the flow”
>>>
>>> And I keep on seeing questions on whether BBR understands ECN - if not…. well I think we see the results.
>>
>> I think geoff goofed in his testing of BBR, starting all flows at the
>> same time, thus syncing up their probing periods. Real traffic is not
>> correlated this way.
>> (I made the same mistake on my initial bbr testing)
> 
> no - I started the flows at 10, 20 and 30 seconds after the initial flow started.

This is nevertheless advantageous for BBR, since it performs its
ProbeRTT phase every 10s. So using 11, 23, and 34 seconds may
make a difference in convergence speed (that was at least observed
in our experiments).
>> I do agree that bbr treats aqm drops as "noise", not backpressure. And
>> bbr scares me.
>>
>> I look forward very much to bbr one day soon doing some sort of sane,
>> conservative, response to ecn marks.
>>
> 
> 
> I’m not sure that I understand this comment.
> 
> Part of the pressure going on here is the issue of whether the endpoints can and should trust the signals and.or manipulation that they get from the network infrastructure. BBR is using a different form of feedback to control its send rate. Essentially BBR is taking a delay variance measurement 1 / 8 of the time to adjust its internal model of the end-to-end delay bandwidth product (every 8th RTT). ECN provides a constant information flow, and this certainly matches the requirements of loss-based TCP, where every ACK contributes to the TCP flow dynamic, but it does not seem to me to be a good match to BBR’s requirements. 

I don't think that this characterization of BBR is completely accurate.
BBR is not measuring the "delay variance" (though it could) in ProbeBW,
but it measures the maximally achieved delivery rate while sending at an
increased data rate. So even after the onset of queueing, a BBR sender
will see a higher achieved delivery rate (at the cost of other competing
flows). BBR also calculates a BDP estimate, but it is only used to
calculated the "inflight cap". Especially the measured RTT is only used
in this calculation and nowhere else in BBRv1.0.

> The idea with BBR is to drive the network path such at the internal routers are sitting just at the initial onset of queuing. In theory ECN will not trigger at the onset of queuing, but will trigger later in the cycle of queue buildup.

Yep, but a BBR sender doesn't detect the onset of queuing and steadily
increases its amount of inflight data until the cap of 2BDP is reached.
In total, the queue can reach 1.5 BDP.

>> PS having fq on the link makes cubic and bbr cohabitate just fine.
>> fq_codel vs bbr behavior was reasonable, though bbr lost a lot more
>> packets before finding a decent state.
> 
> I guess that "It Depends" - my long delay experiments in the presentation referenced above showed cubic being completely crowded out by BBR.

Yes. This will always happen in case the bottleneck buffer is smaller
than 1 BDP (see slide 9 of
https://datatracker.ietf.org/meeting/100/materials/slides-100-iccrg-an-experimental-evaluation-of-bbr-congestion-control-00).

Regards,
 Roland

Re: [Bloat] geoff huston's take on BBR

[Geoff Huston]

> On 13 Jun 2018, at 1:58 am, Bless, Roland (TM) <roland.bless@kit.edu> wrote:
>> 
>> no - I started the flows at 10, 20 and 30 seconds after the initial flow started.
> 
> This is nevertheless advantageous for BBR, since it performs its
> ProbeRTT phase every 10s. So using 11, 23, and 34 seconds may
> make a difference in convergence speed (that was at least observed
> in our experiments).


I’m not sure that I follow this - It was my understanding that BBR used the RTT interval as its control timer, and maintained a constant send rate for 6 successive RTTY intervals, then inflated the sending r4ate by 25% for the next RTT interval and deflated it by the same amount for the next RTT interval. I don’t believe that there is any absolute timer in BBR along the lines of a 10 second timer that you appear to be suggesting here.

regards,

   Geoff

Re: [Bloat] geoff huston's take on BBR

[Bless, Roland (TM)]

Hi Geoff,

Am 13.06.2018 um 01:02 schrieb Geoff Huston:
>> On 13 Jun 2018, at 1:58 am, Bless, Roland (TM) <roland.bless@kit.edu> wrote:
>>>
>>> no - I started the flows at 10, 20 and 30 seconds after the initial flow started.
>>
>> This is nevertheless advantageous for BBR, since it performs its
>> ProbeRTT phase every 10s. So using 11, 23, and 34 seconds may
>> make a difference in convergence speed (that was at least observed
>> in our experiments).
> 
> 
> I’m not sure that I follow this - It was my understanding that BBR used the RTT interval as its control timer, and maintained a constant send rate for 6 successive RTTY intervals, then inflated the sending r4ate by 25% for the next RTT interval and deflated it by the same amount for the next RTT interval. I don’t believe that there is any absolute timer in BBR along the lines of a 10 second timer that you appear to be suggesting here.

I didn't write that BBR uses a 10s timer, but BBR actually uses a time
window of 10s. What you describe is BBR's ProbeBW phase, where
it probes for more bandwidth. However, your understanding is not
quite correct. Since BBR uses a maximum filter (windowed by 10RTTs),
it actually increases its sending rate _immediately_ if the 1.25
pacing gain achieved a higher delivery rate (which is nearly always the
case if multiple flows are present).
So depending on where the pacing_gain is raised in the gain cycle (it
starts randomly), BBR will increase its sending rate even within this
gain cycle of 8 RTTs and keep it for further 10 RTTs. BBR will
decrease its sending rate only after 10 RTTs. That is why
multiple BBR flows together are sending always faster than the
bottleneck rate in total.
I would recommend that you check https://queue.acm.org/detail.cfm?id=3022184
again.

Now for the 10s interval. You are correct that it's not
a 10s timer, but I wasn't suggesting that either.
The RTprop estimate is defined on page 7 of the article
and  described as
"Since path changes happen on time scales >>RTprop, an unbiased,
efficient estimator at time T is
...= min(RTT_t) \forall t \in [T-W_R,T]
i.e., a running min over time window W_R (which is typically
tens of seconds to minutes)."
and on p. 31:
"BBR flows cooperate to periodically drain the
bottleneck queue using a state called ProbeRTT. In any
state other than ProbeRTT itself, if the RTProp estimate
has not been updated (i.e., by getting a lower RTT
measurement) for more than 10 seconds, then BBR enters
ProbeRTT and reduces the cwnd to a very small value
(four packets)."
So it's basically a time window and also implemented like that.
A concise BBR description can be also found in our paper in
section II (http://doc.tm.kit.edu/2017-kit-icnp-bbr-authors-copy.pdf).

Regards,
 Roland

Re: [Bloat] geoff huston's take on BBR

[Greg White]

On 6/12/18, 8:39 AM, "Bloat on behalf of Geoff Huston" <bloat-bounces@lists.bufferbloat.net on behalf of gih@apnic.net> wrote:

    >
    >I do agree that bbr treats aqm drops as "noise", not backpressure. And
    >bbr scares me.
    >I look forward very much to bbr one day soon doing some sort of sane,
    >conservative, response to ecn marks.
    
    
    I’m not sure that I understand this comment.
    
    Part of the pressure going on here is the issue of whether the endpoints can and should trust the signals and.or manipulation that they get from the network infrastructure. BBR is using a different form of feedback to control its send rate. Essentially BBR is taking a delay variance measurement 1 / 8 of the time to adjust its internal model of the end-to-end delay bandwidth product (every 8th RTT). ECN provides a constant information flow, and this certainly matches the requirements of loss-based TCP, where every ACK contributes to the TCP flow dynamic, but it does not seem to me to be a good match to BBR’s requirements. 
    
    The idea with BBR is to drive the network path such at the internal routers are sitting just at the initial onset of queuing. In theory ECN will not trigger at the onset of queuing, but will trigger later in the cycle of queue buildup.
    
[GW]  That's "Classic" ECN.  The proposed new version https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-02 starts marking right at the onset of queuing, which would allow a congestion controller to much more accurately stay right at the optimal congestion window, without all of the cycling phenomena and inferences that BBR uses.



    
    

Re: [Bloat] geoff huston's take on BBR

[Anna Brunstrom]

On 2018-06-13 00:28, Greg White wrote:

>
> On 6/12/18, 8:39 AM, "Bloat on behalf of Geoff Huston" <bloat-bounces@lists.bufferbloat.net on behalf of gih@apnic.net> wrote:
>
>      >
>      >I do agree that bbr treats aqm drops as "noise", not backpressure. And
>      >bbr scares me.
>      >I look forward very much to bbr one day soon doing some sort of sane,
>      >conservative, response to ecn marks.
>      
>      
>      I’m not sure that I understand this comment.
>      
>      Part of the pressure going on here is the issue of whether the endpoints can and should trust the signals and.or manipulation that they get from the network infrastructure. BBR is using a different form of feedback to control its send rate. Essentially BBR is taking a delay variance measurement 1 / 8 of the time to adjust its internal model of the end-to-end delay bandwidth product (every 8th RTT). ECN provides a constant information flow, and this certainly matches the requirements of loss-based TCP, where every ACK contributes to the TCP flow dynamic, but it does not seem to me to be a good match to BBR’s requirements.
>      
>      The idea with BBR is to drive the network path such at the internal routers are sitting just at the initial onset of queuing. In theory ECN will not trigger at the onset of queuing, but will trigger later in the cycle of queue buildup.
>      
> [GW]  That's "Classic" ECN.  The proposed new version https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-02 starts marking right at the onset of queuing, which would allow a congestion controller to much more accurately stay right at the optimal congestion window, without all of the cycling phenomena and inferences that BBR uses.

See 
https://datatracker.ietf.org/meeting/101/materials/slides-101-iccrg-bbr-congestion-control-with-l4s-support-02 
for work on BBR with L4S support by Ingemar Johansson. Or the second 
talk in the video from the session 
https://www.youtube.com/watch?v=rHH9wFbms80&list=PLC86T-6ZTP5j_HaBNdfPbgxGIp22cnaWS

Cheers,
Anna

>
>
>
>      
>      
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] geoff huston's take on BBR

[Jim Gettys]

[-- Attachment #1: Type: text/plain, Size: 1428 bytes --]

On Tue, Jun 12, 2018 at 2:55 AM Dave Taht <dave.taht@gmail.com> wrote:

> On Mon, Jun 11, 2018 at 10:58 PM, Kevin Darbyshire-Bryant
> <kevin@darbyshire-bryant.me.uk> wrote:
> >
> >> On 11 Jun 2018, at 22:27, Dave Taht <dave.taht@gmail.com> wrote:
> >>
> >> https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf
> >
> > Fascinating!
> >
> > "       • BBR changes all those assumptions, and could potentially push
> many networks into sustained instability
> >
> >         • –  We cannot use the conventional network control mechanisms
> to regulate BBR flows
> >
> > • Selective packet drop just won’t create back pressure on the flow”
> >
> > And I keep on seeing questions on whether BBR understands ECN - if not….
> well I think we see the results.
>
> I think geoff goofed in his testing of BBR, starting all flows at the
> same time, thus syncing up their probing periods. Real traffic is not
> correlated this way.
> (I made the same mistake on my initial bbr testing)
>

​Beware of self synchronization...  In other situations, there have been
situations in which aggregate behavior occurs due to this phenomena.

So both synchronized and unsynchronized startup of flows under test is
actually a good idea (and watching to see if anything in the aggregate
behavior over time results in such synchronization).
                                                  - Jim

[-- Attachment #2: Type: text/html, Size: 2387 bytes --]