[Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Maciej Soltysiak]

[-- Attachment #1: Type: text/plain, Size: 700 bytes --]

Hi,

Just saw this: https://patchwork.ozlabs.org/patch/671069/

Interested to see how BBR would play out with things like fq_codel or cake.

"loss-based congestion control is unfortunately out-dated in today's
networks. On
today's Internet, loss-based congestion control causes the infamous
bufferbloat problem"

So, instead of waiting for packet loss they probe and measure, e.g. when
doing slow start (here called STARTUP) they don't speed up until packet
loss, but slow down before reaching estimated bandwidth level.

Cake and fq_codel work on all packets and aim to signal packet loss early
to network stacks by dropping; BBR works on TCP and aims to prevent packet
loss.


Best regards,
Maciej

[-- Attachment #2: Type: text/html, Size: 999 bytes --]

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Dave Taht]

BBR is pretty awesome, and it's one of the reasons why I stopped
sweating inbound rate limiting + fq_codel as much as I used to. I have
a blog entry pending on this but wasn't expecting the code to be
released before the paper was... and all I had to go on til yesterday
was Nowlan's dissertation:

http://blog.cerowrt.org/papers/bbr_thesis.pdf

which seemed closer to good than anything I'd read before, but still
wrong in a few respects, which has taken a few years to sort out. I
think reading the upcoming acm queue paper is going to be fun!

I think they have identified the right variables to probe - RTT and
bandwidth, in sequence - for modern congestion control to work much
better.

Still BBR makes a few assumptions that do not hold (or so I think) -
with wifi in the control loop, and it needs wider testing in more
circumstances than just google facing out - like on itty bitty nas's
and media servers - and especially seeing what happens when it
interacts with fq_codel and cake would be good to see. I've watched
youtube be *excellent* for 2 years now, and only had the faintest
hints as to why.

It was quite amusing that the original patchset didn't compile on 32
bit platforms.

And make no mistake - it still makes plenty of sense to apply
fq_codel-like algorithms to routers, and the stuff we just did to wifi
for fq_codel and airtime fairness. Had I thought BBR solved everything
I'd have quit years ago.



On Sat, Sep 17, 2016 at 11:34 AM, Maciej Soltysiak <maciej@soltysiak.com> wrote:
> Hi,
>
> Just saw this: https://patchwork.ozlabs.org/patch/671069/
>
> Interested to see how BBR would play out with things like fq_codel or cake.
>
> "loss-based congestion control is unfortunately out-dated in today's
> networks. On
> today's Internet, loss-based congestion control causes the infamous
> bufferbloat problem"
>
> So, instead of waiting for packet loss they probe and measure, e.g. when
> doing slow start (here called STARTUP) they don't speed up until packet
> loss, but slow down before reaching estimated bandwidth level.
>
> Cake and fq_codel work on all packets and aim to signal packet loss early to
> network stacks by dropping; BBR works on TCP and aims to prevent packet
> loss.
>
>
> Best regards,
> Maciej
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Alan Jenkins]

On 17/09/16 19:53, Dave Taht wrote:
> BBR is pretty awesome, and it's one of the reasons why I stopped
> sweating inbound rate limiting + fq_codel as much as I used to. I have
> a blog entry pending on this but wasn't expecting the code to be
> released before the paper was... and all I had to go on til yesterday
> was Nowlan's dissertation:
>
> http://blog.cerowrt.org/papers/bbr_thesis.pdf

are we sure - that's a fairly different algorithm and different 
expansion of the acronym...

 > video  streaming  experiments  ran  on  a  live,  production  CDN, 
where clients included real mobile and desktop users

 > large, multinational production network

hmm, I suppose...

 > ## Acknowledgments ##
 >
 > Van
 > ...
 > Eric
 > ...


ok, there's clearly some overlap here :-D.


> which seemed closer to good than anything I'd read before, but still
> wrong in a few respects, which has taken a few years to sort out. I
> think reading the upcoming acm queue paper is going to be fun!
>
> I think they have identified the right variables to probe - RTT and
> bandwidth, in sequence - for modern congestion control to work much
> better.
>
> Still BBR makes a few assumptions that do not hold (or so I think) -
> with wifi in the control loop, and it needs wider testing in more
> circumstances than just google facing out - like on itty bitty nas's
> and media servers - and especially seeing what happens when it
> interacts with fq_codel and cake would be good to see. I've watched
> youtube be *excellent* for 2 years now, and only had the faintest
> hints as to why.
>
> It was quite amusing that the original patchset didn't compile on 32
> bit platforms.
>
> And make no mistake - it still makes plenty of sense to apply
> fq_codel-like algorithms to routers, and the stuff we just did to wifi
> for fq_codel and airtime fairness. Had I thought BBR solved everything
> I'd have quit years ago.
>
>
>
> On Sat, Sep 17, 2016 at 11:34 AM, Maciej Soltysiak <maciej@soltysiak.com> wrote:
>> Hi,
>>
>> Just saw this: https://patchwork.ozlabs.org/patch/671069/
>>
>> Interested to see how BBR would play out with things like fq_codel or cake.
>>
>> "loss-based congestion control is unfortunately out-dated in today's
>> networks. On
>> today's Internet, loss-based congestion control causes the infamous
>> bufferbloat problem"
>>
>> So, instead of waiting for packet loss they probe and measure, e.g. when
>> doing slow start (here called STARTUP) they don't speed up until packet
>> loss, but slow down before reaching estimated bandwidth level.
>>
>> Cake and fq_codel work on all packets and aim to signal packet loss early to
>> network stacks by dropping; BBR works on TCP and aims to prevent packet
>> loss.

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Dave Taht]

On Wed, Sep 21, 2016 at 2:06 AM, Alan Jenkins
<alan.christopher.jenkins@gmail.com> wrote:
> On 17/09/16 19:53, Dave Taht wrote:
>>
>> BBR is pretty awesome, and it's one of the reasons why I stopped
>> sweating inbound rate limiting + fq_codel as much as I used to. I have
>> a blog entry pending on this but wasn't expecting the code to be
>> released before the paper was... and all I had to go on til yesterday
>> was Nowlan's dissertation:
>>
>> http://blog.cerowrt.org/papers/bbr_thesis.pdf
>
>
> are we sure - that's a fairly different algorithm and different expansion of
> the acronym...

Yes it is very different, but it appears to have had the germ of at
least one of the good ideas in the soon to be published BBR.

"BBR detects bufferbloat by comparing its calculated correlation to a
configurable threshold (e.g., 90%) and declaring bufferbloat whenever
the value exceeds the threshold."

"Upon detecting bufferbloat, BBR immediately sets the TCP cwnd to the
window estimate, rather than gradually reducing the sending rate as
in, for example, Proportional Rate Reduction [19]. Immediately
changing the cwnd has two advantages. 27 First, it stops packet
transmissions immediately and prevents further exacerbating the delay.
When the transmissions resume with the reduced cwnd, they should
experience shorter queuing delay.

*Second, drastic changes in sending rate should result in sharp
differences in observed RTT, increasing the signal-to-noise ratio in
the observations and improving the correlation calculation*."

I dunno, I'm just reading tea leaves here!

can't wait for the paper!



>> video  streaming  experiments  ran  on  a  live,  production  CDN, where
>> clients included real mobile and desktop users
>
>> large, multinational production network
>
> hmm, I suppose...
>
>> ## Acknowledgments ##
>>
>> Van
>> ...
>> Eric
>> ...
>
>
> ok, there's clearly some overlap here :-D.
>
>
>
>> which seemed closer to good than anything I'd read before, but still
>> wrong in a few respects, which has taken a few years to sort out. I
>> think reading the upcoming acm queue paper is going to be fun!
>>
>> I think they have identified the right variables to probe - RTT and
>> bandwidth, in sequence - for modern congestion control to work much
>> better.
>>
>> Still BBR makes a few assumptions that do not hold (or so I think) -
>> with wifi in the control loop, and it needs wider testing in more
>> circumstances than just google facing out - like on itty bitty nas's
>> and media servers - and especially seeing what happens when it
>> interacts with fq_codel and cake would be good to see. I've watched
>> youtube be *excellent* for 2 years now, and only had the faintest
>> hints as to why.
>>
>> It was quite amusing that the original patchset didn't compile on 32
>> bit platforms.
>>
>> And make no mistake - it still makes plenty of sense to apply
>> fq_codel-like algorithms to routers, and the stuff we just did to wifi
>> for fq_codel and airtime fairness. Had I thought BBR solved everything
>> I'd have quit years ago.
>>
>>
>>
>> On Sat, Sep 17, 2016 at 11:34 AM, Maciej Soltysiak <maciej@soltysiak.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> Just saw this: https://patchwork.ozlabs.org/patch/671069/
>>>
>>> Interested to see how BBR would play out with things like fq_codel or
>>> cake.
>>>
>>> "loss-based congestion control is unfortunately out-dated in today's
>>> networks. On
>>> today's Internet, loss-based congestion control causes the infamous
>>> bufferbloat problem"
>>>
>>> So, instead of waiting for packet loss they probe and measure, e.g. when
>>> doing slow start (here called STARTUP) they don't speed up until packet
>>> loss, but slow down before reaching estimated bandwidth level.
>>>
>>> Cake and fq_codel work on all packets and aim to signal packet loss early
>>> to
>>> network stacks by dropping; BBR works on TCP and aims to prevent packet
>>> loss.



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Alan Jenkins]

[-- Attachment #1: Type: text/plain, Size: 2302 bytes --]

On 21/09/16 10:39, Dave Taht wrote:
> On Wed, Sep 21, 2016 at 2:06 AM, Alan Jenkins
> <alan.christopher.jenkins@gmail.com> wrote:
>> are we sure - that's a fairly different algorithm and different expansion of
>> the acronym...
> Yes it is very different,

My cryptic conclusion was a common group of names were involved in both 
projects (including "Neal" as well).  If there wasn't _some_ connection, 
I think they'd have suggested changing the project name to avoid confusion.

>   but it appears to have had the germ of at
> least one of the good ideas in the soon to be published BBR.
>
> "BBR detects bufferbloat by comparing its calculated correlation to a
> configurable threshold (e.g., 90%) and declaring bufferbloat whenever
> the value exceeds the threshold."
>
> "Upon detecting bufferbloat, BBR immediately sets the TCP cwnd to the
> window estimate, rather than gradually reducing the sending rate as
> in, for example, Proportional Rate Reduction [19]. Immediately
> changing the cwnd has two advantages. 27 First, it stops packet
> transmissions immediately and prevents further exacerbating the delay.
> When the transmissions resume with the reduced cwnd, they should
> experience shorter queuing delay.
>
> *Second, drastic changes in sending rate should result in sharp
> differences in observed RTT, increasing the signal-to-noise ratio in
> the observations and improving the correlation calculation*."
>
> I dunno, I'm just reading tea leaves here!

I guess you're referring to PROBE_RTT.  PROBE_RTT seems so obvious 
though, but I guess that's true of all the best ideas :).

I thought I saw a few other technical links.  The rate measurement is a 
similar idea.

> can't wait for the paper!

excite!

Aside: just reading the code I think PROBE_RTT will synchronize with 
other BBR instances sharing the same bottleneck.  Very cool if it works 
that way.

(And after PROBE_RTT it randomizes the next PROBE_BW phase, to avoid the 
harmful kind of synchronization).

>
>>> video  streaming  experiments  ran  on  a  live,  production  CDN, where
>>> clients included real mobile and desktop users
>>> large, multinational production network
>> hmm, I suppose...
>>
>>> ## Acknowledgments ##
>>>
>>> Van
>>> ...
>>> Eric
>>> ...
>>
>> ok, there's clearly some overlap here :-D.



[-- Attachment #2: Type: text/html, Size: 3753 bytes --]

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Mikael Abrahamsson]

On Wed, 21 Sep 2016, Dave Taht wrote:

> I dunno, I'm just reading tea leaves here!
>
> can't wait for the paper!

+1.

I would like to understand how BBR interacts with a window-fully-open 
classic TCP session and FIFO induced delay that is in steady-state before 
the BBR session starts.

So let's say I have 100ms of lightspeed-in-fiber RTT, and I am then 
running a file transfer with some other TCP algorithm which is sitting 
there, window fully open, creating an additional 100ms of 
stupid-router-FIFO-buffering delay.

So new BBR TCP session comes along, sees 200ms of RTT, and starts sending. 
I guess the classic TCP algorithm still keeps its window fully open, and 
doesn't care that RTT now increased to 210ms by the BBR flow packets.

Now what? BBR flow sees increased latency, and backs off, right? So how 
much of the bandwidth will each flow get? How do these interact?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Alan Jenkins]

On 21/09/2016, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Wed, 21 Sep 2016, Dave Taht wrote:
>
>> I dunno, I'm just reading tea leaves here!
>>
>> can't wait for the paper!
>
> +1.
>
> I would like to understand how BBR interacts with a window-fully-open
> classic TCP session and FIFO induced delay that is in steady-state before
> the BBR session starts.
>
> So let's say I have 100ms of lightspeed-in-fiber RTT, and I am then
> running a file transfer with some other TCP algorithm which is sitting
> there, window fully open, creating an additional 100ms of
> stupid-router-FIFO-buffering delay.
>
> So new BBR TCP session comes along, sees 200ms of RTT, and starts sending.
> I guess the classic TCP algorithm still keeps its window fully open, and
> doesn't care that RTT now increased to 210ms by the BBR flow packets.
>
> Now what? BBR flow sees increased latency, and backs off, right? So how
> much of the bandwidth will each flow get? How do these interact?

I don't know what you're reading, but the BBR code that's just been
submitted does not back off when latency increases.  Period.

If less well-behaved flows drove up the minimum latency measured over
the 10 second interval, BBR would treat this as a longer pipe, and
will seek to fill it.  It would *increase* the number of packets
in-flight.

That assumes the measured maximum bandwidth (over an interval of
10*rtt) remains constant.  (Say there were 100 BBR flows, then you
added one CUBIC flow to bloat the buffer).  I don't have a good
intuition for how the bandwidth estimation behaves in general.

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Mikael Abrahamsson]

On Wed, 21 Sep 2016, Alan Jenkins wrote:

> That assumes the measured maximum bandwidth (over an interval of 10*rtt) 
> remains constant.  (Say there were 100 BBR flows, then you added one 
> CUBIC flow to bloat the buffer).  I don't have a good intuition for how 
> the bandwidth estimation behaves in general.

My example was single CUBIC flow already existing, steady state, full 
speed, filling pipe and buffer with 100ms of packets, and it's not filling 
buffer to 110ms of packets because its window is full. There is no packet 
loss currently.

Now you add single BBR flow. What happens, what's the ratio between BBR 
and CUBIC after 20 seconds of letting things settle down to new 
equilibrium?

Consider the FIFO buffer practically infinite, or at least able to buffer 
5 seconds of packets without drop. It's an interface with a single, 
stupid, huge FIFO buffer.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Dave Taht]

On Wed, Sep 21, 2016 at 3:15 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Wed, 21 Sep 2016, Dave Taht wrote:
>
>> I dunno, I'm just reading tea leaves here!
>>
>> can't wait for the paper!
>
>
> +1.
>
> I would like to understand how BBR interacts with a window-fully-open
> classic TCP session and FIFO induced delay that is in steady-state before
> the BBR session starts.

I did a fairly comprehensive string of tests today, comparing it at
20Mbits, 48ms RTT, to cubic and competing with cubic, against a byte
fifo of 256k, pie, cake, cake flowblind, and fq_codel.

The flent datasets are now at:
http://blog.cerowrt.org/flent/bbr-comprehensive.tgz

You will need the latest flent from git to plot the new
tcp_4up_squarewave test. That test starts 1 BBR flow, then 3 seconds
later, cubic, 3 seconds after that, bbr again, 3 seconds after cubic
again.

It's 4am here, and we've also been busy with something else due today,
but some observations:

* respecting ecn appears unimplemented, so if you compare cubic with
ecn against bbr not respecting it, bad things happen. I bug reported
this to the bbr mailing list. To some extent cake's and pie's overload
on ecn drop mechanism helps. fq_codel/cake with fq is fine, so long as
there are no collisions.

* It seriously outcompetes cubic, particularly on the single queue
aqms. fq_codel is fine. I need to take apart the captures to see how
well it is behaving in this case. My general hope was that with fq in
place, anything that was delay based worked better as it was only
competing against itself.

* It does the right things against bfifo when multiple streams are
present, but in the presence of reverse traffic, cubic starves. (the
dataset for this is kind of wrong in that the legend of the
"smackdown" chart claims BBR is on on the download, but it was
unimplemented in that direction, I only had it running on the upload.)

Did I mention fq* was fine? :)

I'm going to bed.

> So let's say I have 100ms of lightspeed-in-fiber RTT, and I am then running
> a file transfer with some other TCP algorithm which is sitting there, window
> fully open, creating an additional 100ms of stupid-router-FIFO-buffering
> delay.
>
> So new BBR TCP session comes along, sees 200ms of RTT, and starts sending. I
> guess the classic TCP algorithm still keeps its window fully open, and
> doesn't care that RTT now increased to 210ms by the BBR flow packets.

I see evidence of the classic latecomer advantage but it subsides in 10-20 sec:

http://blog.cerowrt.org/flent/bbr-comprehensive/latecomer_advantage.svg

>
> Now what? BBR flow sees increased latency, and backs off, right? So how much
> of the bandwidth will each flow get? How do these interact?

Plot it in flent.

>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Mikael Abrahamsson]

On Wed, 21 Sep 2016, Dave Taht wrote:

> I did a fairly comprehensive string of tests today, comparing it at 
> 20Mbits, 48ms RTT, to cubic and competing with cubic, against a byte 
> fifo of 256k, pie, cake, cake flowblind, and fq_codel.

20 megabit/s is 2.5 megabyte/s, so that 256k FIFO is only 100ms worth of 
buffering. I guess you see packet drop in steady state here, ie buffer is 
full?

I'd be interested in seeing same experiment start with 10MB FIFO, and 
having CUBIC flow start first and give it proper head start.

My intuition and understanding of what's going to happen might very well 
be completely off, but I think it'd be interesting to know.

I'll take a look at your flent data, thanks for posting them!

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Mikael Abrahamsson]

On Wed, 21 Sep 2016, Dave Taht wrote:

> * It seriously outcompetes cubic, particularly on the single queue aqms. 
> fq_codel is fine. I need to take apart the captures to see how well it 
> is behaving in this case. My general hope was that with fq in place, 
> anything that was delay based worked better as it was only competing 
> against itself.

I'm looking at 4up-sqwave-fq_bfifo-256k. Is this really fq_bfifo, or just 
bfifo? Looks like there is no fq.

If someone doesn't have the correct Flent available, I posted two 
screenshots here: http://imgur.com/a/cFtMd

What I think I see:

The flows are started in order: "BBR1, CUBIC2, BBR4, CUBIC3" (a bit 
confusing, but according to your description).

So it looks like BBR1 fills the pipe within half a second or so, nice 
steady state. Then CUBIC2 starts, and slowly over a few seconds, starts to 
starve BBR1 of BW, it looks like steady state here would be that CUBIC2 
would end up with around 65-70% of the BW, and BBR1 getting 30-35%. Then 
BBR4 comes along (10 seconds in), and just KILLS them both, smacks them 
over the head with a hammer, taking 90% of the BW, wildly oscillating in 
BW way above 20 megabit/s down to 10. The ping here goes up to around 
150-160ms. CUBIC3 starts at 15 seconds and get basically no bw at all.

Then at around 22 seconds in, I guess pretty close to 12-13 seconds after 
BBR4 was started, BBR4 starts to calm down, slowly letting the other 
streams come back to life. At around 30 seconds, they all seem to get at 
least a bit of the bw each and nobody is completely starved, but BBR1 
seems to not get much BW at all (very dotted line).

When at the end there is only CUBIC3 and BBR4 left, it looks like BBR4 has 
a 2/3 to 1/3 advantage.

Looking at cake_flowblind_noecn, BBR1 and BBR4 just kills both CUBIC 
flows. Same with PIE.

So it seems my intuition was wrong, at least for these scenarios. It 
wasn't CUBIC that would kill BBR, it's the other way around. Great to have 
testing tools! Thanks Flent!

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] [Bloat] BBR congestion control algorithm for TCP in net-next

[Alan Jenkins]

[-- Attachment #1: Type: text/plain, Size: 2811 bytes --]

On 21/09/16 13:40, Mikael Abrahamsson wrote:
> On Wed, 21 Sep 2016, Dave Taht wrote:
>
>> * It seriously outcompetes cubic, particularly on the single queue 
>> aqms. fq_codel is fine. I need to take apart the captures to see how 
>> well it is behaving in this case. My general hope was that with fq in 
>> place, anything that was delay based worked better as it was only 
>> competing against itself.
>
> I'm looking at 4up-sqwave-fq_bfifo-256k. Is this really fq_bfifo, or 
> just bfifo? Looks like there is no fq.

the queue before the (simulated) _bottleneck_ is simple bfifo.

sch_fq is used on the _endpoint_, because it's a requirement of tcp-bbr

It's disambiguated by knowing you don't add child qdiscs to sch_fq, and 
no-one has written a linux qdisc called fq_bfifo.

>
> If someone doesn't have the correct Flent available, I posted two 
> screenshots here: http://imgur.com/a/cFtMd
>
> What I think I see:
>
> The flows are started in order: "BBR1, CUBIC2, BBR4, CUBIC3" (a bit 
> confusing, but according to your description).
>
> So it looks like BBR1 fills the pipe within half a second or so, nice 
> steady state. Then CUBIC2 starts, and slowly over a few seconds, 
> starts to starve BBR1 of BW, it looks like steady state here would be 
> that CUBIC2 would end up with around 65-70% of the BW, and BBR1 
> getting 30-35%. Then BBR4 comes along (10 seconds in), and just KILLS 
> them both, smacks them over the head with a hammer, taking 90% of the 
> BW, wildly oscillating in BW way above 20 megabit/s down to 10. The 
> ping here goes up to around 150-160ms. CUBIC3 starts at 15 seconds and 
> get basically no bw at all.
>
> Then at around 22 seconds in, I guess pretty close to 12-13 seconds 
> after BBR4 was started, BBR4 starts to calm down, slowly letting the 
> other streams come back to life. At around 30 seconds, they all seem 
> to get at least a bit of the bw each and nobody is completely starved, 
> but BBR1 seems to not get much BW at all (very dotted line).
>
> When at the end there is only CUBIC3 and BBR4 left, it looks like BBR4 
> has a 2/3 to 1/3 advantage.
>
> Looking at cake_flowblind_noecn, BBR1 and BBR4 just kills both CUBIC 
> flows. Same with PIE.
>
> So it seems my intuition was wrong, at least for these scenarios. It 
> wasn't CUBIC that would kill BBR, it's the other way around. Great to 
> have testing tools! Thanks Flent!
>

wow.  I was wondering how BBR ever relinquished bandwidth.  "With some 
difficulty".

Though not in the second screenshot where there's a strict AQM (cake 
flowblind).  Then it shares with other BBRs while never letting CUBIC 
get established.  (I think if you look at drops, they'd be higher than 
what 4xCUBIC  converges to on that link.  See 
https://groups.google.com/forum/#!msg/bbr-dev/EXgiWxHvaEU/wl3Q39svAAAJ ).

[-- Attachment #2: Type: text/html, Size: 3923 bytes --]

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Jonathan Morton]

> On 17 Sep, 2016, at 21:34, Maciej Soltysiak <maciej@soltysiak.com> wrote:
> 
> Cake and fq_codel work on all packets and aim to signal packet loss early to network stacks by dropping; BBR works on TCP and aims to prevent packet loss. 

By dropping, *or* by ECN marking.  The latter avoids packet loss.

 - Jonathan Morton

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1162 bytes --]

Dumb question on this:  The tcp_bbr_info struct for a socket can be
inspected at runtime through the ss utility or through a get socket opts
call, right?

-Aaron

On Sat, Sep 17, 2016 at 11:34 AM, Maciej Soltysiak <maciej@soltysiak.com>
wrote:

> Hi,
>
> Just saw this: https://patchwork.ozlabs.org/patch/671069/
>
> Interested to see how BBR would play out with things like fq_codel or cake.
>
> "loss-based congestion control is unfortunately out-dated in today's
> networks. On
> today's Internet, loss-based congestion control causes the infamous
> bufferbloat problem"
>
> So, instead of waiting for packet loss they probe and measure, e.g. when
> doing slow start (here called STARTUP) they don't speed up until packet
> loss, but slow down before reaching estimated bandwidth level.
>
> Cake and fq_codel work on all packets and aim to signal packet loss early
> to network stacks by dropping; BBR works on TCP and aims to prevent packet
> loss.
>
>
> Best regards,
> Maciej
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>

[-- Attachment #2: Type: text/html, Size: 1950 bytes --]

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Neal Cardwell]

On Mon, Sep 26, 2016 at 2:47 PM, Aaron Wood <woody77@gmail.com> wrote:
> Dumb question on this:  The tcp_bbr_info struct for a socket can be
> inspected at runtime through the ss utility or through a get socket opts
> call, right?

Yes, you can use either approach:

(1) from code you can use TCP_CC_INFO socket option; there is sample
code in the original kernel patch for TCP_CC_INFO:
   https://patchwork.ozlabs.org/patch/465806/

(2) from ss: if you download and build the net-next branch of the
iproute2 package:
    http://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/log/?h=net-next
  then you will get support to print out the main parameters for a BBR
connection, eg:

  The patch with BBR support for ss is here:
    http://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/commit/?h=net-next&id=2f0f9aef94129643133363b4503468cdccc481cc

  As the commit notes, the BBR output looks like:
    bbr:(bw:1.2Mbps,mrtt:18.965,pacing_gain:2.88672,cwnd_gain:2.88672)

Hope that helps,
neal

>
> -Aaron
>
> On Sat, Sep 17, 2016 at 11:34 AM, Maciej Soltysiak <maciej@soltysiak.com>
> wrote:
>>
>> Hi,
>>
>> Just saw this: https://patchwork.ozlabs.org/patch/671069/
>>
>> Interested to see how BBR would play out with things like fq_codel or
>> cake.
>>
>> "loss-based congestion control is unfortunately out-dated in today's
>> networks. On
>> today's Internet, loss-based congestion control causes the infamous
>> bufferbloat problem"
>>
>> So, instead of waiting for packet loss they probe and measure, e.g. when
>> doing slow start (here called STARTUP) they don't speed up until packet
>> loss, but slow down before reaching estimated bandwidth level.
>>
>> Cake and fq_codel work on all packets and aim to signal packet loss early
>> to network stacks by dropping; BBR works on TCP and aims to prevent packet
>> loss.
>>
>>
>> Best regards,
>> Maciej
>>
>>
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 2519 bytes --]

Thanks! And sorry that I missed the sample code in the patch.
On Mon, Sep 26, 2016 at 12:30 Neal Cardwell <ncardwell@google.com> wrote:

> On Mon, Sep 26, 2016 at 2:47 PM, Aaron Wood <woody77@gmail.com> wrote:
> > Dumb question on this:  The tcp_bbr_info struct for a socket can be
> > inspected at runtime through the ss utility or through a get socket opts
> > call, right?
>
> Yes, you can use either approach:
>
> (1) from code you can use TCP_CC_INFO socket option; there is sample
> code in the original kernel patch for TCP_CC_INFO:
>    https://patchwork.ozlabs.org/patch/465806/
>
> (2) from ss: if you download and build the net-next branch of the
> iproute2 package:
>
> http://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/log/?h=net-next
>   then you will get support to print out the main parameters for a BBR
> connection, eg:
>
>   The patch with BBR support for ss is here:
>
> http://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/commit/?h=net-next&id=2f0f9aef94129643133363b4503468cdccc481cc
>
>   As the commit notes, the BBR output looks like:
>     bbr:(bw:1.2Mbps,mrtt:18.965,pacing_gain:2.88672,cwnd_gain:2.88672)
>
> Hope that helps,
> neal
>
> >
> > -Aaron
> >
> > On Sat, Sep 17, 2016 at 11:34 AM, Maciej Soltysiak <maciej@soltysiak.com
> >
> > wrote:
> >>
> >> Hi,
> >>
> >> Just saw this: https://patchwork.ozlabs.org/patch/671069/
> >>
> >> Interested to see how BBR would play out with things like fq_codel or
> >> cake.
> >>
> >> "loss-based congestion control is unfortunately out-dated in today's
> >> networks. On
> >> today's Internet, loss-based congestion control causes the infamous
> >> bufferbloat problem"
> >>
> >> So, instead of waiting for packet loss they probe and measure, e.g. when
> >> doing slow start (here called STARTUP) they don't speed up until packet
> >> loss, but slow down before reaching estimated bandwidth level.
> >>
> >> Cake and fq_codel work on all packets and aim to signal packet loss
> early
> >> to network stacks by dropping; BBR works on TCP and aims to prevent
> packet
> >> loss.
> >>
> >>
> >> Best regards,
> >> Maciej
> >>
> >>
> >> _______________________________________________
> >> Cerowrt-devel mailing list
> >> Cerowrt-devel@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >>
> >
> >
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-devel@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >
>

[-- Attachment #2: Type: text/html, Size: 4183 bytes --]

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Dave Taht]

I just put a netperf server up in linode's freemont, ca, cloud (kvm
paravirtualized hardware), with sch_fq enabled, ecn disabled, and bbr
as the default cc. (reno and cubic are also allowed)

I am curious if y'all hit it with your rrul, tcp_ndown,
rtt_fair_var_down (vs flent-freemont in the same dc) etc, that you get
sane results - with and without sqm-scripts moderating your
connection.

I've always kind of worried that sch_fq would misbehave in a
virtualized environment, and they seem to do some odd things with
policing/shaping in their world that I've not sorted out (speeds >
200Mbit)

DNS for flent-bbr-west.bufferbloat.net is propagating (both IPv4 and IPv6)

Til then:
the ipv4 for the thing is: 173 dot 230 dot 156 dot 252

I've put a broad sweep of results up in my github, haven't looked at
them to a huge extent yet.


On Mon, Sep 26, 2016 at 12:45 PM, Aaron Wood <woody77@gmail.com> wrote:
> Thanks! And sorry that I missed the sample code in the patch.
>
> On Mon, Sep 26, 2016 at 12:30 Neal Cardwell <ncardwell@google.com> wrote:
>>
>> On Mon, Sep 26, 2016 at 2:47 PM, Aaron Wood <woody77@gmail.com> wrote:
>> > Dumb question on this:  The tcp_bbr_info struct for a socket can be
>> > inspected at runtime through the ss utility or through a get socket opts
>> > call, right?
>>
>> Yes, you can use either approach:
>>
>> (1) from code you can use TCP_CC_INFO socket option; there is sample
>> code in the original kernel patch for TCP_CC_INFO:
>>    https://patchwork.ozlabs.org/patch/465806/
>>
>> (2) from ss: if you download and build the net-next branch of the
>> iproute2 package:
>>
>> http://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/log/?h=net-next
>>   then you will get support to print out the main parameters for a BBR
>> connection, eg:
>>
>>   The patch with BBR support for ss is here:
>>
>> http://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/commit/?h=net-next&id=2f0f9aef94129643133363b4503468cdccc481cc
>>
>>   As the commit notes, the BBR output looks like:
>>     bbr:(bw:1.2Mbps,mrtt:18.965,pacing_gain:2.88672,cwnd_gain:2.88672)
>>
>> Hope that helps,
>> neal
>>
>> >
>> > -Aaron
>> >
>> > On Sat, Sep 17, 2016 at 11:34 AM, Maciej Soltysiak
>> > <maciej@soltysiak.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> Just saw this: https://patchwork.ozlabs.org/patch/671069/
>> >>
>> >> Interested to see how BBR would play out with things like fq_codel or
>> >> cake.
>> >>
>> >> "loss-based congestion control is unfortunately out-dated in today's
>> >> networks. On
>> >> today's Internet, loss-based congestion control causes the infamous
>> >> bufferbloat problem"
>> >>
>> >> So, instead of waiting for packet loss they probe and measure, e.g.
>> >> when
>> >> doing slow start (here called STARTUP) they don't speed up until packet
>> >> loss, but slow down before reaching estimated bandwidth level.
>> >>
>> >> Cake and fq_codel work on all packets and aim to signal packet loss
>> >> early
>> >> to network stacks by dropping; BBR works on TCP and aims to prevent
>> >> packet
>> >> loss.
>> >>
>> >>
>> >> Best regards,
>> >> Maciej
>> >>
>> >>
>> >> _______________________________________________
>> >> Cerowrt-devel mailing list
>> >> Cerowrt-devel@lists.bufferbloat.net
>> >> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> >>
>> >
>> >
>> > _______________________________________________
>> > Cerowrt-devel mailing list
>> > Cerowrt-devel@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> >
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 4365 bytes --]

I'll hit it from Comcast's ~150Mbps service on the peninsula when I get
home today (with and without sqm)

-Aaron

On Mon, Sep 26, 2016 at 2:38 PM, Dave Taht <dave.taht@gmail.com> wrote:

> I just put a netperf server up in linode's freemont, ca, cloud (kvm
> paravirtualized hardware), with sch_fq enabled, ecn disabled, and bbr
> as the default cc. (reno and cubic are also allowed)
>
> I am curious if y'all hit it with your rrul, tcp_ndown,
> rtt_fair_var_down (vs flent-freemont in the same dc) etc, that you get
> sane results - with and without sqm-scripts moderating your
> connection.
>
> I've always kind of worried that sch_fq would misbehave in a
> virtualized environment, and they seem to do some odd things with
> policing/shaping in their world that I've not sorted out (speeds >
> 200Mbit)
>
> DNS for flent-bbr-west.bufferbloat.net is propagating (both IPv4 and IPv6)
>
> Til then:
> the ipv4 for the thing is: 173 dot 230 dot 156 dot 252
>
> I've put a broad sweep of results up in my github, haven't looked at
> them to a huge extent yet.
>
>
> On Mon, Sep 26, 2016 at 12:45 PM, Aaron Wood <woody77@gmail.com> wrote:
> > Thanks! And sorry that I missed the sample code in the patch.
> >
> > On Mon, Sep 26, 2016 at 12:30 Neal Cardwell <ncardwell@google.com>
> wrote:
> >>
> >> On Mon, Sep 26, 2016 at 2:47 PM, Aaron Wood <woody77@gmail.com> wrote:
> >> > Dumb question on this:  The tcp_bbr_info struct for a socket can be
> >> > inspected at runtime through the ss utility or through a get socket
> opts
> >> > call, right?
> >>
> >> Yes, you can use either approach:
> >>
> >> (1) from code you can use TCP_CC_INFO socket option; there is sample
> >> code in the original kernel patch for TCP_CC_INFO:
> >>    https://patchwork.ozlabs.org/patch/465806/
> >>
> >> (2) from ss: if you download and build the net-next branch of the
> >> iproute2 package:
> >>
> >> http://git.kernel.org/cgit/linux/kernel/git/shemminger/
> iproute2.git/log/?h=net-next
> >>   then you will get support to print out the main parameters for a BBR
> >> connection, eg:
> >>
> >>   The patch with BBR support for ss is here:
> >>
> >> http://git.kernel.org/cgit/linux/kernel/git/shemminger/
> iproute2.git/commit/?h=net-next&id=2f0f9aef94129643133363b4503468
> cdccc481cc
> >>
> >>   As the commit notes, the BBR output looks like:
> >>     bbr:(bw:1.2Mbps,mrtt:18.965,pacing_gain:2.88672,cwnd_gain:2.88672)
> >>
> >> Hope that helps,
> >> neal
> >>
> >> >
> >> > -Aaron
> >> >
> >> > On Sat, Sep 17, 2016 at 11:34 AM, Maciej Soltysiak
> >> > <maciej@soltysiak.com>
> >> > wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> Just saw this: https://patchwork.ozlabs.org/patch/671069/
> >> >>
> >> >> Interested to see how BBR would play out with things like fq_codel or
> >> >> cake.
> >> >>
> >> >> "loss-based congestion control is unfortunately out-dated in today's
> >> >> networks. On
> >> >> today's Internet, loss-based congestion control causes the infamous
> >> >> bufferbloat problem"
> >> >>
> >> >> So, instead of waiting for packet loss they probe and measure, e.g.
> >> >> when
> >> >> doing slow start (here called STARTUP) they don't speed up until
> packet
> >> >> loss, but slow down before reaching estimated bandwidth level.
> >> >>
> >> >> Cake and fq_codel work on all packets and aim to signal packet loss
> >> >> early
> >> >> to network stacks by dropping; BBR works on TCP and aims to prevent
> >> >> packet
> >> >> loss.
> >> >>
> >> >>
> >> >> Best regards,
> >> >> Maciej
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> Cerowrt-devel mailing list
> >> >> Cerowrt-devel@lists.bufferbloat.net
> >> >> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > Cerowrt-devel mailing list
> >> > Cerowrt-devel@lists.bufferbloat.net
> >> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >> >
> >
> >
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-devel@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >
>
>
>
> --
> Dave Täht
> Let's go make home routers and wifi faster! With better software!
> http://blog.cerowrt.org
>

[-- Attachment #2: Type: text/html, Size: 7213 bytes --]