[Bloat] "BBR" TCP patches submitted to linux kernel

[Bloat] "BBR" TCP patches submitted to linux kernel

[Dave Taht]

I'm really looking forward to trying them out and reading the upcoming paper.

https://patchwork.ozlabs.org/patch/671069/

-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Steinar H. Gunderson]

On Fri, Sep 16, 2016 at 02:04:37PM -0700, Dave Taht wrote:
> I'm really looking forward to trying them out and reading the upcoming paper.
> 
> https://patchwork.ozlabs.org/patch/671069/

BBR opensourced! I'm enthustiastic.

Eric: Congrats!

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Eric Dumazet]

Yeah, team is excited !

But really I am a small contributor in this work (mostly reviewing patches)

Neal, Van, Yuchung and Soheil really worked hard on it.


On Fri, Sep 16, 2016 at 2:11 PM, Steinar H. Gunderson
<sgunderson@bigfoot.com> wrote:
> On Fri, Sep 16, 2016 at 02:04:37PM -0700, Dave Taht wrote:
>> I'm really looking forward to trying them out and reading the upcoming paper.
>>
>> https://patchwork.ozlabs.org/patch/671069/
>
> BBR opensourced! I'm enthustiastic.
>
> Eric: Congrats!
>
> /* Steinar */
> --
> Homepage: https://www.sesse.net/

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Dave Taht]

Thank you for this, in particular...

http://www.spinics.net/lists/netdev/msg395552.html

still reading over the code...

TCP Pr0n, this is!

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Mário Sérgio Fujikawa Ferreira]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

Is there a mailing list I can lurk in to follow on the development?

I'm most interested on a delta to apply to Android 6.x Franco Kernel 
(https://github.com/franciscofranco/angler) and Debian Jessie.

FreeBSD support would be excellent but I don't even have reviewed the 
license to be sure if we need a clean room development (hence I haven't 
read the code) or we can try to relocate it directly. On a side FreeBSD 
note, I'm hopeful for

 1. Packet pacing (https://wiki.freebsd.org/DevSummit/201606/Transport)
 2. ipfw/dummynet AQM with FQ-Codel
    (https://lists.freebsd.org/pipermail/freebsd-ipfw/2016-February/006026.html)

Most excellent news.

On 16/09/2016 19:29, Dave Taht wrote:
> Thank you for this, in particular...
>
> http://www.spinics.net/lists/netdev/msg395552.html
>
> still reading over the code...
>
> TCP Pr0n, this is!
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat


[-- Attachment #2: Type: text/html, Size: 2243 bytes --]

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Dave Täht]

On 9/29/16 4:24 AM, Mário Sérgio Fujikawa Ferreira wrote:
> Is there a mailing list I can lurk in to follow on the development?
> 
> I'm most interested on a delta to apply to Android 6.x Franco Kernel
> (https://github.com/franciscofranco/angler)

Android is still shipping linux 3.10? How... quaint.

> and Debian Jessie.

I have built .debs for ubuntu and am going to go build debs for jessie
on arm soon.

> FreeBSD support would be excellent but I don't even have reviewed the
> license to be sure if we need a clean room development (hence I haven't
> read the code)

The bbr code is licensed dual bsd/gpl. It's dependent on a bunch of apis
though...

 or we can try to relocate it directly. On a side FreeBSD
> note, I'm hopeful for
> 
>  1. Packet pacing (https://wiki.freebsd.org/DevSummit/201606/Transport)

was good to read up on the progress there, although I think 1ms timing
in bsd is needing to be upgraded.

>  2. ipfw/dummynet AQM with FQ-Codel
>     (https://lists.freebsd.org/pipermail/freebsd-ipfw/2016-February/006026.html)

I made some comments on this that I'd like people to look at what is
really going on in bsd + fq_codel at > 100mbit.

https://github.com/opnsense/core/issues/505

> Most excellent news.
> 
> On 16/09/2016 19:29, Dave Taht wrote:
>> Thank you for this, in particular...
>>
>> http://www.spinics.net/lists/netdev/msg395552.html
>>
>> still reading over the code...
>>
>> TCP Pr0n, this is!
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> 
> 
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
> 

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]

On Thu, Sep 29, 2016 at 12:43 PM, Dave Täht <dave@taht.net> wrote:

>
>
> On 9/29/16 4:24 AM, Mário Sérgio Fujikawa Ferreira wrote:
> > Is there a mailing list I can lurk in to follow on the development?
> >
> > I'm most interested on a delta to apply to Android 6.x Franco Kernel
> > (https://github.com/franciscofranco/angler)
>
> Android is still shipping linux 3.10? How... quaint.
>

While you think 3.10 is old, in my experience it's still seen as cutting
edge by many.  RHEL is still only at 3.10.  And routers are using much
older 3.x kernels.  There's a huge lag between what the "enterprise" crowd
is running in production, and what you guys are developing on.  Because
"stability".

It's been one of my major frustrations (especially on the embedded side
where 3.x kernels are still considered 'new' and 2.6.x is 'trusted').

-Aaron

[-- Attachment #2: Type: text/html, Size: 1411 bytes --]

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Mikael Abrahamsson]

On Thu, 29 Sep 2016, Aaron Wood wrote:

> While you think 3.10 is old, in my experience it's still seen as cutting 
> edge by many.  RHEL is still only at 3.10.  And routers are using much 
> older 3.x kernels.  There's a huge lag between what the "enterprise" 
> crowd is running in production, and what you guys are developing on. 
> Because "stability".
>
> It's been one of my major frustrations (especially on the embedded side 
> where 3.x kernels are still considered 'new' and 2.6.x is 'trusted').

State of affairs are actually improving. What I'm seeing from several SoC 
vendors is that they're moving from a "new kernel every 3 years, and we'll 
choose a 2 year old kernel when doing the work so it'll be 5 years old by 
the time a new one comes around, with the result that a lot of devices 
are on 2.6.26, 3.2 and 3.4), to a model where they actually do a new 
kernel every 6 months, and they'll choose a kernel that's around 12-18 
months old at that time.

This is of course not great, but it's an improvement. I'm pushing for SoC 
vendors to actually upstream their patches as much as possible and support 
creation of kernel version independent HAL/API in the kernel that they can 
write their drivers for.

So if you know any netdev people, please tell them to be supportive when 
SoC vendors come and want changes done to the kernel to support for 
instance hw packet accelerators. We want this done right of course (so we 
can live with it for the next 5-10 years at least), but this is very 
important that it gets done.

This of course has interesting effects for AQM, since with packet 
accelerators you're taking the kernel pretty much out of the data path as 
soon as the hardware is programmed... but that's a different but related 
struggle to make sure that these aren't as bloated as yesteryears 
implementations.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1466 bytes --]

On Fri, Sep 30, 2016 at 1:12 AM, Mikael Abrahamsson <swmike@swm.pp.se>
wrote:

> On Thu, 29 Sep 2016, Aaron Wood wrote:
>
> While you think 3.10 is old, in my experience it's still seen as cutting
>> edge by many.  RHEL is still only at 3.10.  And routers are using much
>> older 3.x kernels.  There's a huge lag between what the "enterprise" crowd
>> is running in production, and what you guys are developing on. Because
>> "stability".
>>
>> It's been one of my major frustrations (especially on the embedded side
>> where 3.x kernels are still considered 'new' and 2.6.x is 'trusted').
>>
>
> State of affairs are actually improving. What I'm seeing from several SoC
> vendors is that they're moving from a "new kernel every 3 years, and we'll
> choose a 2 year old kernel when doing the work so it'll be 5 years old by
> the time a new one comes around, with the result that a lot of devices are
> on 2.6.26, 3.2 and 3.4), to a model where they actually do a new kernel
> every 6 months, and they'll choose a kernel that's around 12-18 months old
> at that time.
>
> This is of course not great, but it's an improvement. I'm pushing for SoC
> vendors to actually upstream their patches as much as possible and support
> creation of kernel version independent HAL/API in the kernel that they can
> write their drivers for.
>

It's a great improvement over where things were.  I hope it continues.  I
know I'll be supporting it professionally when I can.

-Aaron

[-- Attachment #2: Type: text/html, Size: 2029 bytes --]

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Steinar H. Gunderson]

On Thu, Sep 29, 2016 at 12:43:33PM -0700, Dave Täht wrote:
> Android is still shipping linux 3.10? How... quaint.

It depends on the device. Generally, you'll find that device makers pick a
relatively recent LTS kernel when launching a device, then stick with that
kernel series forever, even through new major versions of Android.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Mario Ferreira]

[-- Attachment #1: Type: text/plain, Size: 1414 bytes --]

On Thu, Sep 29, 2016, 16:43 Dave Täht <dave@taht.net> wrote:

>
>
> On 9/29/16 4:24 AM, Mário Sérgio Fujikawa Ferreira wrote:
> > Is there a mailing list I can lurk in to follow on the development?
> >
> > I'm most interested on a delta to apply to Android 6.x Franco Kernel
> > (https://github.com/franciscofranco/angler)
>
> Android is still shipping linux 3.10? How... quaint.
>

That's for Huawei Nexus 6P. Google's flagship till Pixel XL arrives. I'll
try getting BBR merged to Franco's kernel (officially or maintained as a
fork)  and add tc-adv with fq_codel/cake.

Since this is mobile, I'm pretty sure it will present a whole new host of
"data points".

I'm swamped right now but I'll have time by november for both FreeBSD and
Android 6 (7 doesn't have source code released just yet) . I'm no expert
but I'll help as I can.

>  2. ipfw/dummynet AQM with FQ-Codel
> >     (
> https://lists.freebsd.org/pipermail/freebsd-ipfw/2016-February/006026.html
> )
>
> I made some comments on this that I'd like people to look at what is
> really going on in bsd + fq_codel at > 100mbit.
>
> https://github.com/opnsense/core/issues/505
>
>
I haven't been involved with the FreeBSD project on recent years but I'll
try to advocate internally.
-- 

Mario S F Ferreira - Brazil - "I guess this is a signature."
feature, n: a documented bug | bug, n: an undocumented feature

[-- Attachment #2: Type: text/html, Size: 2407 bytes --]

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Dave Täht]

On 9/29/16 6:54 PM, Mario Ferreira wrote:
> On Thu, Sep 29, 2016, 16:43 Dave Täht <dave@taht.net
> <mailto:dave@taht.net>> wrote:
> 
> 
> 
>     On 9/29/16 4:24 AM, Mário Sérgio Fujikawa Ferreira wrote:
>     > Is there a mailing list I can lurk in to follow on the development?
>     >
>     > I'm most interested on a delta to apply to Android 6.x Franco Kernel
>     > (https://github.com/franciscofranco/angler)
> 
>     Android is still shipping linux 3.10? How... quaint.
> 
> 
> That's for Huawei Nexus 6P. Google's flagship till Pixel XL arrives.
> I'll try getting BBR merged to Franco's kernel (officially or maintained
> as a fork)  and add tc-adv with fq_codel/cake.
> 
> Since this is mobile, I'm pretty sure it will present a whole new host
> of "data points". 

yes! (there have been a few studies of this, btw)

The part that we don't really know on the android handsets is how
much buffering there really is between qdisc, driver, and firmware,
which no doubt varies between models - and within a model on the
different wifi and 4G stacks. Odds are - just as we just ripped out on
the ath9k/ath10k - so much intermediate buffering exists as to make
applying the latency managing qdisc on top of marginal effectiveness.

In the case of BBR, well, there is some hope that would regulate
TCP on the uplink, but it will have no effect on the down (neither will
the qdiscs) - and it requires sch_fq to work properly - which
means that you'd have a choice between bbr + sch_fq, or
sch_cake/sch_fq_codel.

In all cases, acquiring more data and backports would be nice.
> 
> I'm swamped right now but I'll have time by november for both FreeBSD
> and Android 6 (7 doesn't have source code released just yet) . I'm no
> expert but I'll help as I can.
> 
>     >  2. ipfw/dummynet AQM with FQ-Codel
>     >   
>      (https://lists.freebsd.org/pipermail/freebsd-ipfw/2016-February/006026.html)
> 
>     I made some comments on this that I'd like people to look at what is
>     really going on in bsd + fq_codel at > 100mbit.
> 
>     https://github.com/opnsense/core/issues/505
> 
> 
> I haven't been involved with the FreeBSD project on recent years but
> I'll try to advocate internally. 
> -- 
> 
> Mario S F Ferreira - Brazil - "I guess this is a signature."
> feature, n: a documented bug | bug, n: an undocumented feature
> 

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1640 bytes --]

On Thu, Sep 29, 2016 at 8:50 PM, Dave Täht <dave@taht.net> wrote:

>
> >     Android is still shipping linux 3.10? How... quaint.
> >
> > Since this is mobile, I'm pretty sure it will present a whole new host
> > of "data points".
>
> yes! (there have been a few studies of this, btw)
>
> The part that we don't really know on the android handsets is how
> much buffering there really is between qdisc, driver, and firmware,
> which no doubt varies between models - and within a model on the
> different wifi and 4G stacks. Odds are - just as we just ripped out on
> the ath9k/ath10k - so much intermediate buffering exists as to make
> applying the latency managing qdisc on top of marginal effectiveness.
>
> In the case of BBR, well, there is some hope that would regulate
> TCP on the uplink, but it will have no effect on the down (neither will
> the qdiscs) - and it requires sch_fq to work properly - which
> means that you'd have a choice between bbr + sch_fq, or
> sch_cake/sch_fq_codel
>

yet...  wouldn't BBR mitigate the local radio fw buffering?  The early rtt
probing should (hopefully) see an un-bloated link, and then tune off of
that?  So any local buffering would look like any other network path
buffers.  True, it doesn't help with downlink data, but it may with the
uplink.

I'm not sure that you can use cake on mobile, not with the sorts of wildly
changing bandwidths that I've been seeing (<5 to >50Mbps within a few
seconds, as one moves through varying lines-of-sight to the tower (say
walking into a building, driving into a tunnel or just past a parking
garage...).

-Aaron

[-- Attachment #2: Type: text/html, Size: 2114 bytes --]

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 1561 bytes --]

On Thu, Sep 29, 2016 at 6:24 AM, Mário Sérgio Fujikawa Ferreira <
liouxbsd@gmail.com> wrote:

> Is there a mailing list I can lurk in to follow on the development?
>
> I'm most interested on a delta to apply to Android 6.x Franco Kernel (
> https://github.com/franciscofranco/angler) and Debian Jessie.
>
> FreeBSD support would be excellent but I don't even have reviewed the
> license to be sure if we need a clean room development (hence I haven't
> read the code) or we can try to relocate it directly. On a side FreeBSD
> note, I'm hopeful for
>
>    1. Packet pacing (https://wiki.freebsd.org/DevSummit/201606/Transport)
>    2. ipfw/dummynet AQM with FQ-Codel (https://lists.freebsd.org/
>    pipermail/freebsd-ipfw/2016-February/006026.html
>    <https://lists.freebsd.org/pipermail/freebsd-ipfw/2016-February/006026.html>
>    )
>
> It's a great foot in the door, but PFSense does not use dummynet for
interface shaping, they use PF. We need someone to get fq-codel into PF.


>
>    1.
>
> Most excellent news.
> On 16/09/2016 19:29, Dave Taht wrote:
>
> Thank you for this, in particular...
> http://www.spinics.net/lists/netdev/msg395552.html
>
> still reading over the code...
>
> TCP Pr0n, this is!
> _______________________________________________
> Bloat mailing listBloat@lists.bufferbloat.nethttps://lists.bufferbloat.net/listinfo/bloat
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>

[-- Attachment #2: Type: text/html, Size: 3155 bytes --]

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Mario Ferreira]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

On Thu, Sep 29, 2016, 20:26 Benjamin Cronce <bcronce@gmail.com> wrote:

> On Thu, Sep 29, 2016 at 6:24 AM, Mário Sérgio Fujikawa Ferreira <
> liouxbsd@gmail.com> wrote:
>
>> [...]

> FreeBSD support would be excellent but I don't even have reviewed the
>> license to be sure if we need a clean room development (hence I haven't
>> read the code) or we can try to relocate it directly. On a side FreeBSD
>> note, I'm hopeful for
>>
>>    1. Packet pacing (https://wiki.freebsd.org/DevSummit/201606/Transport)
>>    2. ipfw/dummynet AQM with FQ-Codel (
>>    https://lists.freebsd.org/pipermail/freebsd-ipfw/2016-February/006026.html
>>    )
>>
>> It's a great foot in the door, but PFSense does not use dummynet for
> interface shaping, they
>
use PF. We need someone to get fq-codel into PF.
>

For pfsense you would need someone to code it into ALTQ if I read it right.

A possible approach then would be contacting the original ALTQ developers

https://www.sonycsl.co.jp/person/kjc/kjc/software.html
-- 

Mario S F Ferreira - Brazil - "I guess this is a signature."
feature, n: a documented bug | bug, n: an undocumented feature

[-- Attachment #2: Type: text/html, Size: 3186 bytes --]

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Steinar H. Gunderson]

On Fri, Sep 16, 2016 at 02:04:37PM -0700, Dave Taht wrote:
> I'm really looking forward to trying them out and reading the upcoming paper.
> 
> https://patchwork.ozlabs.org/patch/671069/

As a random data point, I tried a single flow from my main server in .no
to my backup server in .nl and compared CUBIC (with sch_fq) to BBR (naturally
also in sch_fq) on the sender side. The results were quite consistent across
runs:

 - BBR ramped up much quicker than CUBIC.
 - BBR gave ~10% higher max speed than CUBIC (~680 Mbit/sec on a gigabit link).
 - CUBIC wavered a bit up and down (~100 Mbit/sec) from the max; BBR stayed
   put (perhaps 2 Mbit/sec difference) the entire time.

http://pastebin.com/0XQaJvD9 for a typical log. I don't have any fancy
graphs, sorry :-)

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Eric Dumazet]

On Fri, 2016-10-21 at 10:47 +0200, Steinar H. Gunderson wrote:
> On Fri, Sep 16, 2016 at 02:04:37PM -0700, Dave Taht wrote:
> > I'm really looking forward to trying them out and reading the upcoming paper.
> > 
> > https://patchwork.ozlabs.org/patch/671069/
> 
> As a random data point, I tried a single flow from my main server in .no
> to my backup server in .nl and compared CUBIC (with sch_fq) to BBR (naturally
> also in sch_fq) on the sender side. The results were quite consistent across
> runs:
> 
>  - BBR ramped up much quicker than CUBIC.
>  - BBR gave ~10% higher max speed than CUBIC (~680 Mbit/sec on a gigabit link).
>  - CUBIC wavered a bit up and down (~100 Mbit/sec) from the max; BBR stayed
>    put (perhaps 2 Mbit/sec difference) the entire time.
> 
> http://pastebin.com/0XQaJvD9 for a typical log. I don't have any fancy
> graphs, sorry :-)
> 
> /* Steinar */

Hi Steinar

Could you provide /proc/sys/net/ipv4/tcp_rmem
and /proc/sys/net/ipv4/tcp_wmem values on your server ?

What is the rtt between your two hosts ?

Thanks

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Steinar H. Gunderson]

On Fri, Oct 21, 2016 at 03:28:01AM -0700, Eric Dumazet wrote:
> Could you provide /proc/sys/net/ipv4/tcp_rmem
> and /proc/sys/net/ipv4/tcp_wmem values on your server ?

I see I have old sysctl.conf settings here, from tuning way back in the 2.6
era:

  net/core/rmem_max = 8738140
  net/core/wmem_max = 8738140
  
  net/ipv4/tcp_rmem = 4096 873814 8738140
  net/ipv4/tcp_wmem = 4096 873814 8738140

Perhaps it's time to kill them, but I don't know what the defaults are :-)

> What is the rtt between your two hosts ?

35.7–36.0 ms.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Steinar H. Gunderson]

On Fri, Oct 21, 2016 at 12:42:17PM +0200, Steinar H. Gunderson wrote:
> Perhaps it's time to kill them, but I don't know what the defaults are :-)

If it matters (for autotuning), the host has 64 GB of physical RAM.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Eric Dumazet]

On Fri, 2016-10-21 at 12:47 +0200, Steinar H. Gunderson wrote:
> On Fri, Oct 21, 2016 at 12:42:17PM +0200, Steinar H. Gunderson wrote:
> > Perhaps it's time to kill them, but I don't know what the defaults are :-)
> 
> If it matters (for autotuning), the host has 64 GB of physical RAM.

I am working on a patch to change how EPOLLOUT events are generated,
so that we can use /proc/sys/net/ipv4/tcp_notsent_lowat = 1000000
to greatly reduce amount of bytes in write queues, without a throughput
reduction in presence of losses.

(There is a bug at this moment, kernel fails to wakeup the writer
if it receives sack blocks, consuming all not sent bytes in write queue)

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Eric Dumazet]

On Fri, 2016-10-21 at 12:42 +0200, Steinar H. Gunderson wrote:
> On Fri, Oct 21, 2016 at 03:28:01AM -0700, Eric Dumazet wrote:
> > Could you provide /proc/sys/net/ipv4/tcp_rmem
> > and /proc/sys/net/ipv4/tcp_wmem values on your server ?
> 
> I see I have old sysctl.conf settings here, from tuning way back in the 2.6
> era:
> 
>   net/core/rmem_max = 8738140
>   net/core/wmem_max = 8738140
>   
>   net/ipv4/tcp_rmem = 4096 873814 8738140
>   net/ipv4/tcp_wmem = 4096 873814 8738140
> 
> Perhaps it's time to kill them, but I don't know what the defaults are :-)
> 
> > What is the rtt between your two hosts ?
> 
> 35.7–36.0 ms.
> 
> /* Steinar */

Ok, you could try 12 MB instead of 8MB to eventually reach line rate.

net/ipv4/tcp_rmem = 4096 873814 12000000
net/ipv4/tcp_wmem = 4096 873814 12000000

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Steinar H. Gunderson]

On Fri, Oct 21, 2016 at 03:52:24AM -0700, Eric Dumazet wrote:
> Ok, you could try 12 MB instead of 8MB to eventually reach line rate.
> 
> net/ipv4/tcp_rmem = 4096 873814 12000000
> net/ipv4/tcp_wmem = 4096 873814 12000000

No difference, really. (I tried increasing both at the server and the
client.)

FWIW, this is over the public Internet, including one 10 -> 1 Gbit/sec
downconversion somewhere. I can't guarantee there's actually a full gigabit
of bandwidth free along the path.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Eric Dumazet]

On Fri, 2016-10-21 at 13:03 +0200, Steinar H. Gunderson wrote:
> On Fri, Oct 21, 2016 at 03:52:24AM -0700, Eric Dumazet wrote:
> > Ok, you could try 12 MB instead of 8MB to eventually reach line rate.
> > 
> > net/ipv4/tcp_rmem = 4096 873814 12000000
> > net/ipv4/tcp_wmem = 4096 873814 12000000
> 
> No difference, really. (I tried increasing both at the server and the
> client.)
> 
> FWIW, this is over the public Internet, including one 10 -> 1 Gbit/sec
> downconversion somewhere. I can't guarantee there's actually a full gigabit
> of bandwidth free along the path.

Yet it looks like they are few drops on this link, which is nice ;)
 

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Steinar H. Gunderson]

On Fri, Oct 21, 2016 at 04:40:13AM -0700, Eric Dumazet wrote:
> Yet it looks like they are few drops on this link, which is nice ;)

I believe the two ISPs peer directly on AMS-IX, so it's pretty good, yes.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Steinar H. Gunderson]

On Fri, Oct 21, 2016 at 10:47:26AM +0200, Steinar H. Gunderson wrote:
> As a random data point, I tried a single flow from my main server in .no
> to my backup server in .nl and compared CUBIC (with sch_fq) to BBR (naturally
> also in sch_fq) on the sender side. The results were quite consistent across
> runs:

Another datapoint: A friend of mine had a different, worse path (of about 40 ms)
and tested with iperf.

CUBIC delivered 20.1 Mbit/sec (highly varying). BBR delivered 485 Mbit/sec.

/* Steinar */
-- 
Homepage: https://www.sesse.net/

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Eric Dumazet]

On Thu, 2016-10-27 at 19:04 +0200, Steinar H. Gunderson wrote:
> On Fri, Oct 21, 2016 at 10:47:26AM +0200, Steinar H. Gunderson wrote:
> > As a random data point, I tried a single flow from my main server in .no
> > to my backup server in .nl and compared CUBIC (with sch_fq) to BBR (naturally
> > also in sch_fq) on the sender side. The results were quite consistent across
> > runs:
> 
> Another datapoint: A friend of mine had a different, worse path (of about 40 ms)
> and tested with iperf.
> 
> CUBIC delivered 20.1 Mbit/sec (highly varying). BBR delivered 485 Mbit/sec.


Yes ;)

I had cases where cubic was delivering 3 Mbits, but BBR could deliver
1000 times more ;)

Thanks !

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Dave Taht]

On Thu, Oct 27, 2016 at 10:04 AM, Steinar H. Gunderson
<sgunderson@bigfoot.com> wrote:
> On Fri, Oct 21, 2016 at 10:47:26AM +0200, Steinar H. Gunderson wrote:
>> As a random data point, I tried a single flow from my main server in .no
>> to my backup server in .nl and compared CUBIC (with sch_fq) to BBR (naturally
>> also in sch_fq) on the sender side. The results were quite consistent across
>> runs:
>
> Another datapoint: A friend of mine had a different, worse path (of about 40 ms)
> and tested with iperf.
>
> CUBIC delivered 20.1 Mbit/sec (highly varying). BBR delivered 485 Mbit/sec.

I mostly live in a world (wifi) where loss is uncommon, unless forced
on it with a AQM.

At the moment my biggest beef with BBR is that it ignores ECN entirely
(and yet negotiates it). BBR is then so efficient at using up all the
pipe that a single queued aqm "marks madly" and everything else
eventually starves. Watch "ping" fade out here...

http://blog.cerowrt.org/flent/bbr-comprehensive/bbr_ecn_eventually_starving_ping.png

somewhat conversely in fq_codel, this means that it ignores codel's
marking attempts entirely and BBR retains it's own dynamics, (while
the non-BBR flows are fine) which is kind of neat to watch.

> /* Steinar */
> --
> Homepage: https://www.sesse.net/
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Eric Dumazet]

On Thu, 2016-10-27 at 10:33 -0700, Dave Taht wrote:

> At the moment my biggest beef with BBR is that it ignores ECN entirely
> (and yet negotiates it). 

Note that switching cubic to any other CC like BBR is allowed at any
time, way after ECN was negotiated.

So BBR can not solve the issue you mention in a reliable way.

There must be a reason sysctl_tcp_ecn default value is 2 on linux [1],
don't you think ???

_You_ chose to change this sysctl, do not blame BBR for being silly !

ECN was a nice attempt, but suffers from implementation bugs.

For a start, linux does not implement RFC 3540.

If someone cares enough of ECN, then it should cook linux patches to
implement RFC 3540. Hint hint hint.

Then you need to make sure all the nodes between your peers are not
messing with ECN.

BBR simply works, because it is a sender side thing.
You do not have to fix everything in the Internet.

[1]
Documentation/networking/ip-sysctl.txt
tcp_ecn - INTEGER
        Control use of Explicit Congestion Notification (ECN) by TCP.
        ECN is used only when both ends of the TCP connection indicate
        support for it.  This feature is useful in avoiding losses due
        to congestion by allowing supporting routers to signal
        congestion before having to drop packets.
        Possible values are:
                0 Disable ECN.  Neither initiate nor accept ECN.
                1 Enable ECN when requested by incoming connections and
                  also request ECN on outgoing connection attempts.
                2 Enable ECN when requested by incoming connections
                  but do not request ECN on outgoing connections.
        Default: 2

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Dave Taht]

On Thu, Oct 27, 2016 at 10:52 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Thu, 2016-10-27 at 10:33 -0700, Dave Taht wrote:
>
>> At the moment my biggest beef with BBR is that it ignores ECN entirely
>> (and yet negotiates it).
>
> Note that switching cubic to any other CC like BBR is allowed at any
> time, way after ECN was negotiated.
>
> So BBR can not solve the issue you mention in a reliable way.
>
> There must be a reason sysctl_tcp_ecn default value is 2 on linux [1],
> don't you think ???
>
> _You_ chose to change this sysctl, do not blame BBR for being silly !

While I chose to do so on portions of my deployment, there is the larger
problem that Apple has enabled it nearly universally on iOS and OSX
over the past year.

> ECN was a nice attempt, but suffers from implementation bugs.

Believe me, I'm not huge on it either!

>
> For a start, linux does not implement RFC 3540.
>
> If someone cares enough of ECN, then it should cook linux patches to
> implement RFC 3540. Hint hint hint.
>
> Then you need to make sure all the nodes between your peers are not
> messing with ECN.
>
> BBR simply works, because it is a sender side thing.
> You do not have to fix everything in the Internet.
>
> [1]
> Documentation/networking/ip-sysctl.txt
> tcp_ecn - INTEGER
>         Control use of Explicit Congestion Notification (ECN) by TCP.
>         ECN is used only when both ends of the TCP connection indicate
>         support for it.  This feature is useful in avoiding losses due
>         to congestion by allowing supporting routers to signal
>         congestion before having to drop packets.
>         Possible values are:
>                 0 Disable ECN.  Neither initiate nor accept ECN.
>                 1 Enable ECN when requested by incoming connections and
>                   also request ECN on outgoing connection attempts.
>                 2 Enable ECN when requested by incoming connections
>                   but do not request ECN on outgoing connections.
>         Default: 2
>
>



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Yuchung Cheng]

On Thu, Oct 27, 2016 at 10:33 AM, Dave Taht <dave.taht@gmail.com> wrote:
> On Thu, Oct 27, 2016 at 10:04 AM, Steinar H. Gunderson
> <sgunderson@bigfoot.com> wrote:
>> On Fri, Oct 21, 2016 at 10:47:26AM +0200, Steinar H. Gunderson wrote:
>>> As a random data point, I tried a single flow from my main server in .no
>>> to my backup server in .nl and compared CUBIC (with sch_fq) to BBR (naturally
>>> also in sch_fq) on the sender side. The results were quite consistent across
>>> runs:
>>
>> Another datapoint: A friend of mine had a different, worse path (of about 40 ms)
>> and tested with iperf.
>>
>> CUBIC delivered 20.1 Mbit/sec (highly varying). BBR delivered 485 Mbit/sec.
>
> I mostly live in a world (wifi) where loss is uncommon, unless forced
> on it with a AQM.
>
> At the moment my biggest beef with BBR is that it ignores ECN entirely
> (and yet negotiates it). BBR is then so efficient at using up all the
> pipe that a single queued aqm "marks madly" and everything else
> eventually starves. Watch "ping" fade out here...
>
> http://blog.cerowrt.org/flent/bbr-comprehensive/bbr_ecn_eventually_starving_ping.png

Thanks Dave for this issue. We design BBR with CoDel in mind b/c Van
co-designs both :) We have tested BBR with CoDel before and it works.
Could you share your tcpdump traces with us (maybe
you already did but no sure) or suggest how to reproduce this.

This is 2 bbr flow or bbr + ecn-cubic? (I am guessing based on caption
in your graph)

>
> somewhat conversely in fq_codel, this means that it ignores codel's
> marking attempts entirely and BBR retains it's own dynamics, (while
> the non-BBR flows are fine) which is kind of neat to watch.
>
>> /* Steinar */
>> --
>> Homepage: https://www.sesse.net/
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
>
>
> --
> Dave Täht
> Let's go make home routers and wifi faster! With better software!
> http://blog.cerowrt.org
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Dave Taht]

On Thu, Oct 27, 2016 at 10:57 AM, Yuchung Cheng <ycheng@google.com> wrote:
> On Thu, Oct 27, 2016 at 10:33 AM, Dave Taht <dave.taht@gmail.com> wrote:
>> On Thu, Oct 27, 2016 at 10:04 AM, Steinar H. Gunderson
>> <sgunderson@bigfoot.com> wrote:
>>> On Fri, Oct 21, 2016 at 10:47:26AM +0200, Steinar H. Gunderson wrote:
>>>> As a random data point, I tried a single flow from my main server in .no
>>>> to my backup server in .nl and compared CUBIC (with sch_fq) to BBR (naturally
>>>> also in sch_fq) on the sender side. The results were quite consistent across
>>>> runs:
>>>
>>> Another datapoint: A friend of mine had a different, worse path (of about 40 ms)
>>> and tested with iperf.
>>>
>>> CUBIC delivered 20.1 Mbit/sec (highly varying). BBR delivered 485 Mbit/sec.
>>
>> I mostly live in a world (wifi) where loss is uncommon, unless forced
>> on it with a AQM.
>>
>> At the moment my biggest beef with BBR is that it ignores ECN entirely
>> (and yet negotiates it). BBR is then so efficient at using up all the
>> pipe that a single queued aqm "marks madly" and everything else
>> eventually starves. Watch "ping" fade out here...
>>
>> http://blog.cerowrt.org/flent/bbr-comprehensive/bbr_ecn_eventually_starving_ping.png
>
> Thanks Dave for this issue. We design BBR with CoDel in mind b/c Van
> co-designs both :)

It works pretty darn good with codel without ecn. I'm pretty darn happy with it.

fq_codel is even more lovely, especially when competing with cubic.

There are issues with single queued aqms with BBR vs cubic

http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-creaming-cubic-flowblind-aqm.svg

> We have tested BBR with CoDel before and it works.

Well, against cubic on the same link in single queue mode, even
without ecn, life looks like this:

http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-creaming-cubic-flowblind-aqm.svg

but fq_codel is fine, so long as there is no ecn vs nonecn collission

http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-ecn-fq.png

> Could you share your tcpdump traces with us (maybe
> you already did but no sure) or suggest how to reproduce this.
>
> This is 2 bbr flow or bbr + ecn-cubic? (I am guessing based on caption
> in your graph)

That's two BBRs with ecn enabled, going through cake in the single
queue aqm mode "flowblind". I have similar plots with pie and codel
with ecn enabled somewhere.

The emulation is 48ms RTT, 20Mbit down, 5Mbit up.

Regrettably I'm on a couple deadlines (a talk tomorrow, and next
thursday), and can't look harder, I do have caps comparing ecn vs
noecn here

http://blog.cerowrt.org/flent/bbr-ecncaps/



>
>>
>> somewhat conversely in fq_codel, this means that it ignores codel's
>> marking attempts entirely and BBR retains it's own dynamics, (while
>> the non-BBR flows are fine) which is kind of neat to watch.
>>
>>> /* Steinar */
>>> --
>>> Homepage: https://www.sesse.net/
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>>
>>
>>
>> --
>> Dave Täht
>> Let's go make home routers and wifi faster! With better software!
>> http://blog.cerowrt.org
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Bloat] [bbr-dev] Re: "BBR" TCP patches submitted to linux kernel

[Yuchung Cheng]

On Thu, Oct 27, 2016 at 11:14 AM, Dave Taht <dave.taht@gmail.com> wrote:
> On Thu, Oct 27, 2016 at 10:57 AM, Yuchung Cheng <ycheng@google.com> wrote:
>> On Thu, Oct 27, 2016 at 10:33 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>> On Thu, Oct 27, 2016 at 10:04 AM, Steinar H. Gunderson
>>> <sgunderson@bigfoot.com> wrote:
>>>> On Fri, Oct 21, 2016 at 10:47:26AM +0200, Steinar H. Gunderson wrote:
>>>>> As a random data point, I tried a single flow from my main server in .no
>>>>> to my backup server in .nl and compared CUBIC (with sch_fq) to BBR (naturally
>>>>> also in sch_fq) on the sender side. The results were quite consistent across
>>>>> runs:
>>>>
>>>> Another datapoint: A friend of mine had a different, worse path (of about 40 ms)
>>>> and tested with iperf.
>>>>
>>>> CUBIC delivered 20.1 Mbit/sec (highly varying). BBR delivered 485 Mbit/sec.
>>>
>>> I mostly live in a world (wifi) where loss is uncommon, unless forced
>>> on it with a AQM.
>>>
>>> At the moment my biggest beef with BBR is that it ignores ECN entirely
>>> (and yet negotiates it). BBR is then so efficient at using up all the
>>> pipe that a single queued aqm "marks madly" and everything else
>>> eventually starves. Watch "ping" fade out here...
>>>
>>> http://blog.cerowrt.org/flent/bbr-comprehensive/bbr_ecn_eventually_starving_ping.png
>>
>> Thanks Dave for this issue. We design BBR with CoDel in mind b/c Van
>> co-designs both :)
>
> It works pretty darn good with codel without ecn. I'm pretty darn happy with it.
>
> fq_codel is even more lovely, especially when competing with cubic.
>
> There are issues with single queued aqms with BBR vs cubic
>
> http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-creaming-cubic-flowblind-aqm.svg
>
>> We have tested BBR with CoDel before and it works.
>
> Well, against cubic on the same link in single queue mode, even
> without ecn, life looks like this:
>
> http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-creaming-cubic-flowblind-aqm.svg
>
> but fq_codel is fine, so long as there is no ecn vs nonecn collission
>
> http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-ecn-fq.png
>
>> Could you share your tcpdump traces with us (maybe
>> you already did but no sure) or suggest how to reproduce this.
>>
>> This is 2 bbr flow or bbr + ecn-cubic? (I am guessing based on caption
>> in your graph)
>
> That's two BBRs with ecn enabled, going through cake in the single
> queue aqm mode "flowblind". I have similar plots with pie and codel
> with ecn enabled somewhere.
>
> The emulation is 48ms RTT, 20Mbit down, 5Mbit up.
>
> Regrettably I'm on a couple deadlines (a talk tomorrow, and next
> thursday), and can't look harder, I do have caps comparing ecn vs
> noecn here
>
> http://blog.cerowrt.org/flent/bbr-ecncaps/
Thanks for the data (and sorry for ignoring that before). Neal and I
think the behaviors you are observing matches BBR's top issue we are
actively pursuing: with N>1 flows, BBR may build 1.5BDP of queue. But
let's separate that from ECN negotiation. Besidethe implementation
complication Eric pointed out, even if BBR refrains from ECN
negotiation, and the test result wouldn't change much I suspect.

We'll get back soon.

>
>
>
>>
>>>
>>> somewhat conversely in fq_codel, this means that it ignores codel's
>>> marking attempts entirely and BBR retains it's own dynamics, (while
>>> the non-BBR flows are fine) which is kind of neat to watch.
>>>
>>>> /* Steinar */
>>>> --
>>>> Homepage: https://www.sesse.net/
>>>> _______________________________________________
>>>> Bloat mailing list
>>>> Bloat@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/bloat
>>>
>>>
>>>
>>> --
>>> Dave Täht
>>> Let's go make home routers and wifi faster! With better software!
>>> http://blog.cerowrt.org
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>
>
>
> --
> Dave Täht
> Let's go make home routers and wifi faster! With better software!
> http://blog.cerowrt.org
>
> --
> You received this message because you are subscribed to the Google Groups "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Re: [Bloat] [bbr-dev] Re: "BBR" TCP patches submitted to linux kernel

[Yuchung Cheng]

[-- Attachment #1: Type: text/plain, Size: 5256 bytes --]

Hi Dave,

We are able to reproduce the cubic starvation issue with codel (but not
fq_codel), regardless of ECN. So clearly ECN isn't the issue but the
single-queued AQM is. We didn't notice that earlier even though you've
clearly indicated that in your report. But Cubic and BBR loss response
functions are nonlinear so that's expected.

We are curious why you choose the single-queued AQM. Is it just for the
sake of testing?



On Thu, Oct 27, 2016 at 2:30 PM, Yuchung Cheng <ycheng@google.com> wrote:

> On Thu, Oct 27, 2016 at 11:14 AM, Dave Taht <dave.taht@gmail.com> wrote:
> > On Thu, Oct 27, 2016 at 10:57 AM, Yuchung Cheng <ycheng@google.com>
> wrote:
> >> On Thu, Oct 27, 2016 at 10:33 AM, Dave Taht <dave.taht@gmail.com>
> wrote:
> >>> On Thu, Oct 27, 2016 at 10:04 AM, Steinar H. Gunderson
> >>> <sgunderson@bigfoot.com> wrote:
> >>>> On Fri, Oct 21, 2016 at 10:47:26AM +0200, Steinar H. Gunderson wrote:
> >>>>> As a random data point, I tried a single flow from my main server in
> .no
> >>>>> to my backup server in .nl and compared CUBIC (with sch_fq) to BBR
> (naturally
> >>>>> also in sch_fq) on the sender side. The results were quite
> consistent across
> >>>>> runs:
> >>>>
> >>>> Another datapoint: A friend of mine had a different, worse path (of
> about 40 ms)
> >>>> and tested with iperf.
> >>>>
> >>>> CUBIC delivered 20.1 Mbit/sec (highly varying). BBR delivered 485
> Mbit/sec.
> >>>
> >>> I mostly live in a world (wifi) where loss is uncommon, unless forced
> >>> on it with a AQM.
> >>>
> >>> At the moment my biggest beef with BBR is that it ignores ECN entirely
> >>> (and yet negotiates it). BBR is then so efficient at using up all the
> >>> pipe that a single queued aqm "marks madly" and everything else
> >>> eventually starves. Watch "ping" fade out here...
> >>>
> >>> http://blog.cerowrt.org/flent/bbr-comprehensive/bbr_ecn_
> eventually_starving_ping.png
> >>
> >> Thanks Dave for this issue. We design BBR with CoDel in mind b/c Van
> >> co-designs both :)
> >
> > It works pretty darn good with codel without ecn. I'm pretty darn happy
> with it.
> >
> > fq_codel is even more lovely, especially when competing with cubic.
> >
> > There are issues with single queued aqms with BBR vs cubic
> >
> > http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-
> creaming-cubic-flowblind-aqm.svg
> >
> >> We have tested BBR with CoDel before and it works.
> >
> > Well, against cubic on the same link in single queue mode, even
> > without ecn, life looks like this:
> >
> > http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-
> creaming-cubic-flowblind-aqm.svg
> >
> > but fq_codel is fine, so long as there is no ecn vs nonecn collission
> >
> > http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-ecn-fq.png
> >
> >> Could you share your tcpdump traces with us (maybe
> >> you already did but no sure) or suggest how to reproduce this.
> >>
> >> This is 2 bbr flow or bbr + ecn-cubic? (I am guessing based on caption
> >> in your graph)
> >
> > That's two BBRs with ecn enabled, going through cake in the single
> > queue aqm mode "flowblind". I have similar plots with pie and codel
> > with ecn enabled somewhere.
> >
> > The emulation is 48ms RTT, 20Mbit down, 5Mbit up.
> >
> > Regrettably I'm on a couple deadlines (a talk tomorrow, and next
> > thursday), and can't look harder, I do have caps comparing ecn vs
> > noecn here
> >
> > http://blog.cerowrt.org/flent/bbr-ecncaps/
> Thanks for the data (and sorry for ignoring that before). Neal and I
> think the behaviors you are observing matches BBR's top issue we are
> actively pursuing: with N>1 flows, BBR may build 1.5BDP of queue. But
> let's separate that from ECN negotiation. Besidethe implementation
> complication Eric pointed out, even if BBR refrains from ECN
> negotiation, and the test result wouldn't change much I suspect.
>
> We'll get back soon.
>
> >
> >
> >
> >>
> >>>
> >>> somewhat conversely in fq_codel, this means that it ignores codel's
> >>> marking attempts entirely and BBR retains it's own dynamics, (while
> >>> the non-BBR flows are fine) which is kind of neat to watch.
> >>>
> >>>> /* Steinar */
> >>>> --
> >>>> Homepage: https://www.sesse.net/
> >>>> _______________________________________________
> >>>> Bloat mailing list
> >>>> Bloat@lists.bufferbloat.net
> >>>> https://lists.bufferbloat.net/listinfo/bloat
> >>>
> >>>
> >>>
> >>> --
> >>> Dave Täht
> >>> Let's go make home routers and wifi faster! With better software!
> >>> http://blog.cerowrt.org
> >>> _______________________________________________
> >>> Bloat mailing list
> >>> Bloat@lists.bufferbloat.net
> >>> https://lists.bufferbloat.net/listinfo/bloat
> >
> >
> >
> > --
> > Dave Täht
> > Let's go make home routers and wifi faster! With better software!
> > http://blog.cerowrt.org
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "BBR Development" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to bbr-dev+unsubscribe@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>

[-- Attachment #2: Type: text/html, Size: 8194 bytes --]

Re: [Bloat] [bbr-dev] Re: "BBR" TCP patches submitted to linux kernel

[Jonathan Morton]

> On 2 Nov, 2016, at 01:13, 'Yuchung Cheng' via BBR Development <bbr-dev@googlegroups.com> wrote:
> 
> We are curious why you choose the single-queued AQM. Is it just for the sake of testing?

Not to speak for them, but single-queue AQM is the most likely implementation for high-speed hardware.  DOCSIS 3.1, for example, specifies PIE for client upstream, not any form of FQ-PIE.  It therefore remains an important use-case for interop testing.

 - Jonathan Morton

Re: [Bloat] [bbr-dev] Re: "BBR" TCP patches submitted to linux kernel

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 734 bytes --]

On Tue, 1 Nov 2016, Yuchung Cheng wrote:

> We are curious why you choose the single-queued AQM. Is it just for the 
> sake of testing?

Non-flow aware AQM is the most commonly deployed "queue management" on the 
Internet today. Most of them are just stupid FIFOs with taildrop, and the 
buffer size can be anywhere from super small to huge depending on 
equipment used and how it's configured.

Any proposed TCP congestion avoidance algorithm to be deployed on the 
wider Internet has to some degree be able to handle this deployment 
scenario without killing everything else it's sharing capacity with.

Dave Tähts testing case where BBR just kills Cubic makes me very 
concerned.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] [bbr-dev] Re: "BBR" TCP patches submitted to linux kernel

[Klatsky, Carl]

> On Tue, 1 Nov 2016, Yuchung Cheng wrote:
> 
> > We are curious why you choose the single-queued AQM. Is it just for
> > the sake of testing?
> 
> Non-flow aware AQM is the most commonly deployed "queue
> management" on the Internet today. Most of them are just stupid FIFOs
> with taildrop, and the buffer size can be anywhere from super small to huge
> depending on equipment used and how it's configured.
> 
> Any proposed TCP congestion avoidance algorithm to be deployed on the
> wider Internet has to some degree be able to handle this deployment
> scenario without killing everything else it's sharing capacity with.
> 
> Dave Tähts testing case where BBR just kills Cubic makes me very concerned.

If I am understanding BBR correctly, that is working in the sender to receiver direction.  In Dave's test running TCP BBR & TCP CUBIC with a single queue AQM, where CUBIC gets crushed.  Silly question, but the single queue AQM was also operating in the in sender to receiver direction for this test, yes? 

Re: [Bloat] [bbr-dev] Re: "BBR" TCP patches submitted to linux kernel

[Dave Täht]

On 11/2/16 11:21 AM, Klatsky, Carl wrote:
>> On Tue, 1 Nov 2016, Yuchung Cheng wrote:
>>
>>> We are curious why you choose the single-queued AQM. Is it just for
>>> the sake of testing?
>>
>> Non-flow aware AQM is the most commonly deployed "queue
>> management" on the Internet today. Most of them are just stupid FIFOs
>> with taildrop, and the buffer size can be anywhere from super small to huge
>> depending on equipment used and how it's configured.
>>
>> Any proposed TCP congestion avoidance algorithm to be deployed on the
>> wider Internet has to some degree be able to handle this deployment
>> scenario without killing everything else it's sharing capacity with.
>>
>> Dave Tähts testing case where BBR just kills Cubic makes me very concerned.
> 
> If I am understanding BBR correctly, that is working in the sender to receiver direction.  In Dave's test running TCP BBR & TCP CUBIC with a single queue AQM, where CUBIC gets crushed.

The scenario as I constructed it was emulating a sender on "home" side
of the link, using BBR and cubic through an emulated cablemodem running pie.


  Silly question, but the single queue AQM was also operating in the in
sender to receiver direction for this test, yes?
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
> 

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Bernd Paysan]

[-- Attachment #1.1: Type: text/plain, Size: 2399 bytes --]

Am Donnerstag, 27. Oktober 2016 20:14:42 UTC+2 schrieb Dave Taht:
>
> On Thu, Oct 27, 2016 at 10:57 AM, Yuchung Cheng <ych...@google.com 
> <javascript:>> wrote: 
> Well, against cubic on the same link in single queue mode, even 
> without ecn, life looks like this: 
>
>
> http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-creaming-cubic-flowblind-aqm.svg 
>
> but fq_codel is fine, so long as there is no ecn vs nonecn collission 
>
> http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-ecn-fq.png 

 
Looks like you are on the right track. I've been done some work on 
flow/congestion control on my net2o protocol, which is not limited by TCP's 
constraints, and I think you should look at what I did. See my 31c3 
presentation.

Flow/congestion control discussion starts at 46:00, page 79 on the slides.

https://fossil.net2o.de/net2o/doc/trunk/wiki/31c3.md

The primary algorithm is really simple and straight-forward: I send a short 
burst out, and measure the bandwidth achieved on the receiver side - this 
time is used to calculate the delays between two bursts, the average 
sending rate.  In TCP, you would measure the ACKs back at the sender, and 
calculate bandwidth achieved based on the delays between the acks; that 
should be a bit less precise, but good enough.  These bursts allow to adapt 
to changing loads quickly, and there is no need to fill the buffer at all 
(you don't need to create more increase in RTD than those few packets in 
the burst take).  This primary algorithm is ignorant of the buffer fill 
state, so it works together with a buffer filled up by normal TCP 
congestion control.

I took a great deal at providing fairness even without a fair queuing 
policy (much more complicated second order regulation; I'm still not happy 
with the results and trying to figure out something better), therefore, FQ 
is really necessary, everywhere, including on the last mile routers (see 
timings of 4 concurrent streams on page 108 w/o FQ and 109 w. net2o's own 
FQ).  The spikes in the last diagrams result from the receivers regularly 
writing the data away to disk, and when they do so, temporarily stopping 
the transmission of their stream for a short period of time.  That shows 
how fast the bursts can react on changed bandwidth.  With just one client, 
I can hide that delay with a second thread, but with 4 clients on one CPU, 
it just shows up.

[-- Attachment #1.2: Type: text/html, Size: 4107 bytes --]

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Zhen Cao]

[-- Attachment #1: Type: text/plain, Size: 206 bytes --]

On Sat, Sep 17, 2016 at 5:04 AM, Dave Taht <dave.taht@gmail.com> wrote:

> I'm really looking forward to trying them out and reading the upcoming
> paper.
>

Has the upcoming paper come?

Many thanks.
zhen

[-- Attachment #2: Type: text/html, Size: 607 bytes --]

Re: [Bloat] "BBR" TCP patches submitted to linux kernel

[Ingemar Johansson S]

Hi Eric + others 
Cubic : I would believe that the main reason why Cubic give such bad results lies in the HyStart algorithm, I have sofar not seen any major issues with the Cubic congestion avoidance algorithm, provided of course that proper AQM is deployed.

RFC3540 : This actually in the process of becoming historic (see https://datatracker.ietf.org/doc/draft-black-tsvwg-ecn-experimentation/ ), also you will notice that I am on favor of this draft as I am among the folks who wish to free up ECT(1) for L4S use.

/Ingemar

> Message: 5
> Date: Thu, 27 Oct 2016 10:52:08 -0700
> From: Eric Dumazet <eric.dumazet@gmail.com>
> To: Dave Taht <dave.taht@gmail.com>
> Cc: "Steinar H. Gunderson" <sgunderson@bigfoot.com>, bloat
> 	<bloat@lists.bufferbloat.net>
> Subject: Re: [Bloat] "BBR" TCP patches submitted to linux kernel
> Message-ID:
> 	<1477590728.7065.231.camel@edumazet-
> glaptop3.roam.corp.google.com>
> Content-Type: text/plain; charset="UTF-8"
> 
> On Thu, 2016-10-27 at 10:33 -0700, Dave Taht wrote:
> 
> > At the moment my biggest beef with BBR is that it ignores ECN entirely
> > (and yet negotiates it).
> 
> Note that switching cubic to any other CC like BBR is allowed at any
> time, way after ECN was negotiated.
> 
> So BBR can not solve the issue you mention in a reliable way.
> 
> There must be a reason sysctl_tcp_ecn default value is 2 on linux [1],
> don't you think ???
> 
> _You_ chose to change this sysctl, do not blame BBR for being silly !
> 
> ECN was a nice attempt, but suffers from implementation bugs.
> 
> For a start, linux does not implement RFC 3540.
> 
> If someone cares enough of ECN, then it should cook linux patches to
> implement RFC 3540. Hint hint hint.
> 
> Then you need to make sure all the nodes between your peers are not
> messing with ECN.
> 
> BBR simply works, because it is a sender side thing.
> You do not have to fix everything in the Internet.
> 
> [1]
> Documentation/networking/ip-sysctl.txt
> tcp_ecn - INTEGER
>         Control use of Explicit Congestion Notification (ECN) by TCP.
>         ECN is used only when both ends of the TCP connection indicate
>         support for it.  This feature is useful in avoiding losses due
>         to congestion by allowing supporting routers to signal
>         congestion before having to drop packets.
>         Possible values are:
>                 0 Disable ECN.  Neither initiate nor accept ECN.
>                 1 Enable ECN when requested by incoming connections and
>                   also request ECN on outgoing connection attempts.
>                 2 Enable ECN when requested by incoming connections
>                   but do not request ECN on outgoing connections.
>         Default: 2
> 
> 
> 
> 
***********************