Re: [Cake] Cake upstream Planning

Re: [Cake] Cake upstream Planning

[Pete Heist]

>>    On Nov 15, 2017, at 9:04 PM, Dave Taht <dave@taht.net> wrote:
>> 
>> 
>>    Dave Taht <dave@taht.net> writes:
>> 
>>                TCP RTT ~= 8ms with default qdisc, throughput ~= 940 Mbit
>>            TCP RTT ~= 4.5ms with ‘cake unlimited’, throughput ~= 920 Mbit
>>            TCP RTT ~= 1ms with ‘cake unlimited lan’, throughput ~= 920 Mbit
>> 
>> 
>>    This was with BQL in play? Monitoring BQL's behavior might help.
>> 
>>    I'd also love to know an exact setting for the shaper as a close as
>>    possible to the underlying bandwidth of ethernet. However, I tend to be
>>    plagued with
>> 
>> 
>> Yes, with BQL (Intel I210 with igb driver on the APU2). An rrul_be test with —
>> socket-stats.
> 
> https://github.com/ffainelli/bqlmon was a tool for looking at bql more
> directly.
> 
> I had forked it for some reason or another:
> 
> https://github.com/dtaht/bqlmon

Nice, that does work for me. It’s interesting that there are four queues for the igb driver, 00 - 03, and when I try an rrul_be_nflows test, not all four queues are necessarily used. Once I get >= 8 flows in each direction they usually are though. I suppose this is the driver deciding when to start using another queue or not.

Cake does seem to visibly reduce the size of the queues. For whatever terminal/ncurses weirdness reason though, the bar graphs may be sometimes blowing off the top of my 45 row screen, but it doesn’t entirely ruin the experience.

Re: [Cake] Cake upstream Planning

[Dave Taht]

Pete Heist <peteheist@gmail.com> writes:

>>>    On Nov 15, 2017, at 9:04 PM, Dave Taht <dave@taht.net> wrote:
>>> 
>>> 
>>>    Dave Taht <dave@taht.net> writes:
>>> 
>>>                TCP RTT ~= 8ms with default qdisc, throughput ~= 940 Mbit
>>>            TCP RTT ~= 4.5ms with ‘cake unlimited’, throughput ~= 920 Mbit
>>>            TCP RTT ~= 1ms with ‘cake unlimited lan’, throughput ~= 920 Mbit
>>> 
>>> 
>>>    This was with BQL in play? Monitoring BQL's behavior might help.
>>> 
>>>    I'd also love to know an exact setting for the shaper as a close as
>>>    possible to the underlying bandwidth of ethernet. However, I tend to be
>>>    plagued with
>>> 
>>> 
>>> Yes, with BQL (Intel I210 with igb driver on the APU2). An rrul_be test with —
>>> socket-stats.
>> 
>> https://github.com/ffainelli/bqlmon was a tool for looking at bql more
>> directly.
>> 
>> I had forked it for some reason or another:
>> 
>> https://github.com/dtaht/bqlmon
>
> Nice, that does work for me. It’s interesting that there are four queues for the
> igb driver, 00 - 03, and when I try an rrul_be_nflows test, not all four queues
> are necessarily used. Once I get >= 8 flows in each direction they usually are
> though. I suppose this is the driver deciding when to start using another queue
> or not.

Usually it is selected via a hash. In more than a few cases, however,
the designer of the hardware intended it as a strict priority queue. In
other cases, it's based on the CPU.

In all cases such a limited number of queues tends to cause oddities.

I think it was the mvneta (?) that had the strict priority queue idea baked
into it, which we ended up disabling entirely and going with just one
hardware queue.

>
> Cake does seem to visibly reduce the size of the queues.

I generally observe that TSO/GRO/etc tends to make BQL's queues 3-5
times larger than they are without those offloads - no way to fix it,
short of doing what cake does to peel those apart.

> For whatever
> terminal/ncurses weirdness reason though, the bar graphs may be sometimes
> blowing off the top of my 45 row screen, but it doesn’t entirely ruin the
> experience.

Maybe that was why I forked it?

Re: [Cake] Cake upstream Planning

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2463 bytes --]


> On Nov 16, 2017, at 5:31 PM, Dave Taht <dave@taht.net> wrote:
> 
> Pete Heist <peteheist@gmail.com <mailto:peteheist@gmail.com>> writes:
> 
>>>>   On Nov 15, 2017, at 9:04 PM, Dave Taht <dave@taht.net> wrote:
>>>> 
>>>> 
>>>>   Dave Taht <dave@taht.net> writes:
>>>> 
>>> https://github.com/ffainelli/bqlmon was a tool for looking at bql more
>>> directly.
>>> 
>>> I had forked it for some reason or another:
>>> 
>>> https://github.com/dtaht/bqlmon
>> 
>> Nice, that does work for me. It’s interesting that there are four queues for the
>> igb driver, 00 - 03, and when I try an rrul_be_nflows test, not all four queues
>> are necessarily used. Once I get >= 8 flows in each direction they usually are
>> though. I suppose this is the driver deciding when to start using another queue
>> or not.
> 
> Usually it is selected via a hash. In more than a few cases, however,
> the designer of the hardware intended it as a strict priority queue. In
> other cases, it's based on the CPU.
> 
> In all cases such a limited number of queues tends to cause oddities.
> 
> I think it was the mvneta (?) that had the strict priority queue idea baked
> into it, which we ended up disabling entirely and going with just one
> hardware queue.

I noticed when I went to buy the APU2 that the two lower-end models (apu2c2 and apu2c0) have I211 NICs instead of a I210. The I211 is a “value part” that among other things has 2 tx and rx queues per port instead of 4. I wasn’t sure of the real effect of this when I purchased them, but for an extra few bucks the I210 seemed worth it. Table 1-6 on page 13:

https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/i210-ethernet-controller-datasheet.pdf

>> Cake does seem to visibly reduce the size of the queues.
> 
> I generally observe that TSO/GRO/etc tends to make BQL's queues 3-5
> times larger than they are without those offloads - no way to fix it,
> short of doing what cake does to peel those apart.

A real nicety of Cake that the world should benefit from.

>> For whatever
>> terminal/ncurses weirdness reason though, the bar graphs may be sometimes
>> blowing off the top of my 45 row screen, but it doesn’t entirely ruin the
>> experience.
> 
> Maybe that was why I forked it?

Looks like you forked it to fix a multi-queue problem. I forked your fork to add a scaling parameter to fix the bar height. -s 4096 works well for me.


[-- Attachment #2: Type: text/html, Size: 14844 bytes --]

Re: [Cake] Cake upstream Planning

[Dave Taht]

Pete Heist <peteheist@gmail.com> writes:

>     On Nov 16, 2017, at 5:31 PM, Dave Taht <dave@taht.net> wrote:
>
>     
>     Pete Heist <peteheist@gmail.com> writes:
>     
>                 
>
>             On Nov 15, 2017, at 9:04 PM, Dave Taht <dave@taht.net> wrote:
>                 
>                 
>                 Dave Taht <dave@taht.net> writes:
>                 
>                 
>
>             https://github.com/ffainelli/bqlmon was a tool for looking at bql
>             more
>             directly.
>             
>             I had forked it for some reason or another:
>             
>             https://github.com/dtaht/bqlmon
>             
>
>         Nice, that does work for me. It’s interesting that there are four queues
>         for the
>         igb driver, 00 - 03, and when I try an rrul_be_nflows test, not all four
>         queues
>         are necessarily used. Once I get >= 8 flows in each direction they
>         usually are
>         though. I suppose this is the driver deciding when to start using
>         another queue
>         or not.
>         
>
>     Usually it is selected via a hash. In more than a few cases, however,
>     the designer of the hardware intended it as a strict priority queue. In
>     other cases, it's based on the CPU.
>     
>     In all cases such a limited number of queues tends to cause oddities.
>     
>     I think it was the mvneta (?) that had the strict priority queue idea baked
>     into it, which we ended up disabling entirely and going with just one
>     hardware queue.
>     
>
> I noticed when I went to buy the APU2 that the two lower-end models (apu2c2 and
> apu2c0) have I211 NICs instead of a I210. The I211 is a “value part” that among
> other things has 2 tx and rx queues per port instead of 4. I wasn’t sure of the
> real effect of this when I purchased them, but for an extra few bucks the I210
> seemed worth it. Table 1-6 on page 13:
>
> https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/i210-ethernet-controller-datasheet.pdf
>
>         Cake does seem to visibly reduce the size of the queues.
>         
>
>     I generally observe that TSO/GRO/etc tends to make BQL's queues 3-5
>     times larger than they are without those offloads - no way to fix it,
>     short of doing what cake does to peel those apart.
>
> A real nicety of Cake that the world should benefit from.

At a rather large cpu cost.

>
>         For whatever
>         terminal/ncurses weirdness reason though, the bar graphs may be
>         sometimes
>         blowing off the top of my 45 row screen, but it doesn’t entirely ruin
>         the
>         experience.
>         
>
>     Maybe that was why I forked it?
>
> Looks like you forked it to fix a multi-queue problem. I forked your fork to add
> a scaling parameter to fix the bar height. -s 4096 works well for me.

I put in a pull request with all the outstanding patches to the
maintainer. Should have done that 2+ years ago.

Re: [Cake] Cake upstream Planning

[Pete Heist]

> From: Dave Taht <dave@taht.net>
> To: Pete Heist <peteheist@gmail.com>
> Cc: cake@lists.bufferbloat.net
> Subject: Re: [Cake] Cake upstream Planning
> Message-ID: <87375f71h3.fsf@nemesis.taht.net>
> Content-Type: text/plain; charset=utf-8
> 
> Dave Taht <dave@taht.net> writes:
> 
>>> So yes, we can lower TCP RTT with these more aggressive settings. But just to
>>> make sure, we’re confident that there are no other side effects from these lower
>>> targets and intervals? Is there anything else I should test for to be sure? For
>>> example, when I rate limit to 950 Mbit and try the same test above, ‘lan’ causes
>>> a 20% drop in throughput vs the defaults. That may be from an overtaxed CPU, but
>>> I don’t know. I also wonder how this affects routed vs local traffic. I’ll try
>>> to test this at some point, as I want to understand it better anyway to know how
>>> backhaul links should be configured...
> 
> One interesting thing that might make tcp behave better is the new
> pacing code which lets us set pacing to a different log value. Presently
> - at 10 - the TSQ algorithm recalculates things at 1ms. The pacing value
> is intended to be changed to, say, 6 or 7 to make wifi work
> better... and I suspect, that if it were upped to, say 12 (250us), on ethernet,
> that might make pacing more effective there.
> 
> Just like breaking the sound barrier, breaking the 1ms barrier looks to
> be hard within conventional kernel thread scheduling.

To make sure I understand, setting pacing means setting a socket option at the sender, right?

A TCP RTT of ~= 1ms with the ‘lan’ keyword is already quite good, and not something likely worth trying to optimize right away anyway.

Re: [Cake] Donation

[Pete Heist]

> On Nov 12, 2017, at 6:00 PM, cake-request@lists.bufferbloat.net wrote:
> 
> I sometimes think we should establish an organization, with a board of
> directors, a bank account, etc, but aside
> from grant money, donated computers and computer time, and all the
> massive efforts of all the volunteers, that's most of the donations
> we've ever got, and it would be, at least, 800 bucks to start a
> non-profit, + an accountant to "do right".
> 
> Any and all thoughts as to how to do better are welcomed.
> 
> We could have a bake sale for cake, to get it mainlined.

Has there been any thought towards monetizing some portion of the bufferbloat project to help pay for it? Here are a couple of ideas:

- Make a network and bloat testing service with a subscription tier or advertising supported free tier. Proceeds could at first fund the server resources needed, and maybe there’d be something left over. I don’t think there’s anything out there that’s more technically oriented and measures one-way delay, or differentiates between upstream and downstream packet loss, or has some of flent’s features like TCP RTT or the myriad of tests it supports, for starters. I really don’t know if anyone would pay a subscription fee for this though, or how much ad revenue would be possible.

- I’m intrigued by the idea of a cooperative or so-called “platform cooperative". In this case, individuals or companies could pay a monthly or annual fee to either just support the effort, or get access to a higher level of support, the bloat testing service, or specific code changes, etc. Any income beyond what’s needed to meet expenses could be distributed to its members somehow. This is a vague idea so far, I know. But I’ve been pretty impressed with how well the cooperative ISP I use works (lbcfree.net), and at its apparent resiliency over time. Though, the service they offer to their customers may be more clearly defined (“become a member, get Internet”).

By the way, what or how much is needed to get Cake mainlined?

Re: [Cake] Donation

[Dave Taht]

Pete Heist <peteheist@gmail.com> writes:

>> On Nov 12, 2017, at 6:00 PM, cake-request@lists.bufferbloat.net wrote:
>> 
>> I sometimes think we should establish an organization, with a board of
>> directors, a bank account, etc, but aside
>> from grant money, donated computers and computer time, and all the
>> massive efforts of all the volunteers, that's most of the donations
>> we've ever got, and it would be, at least, 800 bucks to start a
>> non-profit, + an accountant to "do right".
>> 
>> Any and all thoughts as to how to do better are welcomed.
>> 
>> We could have a bake sale for cake, to get it mainlined.
>
> Has there been any thought towards monetizing some portion of the bufferbloat
> project to help pay for it? Here are a couple of ideas:

I'm going to skip commenting in the hope that others chime in first.

> By the way, what or how much is needed to get Cake mainlined?

I'd like us to give it a go when net-next reopens in two weeks,
we'd then have 6 weeks or so to get it right.

We need:

* Someone to do the heavy lifting. Which I suspect would be me.
* Someones with various hardware platforms that current kernels can be
  run on. qemu?
* I'd like to see the ack filtering work get tested on lede at low
  bandwidths on dsl especially.
* A whole lotta tests at various RTTs

Blockers:

* Ripping out all the backward compatability cruft for submission to
  mainline and following netdev formatting conventions for comments and
  indentation. I'd like any new features in the backport to get
  backported, though (sigh), as lede looks to be shipping a 4.9 based
  kernel.
  
* tc-cake man page needs to be updated.

* tc-adv related code updated to latest iproute2

* There is some work going on here to add ack filtering to cake, which
looks VERY promising: https://github.com/dtaht/sch_cake/pull/63

I'm going to add something like this to netem also. It may be that
merely leveraging the hash would be enough in cake's case.

* Testing against the net-next kernel on x86, x86_64, arm, mips, and
aarch architectures. (I just got bit by not testing 32 bit arches, sigh)

Non-Blockers:

* I don't believe in cobalt, or rather, I won't believe in it until we
have data at many RTTs. That said, what I'd propose would be a
monolithic cobalt.h file rather than codel5.h.

The netns stuff will make simulating RTTs and bandwidths much easier....

* I think the fq_codel batch drop facility is better than what cake uses
in case of floods. Partially due to the need to handle backports the
mechanism fq_codel uses is hard to use - but going mainline we could add this.

* The autorate_ingress code should be marked experimental. I keep hoping
it can be improved by better looking for "smoothness" inbound, but
algorithms escape me. This doesn't bother me much, as tcp continues to
be improved over the past 50 years, perhaps we can find ways to improve
this with more users.

* It is possible to tune the quantum and peeling functions to not peel
to the extent they do. Particularly there is usually no need (aside from
wanting accurate statistics) to peel below 1500 bytes (except perhaps
with the new ack filter mode). We experimented a lot with this in the
early days but could never come to a resolution.

* I don't have any use for precidence mode and would like to remove it.

Re: [Cake] Donation

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 5821 bytes --]


> On Nov 14, 2017, at 9:10 PM, Dave Taht <dave@taht.net> wrote:
> 
> Pete Heist <peteheist@gmail.com <mailto:peteheist@gmail.com>> writes:
> 
>> By the way, what or how much is needed to get Cake mainlined?
> 
> I'd like us to give it a go when net-next reopens in two weeks,
> we'd then have 6 weeks or so to get it right.
> 
> We need:
> 
> * Someone to do the heavy lifting. Which I suspect would be me.
> * Someones with various hardware platforms that current kernels can be
>  run on. qemu?
> * I'd like to see the ack filtering work get tested on lede at low
>  bandwidths on dsl especially.
> * A whole lotta tests at various RTTs

I can offer some testing time, and can script or batch a range of RTTs. netns would be useful here. For completeness, I suggest a product of rrul_be runs:

Rates: 128 / 256 / 512Kbit, 1 / 2 / 4 / 8 / 16 / 32 / 64 / 128 / 256 / 512Mbit, 1Gbit

RTTs: 150 / 300 / 600us, 1 / 2 / 4 / 8 / 16 / 32 / 64 / 128 / 256 / 512 / 1024ms

Opinions? Some of those might be rough (I’m looking at you 128Kbit / 1024ms), but it would be good to know what happens. For hardware, I could turn my Mac Mini into a qemu box. I guess this list is about right: https://www.debian.org/releases/stable/i386/ch02s01.html.en <https://www.debian.org/releases/stable/i386/ch02s01.html.en>. I don’t know if all tests need to be tried on all platforms.

Testing could go much further, with host fairness, diffserv keywords, rtt settings (more on that later), overheads, nat, etc. We could also test underpowered hardware with rate limiting to see if it degrades gracefully. For sanity, we could just test a smattering of these things.

> Blockers:
> 
> * Ripping out all the backward compatability cruft for submission to
>  mainline and following netdev formatting conventions for comments and
>  indentation. I'd like any new features in the backport to get
>  backported, though (sigh), as lede looks to be shipping a 4.9 based
>  kernel.

Argh, but probably has to be done.

> * tc-cake man page needs to be updated.
> 
> * tc-adv related code updated to latest iproute2
> 
> * There is some work going on here to add ack filtering to cake, which
> looks VERY promising: https://github.com/dtaht/sch_cake/pull/63 <https://github.com/dtaht/sch_cake/pull/63>
> 
> I'm going to add something like this to netem also. It may be that
> merely leveraging the hash would be enough in cake's case.
> 
> * Testing against the net-next kernel on x86, x86_64, arm, mips, and
> aarch architectures. (I just got bit by not testing 32 bit arches, sigh)

Regarding the target and interval settings Cake uses, here are the current keywords available and their settings:

datacentre: 19us / 114us (us yanks might like ‘datacenter' as a synonym)
lan: 50us / 1ms
metro: 500us / 10ms
regional: 1.5ms / 30ms
internet: 5ms / 100ms
oceanic: 15ms / 300ms
satellite: 50ms / 1s
interplanetary: 5ms / 3600s

About a year ago I raised a concern that these values were outside what the CoDel authors intended. The counter-argument at the time was that experimentally, we can show that TCP RTT can be reduced on a Gbit LAN with the ‘lan’ keyword. And that argument seems to hold, so far. On two BQLd systems (2x PCEngines APU2s) connected with GigE, I can run the same experiment now and show that:

TCP RTT ~= 8ms with default qdisc, throughput ~= 940 Mbit
TCP RTT ~= 4.5ms with ‘cake unlimited’, throughput ~= 920 Mbit
TCP RTT ~= 1ms with ‘cake unlimited lan’, throughput ~= 920 Mbit

So yes, we can lower TCP RTT with these more aggressive settings. But just to make sure, we’re confident that there are no other side effects from these lower targets and intervals? Is there anything else I should test for to be sure? For example, when I rate limit to 950 Mbit and try the same test above, ‘lan’ causes a 20% drop in throughput vs the defaults. That may be from an overtaxed CPU, but I don’t know. I also wonder how this affects routed vs local traffic. I’ll try to test this at some point, as I want to understand it better anyway to know how backhaul links should be configured...

> Non-Blockers:
> 
> * I don't believe in cobalt, or rather, I won't believe in it until we
> have data at many RTTs. That said, what I'd propose would be a
> monolithic cobalt.h file rather than codel5.h.
> 
> The netns stuff will make simulating RTTs and bandwidths much easier….
> 
> * I think the fq_codel batch drop facility is better than what cake uses
> in case of floods. Partially due to the need to handle backports the
> mechanism fq_codel uses is hard to use - but going mainline we could add this.
> 
> * The autorate_ingress code should be marked experimental. I keep hoping
> it can be improved by better looking for "smoothness" inbound, but
> algorithms escape me. This doesn't bother me much, as tcp continues to
> be improved over the past 50 years, perhaps we can find ways to improve
> this with more users.
> 
> * It is possible to tune the quantum and peeling functions to not peel
> to the extent they do. Particularly there is usually no need (aside from
> wanting accurate statistics) to peel below 1500 bytes (except perhaps
> with the new ack filter mode). We experimented a lot with this in the
> early days but could never come to a resolution.
> 
> * I don't have any use for precidence mode and would like to remove it.

Regarding non-blockers, for FreeNet’s purposes, I wanted to see if I could add the option to use packet marks as one of the identifiers for host isolation, but I’ve not had time to explore it yet. This would be helpful for ISPs that want to ensure fairness when there isn’t a one-to-one mapping between IP address and customer. I’ll see if I can at least try it.

[-- Attachment #2: Type: text/html, Size: 41479 bytes --]

[Cake] Cake upstream Planning

[Dave Taht]

(note changed topic thread)

I dearly would like to try and submit cake to mainline linux in
december. Getting it done is going to take group effort.

And trying to cover all the corner cases, is going to take co-ordination
and scripting, and perhaps we should switch to google docs to pull together.

Also, it might be fun to schedule a dramatic reading of the source code
via videoconference because theres a lot in cake that not enough people
(except maybe jonathan) understand.

Pete Heist <peteheist@gmail.com> writes:

>     On Nov 14, 2017, at 9:10 PM, Dave Taht <dave@taht.net> wrote:
>
>     
>     Pete Heist <peteheist@gmail.com> writes:
>     
>         By the way, what or how much is needed to get Cake mainlined?
>         
>
>     I'd like us to give it a go when net-next reopens in two weeks,
>     we'd then have 6 weeks or so to get it right.
>     
>     We need:
>     
>     * Someone to do the heavy lifting. Which I suspect would be me.
>     * Someones with various hardware platforms that current kernels can be
>     run on. qemu?
>     * I'd like to see the ack filtering work get tested on lede at low
>     bandwidths on dsl especially.
>     * A whole lotta tests at various RTTs
>     
>
> I can offer some testing time, and can script or batch a range of RTTs. netns
> would be useful here. For completeness, I suggest a product of rrul_be runs:
>
> Rates: 128 / 256 / 512Kbit, 1 / 2 / 4 / 8 / 16 / 32 / 64 / 128 / 256 / 512Mbit,
> 1Gbit
>
> RTTs: 150 / 300 / 600us, 1 / 2 / 4 / 8 / 16 / 32 / 64 / 128 / 256 / 512 / 1024ms

Well, we need simple basic single tcp download tests, I would love to
also reuse the http and voip tests toke used in the first paper.

> Opinions? Some of those might be rough (I’m looking at you 128Kbit / 1024ms),
> but it would be good to know what happens. For hardware, I could turn my Mac
> Mini into a qemu box. I guess this list is about right:

Doing a few qemu setups would be good. In particular it helps with
letting us test a net-next kernel. If we could make available qemu
images all the better.

> https://www.debian.org/releases/stable/i386/ch02s01.html.en. I don’t know if all
> tests need to be tried on all platforms.

My principal requirement for multi-arch testing is that it "not crash"
and "compile". More direct testing - like with the mvneta and other odd
ethernet devices, kind of requires real hardware.

> Testing could go much further, with host fairness, diffserv keywords, rtt
> settings (more on that later), overheads, nat, etc. We could also test
> underpowered hardware with rate limiting to see if it degrades gracefully. For
> sanity, we could just test a smattering of these things.

This is a case where flent's batch facility would help. And we can divvy
up the load among servers using the new netns technique. Assuming I get
a bit of funding we can also grab some servers in the cloud, but I'm not
expecting that, so...

I do plan on getting a box to replace snapon also in this timeframe.

>
>     Blockers:
>     
>     * Ripping out all the backward compatability cruft for submission to
>     mainline and following netdev formatting conventions for comments and
>     indentation. I'd like any new features in the backport to get
>     backported, though (sigh), as lede looks to be shipping a 4.9 based
>     kernel.
>     
>
> Argh, but probably has to be done.

That turned out to not be hard. I'm about to test that result today.

Folding the result sanely back into the main repo did turn out to be
hard. I also have no idea how to fold together the cobalt and regular
cake branches at the moment, so I'm sticking with cobalt.

>
>     * tc-cake man page needs to be updated.
>     
>     * tc-adv related code updated to latest iproute2

I will start a repo for this.

>     * There is some work going on here to add ack filtering to cake, which
>     looks VERY promising: https://github.com/dtaht/sch_cake/pull/63
>     
>     I'm going to add something like this to netem also. It may be that
>     merely leveraging the hash would be enough in cake's case.
>     
>     * Testing against the net-next kernel on x86, x86_64, arm, mips, and
>     aarch architectures. (I just got bit by not testing 32 bit arches, sigh)
>     
>
> Regarding the target and interval settings Cake uses, here are the current
> keywords available and their settings:
>
> datacentre: 19us / 114us (us yanks might like ‘datacenter' as a synonym)
> lan: 50us / 1ms
> metro: 500us / 10ms
> regional: 1.5ms / 30ms
> internet: 5ms / 100ms
> oceanic: 15ms / 300ms
> satellite: 50ms / 1s
> interplanetary: 5ms / 3600s
>
> About a year ago I raised a concern that these values were outside what the
> CoDel authors intended. The counter-argument at the time was that
> experimentally, we can show that TCP RTT can be reduced on a Gbit LAN with the
> ‘lan’ keyword. And that argument seems to hold, so far. On two BQLd systems (2x
> PCEngines APU2s) connected with GigE, I can run the same experiment now and show
> that:
>
> TCP RTT ~= 8ms with default qdisc, throughput ~= 940 Mbit
> TCP RTT ~= 4.5ms with ‘cake unlimited’, throughput ~= 920 Mbit
> TCP RTT ~= 1ms with ‘cake unlimited lan’, throughput ~= 920 Mbit
>
> So yes, we can lower TCP RTT with these more aggressive settings. But just to
> make sure, we’re confident that there are no other side effects from these lower
> targets and intervals? Is there anything else I should test for to be sure? For
> example, when I rate limit to 950 Mbit and try the same test above, ‘lan’ causes
> a 20% drop in throughput vs the defaults. That may be from an overtaxed CPU, but
> I don’t know. I also wonder how this affects routed vs local traffic. I’ll try
> to test this at some point, as I want to understand it better anyway to know how
> backhaul links should be configured...
>
>     Non-Blockers:
>     
>     * I don't believe in cobalt, or rather, I won't believe in it until we
>     have data at many RTTs. That said, what I'd propose would be a
>     monolithic cobalt.h file rather than codel5.h.
>     
>     The netns stuff will make simulating RTTs and bandwidths much easier….
>
>     
>     * I think the fq_codel batch drop facility is better than what cake uses
>     in case of floods. Partially due to the need to handle backports the
>     mechanism fq_codel uses is hard to use - but going mainline we could add
>     this.
>     
>     * The autorate_ingress code should be marked experimental. I keep hoping
>     it can be improved by better looking for "smoothness" inbound, but
>     algorithms escape me. This doesn't bother me much, as tcp continues to
>     be improved over the past 50 years, perhaps we can find ways to improve
>     this with more users.
>     
>     * It is possible to tune the quantum and peeling functions to not peel
>     to the extent they do. Particularly there is usually no need (aside from
>     wanting accurate statistics) to peel below 1500 bytes (except perhaps
>     with the new ack filter mode). We experimented a lot with this in the
>     early days but could never come to a resolution.
>     
>     * I don't have any use for precidence mode and would like to remove it.
>
> Regarding non-blockers, for FreeNet’s purposes, I wanted to see if I could add
> the option to use packet marks as one of the identifiers for host isolation, but
> I’ve not had time to explore it yet. This would be helpful for ISPs that want to
> ensure fairness when there isn’t a one-to-one mapping between IP address and
> customer. I’ll see if I can at least try it.

Re: [Cake] Cake upstream Planning

[Dave Taht]

Dave Taht <dave@taht.net> writes:

>>
>> TCP RTT ~= 8ms with default qdisc, throughput ~= 940 Mbit
>> TCP RTT ~= 4.5ms with ‘cake unlimited’, throughput ~= 920 Mbit
>> TCP RTT ~= 1ms with ‘cake unlimited lan’, throughput ~= 920 Mbit

This was with BQL in play? Monitoring BQL's behavior might help.

I'd also love to know an exact setting for the shaper as a close as
possible to the underlying bandwidth of ethernet. However, I tend to be
plagued with

>>
>> So yes, we can lower TCP RTT with these more aggressive settings. But just to
>> make sure, we’re confident that there are no other side effects from these lower
>> targets and intervals? Is there anything else I should test for to be sure? For
>> example, when I rate limit to 950 Mbit and try the same test above, ‘lan’ causes
>> a 20% drop in throughput vs the defaults. That may be from an overtaxed CPU, but
>> I don’t know. I also wonder how this affects routed vs local traffic. I’ll try
>> to test this at some point, as I want to understand it better anyway to know how
>> backhaul links should be configured...

One interesting thing that might make tcp behave better is the new
pacing code which lets us set pacing to a different log value. Presently
- at 10 - the TSQ algorithm recalculates things at 1ms. The pacing value
is intended to be changed to, say, 6 or 7 to make wifi work
better... and I suspect, that if it were upped to, say 12 (250us), on ethernet,
that might make pacing more effective there.

Just like breaking the sound barrier, breaking the 1ms barrier looks to
be hard within conventional kernel thread scheduling.

>>
>>     Non-Blockers:
>>     
>>     * I don't believe in cobalt, or rather, I won't believe in it until we
>>     have data at many RTTs. That said, what I'd propose would be a
>>     monolithic cobalt.h file rather than codel5.h.
>>     
>>     The netns stuff will make simulating RTTs and bandwidths much easier….
>>
>>     
>>     * I think the fq_codel batch drop facility is better than what cake uses
>>     in case of floods. Partially due to the need to handle backports the
>>     mechanism fq_codel uses is hard to use - but going mainline we could add
>>     this.
>>     
>>     * The autorate_ingress code should be marked experimental. I keep hoping
>>     it can be improved by better looking for "smoothness" inbound, but
>>     algorithms escape me. This doesn't bother me much, as tcp continues to
>>     be improved over the past 50 years, perhaps we can find ways to improve
>>     this with more users.
>>     
>>     * It is possible to tune the quantum and peeling functions to not peel
>>     to the extent they do. Particularly there is usually no need (aside from
>>     wanting accurate statistics) to peel below 1500 bytes (except perhaps
>>     with the new ack filter mode). We experimented a lot with this in the
>>     early days but could never come to a resolution.
>>     
>>     * I don't have any use for precidence mode and would like to remove it.
>>
>> Regarding non-blockers, for FreeNet’s purposes, I wanted to see if I could add
>> the option to use packet marks as one of the identifiers for host isolation, but
>> I’ve not had time to explore it yet. This would be helpful for ISPs that want to
>> ensure fairness when there isn’t a one-to-one mapping between IP address and
>> customer. I’ll see if I can at least try it.
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Cake upstream Planning

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]


> On Nov 15, 2017, at 9:04 PM, Dave Taht <dave@taht.net> wrote:
> 
> Dave Taht <dave@taht.net <mailto:dave@taht.net>> writes:
> 
>>> TCP RTT ~= 8ms with default qdisc, throughput ~= 940 Mbit
>>> TCP RTT ~= 4.5ms with ‘cake unlimited’, throughput ~= 920 Mbit
>>> TCP RTT ~= 1ms with ‘cake unlimited lan’, throughput ~= 920 Mbit
> 
> This was with BQL in play? Monitoring BQL's behavior might help.
> 
> I'd also love to know an exact setting for the shaper as a close as
> possible to the underlying bandwidth of ethernet. However, I tend to be
> plagued with

Yes, with BQL (Intel I210 with igb driver on the APU2). An rrul_be test with —socket-stats.


[-- Attachment #2: Type: text/html, Size: 5925 bytes --]

Re: [Cake] Cake upstream Planning

[Dave Taht]

Pete Heist <peteheist@gmail.com> writes:

>     On Nov 15, 2017, at 9:04 PM, Dave Taht <dave@taht.net> wrote:
>
>     
>     Dave Taht <dave@taht.net> writes:
>     
>                 TCP RTT ~= 8ms with default qdisc, throughput ~= 940 Mbit
>             TCP RTT ~= 4.5ms with ‘cake unlimited’, throughput ~= 920 Mbit
>             TCP RTT ~= 1ms with ‘cake unlimited lan’, throughput ~= 920 Mbit
>             
>
>     This was with BQL in play? Monitoring BQL's behavior might help.
>     
>     I'd also love to know an exact setting for the shaper as a close as
>     possible to the underlying bandwidth of ethernet. However, I tend to be
>     plagued with
>     
>
> Yes, with BQL (Intel I210 with igb driver on the APU2). An rrul_be test with —
> socket-stats.

https://github.com/ffainelli/bqlmon was a tool for looking at bql more
directly.

I had forked it for some reason or another:

https://github.com/dtaht/bqlmon

Re: [Cake] Cake upstream Planning

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1145 bytes --]


> On Nov 15, 2017, at 8:44 PM, Dave Taht <dave@taht.net> wrote:
> 
> I dearly would like to try and submit cake to mainline linux in
> december. Getting it done is going to take group effort.
> 
> And trying to cover all the corner cases, is going to take co-ordination
> and scripting, and perhaps we should switch to google docs to pull together.
> 
> Also, it might be fun to schedule a dramatic reading of the source code
> via videoconference because theres a lot in cake that not enough people
> (except maybe jonathan) understand.

That sounds good. Rather than my making some test plan, if you start a doc just plug me in where you think it’s best.

There’s also been a Cake support feature request hanging around in the EdgeOS forums for a while after Lochnair’s successful work to get it built for the EdgeRouter firmware:

https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/Cake-shaper-support/idi-p/1885749 <https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/Cake-shaper-support/idi-p/1885749>

Maybe this would help get it pushed through into a device that I think has pretty wide deployment…


[-- Attachment #2: Type: text/html, Size: 1872 bytes --]

Re: [Cake] Cake upstream Planning

[Dave Taht]

Pete Heist <peteheist@gmail.com> writes:

>     On Nov 15, 2017, at 8:44 PM, Dave Taht <dave@taht.net> wrote:
>
>     
>     
>     I dearly would like to try and submit cake to mainline linux in
>     december. Getting it done is going to take group effort.
>     
>     And trying to cover all the corner cases, is going to take co-ordination
>     and scripting, and perhaps we should switch to google docs to pull together.
>     
>     Also, it might be fun to schedule a dramatic reading of the source code
>     via videoconference because theres a lot in cake that not enough people
>     (except maybe jonathan) understand.
>     
>
> That sounds good. Rather than my making some test plan, if you start a doc just
> plug me in where you think it’s best.

I'll try to review what's been discussed so far and get a document out
by sunday.

BTW, all, I have reserved time on the "vuc" conference december 15th to
discuss where we are in the bufferbloat project. That's more of a "meet
the press" kind of thing than the internal discussion here.

>
> There’s also been a Cake support feature request hanging around in the EdgeOS
> forums for a while after Lochnair’s successful work to get it built for the
> EdgeRouter firmware:
>
> https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/Cake-shaper-support/idi-p/1885749

I had viewed those folk as a key path to commercialization. They have
a great user community and great products.

> Maybe this would help get it pushed through into a device that I think has
> pretty wide deployment…

The problem is, they were stuck on kernel 3.10, when last I looked and
oy...

I keep hoping we can get cavium to pay attention directly.

Re: [Cake] Cake upstream Planning

[Nils Andreas Svee]

On Wed, Nov 15, 2017, at 21:19, Dave Taht wrote:
> 
> > There’s also been a Cake support feature request hanging around in the EdgeOS
> > forums for a while after Lochnair’s successful work to get it built for the
> > EdgeRouter firmware:
> >
> > https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/Cake-shaper-support/idi-p/1885749
> 
> I had viewed those folk as a key path to commercialization. They have
> a great user community and great products.
> 
> > Maybe this would help get it pushed through into a device that I think has
> > pretty wide deployment…
> 
> The problem is, they were stuck on kernel 3.10, when last I looked and
> oy...
> 
> I keep hoping we can get cavium to pay attention directly.
Sadly they're still stuck 3.10. The next FW will be based on the
3.10.107 kernel, instead of 3.10.14/3.10.20. So that's something, but
it's still ancient and EOL.

Re: [Cake] Cake upstream Planning

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1404 bytes --]


> On Nov 15, 2017, at 9:28 PM, Nils Andreas Svee <nils@stokkdalen.no> wrote:
> 
> On Wed, Nov 15, 2017, at 21:19, Dave Taht wrote:
>> 
>>> There’s also been a Cake support feature request hanging around in the EdgeOS
>>> forums for a while after Lochnair’s successful work to get it built for the
>>> EdgeRouter firmware:
>>> 
>>> https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/Cake-shaper-support/idi-p/1885749
>> 
>> I had viewed those folk as a key path to commercialization. They have
>> a great user community and great products.
>> 
>>> Maybe this would help get it pushed through into a device that I think has
>>> pretty wide deployment…
>> 
>> The problem is, they were stuck on kernel 3.10, when last I looked and
>> oy...
>> 
>> I keep hoping we can get cavium to pay attention directly.
> Sadly they're still stuck 3.10. The next FW will be based on the
> 3.10.107 kernel, instead of 3.10.14/3.10.20. So that's something, but
> it's still ancient and EOL.

Is the tie to 3.10 due to the dependency on the Cavium SDK for hardware support?

Surprisingly I don’t see a feature request so far for a newer kernel, at least among the first half dozen pages or so:

https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/idb-p/EdgeMAX_Ideas/tab/most-kudoed <https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/idb-p/EdgeMAX_Ideas/tab/most-kudoed>


[-- Attachment #2: Type: text/html, Size: 2421 bytes --]

Re: [Cake] Cake upstream Planning

[Nils Andreas Svee]

[-- Attachment #1: Type: text/plain, Size: 1715 bytes --]

Mostly because of the Cavium and MediaTek SDKs yes. However in my quest
to use a newer kernel I found that EdgeOS/Vyatta uses UnionFS which is
not supported on newer kernels than 3.16, so that's gotta be fixed first
too. In VyOS they've solved this by using UnionFS-Fuse.
On Wed, Nov 15, 2017, at 21:58, Pete Heist wrote:
> 
>> On Nov 15, 2017, at 9:28 PM, Nils Andreas Svee
>> <nils@stokkdalen.no> wrote:>> 
>> On Wed, Nov 15, 2017, at 21:19, Dave Taht wrote:
>>> 
>>>> There’s also been a Cake support feature request hanging around in
>>>> the EdgeOS>>>> forums for a while after Lochnair’s successful work to get it built
>>>> for the>>>> EdgeRouter firmware:
>>>> 
>>>> https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/Cake-shaper-support/idi-p/1885749>>> 
>>> I had viewed those folk as a key path to commercialization.
>>> They have>>> a great user community and great products.
>>> 
>>>> Maybe this would help get it pushed through into a device that I
>>>> think has>>>> pretty wide deployment…
>>> 
>>> The problem is, they were stuck on kernel 3.10, when last I
>>> looked and>>> oy...
>>> 
>>> I keep hoping we can get cavium to pay attention directly.
>> Sadly they're still stuck 3.10. The next FW will be based on the
>> 3.10.107 kernel, instead of 3.10.14/3.10.20. So that's something, but>> it's still ancient and EOL.
> 
> Is the tie to 3.10 due to the dependency on the Cavium SDK for
> hardware support?> 
> Surprisingly I don’t see a feature request so far for a newer kernel,
> at least among the first half dozen pages or so:> 
> https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/idb-p/EdgeMAX_Ideas/tab/most-kudoed> 
Best Regards
Nils


[-- Attachment #2: Type: text/html, Size: 2771 bytes --]

Re: [Cake] Cake upstream Planning

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2262 bytes --]

Thanks for that info. I went ahead and added a feature request, in case anyway wants to add some kudos to it:

https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/Upgrade-Linux-kernel-to-at-least-4-4/idi-p/2140663

> On Nov 15, 2017, at 11:40 PM, Nils Andreas Svee <me@lochnair.net> wrote:
> 
> Mostly because of the Cavium and MediaTek SDKs yes. However in my quest to use a newer kernel I found that EdgeOS/Vyatta uses UnionFS which is not supported on newer kernels than 3.16, so that's gotta be fixed first too. In VyOS they've solved this by using UnionFS-Fuse.
> 
> On Wed, Nov 15, 2017, at 21:58, Pete Heist wrote:
>> 
>>> On Nov 15, 2017, at 9:28 PM, Nils Andreas Svee <nils@stokkdalen.no <mailto:nils@stokkdalen.no>> wrote:
>>> 
>>> On Wed, Nov 15, 2017, at 21:19, Dave Taht wrote:
>>>> 
>>>>> There’s also been a Cake support feature request hanging around in the EdgeOS
>>>>> forums for a while after Lochnair’s successful work to get it built for the
>>>>> EdgeRouter firmware:
>>>>> 
>>>>> https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/Cake-shaper-support/idi-p/1885749 <https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/Cake-shaper-support/idi-p/1885749>
>>>> 
>>>> I had viewed those folk as a key path to commercialization. They have
>>>> a great user community and great products.
>>>> 
>>>>> Maybe this would help get it pushed through into a device that I think has
>>>>> pretty wide deployment…
>>>> 
>>>> The problem is, they were stuck on kernel 3.10, when last I looked and
>>>> oy...
>>>> 
>>>> I keep hoping we can get cavium to pay attention directly.
>>> Sadly they're still stuck 3.10. The next FW will be based on the
>>> 3.10.107 kernel, instead of 3.10.14/3.10.20. So that's something, but
>>> it's still ancient and EOL.
>> 
>> 
>> Is the tie to 3.10 due to the dependency on the Cavium SDK for hardware support?
>> 
>> Surprisingly I don’t see a feature request so far for a newer kernel, at least among the first half dozen pages or so:
>> 
>> https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/idb-p/EdgeMAX_Ideas/tab/most-kudoed <https://community.ubnt.com/t5/EdgeMAX-Feature-Requests/idb-p/EdgeMAX_Ideas/tab/most-kudoed>
>> 
> Best Regards
> Nils
> 


[-- Attachment #2: Type: text/html, Size: 4433 bytes --]