[Cake] cake in dd-wrt

[Cake] cake in dd-wrt

[Dave Taht]

https://svn.dd-wrt.com/ticket/5796

-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

:-) i'm following this list and yes we are working on bringing cake in :-)
is there any question behind this link from your side?

Sebastian

Am 18.08.2019 um 18:33 schrieb Dave Taht:
> https://svn.dd-wrt.com/ticket/5796
>

Re: [Cake] cake in dd-wrt

[Dave Taht]

On Tue, Aug 20, 2019 at 5:09 AM Sebastian Gottschall
<s.gottschall@newmedia-net.de> wrote:
>
> :-) i'm following this list and yes we are working on bringing cake in :-)

Yea! thx for being on the list!

> is there any question behind this link from your side?

I just wanted to make people here aware that it was happening.

Is there a build now?

If I had any one principal request it would be to make sure the dd-wrt
gui (if one is made) exposes the link layer parameters. Getting the
framing wrong is about the biggest error I see in the deployment:
https://www.youtube.com/watch?v=LjJW_s5gQ9Y

Other nifty cake features:

"wash" and "besteffort" are important on some cablecos that remark
traffic to CS1
"nat" is dang useful if you are natting
ack-filter helps on really slow asymmetric networks on the slow side only.

So, like, my defaults would be

in: nat wash besteffort # triple-isolate bandwidth X etc
out: nat ack-filter # if > 10x1 down/up ratio

And make sure the link layer settings are exposed in the gui. I really
don't know how much
"washing" is needed outside the cablecos, taking packet captures of
various isps to see how often
dscp bits are mangled nowadays would be good. besteffort on inbound
saves some on cpu.

Are you using the out of tree version or mainline? Out of tree has
some experimental SCE work
that I'd love to see tested at more scale but not actually shipped at this time.

Due to how cpu intensive cake can be (on inbound) I've been working on
a faster, less feature-full fq_codel, it's here:

https://github.com/dtaht/fq_codel_fast

I hate the idea of fq_codel one way, cake the other, but tbf +
fq_codel_fast seems to work well at
~1gbit on my apu2 and cake doesn't. I'd originally planned to try and
make a multi-core shaper out
of it, but sce distracted me....

Having more folk on board benching stuff on modern non-x86 platforms
would be good.

Another cake feature is that you can get all the benefits on a normal
ethernet (with bql) *without turning the shaper on* but we have not
benchmarked that either vs a vs fq_codel or fq_codel_fast

Have fun!

Here's the first paper: https://arxiv.org/pdf/1804.07617.pdf

> Sebastian
>
> Am 18.08.2019 um 18:33 schrieb Dave Taht:
> > https://svn.dd-wrt.com/ticket/5796
> >



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

Am 20.08.2019 um 18:24 schrieb Dave Taht:
> On Tue, Aug 20, 2019 at 5:09 AM Sebastian Gottschall
> <s.gottschall@newmedia-net.de> wrote:
>> :-) i'm following this list and yes we are working on bringing cake in :-)
> Yea! thx for being on the list!
>
>> is there any question behind this link from your side?
> I just wanted to make people here aware that it was happening.
>
> Is there a build now?
the first builds with cake are already out yes, but unfinished. we 
started then to rewrite major parts of the qos code. i expect to push out
a new build tomorrow. it will still not use the full potential of cake 
since we have to bring all together with the priority and ndpi and 
filter based filter together
with the cake scheduler. but it works so far and will be better in the 
next days and weeks after we have found a solution for it. cake is more 
a all in wonder solution
so we have to use the features of cake itself to reimplement the 
priorisation classes
>
> If I had any one principal request it would be to make sure the dd-wrt
> gui (if one is made) exposes the link layer parameters. Getting the
> framing wrong is about the biggest error I see in the deployment:
> https://www.youtube.com/watch?v=LjJW_s5gQ9Y
i have seen this already. out plan here is that the user specifies the 
internet connection type like vdsl2, cable, whatever in case of cake 
which then will be used
as argument
>
> Other nifty cake features:
>
> "wash" and "besteffort" are important on some cablecos that remark
> traffic to CS1
> "nat" is dang useful if you are natting
> ack-filter helps on really slow asymmetric networks on the slow side only.
>
> So, like, my defaults would be
>
> in: nat wash besteffort # triple-isolate bandwidth X etc
> out: nat ack-filter # if > 10x1 down/up ratio
my finding for damn slow internet < 2000 kbit upstream that we need to 
handle rtt like codel with the formula

rtt = 50 - (uprate / 50)

which gives a rtt of 10 at 2000 kbit. otherwise its not working good. 
for the rest i will consider your defaults for our tests.

> And make sure the link layer settings are exposed in the gui. I really
> don't know how much
> "washing" is needed outside the cablecos, taking packet captures of
> various isps to see how often
> dscp bits are mangled nowadays would be good. besteffort on inbound
> saves some on cpu.
dscp might be problematic for isps like orange in france and other fiber 
isps i have found in spain and netherlands.
all these isps expect dscp classes to be set for outbound traffic 
depending on the traffic type. (iptv, voice and internet). this together 
is also
mapped to vlan priorities again internally. so using the dscp bits might 
be problematic for cake since it clears out the dscp flags and so 
internet simply doesnt work anymore
for these isps (or just with modem 56k speed)
>
> Are you using the out of tree version or mainline? Out of tree has
> some experimental SCE work
> that I'd love to see tested at more scale but not actually shipped at this time.
out of tree straight from git with modifications to be compatible to my 
kernels since your compatiblity layer is mmh not perfect.
some compat layer you made doesnt work since they already got backported 
to the latest upstream kernel versions. so just some refinement.
nothing special or much work
>
> Due to how cpu intensive cake can be (on inbound) I've been working on
> a faster, less feature-full fq_codel, it's here:
>
> https://github.com/dtaht/fq_codel_fast
i reviewed it and also tried to compile it with success. problem is, 
that code parts of it are looking unfinished with debug printks etc.
this is why i stopped at that point since i thought its unfinished. but 
i have added it already to my code tree and added the required parts to 
compile
it with all of my kernels (compat layer for older kernels or lets say 
#ifdef hell)
>
> I hate the idea of fq_codel one way, cake the other, but tbf +
> fq_codel_fast seems to work well at
> ~1gbit on my apu2 and cake doesn't. I'd originally planned to try and
> make a multi-core shaper out
> of it, but sce distracted me....
multicore would be nice since alot of home routers are coming with up to 
4 native cpus right now. (ipq8065 has 1.8 ghz and dual core. ipq40xx has 
4 cores but just 700 mhz and is 4 times slower per core and mhz, so it 
makes sense here)
i'm using a APU as internet router as well btw
>
> Having more folk on board benching stuff on modern non-x86 platforms
> would be good.
yeah no problem. we mainly test on a gateworks laguna armv6 platform in 
my company, but most reponse will come from broadcom arm northstar and 
qca ipq platforms from the community
but i hope i also get feedback from slower platforms like mips ar7240 etc
> Another cake feature is that you can get all the benefits on a normal
> ethernet (with bql) *without turning the shaper on* but we have not
> benchmarked that either vs a vs fq_codel or fq_codel_fast
from the response i got from the early experiments is that cake performs 
much better in typical bufferfloat tests than fq_codel. but i will see 
how my latest version now works

which fully integrated it.


Sebastian

>
> Have fun!
>
> Here's the first paper: https://arxiv.org/pdf/1804.07617.pdf
>
>> Sebastian
>>
>> Am 18.08.2019 um 18:33 schrieb Dave Taht:
>>> https://svn.dd-wrt.com/ticket/5796
>>>
>
>

Re: [Cake] cake in dd-wrt

[Toke Høiland-Jørgensen]

Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:

> Am 20.08.2019 um 18:24 schrieb Dave Taht:
>> On Tue, Aug 20, 2019 at 5:09 AM Sebastian Gottschall
>> <s.gottschall@newmedia-net.de> wrote:
>>> :-) i'm following this list and yes we are working on bringing cake in :-)
>> Yea! thx for being on the list!
>>
>>> is there any question behind this link from your side?
>> I just wanted to make people here aware that it was happening.
>>
>> Is there a build now?
> the first builds with cake are already out yes, but unfinished. we 
> started then to rewrite major parts of the qos code. i expect to push out
> a new build tomorrow. it will still not use the full potential of cake 
> since we have to bring all together with the priority and ndpi and 
> filter based filter together
> with the cake scheduler.

Are you aware that you can use the tc filtering functionality to make
this play along with cake's tiers?

>> Are you using the out of tree version or mainline? Out of tree has
>> some experimental SCE work
>> that I'd love to see tested at more scale but not actually shipped at this time.
> out of tree straight from git with modifications to be compatible to my 
> kernels since your compatiblity layer is mmh not perfect.

Please do feel free to send a pull request with your fixes for the
compatibility stuff! :)

-Toke

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

Am 20.08.2019 um 18:58 schrieb Toke Høiland-Jørgensen:
> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>
>> Am 20.08.2019 um 18:24 schrieb Dave Taht:
>>> On Tue, Aug 20, 2019 at 5:09 AM Sebastian Gottschall
>>> <s.gottschall@newmedia-net.de> wrote:
>>>> :-) i'm following this list and yes we are working on bringing cake in :-)
>>> Yea! thx for being on the list!
>>>
>>>> is there any question behind this link from your side?
>>> I just wanted to make people here aware that it was happening.
>>>
>>> Is there a build now?
>> the first builds with cake are already out yes, but unfinished. we
>> started then to rewrite major parts of the qos code. i expect to push out
>> a new build tomorrow. it will still not use the full potential of cake
>> since we have to bring all together with the priority and ndpi and
>> filter based filter together
>> with the cake scheduler.
> Are you aware that you can use the tc filtering functionality to make
> this play along with cake's tiers?
we are already using filters. yes. its just that cake is acting always 
as root and we have different sorts of qos configurations. so you have 
wan. but we may have multiple lan interfaces with individual qos 
settings. the same for mac / ip based user settings. so in fact we need 
to create a individual qdisc for each of these setting types in worst 
case, but in that case we cannot take in account the global available 
bandwidth anymore.
>
>>> Are you using the out of tree version or mainline? Out of tree has
>>> some experimental SCE work
>>> that I'd love to see tested at more scale but not actually shipped at this time.
>> out of tree straight from git with modifications to be compatible to my
>> kernels since your compatiblity layer is mmh not perfect.
> Please do feel free to send a pull request with your fixes for the
> compatibility stuff! :)
will do after i have taken out specific mods for my kernels again. its 
really nothing much. just some kernel minor versions for 3.18 and 4.4 
got some features backported which collided with the cobalt header.
i'm now already ahead with doing a out of tree version of fq_codel_fast 
for multiple kernel versions
>
> -Toke
>

Re: [Cake] cake in dd-wrt

[Toke Høiland-Jørgensen]

Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:

> Am 20.08.2019 um 18:58 schrieb Toke Høiland-Jørgensen:
>> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>>
>>> Am 20.08.2019 um 18:24 schrieb Dave Taht:
>>>> On Tue, Aug 20, 2019 at 5:09 AM Sebastian Gottschall
>>>> <s.gottschall@newmedia-net.de> wrote:
>>>>> :-) i'm following this list and yes we are working on bringing cake in :-)
>>>> Yea! thx for being on the list!
>>>>
>>>>> is there any question behind this link from your side?
>>>> I just wanted to make people here aware that it was happening.
>>>>
>>>> Is there a build now?
>>> the first builds with cake are already out yes, but unfinished. we
>>> started then to rewrite major parts of the qos code. i expect to push out
>>> a new build tomorrow. it will still not use the full potential of cake
>>> since we have to bring all together with the priority and ndpi and
>>> filter based filter together
>>> with the cake scheduler.
>> Are you aware that you can use the tc filtering functionality to make
>> this play along with cake's tiers?
> we are already using filters. yes. its just that cake is acting always 
> as root and we have different sorts of qos configurations. so you have 
> wan. but we may have multiple lan interfaces with individual qos 
> settings. the same for mac / ip based user settings. so in fact we need 
> to create a individual qdisc for each of these setting types in worst 
> case, but in that case we cannot take in account the global available 
> bandwidth anymore.

Ah, right, I see. So this is things like users wanting to limit a
specific type of traffic to a certain bandwidth?

>>>> Are you using the out of tree version or mainline? Out of tree has
>>>> some experimental SCE work
>>>> that I'd love to see tested at more scale but not actually shipped at this time.
>>> out of tree straight from git with modifications to be compatible to my
>>> kernels since your compatiblity layer is mmh not perfect.
>> Please do feel free to send a pull request with your fixes for the
>> compatibility stuff! :)
> will do after i have taken out specific mods for my kernels again.

Great, thanks!

-Toke

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

>> we are already using filters. yes. its just that cake is acting always
>> as root and we have different sorts of qos configurations. so you have
>> wan. but we may have multiple lan interfaces with individual qos
>> settings. the same for mac / ip based user settings. so in fact we need
>> to create a individual qdisc for each of these setting types in worst
>> case, but in that case we cannot take in account the global available
>> bandwidth anymore.
> Ah, right, I see. So this is things like users wanting to limit a
> specific type of traffic to a certain bandwidth?
basicly yes. there are multiple ways. plain traffic shaping by local 
interface name, by local mac, by local ip/net
and in addition there is shaping by port based or dpi based packet 
detection and since each instance of cake doesnt know of any other
use of cake qdiscs its getting complicated. but we just started with 
working on it. i'm sure i find a solution for it


Sebastian

Re: [Cake] cake in dd-wrt

[Toke Høiland-Jørgensen]

Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:

>>> we are already using filters. yes. its just that cake is acting always
>>> as root and we have different sorts of qos configurations. so you have
>>> wan. but we may have multiple lan interfaces with individual qos
>>> settings. the same for mac / ip based user settings. so in fact we need
>>> to create a individual qdisc for each of these setting types in worst
>>> case, but in that case we cannot take in account the global available
>>> bandwidth anymore.
>> Ah, right, I see. So this is things like users wanting to limit a
>> specific type of traffic to a certain bandwidth?
> basicly yes. there are multiple ways. plain traffic shaping by local 
> interface name, by local mac, by local ip/net
> and in addition there is shaping by port based or dpi based packet 
> detection and since each instance of cake doesnt know of any other
> use of cake qdiscs its getting complicated. but we just started with 
> working on it. i'm sure i find a solution for it

Do let us know if you do :)

However, I'd also point out that when running CAKE a lot of these kinds
of setups become simply redundant. For home networks most of the setups
I have seen with such rule-based shaping is simply there to paper over
the underlying bufferbloat issue. Once you solve that you don't really
need all the policy-based stuff.

Now, there are of course exceptions to this where a strict rule-based
shaping *is* really needed; but HTB already provides this in the kernel,
and we don't want to re-invent that, so I'm not sure we'll ever support
this properly in CAKE, sadly...

-Toke

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

Am 20.08.2019 um 20:31 schrieb Toke Høiland-Jørgensen:
> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>
>>>> we are already using filters. yes. its just that cake is acting always
>>>> as root and we have different sorts of qos configurations. so you have
>>>> wan. but we may have multiple lan interfaces with individual qos
>>>> settings. the same for mac / ip based user settings. so in fact we need
>>>> to create a individual qdisc for each of these setting types in worst
>>>> case, but in that case we cannot take in account the global available
>>>> bandwidth anymore.
>>> Ah, right, I see. So this is things like users wanting to limit a
>>> specific type of traffic to a certain bandwidth?
>> basicly yes. there are multiple ways. plain traffic shaping by local
>> interface name, by local mac, by local ip/net
>> and in addition there is shaping by port based or dpi based packet
>> detection and since each instance of cake doesnt know of any other
>> use of cake qdiscs its getting complicated. but we just started with
>> working on it. i'm sure i find a solution for it
> Do let us know if you do :)
>
> However, I'd also point out that when running CAKE a lot of these kinds
> of setups become simply redundant. For home networks most of the setups
> I have seen with such rule-based shaping is simply there to paper over
> the underlying bufferbloat issue. Once you solve that you don't really
> need all the policy-based stuff.
its not just about policy to get all managed. but the point is that a 
heavy bittorrent downloader will still steal the bandwidth of my scp 
session.
so its about control and not just about the flow management
is about limiting ports to a specific bandwidth. for instance. i have a 
concert venue and i limit the backstage network to a certain maximum 
rate since a need a budged for other networks
so i limit the ethernet port of this network on the main router to lets 
say 10 mbit or something like that priorize torrent and other bad 
services to bulk. which just works good for internet.
so we have enough bandwidth on our other cables for doing 4k streams.
dd-wrt is not just used on these plastic routers for home users. this is 
one option and works great without much qos configuration. you're right. 
but if its turning more complex and professional
its not enough anymore.
> Now, there are of course exceptions to this where a strict rule-based
> shaping *is* really needed; but HTB already provides this in the kernel,
> and we don't want to re-invent that, so I'm not sure we'll ever support
> this properly in CAKE, sadly...
this is what we are also doing. cake is finally just a option. you can 
select multiple schedulers at the gui. including codel. fq_codel, 
fq_codel_fast, cake , pie etc.
>
> -Toke
>

Re: [Cake] cake in dd-wrt

[Toke Høiland-Jørgensen]

Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:

> Am 20.08.2019 um 20:31 schrieb Toke Høiland-Jørgensen:
>> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>>
>>>>> we are already using filters. yes. its just that cake is acting always
>>>>> as root and we have different sorts of qos configurations. so you have
>>>>> wan. but we may have multiple lan interfaces with individual qos
>>>>> settings. the same for mac / ip based user settings. so in fact we need
>>>>> to create a individual qdisc for each of these setting types in worst
>>>>> case, but in that case we cannot take in account the global available
>>>>> bandwidth anymore.
>>>> Ah, right, I see. So this is things like users wanting to limit a
>>>> specific type of traffic to a certain bandwidth?
>>> basicly yes. there are multiple ways. plain traffic shaping by local
>>> interface name, by local mac, by local ip/net
>>> and in addition there is shaping by port based or dpi based packet
>>> detection and since each instance of cake doesnt know of any other
>>> use of cake qdiscs its getting complicated. but we just started with
>>> working on it. i'm sure i find a solution for it
>> Do let us know if you do :)
>>
>> However, I'd also point out that when running CAKE a lot of these kinds
>> of setups become simply redundant. For home networks most of the setups
>> I have seen with such rule-based shaping is simply there to paper over
>> the underlying bufferbloat issue. Once you solve that you don't really
>> need all the policy-based stuff.
> its not just about policy to get all managed. but the point is that a 
> heavy bittorrent downloader will still steal the bandwidth of my scp 
> session.

This is indeed a concern sometimes, yeah, and actually this exact case
is the original motivation for the host-based fairness feature in CAKE :)

> so its about control and not just about the flow management
> is about limiting ports to a specific bandwidth. for instance. i have a 
> concert venue and i limit the backstage network to a certain maximum 
> rate since a need a budged for other networks
> so i limit the ethernet port of this network on the main router to lets 
> say 10 mbit or something like that priorize torrent and other bad 
> services to bulk. which just works good for internet.
> so we have enough bandwidth on our other cables for doing 4k streams.
> dd-wrt is not just used on these plastic routers for home users. this is 
> one option and works great without much qos configuration. you're right. 
> but if its turning more complex and professional
> its not enough anymore.

Sure, but those are not CAKE's target audience, so to speak.

I'm not saying those use cases are not legitimate, I'm just saying you
may find it difficult to shoe-horn CAKE into them; which is kinda
deliberate...

>> Now, there are of course exceptions to this where a strict rule-based
>> shaping *is* really needed; but HTB already provides this in the kernel,
>> and we don't want to re-invent that, so I'm not sure we'll ever support
>> this properly in CAKE, sadly...
> this is what we are also doing. cake is finally just a option. you can 
> select multiple schedulers at the gui. including codel. fq_codel, 
> fq_codel_fast, cake , pie etc.

Right. What are you setting as the default, BTW?

-Toke

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

e of course exceptions to this where a strict rule-based
>>> shaping *is* really needed; but HTB already provides this in the kernel,
>>> and we don't want to re-invent that, so I'm not sure we'll ever support
>>> this properly in CAKE, sadly...
>> this is what we are also doing. cake is finally just a option. you can
>> select multiple schedulers at the gui. including codel. fq_codel,
>> fq_codel_fast, cake , pie etc.
> Right. What are you setting as the default, BTW?
default QOS is off :-) and the old historical default settings are htb 
with sfq
>
> -Toke
>

Re: [Cake] cake in dd-wrt

[Jonathan Morton]

> On 20 Aug, 2019, at 9:39 pm, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
> 
> …a heavy bittorrent downloader will still steal the bandwidth of my scp session.

If you can identify the Bittorrent packets, you can mark them CS1, and switch on Cake's "diffserv3" mode (as it is by default).  Then the Bittorrent packets will still be able to use full bandwidth if it's available, but will be limited to 1/16th of the total if there is contention.

 - Jonathan Morton

Re: [Cake] cake in dd-wrt

[Dave Taht]

Jonathan Morton <chromatix99@gmail.com> writes:

>> On 20 Aug, 2019, at 9:39 pm, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>> 
>> …a heavy bittorrent downloader will still steal the bandwidth of my scp session.
>
> If you can identify the Bittorrent packets, you can mark them CS1, and
> switch on Cake's "diffserv3" mode (as it is by default).  Then the
> Bittorrent packets will still be able to use full bandwidth if it's
> available, but will be limited to 1/16th of the total if there is
> contention.

I regard the whole CS1 thing as having never been particularly
successful for a variety of reasons - in particular because
we seemed to be the only ones attempting to use it with rigor.

I would like to patch in and submit "LE" support to mainline cake.

The RFC retires CS1 - which I wouldn't retire - but see:

https://www.rfc-editor.org/rfc/rfc8622.html

Also it seems like a good idea to also submit the NS bit
exclusion from the ack filter to mainline as well.

>
>  - Jonathan Morton
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake in dd-wrt

[Toke Høiland-Jørgensen]

Dave Taht <dave@taht.net> writes:

> Jonathan Morton <chromatix99@gmail.com> writes:
>
>>> On 20 Aug, 2019, at 9:39 pm, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>>> 
>>> …a heavy bittorrent downloader will still steal the bandwidth of my scp session.
>>
>> If you can identify the Bittorrent packets, you can mark them CS1, and
>> switch on Cake's "diffserv3" mode (as it is by default).  Then the
>> Bittorrent packets will still be able to use full bandwidth if it's
>> available, but will be limited to 1/16th of the total if there is
>> contention.
>
> I regard the whole CS1 thing as having never been particularly
> successful for a variety of reasons - in particular because
> we seemed to be the only ones attempting to use it with rigor.
>
> I would like to patch in and submit "LE" support to mainline cake.
>
> The RFC retires CS1 - which I wouldn't retire - but see:
>
> https://www.rfc-editor.org/rfc/rfc8622.html

Yeah, getting support for that upstream might be a good idea :)

> Also it seems like a good idea to also submit the NS bit
> exclusion from the ack filter to mainline as well.

What's that?

-Toke

Re: [Cake] cake in dd-wrt

[Sebastian Moeller]

> On Aug 21, 2019, at 12:21, Toke Høiland-Jørgensen <toke@redhat.com> wrote:
> 
> Dave Taht <dave@taht.net> writes:
> 
>> Jonathan Morton <chromatix99@gmail.com> writes:
>> 
>>>> On 20 Aug, 2019, at 9:39 pm, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>>>> 
>>>> …a heavy bittorrent downloader will still steal the bandwidth of my scp session.
>>> 
>>> If you can identify the Bittorrent packets, you can mark them CS1, and
>>> switch on Cake's "diffserv3" mode (as it is by default).  Then the
>>> Bittorrent packets will still be able to use full bandwidth if it's
>>> available, but will be limited to 1/16th of the total if there is
>>> contention.
>> 
>> I regard the whole CS1 thing as having never been particularly
>> successful for a variety of reasons - in particular because
>> we seemed to be the only ones attempting to use it with rigor.
>> 
>> I would like to patch in and submit "LE" support to mainline cake.
>> 
>> The RFC retires CS1 - which I wouldn't retire - but see:

Does it really do that? I see a section requesting domains using CS1 to remark to LE on egress, which, given that hardware often treats CS1 to higher priority than CS0 seems like the right thing to do... I also see changes to all RFCs that recommended CS1 as LE-DSCP, but it specifically caters to dscp domains that actively use CS1. The bigger issue I see in the request to never bleach/nor re-mark 000001, but if this can be achieved for any codepoint the for LE, assuming every domain owner might actually be interested to use this information for dropping/queueing decisions, no?

Best Regards
	Sebastian


>> 
>> https://www.rfc-editor.org/rfc/rfc8622.html
> 
> Yeah, getting support for that upstream might be a good idea :)
> 
>> Also it seems like a good idea to also submit the NS bit
>> exclusion from the ack filter to mainline as well.
> 
> What's that?
> 
> -Toke
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake in dd-wrt

[Dave Taht]

On Wed, Aug 21, 2019 at 3:21 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Dave Taht <dave@taht.net> writes:
>
> > Jonathan Morton <chromatix99@gmail.com> writes:
> >
> >>> On 20 Aug, 2019, at 9:39 pm, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
> >>>
> >>> …a heavy bittorrent downloader will still steal the bandwidth of my scp session.
> >>
> >> If you can identify the Bittorrent packets, you can mark them CS1, and
> >> switch on Cake's "diffserv3" mode (as it is by default).  Then the
> >> Bittorrent packets will still be able to use full bandwidth if it's
> >> available, but will be limited to 1/16th of the total if there is
> >> contention.
> >
> > I regard the whole CS1 thing as having never been particularly
> > successful for a variety of reasons - in particular because
> > we seemed to be the only ones attempting to use it with rigor.
> >
> > I would like to patch in and submit "LE" support to mainline cake.
> >
> > The RFC retires CS1 - which I wouldn't retire - but see:
> >
> > https://www.rfc-editor.org/rfc/rfc8622.html
>
> Yeah, getting support for that upstream might be a good idea :)

I'd put out a patch on a endian-brain-fart day, which I think was
correct?, but didn't get back to it.

Another cleanup thought is to constify the cake invsqrt cache. (and actually
put in totally correct values)

> > Also it seems like a good idea to also submit the NS bit
> > exclusion from the ack filter to mainline as well.
>
> What's that?

https://github.com/chromi/sce/blob/sce/net/sched/sch_cake.c#L1274

A cleaner way would be to have it be

#ifndef TCP_FLAG_ESCE
#define it (I forget where it's defined)
#endif

#define CAKE_FILTER_FLAGS (TCP_FLAG_ECE | TCP_FLAG_CWR | TCP_FLAG_ESCE)

and use that.

>
> -Toke
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] cake in dd-wrt

[Jonathan Morton]

> On 21 Aug, 2019, at 1:21 pm, Toke Høiland-Jørgensen <toke@redhat.com> wrote:
> 
>> Also it seems like a good idea to also submit the NS bit
>> exclusion from the ack filter to mainline as well.
> 
> What's that?

The ack-filter has rules about which bits and options changing need to be preserved to avoid losing information from the ack stream, and therefore prevent two acks from being coalesced.

The version of Cake in the SCE tree adds the former NS bit (which SCE uses as ESCE) to that list.  SCE still works fine without that patch, it just preserves more information on the reverse path and thus makes the behaviour a bit smoother.

 - Jonathan Morton

[Cake] pie in dd-wrt

[Dave Taht]

this shows some good results with pie on the download

https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=313886&postdays=0&postorder=asc&start=30&sid=4d4d2a583afad73759cbeee1a8f4b329

Re: [Cake] pie in dd-wrt

[Sebastian Gottschall]

thats rather old. i rewrote all the qos code in the last 4 or 5 days. so 
things might be changed. next phase is bringing all the link level 
detail configuration stuff into the gui which will be done
tomorrow or at least still within this week.
i also added now cake to some smaller low budged routers with limited 
resources, so see how it runs. i had bad experiences with fq_codel in 
the past due some high memory usage.
especially since its hard coded somewhat into the wireless ath9k driver. 
so i already modded it for more efficient handling. 4 mb max per queue 
is simply too much for  a 32 mb ram based router.


Sebastian

Am 21.08.2019 um 17:42 schrieb Dave Taht:
> this shows some good results with pie on the download
>
> https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=313886&postdays=0&postorder=asc&start=30&sid=4d4d2a583afad73759cbeee1a8f4b329
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] pie in dd-wrt

[Sebastian Moeller]

On August 21, 2019 6:12:03 PM GMT+02:00, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>thats rather old. i rewrote all the qos code in the last 4 or 5 days.
>so 
>things might be changed. next phase is bringing all the link level 
>detail configuration stuff into the gui which will be done
>tomorrow or at least still within this week.
>i also added now cake to some smaller low budged routers with limited 
>resources, so see how it runs. i had bad experiences with fq_codel in 
>the past due some high memory usage.
>especially since its hard coded somewhat into the wireless ath9k
>driver. 
>so i already modded it for more efficient handling. 4 mb max per queue 
>is simply too much for  a 32 mb ram based router.

This is why I'm sqm we reduced the queued packet maximum m to around 1000, and also why cake has an explicit memlimit keyword.

Best Regards 
        Sebastian


>
>
>Sebastian
>
>Am 21.08.2019 um 17:42 schrieb Dave Taht:
>> this shows some good results with pie on the download
>>
>>
>https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=313886&postdays=0&postorder=asc&start=30&sid=4d4d2a583afad73759cbeee1a8f4b329
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>_______________________________________________
>Cake mailing list
>Cake@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/cake

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Cake] pie in dd-wrt

[Sebastian Gottschall]

Am 21.08.2019 um 18:21 schrieb Sebastian Moeller:
>
> On August 21, 2019 6:12:03 PM GMT+02:00, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>> thats rather old. i rewrote all the qos code in the last 4 or 5 days.
>> so
>> things might be changed. next phase is bringing all the link level
>> detail configuration stuff into the gui which will be done
>> tomorrow or at least still within this week.
>> i also added now cake to some smaller low budged routers with limited
>> resources, so see how it runs. i had bad experiences with fq_codel in
>> the past due some high memory usage.
>> especially since its hard coded somewhat into the wireless ath9k
>> driver.
>> so i already modded it for more efficient handling. 4 mb max per queue
>> is simply too much for  a 32 mb ram based router.
> This is why I'm sqm we reduced the queued packet maximum m to around 1000, and also why cake has an explicit memlimit keyword.
yeah but does this help if fq_codel is hardcoded into mac80211? fq_codel 
has a memlimit  keyword too btw. its fixed to 4MB. i reduced it to 256kb 
on low memory architectures. no other way to get around OOM problems
mac80211  does always make use of fq_codel. it has a own build in 
implementation
>
> Best Regards
>          Sebastian
>
>
>>
>> Sebastian
>>
>> Am 21.08.2019 um 17:42 schrieb Dave Taht:
>>> this shows some good results with pie on the download
>>>
>>>
>> https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=313886&postdays=0&postorder=asc&start=30&sid=4d4d2a583afad73759cbeee1a8f4b329
>>> _______________________________________________
>>> Cake mailing list
>>> Cake@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cake
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] pie in dd-wrt

[Dave Taht]

Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:

> Am 21.08.2019 um 18:21 schrieb Sebastian Moeller:
>>
>> On August 21, 2019 6:12:03 PM GMT+02:00, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>>> thats rather old. i rewrote all the qos code in the last 4 or 5 days.
>>> so
>>> things might be changed. next phase is bringing all the link level
>>> detail configuration stuff into the gui which will be done
>>> tomorrow or at least still within this week.
>>> i also added now cake to some smaller low budged routers with limited
>>> resources, so see how it runs. i had bad experiences with fq_codel in
>>> the past due some high memory usage.
>>> especially since its hard coded somewhat into the wireless ath9k
>>> driver.
>>> so i already modded it for more efficient handling. 4 mb max per queue
>>> is simply too much for  a 32 mb ram based router.
>> This is why I'm sqm we reduced the queued packet maximum m to around 1000, and also why cake has an explicit memlimit keyword.
> yeah but does this help if fq_codel is hardcoded into mac80211?
> fq_codel has a memlimit  keyword too btw. its fixed to 4MB. i reduced
> it to 256kb on low memory architectures. no other way to get around
> OOM problems
> mac80211  does always make use of fq_codel. it has a own build in
> implementation

Certainly memory limits are a huge problem throughout complex qdiscs,
especially when stacked up (eg hfsc 1 -> qdiscx hfsc 2 -> qdisc x),
and 

OOMs suck. Particularly as few test packet flooding their devices
after setting up a complex qdisc system. 

Bytes = time. It doesn't matter how many queues you have. This
to me has always been one of the biggest flaws of the tc subsystem
in general is that the total amount of memory in use on
a given physical interface should be managed by the topmost layer.

Same problem for wifi in multiple SSIDs... it's still just one device.

However we do need enough bytes to keep the device busy and there
are other problems with per packet limits leading me to prefer
using just memory limits. One is, that your typical ack packet
coming off the rx ring is actually 2k in size, not 64 bytes.
I had at one point (in openwrt) something that took small packets
and copied them to a smaller allocation which took pressure off the
memory allocator.

I've kind of lost track, did the ath9k wifi stuff use 1 allocation for
all 4 hw queues? I'm afraid to look this morning... (and I'm not big
on the 4 hw queues either)

One advantage of cake as cake as the shaper is it uses one allocation
for all.  hfsc -> cake/fq_codel/pie/etc, not so much.

similarly, my old "plan" for a sch_mq_shaper (for hardware mq) was that
the sub-qdiscs would all share the same bandwidth and memory allocation.



>>
>> Best Regards
>>          Sebastian
>>
>>
>>>
>>> Sebastian
>>>
>>> Am 21.08.2019 um 17:42 schrieb Dave Taht:
>>>> this shows some good results with pie on the download
>>>>
>>>>
>>> https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=313886&postdays=0&postorder=asc&start=30&sid=4d4d2a583afad73759cbeee1a8f4b329
>>>> _______________________________________________
>>>> Cake mailing list
>>>> Cake@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/cake
>>> _______________________________________________
>>> Cake mailing list
>>> Cake@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cake
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] pie in dd-wrt

[Toke Høiland-Jørgensen]

Dave Taht <dave@taht.net> writes:

> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>
>> Am 21.08.2019 um 18:21 schrieb Sebastian Moeller:
>>>
>>> On August 21, 2019 6:12:03 PM GMT+02:00, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>>>> thats rather old. i rewrote all the qos code in the last 4 or 5 days.
>>>> so
>>>> things might be changed. next phase is bringing all the link level
>>>> detail configuration stuff into the gui which will be done
>>>> tomorrow or at least still within this week.
>>>> i also added now cake to some smaller low budged routers with limited
>>>> resources, so see how it runs. i had bad experiences with fq_codel in
>>>> the past due some high memory usage.
>>>> especially since its hard coded somewhat into the wireless ath9k
>>>> driver.
>>>> so i already modded it for more efficient handling. 4 mb max per queue
>>>> is simply too much for  a 32 mb ram based router.
>>> This is why I'm sqm we reduced the queued packet maximum m to around 1000, and also why cake has an explicit memlimit keyword.
>> yeah but does this help if fq_codel is hardcoded into mac80211?
>> fq_codel has a memlimit  keyword too btw. its fixed to 4MB. i reduced
>> it to 256kb on low memory architectures. no other way to get around
>> OOM problems
>> mac80211  does always make use of fq_codel. it has a own build in
>> implementation
>
> Certainly memory limits are a huge problem throughout complex qdiscs,
> especially when stacked up (eg hfsc 1 -> qdiscx hfsc 2 -> qdisc x),
> and 
>
> OOMs suck. Particularly as few test packet flooding their devices
> after setting up a complex qdisc system. 
>
> Bytes = time. It doesn't matter how many queues you have. This
> to me has always been one of the biggest flaws of the tc subsystem
> in general is that the total amount of memory in use on
> a given physical interface should be managed by the topmost layer.
>
> Same problem for wifi in multiple SSIDs... it's still just one device.
>
> However we do need enough bytes to keep the device busy and there
> are other problems with per packet limits leading me to prefer
> using just memory limits. One is, that your typical ack packet
> coming off the rx ring is actually 2k in size, not 64 bytes.
> I had at one point (in openwrt) something that took small packets
> and copied them to a smaller allocation which took pressure off the
> memory allocator.
>
> I've kind of lost track, did the ath9k wifi stuff use 1 allocation for
> all 4 hw queues? I'm afraid to look this morning... (and I'm not big
> on the 4 hw queues either)

The memory limit in mac80211 is global per phy.

-Toke

Re: [Cake] pie in dd-wrt

[Dave Taht]

On Wed, Aug 21, 2019 at 2:40 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Dave Taht <dave@taht.net> writes:
>
> > Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
> >
> >> Am 21.08.2019 um 18:21 schrieb Sebastian Moeller:
> >>>
> >>> On August 21, 2019 6:12:03 PM GMT+02:00, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
> >>>> thats rather old. i rewrote all the qos code in the last 4 or 5 days.
> >>>> so
> >>>> things might be changed. next phase is bringing all the link level
> >>>> detail configuration stuff into the gui which will be done
> >>>> tomorrow or at least still within this week.
> >>>> i also added now cake to some smaller low budged routers with limited
> >>>> resources, so see how it runs. i had bad experiences with fq_codel in
> >>>> the past due some high memory usage.
> >>>> especially since its hard coded somewhat into the wireless ath9k
> >>>> driver.
> >>>> so i already modded it for more efficient handling. 4 mb max per queue
> >>>> is simply too much for  a 32 mb ram based router.
> >>> This is why I'm sqm we reduced the queued packet maximum m to around 1000, and also why cake has an explicit memlimit keyword.
> >> yeah but does this help if fq_codel is hardcoded into mac80211?
> >> fq_codel has a memlimit  keyword too btw. its fixed to 4MB. i reduced
> >> it to 256kb on low memory architectures. no other way to get around
> >> OOM problems
> >> mac80211  does always make use of fq_codel. it has a own build in
> >> implementation
> >
> > Certainly memory limits are a huge problem throughout complex qdiscs,
> > especially when stacked up (eg hfsc 1 -> qdiscx hfsc 2 -> qdisc x),
> > and
> >
> > OOMs suck. Particularly as few test packet flooding their devices
> > after setting up a complex qdisc system.
> >
> > Bytes = time. It doesn't matter how many queues you have. This
> > to me has always been one of the biggest flaws of the tc subsystem
> > in general is that the total amount of memory in use on
> > a given physical interface should be managed by the topmost layer.
> >
> > Same problem for wifi in multiple SSIDs... it's still just one device.
> >
> > However we do need enough bytes to keep the device busy and there
> > are other problems with per packet limits leading me to prefer
> > using just memory limits. One is, that your typical ack packet
> > coming off the rx ring is actually 2k in size, not 64 bytes.
> > I had at one point (in openwrt) something that took small packets
> > and copied them to a smaller allocation which took pressure off the
> > memory allocator.
> >
> > I've kind of lost track, did the ath9k wifi stuff use 1 allocation for
> > all 4 hw queues? I'm afraid to look this morning... (and I'm not big
> > on the 4 hw queues either)
>
> The memory limit in mac80211 is global per phy.

yea! win! So much better than four instances per ssid.

> -Toke
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] pie in dd-wrt

[Sebastian Gottschall]

Am 21.08.2019 um 23:53 schrieb Dave Taht:
> On Wed, Aug 21, 2019 at 2:40 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>> Dave Taht <dave@taht.net> writes:
>>
>>> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>>>
>>>> Am 21.08.2019 um 18:21 schrieb Sebastian Moeller:
>>>>> On August 21, 2019 6:12:03 PM GMT+02:00, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>>>>>> thats rather old. i rewrote all the qos code in the last 4 or 5 days.
>>>>>> so
>>>>>> things might be changed. next phase is bringing all the link level
>>>>>> detail configuration stuff into the gui which will be done
>>>>>> tomorrow or at least still within this week.
>>>>>> i also added now cake to some smaller low budged routers with limited
>>>>>> resources, so see how it runs. i had bad experiences with fq_codel in
>>>>>> the past due some high memory usage.
>>>>>> especially since its hard coded somewhat into the wireless ath9k
>>>>>> driver.
>>>>>> so i already modded it for more efficient handling. 4 mb max per queue
>>>>>> is simply too much for  a 32 mb ram based router.
>>>>> This is why I'm sqm we reduced the queued packet maximum m to around 1000, and also why cake has an explicit memlimit keyword.
>>>> yeah but does this help if fq_codel is hardcoded into mac80211?
>>>> fq_codel has a memlimit  keyword too btw. its fixed to 4MB. i reduced
>>>> it to 256kb on low memory architectures. no other way to get around
>>>> OOM problems
>>>> mac80211  does always make use of fq_codel. it has a own build in
>>>> implementation
>>> Certainly memory limits are a huge problem throughout complex qdiscs,
>>> especially when stacked up (eg hfsc 1 -> qdiscx hfsc 2 -> qdisc x),
>>> and
>>>
>>> OOMs suck. Particularly as few test packet flooding their devices
>>> after setting up a complex qdisc system.
>>>
>>> Bytes = time. It doesn't matter how many queues you have. This
>>> to me has always been one of the biggest flaws of the tc subsystem
>>> in general is that the total amount of memory in use on
>>> a given physical interface should be managed by the topmost layer.
>>>
>>> Same problem for wifi in multiple SSIDs... it's still just one device.
>>>
>>> However we do need enough bytes to keep the device busy and there
>>> are other problems with per packet limits leading me to prefer
>>> using just memory limits. One is, that your typical ack packet
>>> coming off the rx ring is actually 2k in size, not 64 bytes.
>>> I had at one point (in openwrt) something that took small packets
>>> and copied them to a smaller allocation which took pressure off the
>>> memory allocator.
>>>
>>> I've kind of lost track, did the ath9k wifi stuff use 1 allocation for
>>> all 4 hw queues? I'm afraid to look this morning... (and I'm not big
>>> on the 4 hw queues either)
>> The memory limit in mac80211 is global per phy.
> yea! win! So much better than four instances per ssid.
bad of some devices have 2 phys and still just 32 mb ram. i mean the 4 
mb limit i was talking about is just a mod by openwrt. the original 
default is 32 mb for fq_codel in mac80211.

>
>> -Toke
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>
>

[Cake] Wifi Memory limits in small platforms

[Dave Taht]

It's very good to know how much folk have been struggling to keep
things from OOMing on 32MB platforms. I'd like to hope that the
unified memory management in cake (vs a collection of QoS qdiscs) and
the new fq_codel for wifi stuff (cutting it down to 1 alloc from four)
help, massively on this issue, but until today I was unaware of how
much the field may have been patching things out.

The default 32MB memory limits in fq_codel comes from the stressing
about 10GigE networking from google. 4MB is limit in openwrt,
which is suitable for ~1Gbit, and is sort of there  due to 802.11ac's
maximum (impossible to hit) of a txop that large.

Something as small as 256K is essentially about 128 full size packets
(and often, acks from an ethernet device's rx ring eat 2k).

The structure of the new fq_codel for wifi subsystem is "one in the
hardware, one ready to go, and the rest accumulating". I
typically see about 13-20 packets in an aggregate. 256k strikes me as
a bit small.

I haven't checked, but does this patch still exist in openwrt/dd-wrt?
It had helped a lot when under memory pressure from
a lot of small packets.

https://github.com/dtaht/cerowrt-3.10/blob/master/target/linux/generic/patches-3.10/657-qdisc_reduce_truesize.patch

Arguably this could be made more aggressive, but it massively reduced
memory burdens at the time I did it when
flooding the device, or having lots of acks, and while it cost cpu it
saved on ooming.

There's two other dubious things in the fq_codel for wifi stack
presently. Right now the codel target is set too high for p2p use
(20ms, where 6ms seems more right), and it also flips up to a really
high target and interval AND turns off ecn when there's more than a
few stations available (rather than "active") - it's an overly
conservative figure we used back when we had major issues with
powersave
and multicast that I'd hoped we could cut back to normal after we got
another round of research funding and feedback from the field (which
didn't happen, and we never got around to making it configurable, and
being 25x better than it was before seemed "enough")

I was puzzled at battlemesh as to why I had dropping at about 50ms
delay rather than ecn, and thought it was something
else, and this morning I'm thinking that folk have been reducing the
memlimit to 256k rather....

Re: [Cake] Wifi Memory limits in small platforms

[Dave Taht]

People have a tendency to try and construct very complicated QoS
systems and then try to run them in limited memory. We see a lot of 6
or 7 class hfsc + sfq or fq_codel systems, which can accumulate
hundreds or thousands of packets before doing a drop. A stress test
like ping -f -s 1500 and ping -f -s 64 hitting every defined queue
(somehow) should be run to make sure you don't OOM, and even, even
then, things like acks coming off an ethernet eat 2k when they are
only 64 bytes in size.

this patch (Not even compile tested) might take some of the memory
pressure off when being flooded in this way. While
it costs cpu, given a choice between ooming and slowing down, I'd
rather slow down. Should have done tbf too.

I wonder if the large number of queues we've seen people try to use
with hfsc has been one of the sources of
frequent reports of flakyness? How many queues do people actually try
to create with it in the field... Certainly sfq's default of 127
packets is better than 1000...

Reducing the truesize could be added to cake and tbf also. As well as
the fq_codel for wifi code, on small platforms.

diff --git a/net/sched/sch_hfsc.c b/net/sched/sch_hfsc.c
index 433f2190960f..c0777ce4a259 100644
--- a/net/sched/sch_hfsc.c
+++ b/net/sched/sch_hfsc.c
@@ -1535,7 +1535,8 @@ hfsc_enqueue(struct sk_buff *skb, struct Qdisc
*sch, struct sk_buff **to_free)
        struct hfsc_class *cl;
        int uninitialized_var(err);
        bool first;
-
+       if (sch->q.qlen > 128)
+               skb = skb_reduce_truesize(skb);
        cl = hfsc_classify(skb, sch, &err);
        if (cl == NULL) {
                if (err & __NET_XMIT_BYPASS)
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
index 7bcf20ef9145..40a27392f88e 100644
--- a/net/sched/sch_htb.c
+++ b/net/sched/sch_htb.c
@@ -584,6 +584,9 @@ static int htb_enqueue(struct sk_buff *skb, struct
Qdisc *sch,
        struct htb_sched *q = qdisc_priv(sch);
        struct htb_class *cl = htb_classify(skb, sch, &ret);

+       if (sch->q.qlen > 128)
+               skb = skb_reduce_truesize(skb);
+
        if (cl == HTB_DIRECT) {
                /* enqueue to helper queue */
                if (q->direct_queue.qlen < q->direct_qlen) {

On Thu, Aug 22, 2019 at 6:15 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> It's very good to know how much folk have been struggling to keep
> things from OOMing on 32MB platforms. I'd like to hope that the
> unified memory management in cake (vs a collection of QoS qdiscs) and
> the new fq_codel for wifi stuff (cutting it down to 1 alloc from four)
> help, massively on this issue, but until today I was unaware of how
> much the field may have been patching things out.
>
> The default 32MB memory limits in fq_codel comes from the stressing
> about 10GigE networking from google. 4MB is limit in openwrt,
> which is suitable for ~1Gbit, and is sort of there  due to 802.11ac's
> maximum (impossible to hit) of a txop that large.
>
> Something as small as 256K is essentially about 128 full size packets
> (and often, acks from an ethernet device's rx ring eat 2k).
>
> The structure of the new fq_codel for wifi subsystem is "one in the
> hardware, one ready to go, and the rest accumulating". I
> typically see about 13-20 packets in an aggregate. 256k strikes me as
> a bit small.
>
> I haven't checked, but does this patch still exist in openwrt/dd-wrt?
> It had helped a lot when under memory pressure from
> a lot of small packets.
>
> https://github.com/dtaht/cerowrt-3.10/blob/master/target/linux/generic/patches-3.10/657-qdisc_reduce_truesize.patch
>
> Arguably this could be made more aggressive, but it massively reduced
> memory burdens at the time I did it when
> flooding the device, or having lots of acks, and while it cost cpu it
> saved on ooming.
>
> There's two other dubious things in the fq_codel for wifi stack
> presently. Right now the codel target is set too high for p2p use
> (20ms, where 6ms seems more right), and it also flips up to a really
> high target and interval AND turns off ecn when there's more than a
> few stations available (rather than "active") - it's an overly
> conservative figure we used back when we had major issues with
> powersave
> and multicast that I'd hoped we could cut back to normal after we got
> another round of research funding and feedback from the field (which
> didn't happen, and we never got around to making it configurable, and
> being 25x better than it was before seemed "enough")
>
> I was puzzled at battlemesh as to why I had dropping at about 50ms
> delay rather than ecn, and thought it was something
> else, and this morning I'm thinking that folk have been reducing the
> memlimit to 256k rather....



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] Wifi Memory limits in small platforms

[Sebastian Gottschall]

Am 22.08.2019 um 15:15 schrieb Dave Taht:
> It's very good to know how much folk have been struggling to keep
> things from OOMing on 32MB platforms. I'd like to hope that the
> unified memory management in cake (vs a collection of QoS qdiscs) and
> the new fq_codel for wifi stuff (cutting it down to 1 alloc from four)
> help, massively on this issue, but until today I was unaware of how
> much the field may have been patching things out.
>
> The default 32MB memory limits in fq_codel comes from the stressing
> about 10GigE networking from google. 4MB is limit in openwrt,
> which is suitable for ~1Gbit, and is sort of there  due to 802.11ac's
> maximum (impossible to hit) of a txop that large.
>
> Something as small as 256K is essentially about 128 full size packets
> (and often, acks from an ethernet device's rx ring eat 2k).

what i miss in mac80211 is the following option "fq_codel = off"
its essential and i will definitly work on a patch to deal with this way 
for low memory 802.11n platforms.

>
> The structure of the new fq_codel for wifi subsystem is "one in the
> hardware, one ready to go, and the rest accumulating". I
> typically see about 13-20 packets in an aggregate. 256k strikes me as
> a bit small.
from the rules its that 256 is used for ht only and if vht is involved 
the limit of 4mb is used.
but now comes the point. all 802.11ac platforms having 64mb ram or more. 
but ath10k chipsets are using
about 40 mb of shared memory. so mmh we are hitting the wall again. most 
routers have 128 mb with 802.11ac, but some (noticable dlink) have just 64mb
>
> I haven't checked, but does this patch still exist in openwrt/dd-wrt?
> It had helped a lot when under memory pressure from
> a lot of small packets.
>
> https://github.com/dtaht/cerowrt-3.10/blob/master/target/linux/generic/patches-3.10/657-qdisc_reduce_truesize.patch
>
> Arguably this could be made more aggressive, but it massively reduced
> memory burdens at the time I did it when
> flooding the device, or having lots of acks, and while it cost cpu it
> saved on ooming.
mmh let me check -> nope its at least not in my tree. but will be soon :-)
>
> There's two other dubious things in the fq_codel for wifi stack
> presently. Right now the codel target is set too high for p2p use
> (20ms, where 6ms seems more right), and it also flips up to a really
> high target and interval AND turns off ecn when there's more than a
> few stations available (rather than "active") - it's an overly
> conservative figure we used back when we had major issues with
> powersave
> and multicast that I'd hoped we could cut back to normal after we got
> another round of research funding and feedback from the field (which
> didn't happen, and we never got around to making it configurable, and
> being 25x better than it was before seemed "enough")
>
> I was puzzled at battlemesh as to why I had dropping at about 50ms
> delay rather than ecn, and thought it was something
> else, and this morning I'm thinking that folk have been reducing the
> memlimit to 256k rather....
>

Re: [Cake] Wifi Memory limits in small platforms

[Dave Taht]

Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:

> Am 22.08.2019 um 15:15 schrieb Dave Taht:
>> It's very good to know how much folk have been struggling to keep
>> things from OOMing on 32MB platforms. I'd like to hope that the
>> unified memory management in cake (vs a collection of QoS qdiscs) and
>> the new fq_codel for wifi stuff (cutting it down to 1 alloc from four)
>> help, massively on this issue, but until today I was unaware of how
>> much the field may have been patching things out.
>>
>> The default 32MB memory limits in fq_codel comes from the stressing
>> about 10GigE networking from google. 4MB is limit in openwrt,
>> which is suitable for ~1Gbit, and is sort of there  due to 802.11ac's
>> maximum (impossible to hit) of a txop that large.

I did kind of conflate "qos + fq_codel" vs wifi in this message. It
looks like yer staying with me. 

>> Something as small as 256K is essentially about 128 full size packets
>> (and often, acks from an ethernet device's rx ring eat 2k).
>
> what i miss in mac80211 is the following option "fq_codel = off"
> its essential and i will definitly work on a patch to deal with this
> way for low memory 802.11n platforms.

Well, it would be my hope that turning it off would A) not help that
much on memory or cpu and B) show such a dramatic reduction in
multi-station performance that you'd immediately turn it on again.

I try to encourage folk to run the rtt_fair tests in flent when
twiddling with wifi. Those really shows how bad things are when you
don't have ATF + FQ + Per station aggregation and lots of
clients. Single threaded tests are misleading.

I gave a good demo of why this is (was!), here: https://www.youtube.com/watch?v=Rb-UnHDw02o&t=1551s

and there's more in the ending the anomaly paper. Perversely though,
now that we can do 25x latency reductions and 2.5x more throughput,
more memory is needed to achieve those goals in some cases, which
is part of my concern about chopping things down to 256k here.

>
>>
>> The structure of the new fq_codel for wifi subsystem is "one in the
>> hardware, one ready to go, and the rest accumulating". I
>> typically see about 13-20 packets in an aggregate. 256k strikes me as
>> a bit small.
> from the rules its that 256 is used for ht only and if vht is involved
> the limit of 4mb is used.
> but now comes the point. all 802.11ac platforms having 64mb ram or
> more. but ath10k chipsets are using
> about 40 mb of shared memory. so mmh we are hitting the wall
> again. most routers have 128 mb with 802.11ac, but some (noticable
> dlink) have just 64mb

Ugh.

Is it just the mips boxes with so little ram? All the arm routers I have
have at least 128, some as much as 512.

Yes, having a wifi chip that can theoretically have 4MB in transit
with so little ram is problematic.

Dear dlink: don't do that. It hurts when you do that.

>>
>> I haven't checked, but does this patch still exist in openwrt/dd-wrt?
>> It had helped a lot when under memory pressure from
>> a lot of small packets.
>>
>> https://github.com/dtaht/cerowrt-3.10/blob/master/target/linux/generic/patches-3.10/657-qdisc_reduce_truesize.patch
>>
>> Arguably this could be made more aggressive, but it massively reduced
>> memory burdens at the time I did it when
>> flooding the device, or having lots of acks, and while it cost cpu it
>> saved on ooming.
> mmh let me check -> nope its at least not in my tree. but will be soon :-)

Well, I sent along a mildly improved version of the idea.

I can really see some sort of "test my qos" script that attempts
to flood every queue on the system. And wider adoption of
cake which is lighter weight than the alterntives.

one idea that's in cake was that: we'd hoped to capture the most typical
qos setups with it with "models". It's very easy to add a new model
(besteffort, diffserv3, diffserv4) (it's a lookup table and bandwidth
allocation call), but lacking feedback on more typical QoS constructs
from the field, that's where it ended. When we started the project,
I figured we'd end up with 20+ models before the end.

It would be good to get a tc class dump or output from more typical
QoS Setups.

In sqm and cake...
we have a terrible tendency to tell people "no, just use the defaults!
they work! trust us!"... 

who generally don't believe us and want to keep doing things the
way they always have.

In more than a few circumstances they are right, but we don't understand
what they are trying to do.

As one case that cake doesn't handle, at least some iptv setups are
visible as a strict priority queue over everything else, below which you
do everything else, so the tv stream never, ever, drops a packet.

We didn't do that, but could *easily* add an iptv model to shape
inbound better - if we knew more about how free, FT etc, construct
their packets.

Similarly some folk in this world want strict priority for EF.

>> There's two other dubious things in the fq_codel for wifi stack
>> presently. Right now the codel target is set too high for p2p use
>> (20ms, where 6ms seems more right), and it also flips up to a really
>> high target and interval AND turns off ecn when there's more than a
>> few stations available (rather than "active") - it's an overly
>> conservative figure we used back when we had major issues with
>> powersave
>> and multicast that I'd hoped we could cut back to normal after we got
>> another round of research funding and feedback from the field (which
>> didn't happen, and we never got around to making it configurable, and
>> being 25x better than it was before seemed "enough")
>>
>> I was puzzled at battlemesh as to why I had dropping at about 50ms
>> delay rather than ecn, and thought it was something
>> else, and this morning I'm thinking that folk have been reducing the
>> memlimit to 256k rather....
>>

Re: [Cake] Wifi Memory limits in small platforms

[Sebastian Gottschall]

Am 22.08.2019 um 19:03 schrieb Dave Taht:
> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>
>> Am 22.08.2019 um 15:15 schrieb Dave Taht:
>>> It's very good to know how much folk have been struggling to keep
>>> things from OOMing on 32MB platforms. I'd like to hope that the
>>> unified memory management in cake (vs a collection of QoS qdiscs) and
>>> the new fq_codel for wifi stuff (cutting it down to 1 alloc from four)
>>> help, massively on this issue, but until today I was unaware of how
>>> much the field may have been patching things out.
>>>
>>> The default 32MB memory limits in fq_codel comes from the stressing
>>> about 10GigE networking from google. 4MB is limit in openwrt,
>>> which is suitable for ~1Gbit, and is sort of there  due to 802.11ac's
>>> maximum (impossible to hit) of a txop that large.
> I did kind of conflate "qos + fq_codel" vs wifi in this message. It
> looks like yer staying with me.
>
>>> Something as small as 256K is essentially about 128 full size packets
>>> (and often, acks from an ethernet device's rx ring eat 2k).
>> what i miss in mac80211 is the following option "fq_codel = off"
>> its essential and i will definitly work on a patch to deal with this
>> way for low memory 802.11n platforms.
> Well, it would be my hope that turning it off would A) not help that
> much on memory or cpu and B) show such a dramatic reduction in
> multi-station performance that you'd immediately turn it on again.
isnt it better to have a working platform with less performance than a 
crashing platform with no performance?
i mean i can user older mac80211 versions without that issue on a 
typical nanostation 2/5 which is often used just as CPE device

but with current mac80211 versions (current means last 2-3 years). they 
are just unstable and running out of memory after a while
the only thing which helped was cutting of the memory limit of fq_codel 
inside mac80211
i also have another fancy testunit which is a linksys wrt400 with 32 mb 
ram and 2 ath9k based wifi chipsets. no hope here for running stable
for only 5 minutes even with a single connection under load (my crashing 
test is running a hdtv iptv stream converted to unicast using a 
stateless eoip tunnel)

> I try to encourage folk to run the rtt_fair tests in flent when
> twiddling with wifi. Those really shows how bad things are when you
> don't have ATF + FQ + Per station aggregation and lots of
> clients. Single threaded tests are misleading.
i know but even single threaded tests arent working good on such 
devices. so there is no need to talk about the benefits of atf,fq_codel etc.
but there is need to talk about configurable use of it which also allows 
to disable it if required. if you just have a cpe device with pppoe 
running on it which is common for wisps
there is no need for much fair queuing. this is a task for the 
accesspoint. another typical use for such devices like nanostation, 
rocket, bullet etc. are simple point to point long range links.
this is the main use for such high gain devices like these is my 
assumption.
so we dont talk about a typical cool and fancy ab. we talk about 
compatibility with low end devices without running out of resources. i'm 
a typical programmer from the 80s. keep it small, simple and resource 
efficient as possible. these coding standards should still be considered 
today even if i dont write tetris clones anymore running on 512 byte 
boot sectors using the msdos builtin debug assembler program
>
> I gave a good demo of why this is (was!), here: https://www.youtube.com/watch?v=Rb-UnHDw02o&t=1551s
>
> and there's more in the ending the anomaly paper. Perversely though,
> now that we can do 25x latency reductions and 2.5x more throughput,
> more memory is needed to achieve those goals in some cases, which
> is part of my concern about chopping things down to 256k here.
>>> The structure of the new fq_codel for wifi subsystem is "one in the
>>> hardware, one ready to go, and the rest accumulating". I
>>> typically see about 13-20 packets in an aggregate. 256k strikes me as
>>> a bit small.
>> from the rules its that 256 is used for ht only and if vht is involved
>> the limit of 4mb is used.
>> but now comes the point. all 802.11ac platforms having 64mb ram or
>> more. but ath10k chipsets are using
>> about 40 mb of shared memory. so mmh we are hitting the wall
>> again. most routers have 128 mb with 802.11ac, but some (noticable
>> dlink) have just 64mb
> Ugh.
>
> Is it just the mips boxes with so little ram? All the arm routers I have
> have at least 128, some as much as 512.
you got it. all the mips routers. most problematic the tplink wr841 (and 
similar series) and ubnt devices of course.
these are 802.11 but just comming with 32 mb ram. but there are others 
too of course and i love to maintain all the older devices
for the community. for newer arm based devices we really dont need to 
care about. broadcom arm cpus are comming with chipsets which are not 
supported by linux/mac80211 anyway
or just bad supported for newer chipsets using brcmfmac. (but the 
original broadcom propertiery driver is unstable too of course)
and all other models based on qca ipq8064 etc. are comming with 256 mb 
and more and we really only need to take care about ath9k and ath10k 
(soon maybe ath11k)
everything else doesnt matter. the linksys wrtXXXX series has a mac80211 
driver, but marvell stopped maintaining it at a point where it still was 
shit and unstable. and its mainly based on a binary firmware blob.


>
> Yes, having a wifi chip that can theoretically have 4MB in transit
> with so little ram is problematic.
>
> Dear dlink: don't do that. It hurts when you do that.
>
i talked alot with dlink about this issue, but dlinks solution was just 
switching to a cheaper mediatek mips based platform. now we have more 
ram, but a featureless chipset.
same for tplink.
>>> I haven't checked, but does this patch still exist in openwrt/dd-wrt?
>>> It had helped a lot when under memory pressure from
>>> a lot of small packets.
>>>
>>> https://github.com/dtaht/cerowrt-3.10/blob/master/target/linux/generic/patches-3.10/657-qdisc_reduce_truesize.patch
>>>
>>> Arguably this could be made more aggressive, but it massively reduced
>>> memory burdens at the time I did it when
>>> flooding the device, or having lots of acks, and while it cost cpu it
>>> saved on ooming.
>> mmh let me check -> nope its at least not in my tree. but will be soon :-)
> Well, I sent along a mildly improved version of the idea.
>
> I can really see some sort of "test my qos" script that attempts
> to flood every queue on the system. And wider adoption of
> cake which is lighter weight than the alterntives.
>
> one idea that's in cake was that: we'd hoped to capture the most typical
> qos setups with it with "models". It's very easy to add a new model
> (besteffort, diffserv3, diffserv4) (it's a lookup table and bandwidth
> allocation call), but lacking feedback on more typical QoS constructs
> from the field, that's where it ended. When we started the project,
> I figured we'd end up with 20+ models before the end.
>
> It would be good to get a tc class dump or output from more typical
> QoS Setups.
>
> In sqm and cake...
> we have a terrible tendency to tell people "no, just use the defaults!
> they work! trust us!"...
yeah i know that feeling .but i can never trust the users. the always do 
what they think is good for them
and everyone thinks he knows better since he was reading something using 
google / reddit
>
> who generally don't believe us and want to keep doing things the
> way they always have.
>
> In more than a few circumstances they are right, but we don't understand
> what they are trying to do.
>
> As one case that cake doesn't handle, at least some iptv setups are
> visible as a strict priority queue over everything else, below which you
> do everything else, so the tv stream never, ever, drops a packet.
as i mentioned before. my solition for iptv is layer 2 tunneling to get 
rid of multicast issues and it also converts everthing to a single 
connection.
i use a rfc compliant ether over ip tunnel for this which is not 
upstream in linux, but in freebsd. but there was a driver for kernel 2.4 
around many years ago and i maintained it up
to the latest kernel. its robust, handles fragmentation and just has 12 
bytes overhead.
>
> We didn't do that, but could *easily* add an iptv model to shape
> inbound better - if we knew more about how free, FT etc, construct
> their packets.

inbound they are marked with tos. typical internet has 0 of course. iptv 
has X and voice has Y. (dont ask me for the numbers, i dont have them in 
mind right now)
but for dhcp leases you need to mark your own packets with another dscp. 
otherwise the isp returns no ip. i dont know why this has been made. but 
it has to be handled.
normally orange ships black boxes as routers and to get it working with 
free systems, some people reverse engineered that shit. my conclusion is 
its some sort of
obfuscation to avoid third party hardware since the EU regulated the 
ISP's in a way that they got forced to allow 3rd party products which 
they still try to avoid. (refusing support for internet problems etc.)

> Similarly some folk in this world want strict priority for EF.
>
>>> There's two other dubious things in the fq_codel for wifi stack
>>> presently. Right now the codel target is set too high for p2p use
>>> (20ms, where 6ms seems more right), and it also flips up to a really
>>> high target and interval AND turns off ecn when there's more than a
>>> few stations available (rather than "active") - it's an overly
>>> conservative figure we used back when we had major issues with
>>> powersave
>>> and multicast that I'd hoped we could cut back to normal after we got
>>> another round of research funding and feedback from the field (which
>>> didn't happen, and we never got around to making it configurable, and
>>> being 25x better than it was before seemed "enough")
>>>
>>> I was puzzled at battlemesh as to why I had dropping at about 50ms
>>> delay rather than ecn, and thought it was something
>>> else, and this morning I'm thinking that folk have been reducing the
>>> memlimit to 256k rather....
>>>

Re: [Cake] Wifi Memory limits in small platforms

[Toke Høiland-Jørgensen]

Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:

> Am 22.08.2019 um 19:03 schrieb Dave Taht:
>> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>>
>>> Am 22.08.2019 um 15:15 schrieb Dave Taht:
>>>> It's very good to know how much folk have been struggling to keep
>>>> things from OOMing on 32MB platforms. I'd like to hope that the
>>>> unified memory management in cake (vs a collection of QoS qdiscs) and
>>>> the new fq_codel for wifi stuff (cutting it down to 1 alloc from four)
>>>> help, massively on this issue, but until today I was unaware of how
>>>> much the field may have been patching things out.
>>>>
>>>> The default 32MB memory limits in fq_codel comes from the stressing
>>>> about 10GigE networking from google. 4MB is limit in openwrt,
>>>> which is suitable for ~1Gbit, and is sort of there  due to 802.11ac's
>>>> maximum (impossible to hit) of a txop that large.
>> I did kind of conflate "qos + fq_codel" vs wifi in this message. It
>> looks like yer staying with me.
>>
>>>> Something as small as 256K is essentially about 128 full size packets
>>>> (and often, acks from an ethernet device's rx ring eat 2k).
>>> what i miss in mac80211 is the following option "fq_codel = off"
>>> its essential and i will definitly work on a patch to deal with this
>>> way for low memory 802.11n platforms.
>> Well, it would be my hope that turning it off would A) not help that
>> much on memory or cpu and B) show such a dramatic reduction in
>> multi-station performance that you'd immediately turn it on again.
> isnt it better to have a working platform with less performance than a 
> crashing platform with no performance?
> i mean i can user older mac80211 versions without that issue on a 
> typical nanostation 2/5 which is often used just as CPE device

So before the queueing patches to mac80211, the maximum packet queue
size for ath9k was 3MB in total, or 2.2MB if only a single AC was used
on the WiFi link (that's 128 packets in the driver + 1000 in the
pfifo_fast qdisc * 2074 bytes for the truesize of a full-size packet).
Whereas now the default is 4MB for a non-vht device. So it's not
actually that big of a difference, and as you've already discovered the
defaults can be changed.

Would it be helpful to add support for setting the memory limit in
hostapd (to avoid having to patch the kernel default)?

> but with current mac80211 versions (current means last 2-3 years). they 
> are just unstable and running out of memory after a while
> the only thing which helped was cutting of the memory limit of fq_codel 
> inside mac80211
> i also have another fancy testunit which is a linksys wrt400 with 32 mb 
> ram and 2 ath9k based wifi chipsets. no hope here fonr running stable
> for only 5 minutes even with a single connection under load (my crashing 
> test is running a hdtv iptv stream converted to unicast using a 
> stateless eoip tunnel)
>
>> I try to encourage folk to run the rtt_fair tests in flent when
>> twiddling with wifi. Those really shows how bad things are when you
>> don't have ATF + FQ + Per station aggregation and lots of
>> clients. Single threaded tests are misleading.
> i know but even single threaded tests arent working good on such 
> devices. so there is no need to talk about the benefits of atf,fq_codel etc.
> but there is need to talk about configurable use of it which also allows 
> to disable it if required.

Disabling the fq part won't actually gain you much in terms of memory
usage, though, as most of it is packet memory which is already
configurable.

The one exception to this is the static overhead of 'struct fq_flow', of
which mac80211 currently allocates 4k. That's 300k of memory which is
currently not configurable. But that could be fixed :)

-Toke

Re: [Cake] Wifi Memory limits in small platforms

[Dave Taht]

On Thu, Aug 22, 2019 at 11:23 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>
> > Am 22.08.2019 um 19:03 schrieb Dave Taht:
> >> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
> >>
> >>> Am 22.08.2019 um 15:15 schrieb Dave Taht:
> >>>> It's very good to know how much folk have been struggling to keep
> >>>> things from OOMing on 32MB platforms. I'd like to hope that the
> >>>> unified memory management in cake (vs a collection of QoS qdiscs) and
> >>>> the new fq_codel for wifi stuff (cutting it down to 1 alloc from four)
> >>>> help, massively on this issue, but until today I was unaware of how
> >>>> much the field may have been patching things out.
> >>>>
> >>>> The default 32MB memory limits in fq_codel comes from the stressing
> >>>> about 10GigE networking from google. 4MB is limit in openwrt,
> >>>> which is suitable for ~1Gbit, and is sort of there  due to 802.11ac's
> >>>> maximum (impossible to hit) of a txop that large.
> >> I did kind of conflate "qos + fq_codel" vs wifi in this message. It
> >> looks like yer staying with me.
> >>
> >>>> Something as small as 256K is essentially about 128 full size packets
> >>>> (and often, acks from an ethernet device's rx ring eat 2k).
> >>> what i miss in mac80211 is the following option "fq_codel = off"
> >>> its essential and i will definitly work on a patch to deal with this
> >>> way for low memory 802.11n platforms.
> >> Well, it would be my hope that turning it off would A) not help that
> >> much on memory or cpu and B) show such a dramatic reduction in
> >> multi-station performance that you'd immediately turn it on again.
> > isnt it better to have a working platform with less performance than a
> > crashing platform with no performance?
> > i mean i can user older mac80211 versions without that issue on a
> > typical nanostation 2/5 which is often used just as CPE device
>
> So before the queueing patches to mac80211, the maximum packet queue
> size for ath9k was 3MB in total, or 2.2MB if only a single AC was used
> on the WiFi link (that's 128 packets in the driver + 1000 in the
> pfifo_fast qdisc * 2074 bytes for the truesize of a full-size packet).
> Whereas now the default is 4MB for a non-vht device. So it's not
> actually that big of a difference, and as you've already discovered the
> defaults can be changed.
>
> Would it be helpful to add support for setting the memory limit in
> hostapd (to avoid having to patch the kernel default)?

hmm. I guess exposing that via netlink, etc is a good idea. Me I just
write the sys/kernel/debug/*/*/aqm files.

btw:

qos_map in my mind, for APs at this point, should default to the best
effort queue only. Not sure how to set
that in openwrt (I just patched it out of the kernel). 4 queues with 4
ready to go is a lot, and I have some ugly pics
from battlemesh when I tested it that I should get around to publishing it.

as for that sys file...

I'd rather like to expose target and interval, stop disabling ecn
dynamically, and have something closer
to an ewma for fiddling with the target in the first place.....

/me hides

> > but with current mac80211 versions (current means last 2-3 years). they
> > are just unstable and running out of memory after a while
> > the only thing which helped was cutting of the memory limit of fq_codel
> > inside mac80211
> > i also have another fancy testunit which is a linksys wrt400 with 32 mb
> > ram and 2 ath9k based wifi chipsets. no hope here fonr running stable
> > for only 5 minutes even with a single connection under load (my crashing
> > test is running a hdtv iptv stream converted to unicast using a
> > stateless eoip tunnel)
> >
> >> I try to encourage folk to run the rtt_fair tests in flent when
> >> twiddling with wifi. Those really shows how bad things are when you
> >> don't have ATF + FQ + Per station aggregation and lots of
> >> clients. Single threaded tests are misleading.
> > i know but even single threaded tests arent working good on such
> > devices. so there is no need to talk about the benefits of atf,fq_codel etc.
> > but there is need to talk about configurable use of it which also allows
> > to disable it if required.

I 110% agree that a system that can stay up for years is much better
than one that is fast for 5 minutes!

However I'd like a chance, in collaborating with you and your upcoming
patches - to try and narrow
down crash bugs to various subsystems and be able to get some
benchmarks done that I simply
couldn't do anymore at the financial conclusion of the make-wifi-fast
and cake projects.

I think I have a lot of gear that is dd-wrt compatible - apu2,
wndr3700s, 3800s....

The reduce truesize patch had helped a lot at the time (2012). There
were all kinds of flaky bugs that disappeared.

the new drop monitor patchset looks WONDERFUL for seeing more about
packet drop behavior in the stack, but
it's a 5.3(?) feature only.

I note that I run 18.06.1 on my 32MB pico and nanostations on the
lupin campus, but I run no gui, few additional applications at all
(except babel, snmpd, netperf, and the other core needed daemons).  My
uptimes are principally governed by power failures. I can't remember
the last  "crash, crash" I had, and I do track memory leaks (none).
That said, I'm painfully aware that I should probably give dd-wrt and
openwrt 19.x some testing just to make sure there's no regressions,
but have been reluctant to get involved again without more partners in
crime, because the scars from deploying 18.x widely are only beginning
to heal... and only last week did the needed babel 1.9 upgrade arrive
so I can finally redeploy ipv6 universally. I fear my current
reliability metrics are so good because I took down ipv6 last year....

Pico:

root@pool2:~# free
             total         used         free       shared      buffers
Mem:         28480        23796         4684           92         1868
-/+ buffers:              21928         6552
Swap:            0            0            0

root@pool2:~# uptime
 11:38:09 up 43 days, 21:37,  load average: 0.04, 0.03, 0.04

Same workload over here, on a wndr3800, almost exactly the same config

root@couch:~# free
             total       used       free     shared    buffers     cached
Mem:         60320      22872      37448         68       1960       6120
-/+ buffers/cache:      14792      45528
Swap:            0          0          0


>
> Disabling the fq part won't actually gain you much in terms of memory
> usage, though, as most of it is packet memory which is already
> configurable.
>
> The one exception to this is the static overhead of 'struct fq_flow', of
> which mac80211 currently allocates 4k. That's 300k of memory which is
> currently not configurable. But that could be fixed :)
>
> -Toke
--

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] [Battlemesh] Wifi Memory limits in small platforms

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> On Thu, Aug 22, 2019 at 11:23 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>>
>> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>>
>> > Am 22.08.2019 um 19:03 schrieb Dave Taht:
>> >> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>> >>
>> >>> Am 22.08.2019 um 15:15 schrieb Dave Taht:
>> >>>> It's very good to know how much folk have been struggling to keep
>> >>>> things from OOMing on 32MB platforms. I'd like to hope that the
>> >>>> unified memory management in cake (vs a collection of QoS qdiscs) and
>> >>>> the new fq_codel for wifi stuff (cutting it down to 1 alloc from four)
>> >>>> help, massively on this issue, but until today I was unaware of how
>> >>>> much the field may have been patching things out.
>> >>>>
>> >>>> The default 32MB memory limits in fq_codel comes from the stressing
>> >>>> about 10GigE networking from google. 4MB is limit in openwrt,
>> >>>> which is suitable for ~1Gbit, and is sort of there  due to 802.11ac's
>> >>>> maximum (impossible to hit) of a txop that large.
>> >> I did kind of conflate "qos + fq_codel" vs wifi in this message. It
>> >> looks like yer staying with me.
>> >>
>> >>>> Something as small as 256K is essentially about 128 full size packets
>> >>>> (and often, acks from an ethernet device's rx ring eat 2k).
>> >>> what i miss in mac80211 is the following option "fq_codel = off"
>> >>> its essential and i will definitly work on a patch to deal with this
>> >>> way for low memory 802.11n platforms.
>> >> Well, it would be my hope that turning it off would A) not help that
>> >> much on memory or cpu and B) show such a dramatic reduction in
>> >> multi-station performance that you'd immediately turn it on again.
>> > isnt it better to have a working platform with less performance than a
>> > crashing platform with no performance?
>> > i mean i can user older mac80211 versions without that issue on a
>> > typical nanostation 2/5 which is often used just as CPE device
>>
>> So before the queueing patches to mac80211, the maximum packet queue
>> size for ath9k was 3MB in total, or 2.2MB if only a single AC was used
>> on the WiFi link (that's 128 packets in the driver + 1000 in the
>> pfifo_fast qdisc * 2074 bytes for the truesize of a full-size packet).
>> Whereas now the default is 4MB for a non-vht device. So it's not
>> actually that big of a difference, and as you've already discovered the
>> defaults can be changed.
>>
>> Would it be helpful to add support for setting the memory limit in
>> hostapd (to avoid having to patch the kernel default)?
>
> hmm. I guess exposing that via netlink, etc is a good idea. Me I just
> write the sys/kernel/debug/*/*/aqm files.

It already is, and you can set it through iw (as I pointed out
up-thread):

iw phy phy0 set txq memory_limit 2097152

But it's not supported in hostapd, so you have to do that manually as it
is now.

> btw:
>
> qos_map in my mind, for APs at this point, should default to the best
> effort queue only. Not sure how to set that in openwrt (I just patched
> it out of the kernel).

Think it's possible to set this in hostapd config; haven't tried it...

-Toke

Re: [Cake] [Battlemesh] Wifi Memory limits in small platforms

[Sebastian Moeller]

> On Aug 22, 2019, at 21:37, Toke Høiland-Jørgensen <toke@redhat.com> wrote:
> 
> Dave Taht <dave.taht@gmail.com> writes:
> 
>> On Thu, Aug 22, 2019 at 11:23 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>>> 
>>> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>>> 
>>>> Am 22.08.2019 um 19:03 schrieb Dave Taht:
>>>>> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>>>>> 
>>>>>> Am 22.08.2019 um 15:15 schrieb Dave Taht:
>>>>>>> It's very good to know how much folk have been struggling to keep
>>>>>>> things from OOMing on 32MB platforms. I'd like to hope that the
>>>>>>> unified memory management in cake (vs a collection of QoS qdiscs) and
>>>>>>> the new fq_codel for wifi stuff (cutting it down to 1 alloc from four)
>>>>>>> help, massively on this issue, but until today I was unaware of how
>>>>>>> much the field may have been patching things out.
>>>>>>> 
>>>>>>> The default 32MB memory limits in fq_codel comes from the stressing
>>>>>>> about 10GigE networking from google. 4MB is limit in openwrt,
>>>>>>> which is suitable for ~1Gbit, and is sort of there  due to 802.11ac's
>>>>>>> maximum (impossible to hit) of a txop that large.
>>>>> I did kind of conflate "qos + fq_codel" vs wifi in this message. It
>>>>> looks like yer staying with me.
>>>>> 
>>>>>>> Something as small as 256K is essentially about 128 full size packets
>>>>>>> (and often, acks from an ethernet device's rx ring eat 2k).
>>>>>> what i miss in mac80211 is the following option "fq_codel = off"
>>>>>> its essential and i will definitly work on a patch to deal with this
>>>>>> way for low memory 802.11n platforms.
>>>>> Well, it would be my hope that turning it off would A) not help that
>>>>> much on memory or cpu and B) show such a dramatic reduction in
>>>>> multi-station performance that you'd immediately turn it on again.
>>>> isnt it better to have a working platform with less performance than a
>>>> crashing platform with no performance?
>>>> i mean i can user older mac80211 versions without that issue on a
>>>> typical nanostation 2/5 which is often used just as CPE device
>>> 
>>> So before the queueing patches to mac80211, the maximum packet queue
>>> size for ath9k was 3MB in total, or 2.2MB if only a single AC was used
>>> on the WiFi link (that's 128 packets in the driver + 1000 in the
>>> pfifo_fast qdisc * 2074 bytes for the truesize of a full-size packet).
>>> Whereas now the default is 4MB for a non-vht device. So it's not
>>> actually that big of a difference, and as you've already discovered the
>>> defaults can be changed.
>>> 
>>> Would it be helpful to add support for setting the memory limit in
>>> hostapd (to avoid having to patch the kernel default)?
>> 
>> hmm. I guess exposing that via netlink, etc is a good idea. Me I just
>> write the sys/kernel/debug/*/*/aqm files.
> 
> It already is, and you can set it through iw (as I pointed out
> up-thread):
> 
> iw phy phy0 set txq memory_limit 2097152
> 
> But it's not supported in hostapd, so you have to do that manually as it
> is now.
> 
>> btw:
>> 
>> qos_map in my mind, for APs at this point, should default to the best
>> effort queue only. Not sure how to set that in openwrt (I just patched
>> it out of the kernel).
> 
> Think it's possible to set this in hostapd config; haven't tried it...

	I believe that OpenWrt's hostapd does not support that feature, at least it did not last year when I looked...

Best Regards
	Sebastian

> 
> -Toke
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Wifi Memory limits in small platforms

[Sebastian Gottschall]

>>> but with current mac80211 versions (current means last 2-3 years). they
>>> are just unstable and running out of memory after a while
>>> the only thing which helped was cutting of the memory limit of fq_codel
>>> inside mac80211
>>> i also have another fancy testunit which is a linksys wrt400 with 32 mb
>>> ram and 2 ath9k based wifi chipsets. no hope here fonr running stable
>>> for only 5 minutes even with a single connection under load (my crashing
>>> test is running a hdtv iptv stream converted to unicast using a
>>> stateless eoip tunnel)
>>>
>>>> I try to encourage folk to run the rtt_fair tests in flent when
>>>> twiddling with wifi. Those really shows how bad things are when you
>>>> don't have ATF + FQ + Per station aggregation and lots of
>>>> clients. Single threaded tests are misleading.
>>> i know but even single threaded tests arent working good on such
>>> devices. so there is no need to talk about the benefits of atf,fq_codel etc.
>>> but there is need to talk about configurable use of it which also allows
>>> to disable it if required.
> I 110% agree that a system that can stay up for years is much better
> than one that is fast for 5 minutes!
>
> However I'd like a chance, in collaborating with you and your upcoming
> patches - to try and narrow
> down crash bugs to various subsystems and be able to get some
> benchmarks done that I simply
> couldn't do anymore at the financial conclusion of the make-wifi-fast
> and cake projects.
>
> I think I have a lot of gear that is dd-wrt compatible - apu2,
> wndr3700s, 3800s....
if its v4, these are having 128 mb (i have them too). and apu2 has 2 gb. 
so its getting real interesting
if you choose such a bad one with 32 mb ram which are still commonly 
used by "freifunk"
> The reduce truesize patch had helped a lot at the time (2012). There
> were all kinds of flaky bugs that disappeared.
i tested and it helped to make ethernet unavailable. it worked for wifi 
interfaces. but the eth0 and eth1 on my ipq8064 based
testboard did not work anymore. no dhcp lease, no ping. but i was able 
to capture inbound packets. (qos was not even enabled while testing, so 
no cake, fq_code letc. just standard sfq scheduler)
so i reverted and all worked again
>
> the new drop monitor patchset looks WONDERFUL for seeing more about
> packet drop behavior in the stack, but
> it's a 5.3(?) feature only.
i love backporting :-)
>
> I note that I run 18.06.1 on my 32MB pico and nanostations on the
> lupin campus, but I run no gui, few additional applications at all
> (except babel, snmpd, netperf, and the other core needed daemons).  My
> uptimes are principally governed by power failures. I can't remember
> the last  "crash, crash" I had, and I do track memory leaks (none).
> That said, I'm painfully aware that I should probably give dd-wrt and
> openwrt 19.x some testing just to make sure there's no regressions,
> but have been reluctant to get involved again without more partners in
> crime, because the scars from deploying 18.x widely are only beginning
> to heal... and only last week did the needed babel 1.9 upgrade arrive
> so I can finally redeploy ipv6 universally. I fear my current
> reliability metrics are so good because I took down ipv6 last year....
my workaround with memory problems is also disabling http normally. i 
have some of these nanostations in the field

just running hostapd, snmp, syslog. but anything else is disabled due 
the oom problematics. it never was a real crash.

but oom. but i never played with babel. ospf etc. all working out of the 
box based on quagga on low end devices and frr on bigger ones.

>
> Pico:
>
> root@pool2:~# free
>               total         used         free       shared      buffers
> Mem:         28480        23796         4684           92         1868
> -/+ buffers:              21928         6552
> Swap:            0            0            0
>
> root@pool2:~# uptime
>   11:38:09 up 43 days, 21:37,  load average: 0.04, 0.03, 0.04
>
> Same workload over here, on a wndr3800, almost exactly the same config
>
> root@couch:~# free
>               total       used       free     shared    buffers     cached
> Mem:         60320      22872      37448         68       1960       6120
> -/+ buffers/cache:      14792      45528
> Swap:            0          0          0

NS2

root@TRO1:~# free

               total        used        free      shared buff/cache   
available
Mem:          29124       19228        3552           0 6344        7752
Swap:             0           0           0

wndr3700v4

root@DD-WRT:~# free
               total        used        free      shared buff/cache   
available
Mem:         125884       23048       92940           0 9896       99824
Swap:             0           0           0
root@DD-WRT:~#


>
>> Disabling the fq part won't actually gain you much in terms of memory
>> usage, though, as most of it is packet memory which is already
>> configurable.
>>
>> The one exception to this is the static overhead of 'struct fq_flow', of
>> which mac80211 currently allocates 4k. That's 300k of memory which is
>> currently not configurable. But that could be fixed :)
>>
>> -Toke
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
>

Re: [Cake] Wifi Memory limits in small platforms

[Dave Taht]

Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:

>>>> but with current mac80211 versions (current means last 2-3 years). they
>>>> are just unstable and running out of memory after a while
>>>> the only thing which helped was cutting of the memory limit of fq_codel
>>>> inside mac80211
>>>> i also have another fancy testunit which is a linksys wrt400 with 32 mb
>>>> ram and 2 ath9k based wifi chipsets. no hope here fonr running stable
>>>> for only 5 minutes even with a single connection under load (my crashing
>>>> test is running a hdtv iptv stream converted to unicast using a
>>>> stateless eoip tunnel)
>>>>
>>>>> I try to encourage folk to run the rtt_fair tests in flent when
>>>>> twiddling with wifi. Those really shows how bad things are when you
>>>>> don't have ATF + FQ + Per station aggregation and lots of
>>>>> clients. Single threaded tests are misleading.
>>>> i know but even single threaded tests arent working good on such
>>>> devices. so there is no need to talk about the benefits of atf,fq_codel etc.
>>>> but there is need to talk about configurable use of it which also allows
>>>> to disable it if required.
>> I 110% agree that a system that can stay up for years is much better
>> than one that is fast for 5 minutes!
>>
>> However I'd like a chance, in collaborating with you and your upcoming
>> patches - to try and narrow
>> down crash bugs to various subsystems and be able to get some
>> benchmarks done that I simply
>> couldn't do anymore at the financial conclusion of the make-wifi-fast
>> and cake projects.
>>
>> I think I have a lot of gear that is dd-wrt compatible - apu2,
>> wndr3700s, 3800s....
> if its v4, these are having 128 mb (i have them too).

These are from the cerowrt era, so, 32 or 64MB of ram.

> and apu2 has 2
> gb. so its getting real interesting
> if you choose such a bad one with 32 mb ram which are still commonly
> used by "freifunk"

One thing we can start doing more 'round here is to boot the x86 boxes
with mem=32MB or something similar (40% larger due to 64 bits? no idea,
maybe look at free mem on a similar config) to see what shows up. 

For example, one of my APU2s has dual ath9/ath10k cards which is a
a reasonable sim of one of your configs. 

>> The reduce truesize patch had helped a lot at the time (2012). There
>> were all kinds of flaky bugs that disappeared.
> i tested and it helped to make ethernet unavailable. it worked for

thx for making me chortle in sad empathy.

> wifi interfaces. but the eth0 and eth1 on my ipq8064 based
> testboard did not work anymore. no dhcp lease, no ping. but i was able
> to capture inbound packets. (qos was not even enabled while testing,
> so no cake, fq_code letc. just standard sfq scheduler)
> so i reverted and all worked again

OK. Thx for trying. there have been so many bugs in gso/gro and hardware
offloads that I figure that that's why the patch was dropped over time.

is cake's gso-splitting working on that same hardware? I'm not sure
to what extent that reduces packet size or not these days.

I'll try that again on x86, maybe it needed to pullskb....

>>
>> the new drop monitor patchset looks WONDERFUL for seeing more about
>> packet drop behavior in the stack, but
>> it's a 5.3(?) feature only.
> i love backporting :-)

I used to but these days I'm content to work out of net-next x.y.0-rc4
or later. I get more sleep that way. Oh, wait, it just hit that....

>>
>> I note that I run 18.06.1 on my 32MB pico and nanostations on the
>> lupin campus, but I run no gui, few additional applications at all
>> (except babel, snmpd, netperf, and the other core needed daemons).  My
>> uptimes are principally governed by power failures. I can't remember
>> the last  "crash, crash" I had, and I do track memory leaks (none).
>> That said, I'm painfully aware that I should probably give dd-wrt and
>> openwrt 19.x some testing just to make sure there's no regressions,
>> but have been reluctant to get involved again without more partners in
>> crime, because the scars from deploying 18.x widely are only beginning
>> to heal... and only last week did the needed babel 1.9 upgrade arrive
>> so I can finally redeploy ipv6 universally. I fear my current
>> reliability metrics are so good because I took down ipv6 last year....
> my workaround with memory problems is also disabling http normally. i
> have some of these nanostations in the field
>
> just running hostapd, snmp, syslog. but anything else is disabled due
> the oom problematics. it never was a real crash.
>
> but oom. but i never played with babel. ospf etc. all working out of
> the box based on quagga on low end devices and frr on bigger ones.
>
>>
>> Pico:
>>
>> root@pool2:~# free
>>               total         used         free       shared      buffers
>> Mem:         28480        23796         4684           92         1868
>> -/+ buffers:              21928         6552
>> Swap:            0            0            0
>>
>> root@pool2:~# uptime
>>   11:38:09 up 43 days, 21:37,  load average: 0.04, 0.03, 0.04
>>
>> Same workload over here, on a wndr3800, almost exactly the same config
>>
>> root@couch:~# free
>>               total       used       free     shared    buffers     cached
>> Mem:         60320      22872      37448         68       1960       6120
>> -/+ buffers/cache:      14792      45528
>> Swap:            0          0          0
>
> NS2
>
> root@TRO1:~# free
>
>               total        used        free      shared buff/cache  
> available
> Mem:          29124       19228        3552           0 6344        7752
> Swap:             0           0           0

It looks like you are running even less stuff than I am. And this
machine is running with 256k bufs?

> wndr3700v4
>
> root@DD-WRT:~# free
>               total        used        free      shared buff/cache  
> available
> Mem:         125884       23048       92940           0 9896       99824
> Swap:             0           0           0
> root@DD-WRT:~#
>
>
>>
>>> Disabling the fq part won't actually gain you much in terms of memory
>>> usage, though, as most of it is packet memory which is already
>>> configurable.
>>>
>>> The one exception to this is the static overhead of 'struct fq_flow', of
>>> which mac80211 currently allocates 4k. That's 300k of memory which is
>>> currently not configurable. But that could be fixed :)
>>>
>>> -Toke
>> --
>>
>> Dave Täht
>> CTO, TekLibre, LLC
>> http://www.teklibre.com
>> Tel: 1-831-205-9740
>>

Re: [Cake] Wifi Memory limits in small platforms

[Sebastian Gottschall]

Am 23.08.2019 um 01:39 schrieb Dave Taht:
> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>
>>>>> but with current mac80211 versions (current means last 2-3 years). they
>>>>> are just unstable and running out of memory after a while
>>>>> the only thing which helped was cutting of the memory limit of fq_codel
>>>>> inside mac80211
>>>>> i also have another fancy testunit which is a linksys wrt400 with 32 mb
>>>>> ram and 2 ath9k based wifi chipsets. no hope here fonr running stable
>>>>> for only 5 minutes even with a single connection under load (my crashing
>>>>> test is running a hdtv iptv stream converted to unicast using a
>>>>> stateless eoip tunnel)
>>>>>
>>>>>> I try to encourage folk to run the rtt_fair tests in flent when
>>>>>> twiddling with wifi. Those really shows how bad things are when you
>>>>>> don't have ATF + FQ + Per station aggregation and lots of
>>>>>> clients. Single threaded tests are misleading.
>>>>> i know but even single threaded tests arent working good on such
>>>>> devices. so there is no need to talk about the benefits of atf,fq_codel etc.
>>>>> but there is need to talk about configurable use of it which also allows
>>>>> to disable it if required.
>>> I 110% agree that a system that can stay up for years is much better
>>> than one that is fast for 5 minutes!
>>>
>>> However I'd like a chance, in collaborating with you and your upcoming
>>> patches - to try and narrow
>>> down crash bugs to various subsystems and be able to get some
>>> benchmarks done that I simply
>>> couldn't do anymore at the financial conclusion of the make-wifi-fast
>>> and cake projects.
>>>
>>> I think I have a lot of gear that is dd-wrt compatible - apu2,
>>> wndr3700s, 3800s....
>> if its v4, these are having 128 mb (i have them too).
> These are from the cerowrt era, so, 32 or 64MB of ram.
>
>> and apu2 has 2
>> gb. so its getting real interesting
>> if you choose such a bad one with 32 mb ram which are still commonly
>> used by "freifunk"
> One thing we can start doing more 'round here is to boot the x86 boxes
> with mem=32MB or something similar (40% larger due to 64 bits? no idea,
> maybe look at free mem on a similar config) to see what shows up.
>
> For example, one of my APU2s has dual ath9/ath10k cards which is a
> a reasonable sim of one of your configs.
since x64 have alot more differen configurations the kernels are much 
bigger (drivers, drivers, drivers) . i'm sure it will not work with just 
32 mb
>
>>> The reduce truesize patch had helped a lot at the time (2012). There
>>> were all kinds of flaky bugs that disappeared.
>> i tested and it helped to make ethernet unavailable. it worked for
> thx for making me chortle in sad empathy.
>
>> wifi interfaces. but the eth0 and eth1 on my ipq8064 based
>> testboard did not work anymore. no dhcp lease, no ping. but i was able
>> to capture inbound packets. (qos was not even enabled while testing,
>> so no cake, fq_code letc. just standard sfq scheduler)
>> so i reverted and all worked again
> OK. Thx for trying. there have been so many bugs in gso/gro and hardware
> offloads that I figure that that's why the patch was dropped over time.
>
> is cake's gso-splitting working on that same hardware? I'm not sure
> to what extent that reduces packet size or not these days.
cake works yes, but i have not checked explicit for gso-splitting. it 
just worked
>
> I'll try that again on x86, maybe it needed to pullskb....
can be hw specific. but who knows.
>
>>> Pico:
>>>
>>> root@pool2:~# free
>>>                total         used         free       shared      buffers
>>> Mem:         28480        23796         4684           92         1868
>>> -/+ buffers:              21928         6552
>>> Swap:            0            0            0
>>>
>>> root@pool2:~# uptime
>>>    11:38:09 up 43 days, 21:37,  load average: 0.04, 0.03, 0.04
>>>
>>> Same workload over here, on a wndr3800, almost exactly the same config
>>>
>>> root@couch:~# free
>>>                total       used       free     shared    buffers     cached
>>> Mem:         60320      22872      37448         68       1960       6120
>>> -/+ buffers/cache:      14792      45528
>>> Swap:            0          0          0
>> NS2
>>
>> root@TRO1:~# free
>>
>>                total        used        free      shared buff/cache
>> available
>> Mem:          29124       19228        3552           0 6344        7752
>> Swap:             0           0           0
> It looks like you are running even less stuff than I am. And this
> machine is running with 256k bufs?
yes. but it may also depend what you're running. i mean openwrt and 
dd-wrt are very different.
the webserver etc. in dd-wrt might be more lightweight. i do not use lua 
or any other slow component
its all written in C including the web code.

thats my process list

   PID USER       VSZ STAT COMMAND
     1 root      1172 S    /sbin/init
     2 root         0 SW   [kthreadd]
     3 root         0 SW   [ksoftirqd/0]
     4 root         0 SW   [kworker/0:0]
     5 root         0 SW<  [kworker/0:0H]
     6 root         0 SW   [kworker/u2:0]
     7 root         0 SW<  [khelper]
     8 root         0 SW<  [writeback]
    10 root         0 SW<  [crypto]
    12 root         0 SW<  [bioset]
    64 root         0 SW<  [kblockd]
    65 root         0 SW   [kswapd0]
    66 root         0 SW   [kworker/0:1]
   108 root         0 SW   [fsnotify_mark]
   120 root         0 SW<  [deferwq]
   121 root         0 SW   [kworker/u2:2]
   503 root      1176 S    /sbin/hotplug2 --set-rules-file 
/etc/hotplug2.rules --persistent
   524 root      1856 S    watchdog
   553 root         0 SW<  [cfg80211]
   574 root      1792 S    /sbin/wlanled -l generic_0:-94 -l 
generic_1:-80 -l generic_11:-73 -l generic_7:-65
   777 root      3780 S    hostapd -B -P /var/run/ath0_hostapd.pid 
/tmp/ath0_hostap.conf
   951 root      1812 S    wland
  1025 root       904 S    cron
  1083 root      1544 S    resetbutton
  1086 root      1856 S    process_monitor
  1217 root      1376 S    syslogd -Z -L -R 192.168.0.117
  1224 root      1376 S    klogd
  1341 root      1112 S    mactelnetd
  1456 root      1224 S    dropbear -b /tmp/loginprompt -r 
/tmp/root/.ssh/ssh_host_rsa_key -p 22 -a
  4449 root      3692 S    httpd -p 80
10770 root      1172 S    dnsmasq -u root -g root 
--conf-file=/tmp/dnsmasq.conf
29786 root      1292 R    dropbear -b /tmp/loginprompt -r 
/tmp/root/.ssh/ssh_host_rsa_key -p 22 -a
29787 root      1376 S    -sh
29799 root      1376 R    ps


>
>> wndr3700v4
>>
>> root@DD-WRT:~# free
>>                total        used        free      shared buff/cache
>> available
>> Mem:         125884       23048       92940           0 9896       99824
>> Swap:             0           0           0
>> root@DD-WRT:~#
>>

Re: [Cake] Wifi Memory limits in small platforms

[Sebastian Moeller]

> On Aug 23, 2019, at 01:39, Dave Taht <dave@taht.net> wrote:
> 
> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
> 
>>>>> but with current mac80211 versions (current means last 2-3 years). they
>>>>> are just unstable and running out of memory after a while
>>>>> the only thing which helped was cutting of the memory limit of fq_codel
>>>>> inside mac80211
>>>>> i also have another fancy testunit which is a linksys wrt400 with 32 mb
>>>>> ram and 2 ath9k based wifi chipsets. no hope here fonr running stable
>>>>> for only 5 minutes even with a single connection under load (my crashing
>>>>> test is running a hdtv iptv stream converted to unicast using a
>>>>> stateless eoip tunnel)
>>>>> 
>>>>>> I try to encourage folk to run the rtt_fair tests in flent when
>>>>>> twiddling with wifi. Those really shows how bad things are when you
>>>>>> don't have ATF + FQ + Per station aggregation and lots of
>>>>>> clients. Single threaded tests are misleading.
>>>>> i know but even single threaded tests arent working good on such
>>>>> devices. so there is no need to talk about the benefits of atf,fq_codel etc.
>>>>> but there is need to talk about configurable use of it which also allows
>>>>> to disable it if required.
>>> I 110% agree that a system that can stay up for years is much better
>>> than one that is fast for 5 minutes!
>>> 
>>> However I'd like a chance, in collaborating with you and your upcoming
>>> patches - to try and narrow
>>> down crash bugs to various subsystems and be able to get some
>>> benchmarks done that I simply
>>> couldn't do anymore at the financial conclusion of the make-wifi-fast
>>> and cake projects.
>>> 
>>> I think I have a lot of gear that is dd-wrt compatible - apu2,
>>> wndr3700s, 3800s....
>> if its v4, these are having 128 mb (i have them too).
> 
> These are from the cerowrt era, so, 32 or 64MB of ram.

	I believe we only used wndr3700v2 (64MB) and wndr3800 (128MB), at least those were the recommended ones. I also remember making these OOM with a simple UDP flood with randomized port addresses quite easily intially. That is, until we used fq_codel's limit keyword to restrict the number of maximally queued packets. This experience also carried into cake and culminated into the memlimit keyword. It seems I completely missed the addition of the "memory_limit BYTES" keyword to fq_codel, which seems a better fit to our needs than the "limit 1001" we currently use (why 1001 instead of 1000, simply to be able to quickly see whether this is our limit or something the user used, pleople ted to leave the last digit alone when playing with these parameters ;)).
I guess I have not bothered to repeat that test since fq_codel became the default qdisc in OpenWrt...

Best Regards
	Sebastian


> 
>> and apu2 has 2
>> gb. so its getting real interesting
>> if you choose such a bad one with 32 mb ram which are still commonly
>> used by "freifunk"
> 
> One thing we can start doing more 'round here is to boot the x86 boxes
> with mem=32MB or something similar (40% larger due to 64 bits? no idea,
> maybe look at free mem on a similar config) to see what shows up. 
> 
> For example, one of my APU2s has dual ath9/ath10k cards which is a
> a reasonable sim of one of your configs. 
> 
>>> The reduce truesize patch had helped a lot at the time (2012). There
>>> were all kinds of flaky bugs that disappeared.
>> i tested and it helped to make ethernet unavailable. it worked for
> 
> thx for making me chortle in sad empathy.
> 
>> wifi interfaces. but the eth0 and eth1 on my ipq8064 based
>> testboard did not work anymore. no dhcp lease, no ping. but i was able
>> to capture inbound packets. (qos was not even enabled while testing,
>> so no cake, fq_code letc. just standard sfq scheduler)
>> so i reverted and all worked again
> 
> OK. Thx for trying. there have been so many bugs in gso/gro and hardware
> offloads that I figure that that's why the patch was dropped over time.
> 
> is cake's gso-splitting working on that same hardware? I'm not sure
> to what extent that reduces packet size or not these days.
> 
> I'll try that again on x86, maybe it needed to pullskb....
> 
>>> 
>>> the new drop monitor patchset looks WONDERFUL for seeing more about
>>> packet drop behavior in the stack, but
>>> it's a 5.3(?) feature only.
>> i love backporting :-)
> 
> I used to but these days I'm content to work out of net-next x.y.0-rc4
> or later. I get more sleep that way. Oh, wait, it just hit that....
> 
>>> 
>>> I note that I run 18.06.1 on my 32MB pico and nanostations on the
>>> lupin campus, but I run no gui, few additional applications at all
>>> (except babel, snmpd, netperf, and the other core needed daemons).  My
>>> uptimes are principally governed by power failures. I can't remember
>>> the last  "crash, crash" I had, and I do track memory leaks (none).
>>> That said, I'm painfully aware that I should probably give dd-wrt and
>>> openwrt 19.x some testing just to make sure there's no regressions,
>>> but have been reluctant to get involved again without more partners in
>>> crime, because the scars from deploying 18.x widely are only beginning
>>> to heal... and only last week did the needed babel 1.9 upgrade arrive
>>> so I can finally redeploy ipv6 universally. I fear my current
>>> reliability metrics are so good because I took down ipv6 last year....
>> my workaround with memory problems is also disabling http normally. i
>> have some of these nanostations in the field
>> 
>> just running hostapd, snmp, syslog. but anything else is disabled due
>> the oom problematics. it never was a real crash.
>> 
>> but oom. but i never played with babel. ospf etc. all working out of
>> the box based on quagga on low end devices and frr on bigger ones.
>> 
>>> 
>>> Pico:
>>> 
>>> root@pool2:~# free
>>>              total         used         free       shared      buffers
>>> Mem:         28480        23796         4684           92         1868
>>> -/+ buffers:              21928         6552
>>> Swap:            0            0            0
>>> 
>>> root@pool2:~# uptime
>>>  11:38:09 up 43 days, 21:37,  load average: 0.04, 0.03, 0.04
>>> 
>>> Same workload over here, on a wndr3800, almost exactly the same config
>>> 
>>> root@couch:~# free
>>>              total       used       free     shared    buffers     cached
>>> Mem:         60320      22872      37448         68       1960       6120
>>> -/+ buffers/cache:      14792      45528
>>> Swap:            0          0          0
>> 
>> NS2
>> 
>> root@TRO1:~# free
>> 
>>               total        used        free      shared buff/cache  
>> available
>> Mem:          29124       19228        3552           0 6344        7752
>> Swap:             0           0           0
> 
> It looks like you are running even less stuff than I am. And this
> machine is running with 256k bufs?
> 
>> wndr3700v4
>> 
>> root@DD-WRT:~# free
>>               total        used        free      shared buff/cache  
>> available
>> Mem:         125884       23048       92940           0 9896       99824
>> Swap:             0           0           0
>> root@DD-WRT:~#
>> 
>> 
>>> 
>>>> Disabling the fq part won't actually gain you much in terms of memory
>>>> usage, though, as most of it is packet memory which is already
>>>> configurable.
>>>> 
>>>> The one exception to this is the static overhead of 'struct fq_flow', of
>>>> which mac80211 currently allocates 4k. That's 300k of memory which is
>>>> currently not configurable. But that could be fixed :)
>>>> 
>>>> -Toke
>>> --
>>> 
>>> Dave Täht
>>> CTO, TekLibre, LLC
>>> http://www.teklibre.com
>>> Tel: 1-831-205-9740
>>> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

[Cake] fq_codel_fast crash/lockup

[Sebastian Gottschall]

if you mind.

running on arm kernel htb+fq_codel_fast

INFO: rcu_preempt self-detected stall on CPU
         0-...: (1 GPs behind) idle=0ab/140000000000001/0 
softirq=2280/2285 fqs=5984
          (t=6000 jiffies g=211 c=210 q=565)
Task dump for CPU 0:
tc              R running      0  1024    890 0x00000002
Backtrace:
[<8001b71c>] (dump_backtrace) from [<8001b9c0>] (show_stack+0x18/0x1c)
  r7:8051ca40 r6:8050f908 r5:0000037a r4:87a8f400
[<8001b9a8>] (show_stack) from [<8005bad0>] (sched_show_task+0xe0/0xe8)
[<8005b9f0>] (sched_show_task) from [<8005cf20>] (dump_cpu_task+0x40/0x44)
  r5:00000000 r4:00000000
[<8005cee0>] (dump_cpu_task) from [<80078534>] 
(rcu_dump_cpu_stacks+0x74/0xac)
  r5:00000000 r4:8051ca40
[<800784c0>] (rcu_dump_cpu_stacks) from [<8007ba20>] 
(rcu_check_callbacks+0x2e8/0x854)
  r9:8051ca40 r8:8050e100 r7:068d8000 r6:00000235 r5:8050f9c0 r4:804fed40
[<8007b738>] (rcu_check_callbacks) from [<8007df44>] 
(update_process_times+0x44/0x6c)
  r10:00000000 r9:00000000 r8:00000000 r7:0000001d r6:87a8f400 r5:00000000
  r4:ffffe000
[<8007df00>] (update_process_times) from [<80089a08>] 
(tick_periodic+0xb0/0xb8)
  r7:0000001d r6:7fffffff r5:86dd9800 r4:8050e144
[<80089958>] (tick_periodic) from [<80089a44>] 
(tick_handle_periodic+0x34/0xac)
  r5:86dd9800 r4:ffffffff
[<80089a10>] (tick_handle_periodic) from [<8001dc04>] 
(twd_handler+0x38/0x40)
  r9:00000000 r8:86dd9800 r7:0000001d r6:87801cc0 r5:805214bc r4:00000001
[<8001dbcc>] (twd_handler) from [<80074c8c>] 
(handle_percpu_devid_irq+0x7c/0x94)
  r5:805214bc r4:805036c0
[<80074c10>] (handle_percpu_devid_irq) from [<80071214>] 
(__handle_domain_irq+0xb8/0xd8)
  r9:00000000 r8:87804800 r7:00000001 r6:804fcfc4 r5:00000000 r4:00000000
[<8007115c>] (__handle_domain_irq) from [<80009388>] 
(gic_handle_irq+0x50/0x7c)
  r9:00000000 r8:b0101100 r7:b0100100 r6:b010010c r5:87147ac8 r4:8050fb20
[<80009338>] (gic_handle_irq) from [<80009ed4>] (__irq_svc+0x54/0x90)
Exception stack(0x87147ac8 to 0x87147b10)
7ac0:                   8696e090 00000000 0000001f 00000020 86740000 
87147b78
7ae0: 86680094 86680094 87147b78 00000000 00000000 87147b24 87147b28 
87147b18
7b00: 7f4cb1ec 80017a28 20000013 ffffffff
  r9:00000000 r8:87147b78 r7:87147afc r6:ffffffff r5:20000013 r4:80017a28
[<800179d8>] (_raw_spin_lock_bh) from [<7f4cb1ec>] 
(fq_codel_dump_stats+0x68/0xd8 [sch_fq_codel_fast])
[<7f4cb184>] (fq_codel_dump_stats [sch_fq_codel_fast]) from [<802786c0>] 
(tc_fill_qdisc+0x1d0/0x284)
  r5:86740000 r4:870e0c00
[<802784f0>] (tc_fill_qdisc) from [<80278a78>] 
(qdisc_notify.isra.0+0xec/0x138)
  r10:86740000 r9:869cb800 r8:870d6308 r7:870d6306 r6:8052b780 r5:00000400
  r4:870e0c00
[<8027898c>] (qdisc_notify.isra.0) from [<80278afc>] 
(notify_and_destroy+0x38/0x50)
  r10:8052b780 r9:00000000 r8:80285a50 r7:869cba00 r6:00000000 r5:8696e000
  r4:869cb800
[<80278ac4>] (notify_and_destroy) from [<80278da8>] 
(qdisc_graft+0x294/0x360)
  r4:86740000
[<80278b14>] (qdisc_graft) from [<8027a424>] (tc_modify_qdisc+0x4a0/0x4f4)
  r10:8052b780 r9:00000000 r8:8696e000 r7:00010100 r6:870e1000 r5:870d6300
  r4:86740000
[<80279f84>] (tc_modify_qdisc) from [<802688b0>] 
(rtnetlink_rcv_msg+0x16c/0x1e4)
  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:870d7b00 r5:870d6300
  r4:000000f0
[<80268744>] (rtnetlink_rcv_msg) from [<802a37e4>] 
(netlink_rcv_skb+0x64/0xc8)
  r8:00000000 r7:870d7b00 r6:870d6300 r5:80268744 r4:870d7b00
[<802a3780>] (netlink_rcv_skb) from [<8026873c>] (rtnetlink_rcv+0x2c/0x34)
  r7:870d7b00 r6:87224000 r5:0000003c r4:870d7b00
[<80268710>] (rtnetlink_rcv) from [<802a3218>] 
(netlink_unicast+0x158/0x20c)
  r5:0000003c r4:878a8400
[<802a30c0>] (netlink_unicast) from [<802a3610>] 
(netlink_sendmsg+0x344/0x38c)
  r8:024000c0 r7:87224000 r6:0000003c r5:870d7b00 r4:87147f4c
[<802a32cc>] (netlink_sendmsg) from [<8023f680>] 
(___sys_sendmsg+0x1e8/0x22c)
  r10:87147e28 r9:87147e28 r8:00000000 r7:00000000 r6:8751ae00 r5:00000000
  r4:87147f4c
[<8023f498>] (___sys_sendmsg) from [<8023fd88>] (__sys_sendmsg+0x44/0x68)
  r10:00000000 r9:87146000 r8:80009724 r7:00000128 r6:00000000 r5:7e950ccc
  r4:8751ae00
[<8023fd44>] (__sys_sendmsg) from [<8023fdbc>] (SyS_sendmsg+0x10/0x14)
  r6:00000000 r5:00000000 r4:00000000
[<8023fdac>] (SyS_sendmsg) from [<80009560>] (ret_fast_syscall+0x0/0x48)

Re: [Cake] pie in dd-wrt

[Toke Høiland-Jørgensen]

Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:

> Am 21.08.2019 um 18:21 schrieb Sebastian Moeller:
>>
>> On August 21, 2019 6:12:03 PM GMT+02:00, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>>> thats rather old. i rewrote all the qos code in the last 4 or 5 days.
>>> so
>>> things might be changed. next phase is bringing all the link level
>>> detail configuration stuff into the gui which will be done
>>> tomorrow or at least still within this week.
>>> i also added now cake to some smaller low budged routers with limited
>>> resources, so see how it runs. i had bad experiences with fq_codel in
>>> the past due some high memory usage.
>>> especially since its hard coded somewhat into the wireless ath9k
>>> driver.
>>> so i already modded it for more efficient handling. 4 mb max per queue
>>> is simply too much for  a 32 mb ram based router.
>> This is why I'm sqm we reduced the queued packet maximum m to around 1000, and also why cake has an explicit memlimit keyword.
> yeah but does this help if fq_codel is hardcoded into mac80211? fq_codel 
> has a memlimit  keyword too btw. its fixed to 4MB. i reduced it to 256kb 
> on low memory architectures. no other way to get around OOM problems
> mac80211  does always make use of fq_codel. it has a own build in 
> implementation

The mac80211 implementation also has a memlimit parameter. You can set
it through debugfs - 2MB would be:

echo 2097152 > /sys/kernel/debug/ieee80211/phy0/aqm

or through iw:

iw phy phy0 set txq memory_limit 2097152

The nl80211 attribute is called NL80211_ATTR_TXQ_MEMORY_LIMIT.

-Toke

Re: [Cake] pie in dd-wrt

[Sebastian Gottschall]

Am 21.08.2019 um 23:39 schrieb Toke Høiland-Jørgensen:
> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>
>> Am 21.08.2019 um 18:21 schrieb Sebastian Moeller:
>>> On August 21, 2019 6:12:03 PM GMT+02:00, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>>>> thats rather old. i rewrote all the qos code in the last 4 or 5 days.
>>>> so
>>>> things might be changed. next phase is bringing all the link level
>>>> detail configuration stuff into the gui which will be done
>>>> tomorrow or at least still within this week.
>>>> i also added now cake to some smaller low budged routers with limited
>>>> resources, so see how it runs. i had bad experiences with fq_codel in
>>>> the past due some high memory usage.
>>>> especially since its hard coded somewhat into the wireless ath9k
>>>> driver.
>>>> so i already modded it for more efficient handling. 4 mb max per queue
>>>> is simply too much for  a 32 mb ram based router.
>>> This is why I'm sqm we reduced the queued packet maximum m to around 1000, and also why cake has an explicit memlimit keyword.
>> yeah but does this help if fq_codel is hardcoded into mac80211? fq_codel
>> has a memlimit  keyword too btw. its fixed to 4MB. i reduced it to 256kb
>> on low memory architectures. no other way to get around OOM problems
>> mac80211  does always make use of fq_codel. it has a own build in
>> implementation
> The mac80211 implementation also has a memlimit parameter. You can set
> it through debugfs - 2MB would be:
>
> echo 2097152 > /sys/kernel/debug/ieee80211/phy0/aqm
>
> or through iw:
>
> iw phy phy0 set txq memory_limit 2097152
>
> The nl80211 attribute is called NL80211_ATTR_TXQ_MEMORY_LIMIT.
as i said i already modified mac80211 in a way that it sets sane memory 
limits depending on the architecture. devices with just 32 mb ram run 
only stable with 256kb memory limit.
so i configured different defaults. but the point is still that for a 
standard user (lets say in openwrt) the current implementation is not 
good. somewhere in the openwrt community i was reading that such devices 
should not be used anymore for openwrt due the memory limitations. but 
thats no solution for me if it was working before the introduction of 
fq_codel in mac80211
>
> -Toke
>

Re: [Cake] pie in dd-wrt

[Toke Høiland-Jørgensen]

Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:

> Am 21.08.2019 um 23:39 schrieb Toke Høiland-Jørgensen:
>> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>>
>>> Am 21.08.2019 um 18:21 schrieb Sebastian Moeller:
>>>> On August 21, 2019 6:12:03 PM GMT+02:00, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>>>>> thats rather old. i rewrote all the qos code in the last 4 or 5 days.
>>>>> so
>>>>> things might be changed. next phase is bringing all the link level
>>>>> detail configuration stuff into the gui which will be done
>>>>> tomorrow or at least still within this week.
>>>>> i also added now cake to some smaller low budged routers with limited
>>>>> resources, so see how it runs. i had bad experiences with fq_codel in
>>>>> the past due some high memory usage.
>>>>> especially since its hard coded somewhat into the wireless ath9k
>>>>> driver.
>>>>> so i already modded it for more efficient handling. 4 mb max per queue
>>>>> is simply too much for  a 32 mb ram based router.
>>>> This is why I'm sqm we reduced the queued packet maximum m to around 1000, and also why cake has an explicit memlimit keyword.
>>> yeah but does this help if fq_codel is hardcoded into mac80211? fq_codel
>>> has a memlimit  keyword too btw. its fixed to 4MB. i reduced it to 256kb
>>> on low memory architectures. no other way to get around OOM problems
>>> mac80211  does always make use of fq_codel. it has a own build in
>>> implementation
>> The mac80211 implementation also has a memlimit parameter. You can set
>> it through debugfs - 2MB would be:
>>
>> echo 2097152 > /sys/kernel/debug/ieee80211/phy0/aqm
>>
>> or through iw:
>>
>> iw phy phy0 set txq memory_limit 2097152
>>
>> The nl80211 attribute is called NL80211_ATTR_TXQ_MEMORY_LIMIT.
> as i said i already modified mac80211 in a way that it sets sane memory 
> limits depending on the architecture. devices with just 32 mb ram run 
> only stable with 256kb memory limit.
> so i configured different defaults. but the point is still that for a 
> standard user (lets say in openwrt) the current implementation is not 
> good. somewhere in the openwrt community i was reading that such devices 
> should not be used anymore for openwrt due the memory limitations. but 
> thats no solution for me if it was working before the introduction of 
> fq_codel in mac80211

... which is why it is configurable?

-Toke

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

Am 20.08.2019 um 21:07 schrieb Jonathan Morton:
>> On 20 Aug, 2019, at 9:39 pm, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>>
>> …a heavy bittorrent downloader will still steal the bandwidth of my scp session.
> If you can identify the Bittorrent packets, you can mark them CS1, and switch on Cake's "diffserv3" mode (as it is by default).  Then the Bittorrent packets will still be able to use full bandwidth if it's available, but will be limited to 1/16th of the total if there is contention.
we have already priorisation class system in the gui which was used in 
the past which offers more classes than offered but this isnt the point 
here.
it wont be limited to a 1/16 if we have multiple lan interfaces or 
different settings per user (specified by mac or ip/net). so we need a 
cake instance for each lan interface but these cake instances dont know 
of each other. so each will use the full bandwidth available
its not just about limiting a single service on a single interface. this 
isnt a problem

>
>   - Jonathan Morton

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

can you explain line

524 in 
https://github.com/dtaht/fq_codel_fast/blob/master/sch_fq_codel_fast.c?


Am 20.08.2019 um 18:47 schrieb Sebastian Gottschall:
>
> Am 20.08.2019 um 18:24 schrieb Dave Taht:
>> On Tue, Aug 20, 2019 at 5:09 AM Sebastian Gottschall
>> <s.gottschall@newmedia-net.de> wrote:
>>> :-) i'm following this list and yes we are working on bringing cake 
>>> in :-)
>> Yea! thx for being on the list!
>>
>>> is there any question behind this link from your side?
>> I just wanted to make people here aware that it was happening.
>>
>> Is there a build now?
> the first builds with cake are already out yes, but unfinished. we 
> started then to rewrite major parts of the qos code. i expect to push out
> a new build tomorrow. it will still not use the full potential of cake 
> since we have to bring all together with the priority and ndpi and 
> filter based filter together
> with the cake scheduler. but it works so far and will be better in the 
> next days and weeks after we have found a solution for it. cake is 
> more a all in wonder solution
> so we have to use the features of cake itself to reimplement the 
> priorisation classes
>>
>> If I had any one principal request it would be to make sure the dd-wrt
>> gui (if one is made) exposes the link layer parameters. Getting the
>> framing wrong is about the biggest error I see in the deployment:
>> https://www.youtube.com/watch?v=LjJW_s5gQ9Y
> i have seen this already. out plan here is that the user specifies the 
> internet connection type like vdsl2, cable, whatever in case of cake 
> which then will be used
> as argument
>>
>> Other nifty cake features:
>>
>> "wash" and "besteffort" are important on some cablecos that remark
>> traffic to CS1
>> "nat" is dang useful if you are natting
>> ack-filter helps on really slow asymmetric networks on the slow side 
>> only.
>>
>> So, like, my defaults would be
>>
>> in: nat wash besteffort # triple-isolate bandwidth X etc
>> out: nat ack-filter # if > 10x1 down/up ratio
> my finding for damn slow internet < 2000 kbit upstream that we need to 
> handle rtt like codel with the formula
>
> rtt = 50 - (uprate / 50)
>
> which gives a rtt of 10 at 2000 kbit. otherwise its not working good. 
> for the rest i will consider your defaults for our tests.
>
>> And make sure the link layer settings are exposed in the gui. I really
>> don't know how much
>> "washing" is needed outside the cablecos, taking packet captures of
>> various isps to see how often
>> dscp bits are mangled nowadays would be good. besteffort on inbound
>> saves some on cpu.
> dscp might be problematic for isps like orange in france and other 
> fiber isps i have found in spain and netherlands.
> all these isps expect dscp classes to be set for outbound traffic 
> depending on the traffic type. (iptv, voice and internet). this 
> together is also
> mapped to vlan priorities again internally. so using the dscp bits 
> might be problematic for cake since it clears out the dscp flags and 
> so internet simply doesnt work anymore
> for these isps (or just with modem 56k speed)
>>
>> Are you using the out of tree version or mainline? Out of tree has
>> some experimental SCE work
>> that I'd love to see tested at more scale but not actually shipped at 
>> this time.
> out of tree straight from git with modifications to be compatible to 
> my kernels since your compatiblity layer is mmh not perfect.
> some compat layer you made doesnt work since they already got 
> backported to the latest upstream kernel versions. so just some 
> refinement.
> nothing special or much work
>>
>> Due to how cpu intensive cake can be (on inbound) I've been working on
>> a faster, less feature-full fq_codel, it's here:
>>
>> https://github.com/dtaht/fq_codel_fast
> i reviewed it and also tried to compile it with success. problem is, 
> that code parts of it are looking unfinished with debug printks etc.
> this is why i stopped at that point since i thought its unfinished. 
> but i have added it already to my code tree and added the required 
> parts to compile
> it with all of my kernels (compat layer for older kernels or lets say 
> #ifdef hell)
>>
>> I hate the idea of fq_codel one way, cake the other, but tbf +
>> fq_codel_fast seems to work well at
>> ~1gbit on my apu2 and cake doesn't. I'd originally planned to try and
>> make a multi-core shaper out
>> of it, but sce distracted me....
> multicore would be nice since alot of home routers are coming with up 
> to 4 native cpus right now. (ipq8065 has 1.8 ghz and dual core. 
> ipq40xx has 4 cores but just 700 mhz and is 4 times slower per core 
> and mhz, so it makes sense here)
> i'm using a APU as internet router as well btw
>>
>> Having more folk on board benching stuff on modern non-x86 platforms
>> would be good.
> yeah no problem. we mainly test on a gateworks laguna armv6 platform 
> in my company, but most reponse will come from broadcom arm northstar 
> and qca ipq platforms from the community
> but i hope i also get feedback from slower platforms like mips ar7240 etc
>> Another cake feature is that you can get all the benefits on a normal
>> ethernet (with bql) *without turning the shaper on* but we have not
>> benchmarked that either vs a vs fq_codel or fq_codel_fast
> from the response i got from the early experiments is that cake 
> performs much better in typical bufferfloat tests than fq_codel. but i 
> will see how my latest version now works
>
> which fully integrated it.
>
>
> Sebastian
>
>>
>> Have fun!
>>
>> Here's the first paper: https://arxiv.org/pdf/1804.07617.pdf
>>
>>> Sebastian
>>>
>>> Am 18.08.2019 um 18:33 schrieb Dave Taht:
>>>> https://svn.dd-wrt.com/ticket/5796
>>>>
>>
>>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake in dd-wrt

[Dave Taht]

Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:

> can you explain line
>
> 524 in
> https://github.com/dtaht/fq_codel_fast/blob/master/sch_fq_codel_fast.c?

typo!!!!!

>
>
> Am 20.08.2019 um 18:47 schrieb Sebastian Gottschall:
>>
>> Am 20.08.2019 um 18:24 schrieb Dave Taht:
>>> On Tue, Aug 20, 2019 at 5:09 AM Sebastian Gottschall
>>> <s.gottschall@newmedia-net.de> wrote:
>>>> :-) i'm following this list and yes we are working on bringing
>>>> cake in :-)
>>> Yea! thx for being on the list!
>>>
>>>> is there any question behind this link from your side?
>>> I just wanted to make people here aware that it was happening.
>>>
>>> Is there a build now?
>> the first builds with cake are already out yes, but unfinished. we
>> started then to rewrite major parts of the qos code. i expect to
>> push out
>> a new build tomorrow. it will still not use the full potential of
>> cake since we have to bring all together with the priority and ndpi
>> and filter based filter together
>> with the cake scheduler. but it works so far and will be better in
>> the next days and weeks after we have found a solution for it. cake
>> is more a all in wonder solution
>> so we have to use the features of cake itself to reimplement the
>> priorisation classes
>>>
>>> If I had any one principal request it would be to make sure the dd-wrt
>>> gui (if one is made) exposes the link layer parameters. Getting the
>>> framing wrong is about the biggest error I see in the deployment:
>>> https://www.youtube.com/watch?v=LjJW_s5gQ9Y
>> i have seen this already. out plan here is that the user specifies
>> the internet connection type like vdsl2, cable, whatever in case of
>> cake which then will be used
>> as argument
>>>
>>> Other nifty cake features:
>>>
>>> "wash" and "besteffort" are important on some cablecos that remark
>>> traffic to CS1
>>> "nat" is dang useful if you are natting
>>> ack-filter helps on really slow asymmetric networks on the slow
>>> side only.
>>>
>>> So, like, my defaults would be
>>>
>>> in: nat wash besteffort # triple-isolate bandwidth X etc
>>> out: nat ack-filter # if > 10x1 down/up ratio
>> my finding for damn slow internet < 2000 kbit upstream that we need
>> to handle rtt like codel with the formula
>>
>> rtt = 50 - (uprate / 50)
>>
>> which gives a rtt of 10 at 2000 kbit. otherwise its not working
>> good. for the rest i will consider your defaults for our tests.
>>
>>> And make sure the link layer settings are exposed in the gui. I really
>>> don't know how much
>>> "washing" is needed outside the cablecos, taking packet captures of
>>> various isps to see how often
>>> dscp bits are mangled nowadays would be good. besteffort on inbound
>>> saves some on cpu.
>> dscp might be problematic for isps like orange in france and other
>> fiber isps i have found in spain and netherlands.
>> all these isps expect dscp classes to be set for outbound traffic
>> depending on the traffic type. (iptv, voice and internet). this
>> together is also
>> mapped to vlan priorities again internally. so using the dscp bits
>> might be problematic for cake since it clears out the dscp flags and
>> so internet simply doesnt work anymore
>> for these isps (or just with modem 56k speed)
>>>
>>> Are you using the out of tree version or mainline? Out of tree has
>>> some experimental SCE work
>>> that I'd love to see tested at more scale but not actually shipped
>>> at this time.
>> out of tree straight from git with modifications to be compatible to
>> my kernels since your compatiblity layer is mmh not perfect.
>> some compat layer you made doesnt work since they already got
>> backported to the latest upstream kernel versions. so just some
>> refinement.
>> nothing special or much work
>>>
>>> Due to how cpu intensive cake can be (on inbound) I've been working on
>>> a faster, less feature-full fq_codel, it's here:
>>>
>>> https://github.com/dtaht/fq_codel_fast
>> i reviewed it and also tried to compile it with success. problem is,
>> that code parts of it are looking unfinished with debug printks etc.
>> this is why i stopped at that point since i thought its
>> unfinished. but i have added it already to my code tree and added
>> the required parts to compile
>> it with all of my kernels (compat layer for older kernels or lets
>> say #ifdef hell)
>>>
>>> I hate the idea of fq_codel one way, cake the other, but tbf +
>>> fq_codel_fast seems to work well at
>>> ~1gbit on my apu2 and cake doesn't. I'd originally planned to try and
>>> make a multi-core shaper out
>>> of it, but sce distracted me....
>> multicore would be nice since alot of home routers are coming with
>> up to 4 native cpus right now. (ipq8065 has 1.8 ghz and dual
>> core. ipq40xx has 4 cores but just 700 mhz and is 4 times slower per
>> core and mhz, so it makes sense here)
>> i'm using a APU as internet router as well btw
>>>
>>> Having more folk on board benching stuff on modern non-x86 platforms
>>> would be good.
>> yeah no problem. we mainly test on a gateworks laguna armv6 platform
>> in my company, but most reponse will come from broadcom arm
>> northstar and qca ipq platforms from the community
>> but i hope i also get feedback from slower platforms like mips ar7240 etc
>>> Another cake feature is that you can get all the benefits on a normal
>>> ethernet (with bql) *without turning the shaper on* but we have not
>>> benchmarked that either vs a vs fq_codel or fq_codel_fast
>> from the response i got from the early experiments is that cake
>> performs much better in typical bufferfloat tests than fq_codel. but
>> i will see how my latest version now works
>>
>> which fully integrated it.
>>
>>
>> Sebastian
>>
>>>
>>> Have fun!
>>>
>>> Here's the first paper: https://arxiv.org/pdf/1804.07617.pdf
>>>
>>>> Sebastian
>>>>
>>>> Am 18.08.2019 um 18:33 schrieb Dave Taht:
>>>>> https://svn.dd-wrt.com/ticket/5796
>>>>>
>>>
>>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake in dd-wrt

[Dave Taht]

Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:

> Am 20.08.2019 um 18:24 schrieb Dave Taht:
>> On Tue, Aug 20, 2019 at 5:09 AM Sebastian Gottschall
>> <s.gottschall@newmedia-net.de> wrote:
>>> :-) i'm following this list and yes we are working on bringing cake
>>> in :-)
>> Yea! thx for being on the list!
>>
>>> is there any question behind this link from your side?
>> I just wanted to make people here aware that it was happening.
>>
>> Is there a build now?
> the first builds with cake are already out yes, but unfinished. we
> started then to rewrite major parts of the qos code. i expect to push
> out
> a new build tomorrow. it will still not use the full potential of cake
> since we have to bring all together with the priority and ndpi and
> filter based filter together
> with the cake scheduler. but it works so far and will be better in the
> next days and weeks after we have found a solution for it. cake is
> more a all in wonder solution
> so we have to use the features of cake itself to reimplement the
> priorisation classes
>>
>> If I had any one principal request it would be to make sure the
>> dd-wrt
>> gui (if one is made) exposes the link layer parameters. Getting the
>> framing wrong is about the biggest error I see in the deployment:
>> https://www.youtube.com/watch?v=LjJW_s5gQ9Y
> i have seen this already. out plan here is that the user specifies the
> internet connection type like vdsl2, cable, whatever in case of cake
> which then will be used
> as argument

Awesome!

>>
>> Other nifty cake features:
>>
>> "wash" and "besteffort" are important on some cablecos that remark
>> traffic to CS1
>> "nat" is dang useful if you are natting
>> ack-filter helps on really slow asymmetric networks on the slow side
>> only.
>>
>> So, like, my defaults would be
>>
>> in: nat wash besteffort # triple-isolate bandwidth X etc
>> out: nat ack-filter # if > 10x1 down/up ratio
> my finding for damn slow internet < 2000 kbit upstream that we need to
> handle rtt like codel with the formula
>
> rtt = 50 - (uprate / 50)
>
> which gives a rtt of 10 at 2000 kbit. otherwise its not working
> good. for the rest i will consider your defaults for our tests.
>
>> And make sure the link layer settings are exposed in the gui. I
>> really
>> don't know how much
>> "washing" is needed outside the cablecos, taking packet captures of
>> various isps to see how often
>> dscp bits are mangled nowadays would be good. besteffort on inbound
>> saves some on cpu.
> dscp might be problematic for isps like orange in france and other
> fiber isps i have found in spain and netherlands.
> all these isps expect dscp classes to be set for outbound traffic
> depending on the traffic type. (iptv, voice and internet). this
> together is also
> mapped to vlan priorities again internally. so using the dscp bits
> might be problematic for cake since it clears out the dscp flags and
> so internet simply doesnt work anymore
> for these isps (or just with modem 56k speed)

I would love to know more about what dscp's are in more common use and
what isps intend for them to be doing. cake's default mappings are the
result of much reading of RFCs, debate, and headbanging. The "model"
concept is easily extended


>> Are you using the out of tree version or mainline? Out of tree has
>> some experimental SCE work
>> that I'd love to see tested at more scale but not actually shipped
>> at this time.
> out of tree straight from git with modifications to be compatible to
> my kernels since your compatiblity layer is mmh not perfect.
> some compat layer you made doesnt work since they already got
> backported to the latest upstream kernel versions. so just some
> refinement.
> nothing special or much work
>>
>> Due to how cpu intensive cake can be (on inbound) I've been working
>> on
>> a faster, less feature-full fq_codel, it's here:
>>
>> https://github.com/dtaht/fq_codel_fast
> i reviewed it and also tried to compile it with success. problem is,
> that code parts of it are looking unfinished with debug printks etc.
> this is why i stopped at that point since i thought its
> unfinished. but i have added it already to my code tree and added the
> required parts to compile
> it with all of my kernels (compat layer for older kernels or lets say
> #ifdef hell)

Thx! It's unfinished, but functional, as I seemed to be the only one
that had interest in revising fq_codel further, and lack time to
actually build and profile embedded targets.

>>
>> I hate the idea of fq_codel one way, cake the other, but tbf +
>> fq_codel_fast seems to work well at
>> ~1gbit on my apu2 and cake doesn't. I'd originally planned to try
>> and
>> make a multi-core shaper out
>> of it, but sce distracted me....
> multicore would be nice since alot of home routers are coming with up
> to 4 native cpus right now. (ipq8065 has 1.8 ghz and dual
> core. ipq40xx has 4 cores but just 700 mhz and is 4 times slower per
> core and mhz, so it makes sense here)
> i'm using a APU as internet router as well btw

That's my primary target at the moment.

>> Having more folk on board benching stuff on modern non-x86 platforms
>> would be good.
> yeah no problem. we mainly test on a gateworks laguna armv6 platform
> in my company, but most reponse will come from broadcom arm northstar
> and qca ipq platforms from the community
> but i hope i also get feedback from slower platforms like mips ar7240
> etc

I've not heard of "northstar"?

I have to admit I'm mostly interested in benchmarks from the newer
platforms. Have not been keeping up with the state of the art here.

It seemed better to make cake as perfect as could be and wait for the hw
to catchup. At the time.

I keep hoping to find the perfect next-gen wifi 6 with fq_codel derived
tech everywhere box ... and to be able to just buy it off the shelf
this christmas.

>> Another cake feature is that you can get all the benefits on a
>> normal
>> ethernet (with bql) *without turning the shaper on* but we have not
>> benchmarked that either vs a vs fq_codel or fq_codel_fast
> from the response i got from the early experiments is that cake
> performs much better in typical bufferfloat tests than fq_codel. but i
> will see how my latest version now works

> which fully integrated it.

Arguably fq_codel could use a few tweaks from the cake/cobalt effort.
It was great for 2012 though!

>
>
> Sebastian
>
>>
>> Have fun!
>>
>> Here's the first paper: https://arxiv.org/pdf/1804.07617.pdf
>>
>>> Sebastian
>>>
>>> Am 18.08.2019 um 18:33 schrieb Dave Taht:
>>>> https://svn.dd-wrt.com/ticket/5796
>>>>
>>
>>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

some on cpu.

>> dscp might be problematic for isps like orange in france and other
>> fiber isps i have found in spain and netherlands.
>> all these isps expect dscp classes to be set for outbound traffic
>> depending on the traffic type. (iptv, voice and internet). this
>> together is also
>> mapped to vlan priorities again internally. so using the dscp bits
>> might be problematic for cake since it clears out the dscp flags and
>> so internet simply doesnt work anymore
>> for these isps (or just with modem 56k speed)
> I would love to know more about what dscp's are in more common use and
> what isps intend for them to be doing. cake's default mappings are the
> result of much reading of RFCs, debate, and headbanging. The "model"
> concept is easily extended

as i said. orange fibre lines in france are using dscp/tos markings for 
voice, iptv and internet. if it would be just inbound nobody would care.

but outgoing traffic must be marked as well. for initial dhcp lease from 
this isp you need to mark dhcp markets with dscp 6 for instance. 
otherwise you get no response
the exact same platform is used in spain and netherlands. but i cannot 
remember the isp names anymore. but it was always for fiber isps

> /github.com/dtaht/fq_codel_fast
>> i reviewed it and also tried to compile it with success. problem is,
>> that code parts of it are looking unfinished with debug printks etc.
>> this is why i stopped at that point since i thought its
>> unfinished. but i have added it already to my code tree and added the
>> required parts to compile
>> it with all of my kernels (compat layer for older kernels or lets say
>> #ifdef hell)
> Thx! It's unfinished, but functional, as I seemed to be the only one
> that had interest in revising fq_codel further, and lack time to
> actually build and profile embedded targets.
>
> I've not heard of "northstar"?
its the codename for the cpu armv7 platform of broadcom which is out 
since many years.

you may know it under the name bcm4709. they are comming almost with 
dualcore from 800 - 1400 mhz


>
> I have to admit I'm mostly interested in benchmarks from the newer
> platforms. Have not been keeping up with the state of the art here.
>
> It seemed better to make cake as perfect as could be and wait for the hw
> to catchup. At the time.
the hw on the home market is already fast enough to beat your apu2. not 
every plastic router of cause
but some of them are really high performing
> I keep hoping to find the perfect next-gen wifi 6 with fq_codel derived
> tech everywhere box ... and to be able to just buy it off the shelf
> this christmas.
wifi 6 the new buzzword. i still call it ax.  everytime a marketing 
idiot creates a new name for existing technology i get gastric spasm
consider that wifi 6 brings almost nothing unless you're close to the 
router and my current 802.11ac routers with qca9984 chipsets
already beating the gigabit limit.

Re: [Cake] cake in dd-wrt

[Sebastian Moeller]

> On Aug 20, 2019, at 18:47, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
> 
> 
> Am 20.08.2019 um 18:24 schrieb Dave Taht:
>> On Tue, Aug 20, 2019 at 5:09 AM Sebastian Gottschall
>> <s.gottschall@newmedia-net.de> wrote:
>>> [...]
>> 
>> If I had any one principal request it would be to make sure the dd-wrt
>> gui (if one is made) exposes the link layer parameters. Getting the
>> framing wrong is about the biggest error I see in the deployment:
>> https://www.youtube.com/watch?v=LjJW_s5gQ9Y
> i have seen this already. out plan here is that the user specifies the internet connection type like vdsl2, cable, whatever in case of cake which then will be used
> as argument

	Good goal, that also is theoretically well supported by cake with its multitude of encapsulation/overhead realated keywords. Unfortunately reality is not as nice and tidy as this collection of keywords implies, There are 8 keywords for ATM/AAL5 based encapsulations (ADSL, ADSL2, ADSL2+, ...), 2 for VDSL2, 1 for DOCSIS, 1 for ethernet, for a total of 12 that all can be combined with one or more VLAN-tag keywords, for a total of 24 to 36 combinations. (And these are not even exhaustive, as e.g. the use of ds-lite can increase the per-packet overhead for IPv4 packets by another 20 bytes).
	Ideally one would just empirically measure the effective overhead and use the "overhead NN mpu NN" keywords instead, but that has issues as measuring overhead empirically is simply hard... The best bet would be to leverage BEREC to require ISPs to explicitly inform their customers of the effective gross-rates and applicable overheads for each link, but I am not holding my breath. Over at https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm we tried to give simplified instructions for setting the overheads for different access technologies, but these are not guaranteed to fit everybody (not even most users, as we have no numbers about the relative distributions of the different encapsulation options).

Best Regards
	"another" Sebastian

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

>> i have seen this already. out plan here is that the user specifies the internet connection type like vdsl2, cable, whatever in case of cake which then will be used
>> as argument
> 	Good goal, that also is theoretically well supported by cake with its multitude of encapsulation/overhead realated keywords. Unfortunately reality is not as nice and tidy as this collection of keywords implies, There are 8 keywords for ATM/AAL5 based encapsulations (ADSL, ADSL2, ADSL2+, ...), 2 for VDSL2, 1 for DOCSIS, 1 for ethernet, for a total of 12 that all can be combined with one or more VLAN-tag keywords, for a total of 24 to 36 combinations. (And these are not even exhaustive, as e.g. the use of ds-lite can increase the per-packet overhead for IPv4 packets by another 20 bytes).
> 	Ideally one would just empirically measure the effective overhead and use the "overhead NN mpu NN" keywords instead, but that has issues as measuring overhead empirically is simply hard... The best bet would be to leverage BEREC to require ISPs to explicitly inform their customers of the effective gross-rates and applicable overheads for each link, but I am not holding my breath. Over at https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm we tried to give simplified instructions for setting the overheads for different access technologies, but these are not guaranteed to fit everybody (not even most users, as we have no numbers about the relative distributions of the different encapsulation options).
>
> Best Regards
> 	"another" Sebastian

as i said. i just started. lets see if i can find a better solution or a 
clever way of auto detecting/measuring the overhead

Sebastian

>
>
>

Re: [Cake] cake in dd-wrt

[Sebastian Moeller]

> On Aug 21, 2019, at 09:50, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
> 
>>> i have seen this already. out plan here is that the user specifies the internet connection type like vdsl2, cable, whatever in case of cake which then will be used
>>> as argument
>> 	Good goal, that also is theoretically well supported by cake with its multitude of encapsulation/overhead realated keywords. Unfortunately reality is not as nice and tidy as this collection of keywords implies, There are 8 keywords for ATM/AAL5 based encapsulations (ADSL, ADSL2, ADSL2+, ...), 2 for VDSL2, 1 for DOCSIS, 1 for ethernet, for a total of 12 that all can be combined with one or more VLAN-tag keywords, for a total of 24 to 36 combinations. (And these are not even exhaustive, as e.g. the use of ds-lite can increase the per-packet overhead for IPv4 packets by another 20 bytes).
>> 	Ideally one would just empirically measure the effective overhead and use the "overhead NN mpu NN" keywords instead, but that has issues as measuring overhead empirically is simply hard... The best bet would be to leverage BEREC to require ISPs to explicitly inform their customers of the effective gross-rates and applicable overheads for each link, but I am not holding my breath. Over at https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm we tried to give simplified instructions for setting the overheads for different access technologies, but these are not guaranteed to fit everybody (not even most users, as we have no numbers about the relative distributions of the different encapsulation options).
>> 
>> Best Regards
>> 	"another" Sebastian
> 
> as i said. i just started. lets see if i can find a better solution or a clever way of auto detecting/measuring the overhead

	If you do find a clever and fat way, please let me know ;). The best I came up with only works for ATM/AAL5 and is neither clever, automated or fast is at https://github.com/moeller0/ATM_overhead_detector (which has the advantage of also confirming ATM/AAL5-quantisation). I have some ideas about how to deduce overhead generically but these require very precise measurements of maximum goodput for different packet sizes and even less fit for general consumption that the atm stuff.

Best Regards
	Sebastian


> 
> Sebastian
> 
>> 
>> 
>> 

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

Am 21.08.2019 um 09:56 schrieb Sebastian Moeller:
>
>> On Aug 21, 2019, at 09:50, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>>
>>>> i have seen this already. out plan here is that the user specifies the internet connection type like vdsl2, cable, whatever in case of cake which then will be used
>>>> as argument
>>> 	Good goal, that also is theoretically well supported by cake with its multitude of encapsulation/overhead realated keywords. Unfortunately reality is not as nice and tidy as this collection of keywords implies, There are 8 keywords for ATM/AAL5 based encapsulations (ADSL, ADSL2, ADSL2+, ...), 2 for VDSL2, 1 for DOCSIS, 1 for ethernet, for a total of 12 that all can be combined with one or more VLAN-tag keywords, for a total of 24 to 36 combinations. (And these are not even exhaustive, as e.g. the use of ds-lite can increase the per-packet overhead for IPv4 packets by another 20 bytes).
>>> 	Ideally one would just empirically measure the effective overhead and use the "overhead NN mpu NN" keywords instead, but that has issues as measuring overhead empirically is simply hard... The best bet would be to leverage BEREC to require ISPs to explicitly inform their customers of the effective gross-rates and applicable overheads for each link, but I am not holding my breath. Over at https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm we tried to give simplified instructions for setting the overheads for different access technologies, but these are not guaranteed to fit everybody (not even most users, as we have no numbers about the relative distributions of the different encapsulation options).
>>>
>>> Best Regards
>>> 	"another" Sebastian
>> as i said. i just started. lets see if i can find a better solution or a clever way of auto detecting/measuring the overhead
> 	If you do find a clever and fat way, please let me know ;). The best I came up with only works for ATM/AAL5 and is neither clever, automated or fast is at https://github.com/moeller0/ATM_overhead_detector (which has the advantage of also confirming ATM/AAL5-quantisation). I have some ideas about how to deduce overhead generically but these require very precise measurements of maximum goodput for different packet sizes and even less fit for general consumption that the atm stuff.
then the only solution is having a good reliable peer for measuring. we 
may missuse speedtest servers :-)
>
> Best Regards
> 	Sebastian
>
>
>> Sebastian
>>
>>>
>>>
>

Re: [Cake] cake in dd-wrt

[Sebastian Moeller]

> On Aug 21, 2019, at 11:04, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
> 
> 
> Am 21.08.2019 um 09:56 schrieb Sebastian Moeller:
>> 
>>> On Aug 21, 2019, at 09:50, Sebastian Gottschall <s.gottschall@newmedia-net.de> wrote:
>>> 
>>>>> i have seen this already. out plan here is that the user specifies the internet connection type like vdsl2, cable, whatever in case of cake which then will be used
>>>>> as argument
>>>> 	Good goal, that also is theoretically well supported by cake with its multitude of encapsulation/overhead realated keywords. Unfortunately reality is not as nice and tidy as this collection of keywords implies, There are 8 keywords for ATM/AAL5 based encapsulations (ADSL, ADSL2, ADSL2+, ...), 2 for VDSL2, 1 for DOCSIS, 1 for ethernet, for a total of 12 that all can be combined with one or more VLAN-tag keywords, for a total of 24 to 36 combinations. (And these are not even exhaustive, as e.g. the use of ds-lite can increase the per-packet overhead for IPv4 packets by another 20 bytes).
>>>> 	Ideally one would just empirically measure the effective overhead and use the "overhead NN mpu NN" keywords instead, but that has issues as measuring overhead empirically is simply hard... The best bet would be to leverage BEREC to require ISPs to explicitly inform their customers of the effective gross-rates and applicable overheads for each link, but I am not holding my breath. Over at https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm we tried to give simplified instructions for setting the overheads for different access technologies, but these are not guaranteed to fit everybody (not even most users, as we have no numbers about the relative distributions of the different encapsulation options).
>>>> 
>>>> Best Regards
>>>> 	"another" Sebastian
>>> as i said. i just started. lets see if i can find a better solution or a clever way of auto detecting/measuring the overhead
>> 	If you do find a clever and fat way, please let me know ;). The best I came up with only works for ATM/AAL5 and is neither clever, automated or fast is at https://github.com/moeller0/ATM_overhead_detector (which has the advantage of also confirming ATM/AAL5-quantisation). I have some ideas about how to deduce overhead generically but these require very precise measurements of maximum goodput for different packet sizes and even less fit for general consumption that the atm stuff.
> then the only solution is having a good reliable peer for measuring. we may missuse speedtest servers :-)

	Nah, barely good enough, breitbandmessung.de might be suited (they have access control to not overwhelm their test servers), but other Speedtests are notoriously inaccurate* (I am looking at you Ookla...) and occasionally report "measured" goodput in excess over the actual goddput achivable over the given gross access rate.
	IMHO the real challenge is that to set our shaper correctly we need both information about the gross rate of the link (which can bei either physically bound, by say a dsl-link's sync-rates, or in softwar, say in a BRAS/BNG-level traffic shaper) and of the worst applicable overhead between user and ISP (which again might be a physical link property or might come from the configuration of the ISPs traffic shaper). Most ISP will giver very little information about the precise value of any of the two. So we need to solve for two unkowns (per direction, even though per-packet-overhead likely is going to be identical for both directions), making the whole endeavor way more complicated than it should be. If we would know at least one of the values precisely, gross-limit-rates or per-packet-overhead, we could "simply" try different values fr the other and measure the resulting bufferbloat, plotting the bloat versus the variable should show a more or less step-like increase once we exceed the parameter's true value. But I digress, we do not know any of the two...



*) I guess they are precise and accurate enough for their intended use-case, but they are somewhat problematic for precise measurements of real maximum goodput.


>> 
>> Best Regards
>> 	Sebastian
>> 
>> 
>>> Sebastian
>>> 
>>>> 
>>>> 
>> 

Re: [Cake] cake in dd-wrt

[Dave Taht]

On Wed, Aug 21, 2019 at 12:51 AM Sebastian Gottschall
<s.gottschall@newmedia-net.de> wrote:
>
> >> i have seen this already. out plan here is that the user specifies the internet connection type like vdsl2, cable, whatever in case of cake which then will be used
> >> as argument
> >       Good goal, that also is theoretically well supported by cake with its multitude of encapsulation/overhead realated keywords. Unfortunately reality is not as nice and tidy as this collection of keywords implies, There are 8 keywords for ATM/AAL5 based encapsulations (ADSL, ADSL2, ADSL2+, ...), 2 for VDSL2, 1 for DOCSIS, 1 for ethernet, for a total of 12 that all can be combined with one or more VLAN-tag keywords, for a total of 24 to 36 combinations. (And these are not even exhaustive, as e.g. the use of ds-lite can increase the per-packet overhead for IPv4 packets by another 20 bytes).
> >       Ideally one would just empirically measure the effective overhead and use the "overhead NN mpu NN" keywords instead, but that has issues as measuring overhead empirically is simply hard... The best bet would be to leverage BEREC to require ISPs to explicitly inform their customers of the effective gross-rates and applicable overheads for each link, but I am not holding my breath. Over at https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm we tried to give simplified instructions for setting the overheads for different access technologies, but these are not guaranteed to fit everybody (not even most users, as we have no numbers about the relative distributions of the different encapsulation options).
> >
> > Best Regards
> >       "another" Sebastian
>
> as i said. i just started. lets see if i can find a better solution or a
> clever way of auto detecting/measuring the overhead

+1. One of my favorite feynman sayings is "disregard" and we need new
thinking here.

I note that I maintain anywhere between 6-16 flent (netperf and irtt)
servers around the world,
and they are mostly underused....

Sometimes I've thought that a "right" approach would be to send a 10
sec full udp burst,
each packet pre-timestamped internally, at, say, 100Mbit...
and then measure "smoothness" at the receiver and ifconfig interface
(accounting for any other
traffic along the way).


>
> Sebastian
>
> >
> >
> >



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

Am 21.08.2019 um 15:20 schrieb Dave Taht:
> On Wed, Aug 21, 2019 at 12:51 AM Sebastian Gottschall
> <s.gottschall@newmedia-net.de> wrote:
>>>> i have seen this already. out plan here is that the user specifies the internet connection type like vdsl2, cable, whatever in case of cake which then will be used
>>>> as argument
>>>        Good goal, that also is theoretically well supported by cake with its multitude of encapsulation/overhead realated keywords. Unfortunately reality is not as nice and tidy as this collection of keywords implies, There are 8 keywords for ATM/AAL5 based encapsulations (ADSL, ADSL2, ADSL2+, ...), 2 for VDSL2, 1 for DOCSIS, 1 for ethernet, for a total of 12 that all can be combined with one or more VLAN-tag keywords, for a total of 24 to 36 combinations. (And these are not even exhaustive, as e.g. the use of ds-lite can increase the per-packet overhead for IPv4 packets by another 20 bytes).
>>>        Ideally one would just empirically measure the effective overhead and use the "overhead NN mpu NN" keywords instead, but that has issues as measuring overhead empirically is simply hard... The best bet would be to leverage BEREC to require ISPs to explicitly inform their customers of the effective gross-rates and applicable overheads for each link, but I am not holding my breath. Over at https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm we tried to give simplified instructions for setting the overheads for different access technologies, but these are not guaranteed to fit everybody (not even most users, as we have no numbers about the relative distributions of the different encapsulation options).
>>>
>>> Best Regards
>>>        "another" Sebastian
>> as i said. i just started. lets see if i can find a better solution or a
>> clever way of auto detecting/measuring the overhead
> +1. One of my favorite feynman sayings is "disregard" and we need new
> thinking here.
>
> I note that I maintain anywhere between 6-16 flent (netperf and irtt)
> servers around the world,
> and they are mostly underused....
>
> Sometimes I've thought that a "right" approach would be to send a 10
> sec full udp burst,
> each packet pre-timestamped internally, at, say, 100Mbit...
> and then measure "smoothness" at the receiver and ifconfig interface
> (accounting for any other
> traffic along the way).
not sure if 16 are enough if we have to handle millions of home routers. 
so a dynamic on the fly approach or more deteministic would be cool too.
i mean mtu measuring is simple by testing the tcp fragmentation, but 
this doesnt help for ipv6 which doesnt allow fragmentation in a easy way
>
>
>> Sebastian
>>
>>>
>>>
>
>

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

to make it short. i added now fq_codel_fast to dd-wrt too.

Am 18.08.2019 um 18:33 schrieb Dave Taht:
> https://svn.dd-wrt.com/ticket/5796
>

Re: [Cake] cake in dd-wrt

[Dave Taht]

Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:

> to make it short. i added now fq_codel_fast to dd-wrt too.

Thank you. I note that it requires a patch that is in tc-adv.

* Fixed (1024) number of queues
* NO tc filter support
* closer to O(1) bulk dropper
* less deterministic bulk drop
* gso splitting with very preliminary SCE support
* ton of stats ripped out

As to whether or not any of these are "killer" features, I don't know. I
certainly wouldn't ship it as a built-in package in a final release -
but I would certainly love to know if it made a difference in cpu use on
lower end platforms on your userbase.

Maybe with some feedback from the field I'd be inspired to do more with
it.

>
> Am 18.08.2019 um 18:33 schrieb Dave Taht:
>> https://svn.dd-wrt.com/ticket/5796
>>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake in dd-wrt

[Sebastian Gottschall]

Am 21.08.2019 um 01:50 schrieb Dave Taht:
> Sebastian Gottschall <s.gottschall@newmedia-net.de> writes:
>
>> to make it short. i added now fq_codel_fast to dd-wrt too.
> Thank you. I note that it requires a patch that is in tc-adv.
like cake too. i already merged the module. i maintain a own version of tc

which is much smaller. so i had not just to merge, i also had to rewrite 
it a little bit.

my tc version does not support all that useless json crap. so its 
stripped down with no useless stuff

>
> * Fixed (1024) number of queues
> * NO tc filter support
> * closer to O(1) bulk dropper
> * less deterministic bulk drop
> * gso splitting with very preliminary SCE support
> * ton of stats ripped out
>
> As to whether or not any of these are "killer" features, I don't know. I
> certainly wouldn't ship it as a built-in package in a final release -
> but I would certainly love to know if it made a difference in cpu use on
> lower end platforms on your userbase.
>
> Maybe with some feedback from the field I'd be inspired to do more with
> it.

i guess i get some feedback soon since i upload a new build to my 
servers right now and some of the users are quick in response


>
>> Am 18.08.2019 um 18:33 schrieb Dave Taht:
>>> https://svn.dd-wrt.com/ticket/5796
>>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake