[Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Alan Jenkins]

Hi

I guess I've done the complementary half to Seb's test :).  Basically
"cake overhead 40" didn't work, but that's the fault of cake in this
build.  Or tc, as Johnathan suggested.  (The "cake atm" part seems to
work, as per my previous test).

"tc qdisc" says "cake overhead 0", as Sebastian noticed.  And the test
results show "cake overhead 40" does not give a measurable
improvement.  But "tc stab overhead 40" does.

I ran this test with the updated sqm-scripts and I think they're doing
the right thing.


Method:

I used the updated files from sqm-scripts,

(once I remembered to mark them executable.  Lacking that causes a
failure with no error messages, because sqm-scripts checks before
running them :)

but didn't bother updating & using luci-app-sqm.

The test was to compare netperf-runner results - ping during combined
upload & download - for "overhead 40" and "overhead 0".  I tested both
values of linklayer_adaptation_mechanism.

I had to repeat 6 times (60s per run for each overhead) because of
random variation in the range of 3-4ms.  I alternated "overhead 40"
and "overhead 0" to try and exclude longer-term variation effects.

With "stab overhead 40", median latency was better by about 3-4ms.
With "cake overhead 40", there is no such effect.

I'm confident of the result.  I just need to set up the scripting
capability in flent now.  Running this manually takes too long!

Alan


On 18/06/2015, Sebastian Moeller <moeller0@gmx.de> wrote:
>
> 	Not sure I can test functionality, but at least I could confirm the
> following:
> 1) using the tc stab link layer adjustment method with cake could work
> (needs testing, td-d qdisc reports all the right things).
>
> 2) the advanced qdisc option strings are passed to cake now, which might
> increase the potential tester pool.
> 2.1) these option strings are truly dangerous: one typo or invalid keyword
> and cake will not start up in that direction without any notice; “no error
> checking” indeed.
>
> 3) somehow the installed cake version has issues with numeric overheads, or
> at least with reporting them.
> 	cake will accept “overhead N” arguments so it clearly understands they are
> valid (otherwise see 2.1).
> 	But it does keep reporting “raw” in “tc -d qdisc” while it should report
> some number istead of raw there.
>
> 	“atm overhead 40” gets reported as “qdisc cake 802d: dev ifb4eth1 root
> refcnt 2 bandwidth 45Mbit besteffort flows atm overhead 0“
> 	So clearly something is off with either cake or tc on that host.
> 	@Jonathan, any idea what this might be caused by?
>
> 4) My changes not made the router go off-line...
>
> 5) I wonder whether we should not give cake its own script cake.qos, that
> would keep simple and simplest more readable and the cake script should be
> refreshingly concise ;)
>
>
>>
>> So you can't count on getting accurate results this way, but can at
>> least test functionality.
>
> 	The main tests that needed doing, is do the changes from the GUI propagate
> into the enabled sqm instances. It seems they do, so my changes are save
> enough for the greater/smaller hard-core tester community. For actual
> functional tests we need an actual ADSL-link where link-layer adjustments
> issues can readily be observed…
>
>>
>> My plan, such as it was, was to get back to 3 source specific gateways
>> but have run into barrier after barrier.
>
> 	Maybe we should not have “forked” cerowrt before BB was released, by the
> looks of it you could use a barrier breaker. (Sorry can not resist a bad
> pun/joke, was a long day today…)
>
> Best Regards
> 	Sebastian
>
>>
>>
>> On Thu, Jun 18, 2015 at 1:24 PM, Sebastian Moeller <moeller0@gmx.de>
>> wrote:
>>> Hi Dave,
>>>
>>>
>>> On Jun 18, 2015, at 22:18 , Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>>> On Thu, Jun 18, 2015 at 1:04 PM, Sebastian Moeller <moeller0@gmx.de>
>>>> wrote:
>>>>> Hi Dave,
>>>>>
>>>>>
>>>>> On Jun 18, 2015, at 21:33 , Dave Taht <dave.taht@gmail.com> wrote:
>>>>>
>>>>>> My present box under test is at 2601:646:8300:2cba::129
>>>>>>
>>>>>> You know the default password for ssh. It is also accessible via
>>>>>> https://[2601:646:8300:2cba::129/]
>>>>>
>>>>>       Just the joy of being able to play with cake, made me commit a
>>>>> new change ;) This changes a get_cake_lla_string to
>>>>> `get_cake_lla_string` which in turn might actually do something. I want
>>>>> to note that currently this only passes the numerically defined
>>>>> overhead variable from the GUI to the cake call and side steps the
>>>>> issue how to present the symbolic options. I think this should be fine
>>>>> as current cake users should be seasoned enough to just shrug off this
>>>>> slight inconvenience ;)
>>>>>       I only noticed that as I just added the tc_stab link layer
>>>>> adjustment method to the cake calls in simple.qos/simplest.qos so that
>>>>> the different methods can be validated against each other (which was
>>>>> helpful when we had issues with HTB’s private link layer adjustments).
>>>>
>>>> My favorite thing with cake is doing a watch tc -s qdisc show dev
>>>> whatever and looking at the sp(arse) statistic under load. My (ietf)
>>>> world is full of people that think 100ms delay is ok, seeing usec
>>>> makes me happy, and knowing that cake is "peeling" apart the often
>>>> huge packets is quite nice also.
>>>
>>>        A good to know.
>>>
>>>>
>>>> High on my list now that I am heads down in getting snmpd to work
>>>> again is to somehow parse various outputs to show drop and CE events
>>>> in mrtg and/or cacti.
>>>>
>>>> I do wish I could find ways to make everyone more productive. I could
>>>> put a box up in my co-lo next week, if that would help, but reflashing
>>>> is a problem no matter where I go.
>>>>
>>>>>>
>>>>>> It is one iteration behind jonathon's cake, and one iteration now
>>>>>> behind sebastian.
>>>>>
>>>>>       Make that two iterations ;)
>>>>
>>>> K. Me busy. I will try to keep this box up on this ip for as long as I
>>>> can. I had hoped to flash 6 routers this week.
>>>
>>>        Ooops, so I just manually hoisted that boxes sqm-scripts and GUI
>>> to my current development state (I overwrote sqm.lua for luck-app-sqm,
>>> and functions.sh under /usr/lib/sqm and added simple_WIP4cake.qos, these
>>> changes should be confined to simple_WIP4cake (the one extra function in
>>> functions.sh is only called by simple_WIP4cake and hence things should
>>> work for you as before)).
>>>        I would love to see wether this can be used to set up cake on
>>> eth0, so I would like to ask for permission to test (for all I know a
>>> reboot might be required, so this only works if you are near that box and
>>> there is nothing important using it right now).
>>>
>>> Best Regards
>>>        Sebastian
>>>
>>>
>>>>
>>>>>
>>>>> This also means I would love experienced testers for the latest
>>>>> sqm-scripts changes...
>>>>>
>>>>> Best Regards
>>>>>       Sebastian
>>>>>
>>>>>> At the moment I am trying to fix snmpd (massive upgrade + musl
>>>>>> support) and won't be touching that box for a while. It IS a dynamic
>>>>>> ipv6 address....
>>>>>>
>>>>>> Can anyone here push out tc-adv, kmod-* into openwrt's routing repo?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 18, 2015 at 12:19 PM, Sebastian Moeller <moeller0@gmx.de>
>>>>>> wrote:
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I just committed a few cake related changes to luci-app-sqm and
>>>>>>> sqm-scripts in ceropackages 3.10. Mainly it is about passing link
>>>>>>> layer and numeric overhead on to cake (if selected as qdisc), but it
>>>>>>> also passed the two string fields ion the GUI aptly named “Advanced
>>>>>>> option string to pass to the [ingress|egress} queueing disciplines;
>>>>>>> no error checking, use very carefully.” to the [ingress|egress] cake
>>>>>>> instances. This should be helpful in testing cake options under
>>>>>>> openwrt builds. BUT this is totally untested, as I have not managed
>>>>>>> to build cake locally let alone install it. So please, anybody,
>>>>>>> please test the changes and report failure and/or success back. Thank
>>>>>>> you very much in advance.
>>>>>>>
>>>>>>> Best Regards
>>>>>>>      Sebastian

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Sebastian Moeller]

Hi Alan,

excellent, thanks a million.

On Jun 19, 2015, at 16:44 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:

> Hi
> 
> I guess I've done the complementary half to Seb's test :).  Basically
> "cake overhead 40" didn't work, but that's the fault of cake in this
> build.  Or tc, as Johnathan suggested.  (The "cake atm" part seems to
> work, as per my previous test).

	Great!

> 
> "tc qdisc" says "cake overhead 0", as Sebastian noticed.  And the test
> results show "cake overhead 40" does not give a measurable
> improvement.  But "tc stab overhead 40" does.
> 
> I ran this test with the updated sqm-scripts and I think they're doing
> the right thing.

	Thanks for testing this, especially as I can not due to a lack of an ADSL-link (and lack of cake actually, last I looked all I could find was cookies in my browser and a promise of pie in my router)

> 
> 
> Method:
> 
> I used the updated files from sqm-scripts,
> 
> (once I remembered to mark them executable.  Lacking that causes a
> failure with no error messages, because sqm-scripts checks before
> running them :)
> 
> but didn't bother updating & using luci-app-sqm.

	Ah, okay, I guess I did test this part with Dave’s help, so this should work with the most recent sqm.lua.

> 
> The test was to compare netperf-runner results - ping during combined
> upload & download - for "overhead 40" and "overhead 0".  I tested both
> values of linklayer_adaptation_mechanism.
> 
> I had to repeat 6 times (60s per run for each overhead) because of
> random variation in the range of 3-4ms.  I alternated "overhead 40"
> and "overhead 0" to try and exclude longer-term variation effects.
> 
> With "stab overhead 40", median latency was better by about 3-4ms.
> With "cake overhead 40", there is no such effect.

	Intersting, when I still had a 6M/1M ADSL link, I saw much larger latency under load increases when setting the per packet overhead to small, but I had my egress shaper running at 100% of line rate, so the system was rigged for maximum effect that way. How are your shapers typically set? 

> 
> I'm confident of the result.  I just need to set up the scripting
> capability in flent now.  Running this manually takes too long!
> 


Best Regards
	Sebastian


> Alan
> 
> 
> On 18/06/2015, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> 	Not sure I can test functionality, but at least I could confirm the
>> following:
>> 1) using the tc stab link layer adjustment method with cake could work
>> (needs testing, td-d qdisc reports all the right things).
>> 
>> 2) the advanced qdisc option strings are passed to cake now, which might
>> increase the potential tester pool.
>> 2.1) these option strings are truly dangerous: one typo or invalid keyword
>> and cake will not start up in that direction without any notice; “no error
>> checking” indeed.
>> 
>> 3) somehow the installed cake version has issues with numeric overheads, or
>> at least with reporting them.
>> 	cake will accept “overhead N” arguments so it clearly understands they are
>> valid (otherwise see 2.1).
>> 	But it does keep reporting “raw” in “tc -d qdisc” while it should report
>> some number istead of raw there.
>> 
>> 	“atm overhead 40” gets reported as “qdisc cake 802d: dev ifb4eth1 root
>> refcnt 2 bandwidth 45Mbit besteffort flows atm overhead 0“
>> 	So clearly something is off with either cake or tc on that host.
>> 	@Jonathan, any idea what this might be caused by?
>> 
>> 4) My changes not made the router go off-line...
>> 
>> 5) I wonder whether we should not give cake its own script cake.qos, that
>> would keep simple and simplest more readable and the cake script should be
>> refreshingly concise ;)
>> 
>> 
>>> 
>>> So you can't count on getting accurate results this way, but can at
>>> least test functionality.
>> 
>> 	The main tests that needed doing, is do the changes from the GUI propagate
>> into the enabled sqm instances. It seems they do, so my changes are save
>> enough for the greater/smaller hard-core tester community. For actual
>> functional tests we need an actual ADSL-link where link-layer adjustments
>> issues can readily be observed…
>> 
>>> 
>>> My plan, such as it was, was to get back to 3 source specific gateways
>>> but have run into barrier after barrier.
>> 
>> 	Maybe we should not have “forked” cerowrt before BB was released, by the
>> looks of it you could use a barrier breaker. (Sorry can not resist a bad
>> pun/joke, was a long day today…)
>> 
>> Best Regards
>> 	Sebastian
>> 
>>> 
>>> 
>>> On Thu, Jun 18, 2015 at 1:24 PM, Sebastian Moeller <moeller0@gmx.de>
>>> wrote:
>>>> Hi Dave,
>>>> 
>>>> 
>>>> On Jun 18, 2015, at 22:18 , Dave Taht <dave.taht@gmail.com> wrote:
>>>> 
>>>>> On Thu, Jun 18, 2015 at 1:04 PM, Sebastian Moeller <moeller0@gmx.de>
>>>>> wrote:
>>>>>> Hi Dave,
>>>>>> 
>>>>>> 
>>>>>> On Jun 18, 2015, at 21:33 , Dave Taht <dave.taht@gmail.com> wrote:
>>>>>> 
>>>>>>> My present box under test is at 2601:646:8300:2cba::129
>>>>>>> 
>>>>>>> You know the default password for ssh. It is also accessible via
>>>>>>> https://[2601:646:8300:2cba::129/]
>>>>>> 
>>>>>>      Just the joy of being able to play with cake, made me commit a
>>>>>> new change ;) This changes a get_cake_lla_string to
>>>>>> `get_cake_lla_string` which in turn might actually do something. I want
>>>>>> to note that currently this only passes the numerically defined
>>>>>> overhead variable from the GUI to the cake call and side steps the
>>>>>> issue how to present the symbolic options. I think this should be fine
>>>>>> as current cake users should be seasoned enough to just shrug off this
>>>>>> slight inconvenience ;)
>>>>>>      I only noticed that as I just added the tc_stab link layer
>>>>>> adjustment method to the cake calls in simple.qos/simplest.qos so that
>>>>>> the different methods can be validated against each other (which was
>>>>>> helpful when we had issues with HTB’s private link layer adjustments).
>>>>> 
>>>>> My favorite thing with cake is doing a watch tc -s qdisc show dev
>>>>> whatever and looking at the sp(arse) statistic under load. My (ietf)
>>>>> world is full of people that think 100ms delay is ok, seeing usec
>>>>> makes me happy, and knowing that cake is "peeling" apart the often
>>>>> huge packets is quite nice also.
>>>> 
>>>>       A good to know.
>>>> 
>>>>> 
>>>>> High on my list now that I am heads down in getting snmpd to work
>>>>> again is to somehow parse various outputs to show drop and CE events
>>>>> in mrtg and/or cacti.
>>>>> 
>>>>> I do wish I could find ways to make everyone more productive. I could
>>>>> put a box up in my co-lo next week, if that would help, but reflashing
>>>>> is a problem no matter where I go.
>>>>> 
>>>>>>> 
>>>>>>> It is one iteration behind jonathon's cake, and one iteration now
>>>>>>> behind sebastian.
>>>>>> 
>>>>>>      Make that two iterations ;)
>>>>> 
>>>>> K. Me busy. I will try to keep this box up on this ip for as long as I
>>>>> can. I had hoped to flash 6 routers this week.
>>>> 
>>>>       Ooops, so I just manually hoisted that boxes sqm-scripts and GUI
>>>> to my current development state (I overwrote sqm.lua for luck-app-sqm,
>>>> and functions.sh under /usr/lib/sqm and added simple_WIP4cake.qos, these
>>>> changes should be confined to simple_WIP4cake (the one extra function in
>>>> functions.sh is only called by simple_WIP4cake and hence things should
>>>> work for you as before)).
>>>>       I would love to see wether this can be used to set up cake on
>>>> eth0, so I would like to ask for permission to test (for all I know a
>>>> reboot might be required, so this only works if you are near that box and
>>>> there is nothing important using it right now).
>>>> 
>>>> Best Regards
>>>>       Sebastian
>>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>> This also means I would love experienced testers for the latest
>>>>>> sqm-scripts changes...
>>>>>> 
>>>>>> Best Regards
>>>>>>      Sebastian
>>>>>> 
>>>>>>> At the moment I am trying to fix snmpd (massive upgrade + musl
>>>>>>> support) and won't be touching that box for a while. It IS a dynamic
>>>>>>> ipv6 address....
>>>>>>> 
>>>>>>> Can anyone here push out tc-adv, kmod-* into openwrt's routing repo?
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, Jun 18, 2015 at 12:19 PM, Sebastian Moeller <moeller0@gmx.de>
>>>>>>> wrote:
>>>>>>>> Hi All,
>>>>>>>> 
>>>>>>>> I just committed a few cake related changes to luci-app-sqm and
>>>>>>>> sqm-scripts in ceropackages 3.10. Mainly it is about passing link
>>>>>>>> layer and numeric overhead on to cake (if selected as qdisc), but it
>>>>>>>> also passed the two string fields ion the GUI aptly named “Advanced
>>>>>>>> option string to pass to the [ingress|egress} queueing disciplines;
>>>>>>>> no error checking, use very carefully.” to the [ingress|egress] cake
>>>>>>>> instances. This should be helpful in testing cake options under
>>>>>>>> openwrt builds. BUT this is totally untested, as I have not managed
>>>>>>>> to build cake locally let alone install it. So please, anybody,
>>>>>>>> please test the changes and report failure and/or success back. Thank
>>>>>>>> you very much in advance.
>>>>>>>> 
>>>>>>>> Best Regards
>>>>>>>>     Sebastian
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Alan Jenkins]

On 19/06/2015, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Alan,
>
> excellent, thanks a million.
>
> On Jun 19, 2015, at 16:44 , Alan Jenkins
> <alan.christopher.jenkins@gmail.com> wrote:
>
>> Hi
>>
>> I guess I've done the complementary half to Seb's test :).  Basically
>> "cake overhead 40" didn't work, but that's the fault of cake in this
>> build.  Or tc, as Johnathan suggested.  (The "cake atm" part seems to
>> work, as per my previous test).
>
> 	Great!
>
>>
>> "tc qdisc" says "cake overhead 0", as Sebastian noticed.  And the test
>> results show "cake overhead 40" does not give a measurable
>> improvement.  But "tc stab overhead 40" does.
>>
>> I ran this test with the updated sqm-scripts and I think they're doing
>> the right thing.
>
> 	Thanks for testing this, especially as I can not due to a lack of an
> ADSL-link (and lack of cake actually, last I looked all I could find was
> cookies in my browser and a promise of pie in my router)
>
>>
>>
>> Method:
>>
>> I used the updated files from sqm-scripts,
>>
>> (once I remembered to mark them executable.  Lacking that causes a
>> failure with no error messages, because sqm-scripts checks before
>> running them :)
>>
>> but didn't bother updating & using luci-app-sqm.
>
> 	Ah, okay, I guess I did test this part with Dave’s help, so this should
> work with the most recent sqm.lua.
>
>>
>> The test was to compare netperf-runner results - ping during combined
>> upload & download - for "overhead 40" and "overhead 0".  I tested both
>> values of linklayer_adaptation_mechanism.
>>
>> I had to repeat 6 times (60s per run for each overhead) because of
>> random variation in the range of 3-4ms.  I alternated "overhead 40"
>> and "overhead 0" to try and exclude longer-term variation effects.
>>
>> With "stab overhead 40", median latency was better by about 3-4ms.
>> With "cake overhead 40", there is no such effect.
>
> 	Intersting, when I still had a 6M/1M ADSL link, I saw much larger latency
> under load increases when setting the per packet overhead to small, but I
> had my egress shaper running at 100% of line rate, so the system was rigged
> for maximum effect that way. How are your shapers typically set?

For this test I try to push it, today I used 95%.  I started trying
100%, which is still much better than unshaped.  I was scared off by
the random variation, I think it was higher at 100%.

For long term use I reduce it, because I've seen the line rate vary
slightly.  (1020k up today, 912 a while back.  Currently it reports a
"max" figure I don't understand, it's about 1100 despite being
rebooted daily.  16390k down).

Alan

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Dave Taht]

re: keywords: it may be that the tc-adv package is not correctly
overriding the tc package. Sigh.

try a opkg update; opkg remove tc-adv; opkg install tc-adv and see if
you get the keywords.

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Alan Jenkins]

On 19/06/15 21:12, Dave Taht wrote:
> re: keywords: it may be that the tc-adv package is not correctly
> overriding the tc package. Sigh.
>
> try a opkg update; opkg remove tc-adv; opkg install tc-adv and see if
> you get the keywords.

Or I screwed up and the box is running the previous build?

$ ssh root@mortar
BusyBox v1.23.2 (2015-06-12 19:22:15 PDT) built-in shell (ash)


I hope so because the alternative is debugging this-

root@mortar:~# strace
-ash: strace: not found
root@mortar:~# echo $?
127

(same happens when installing tc-adv)

Alan

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Dave Taht]

hmm?

opkg update; opkg install strace ?

I have done several builds in this dir in the last few days, however,
and that could be messing you up. I am trying to get a good build now,
but it is blowing up on needing libssp for some reason.

I really picked the wrong time to try and get something stable built.
This was my last big chance to deploy a few routers in test before
starting to travel extensively.



On Fri, Jun 19, 2015 at 1:40 PM, Alan Jenkins
<alan.christopher.jenkins@gmail.com> wrote:
> On 19/06/15 21:12, Dave Taht wrote:
>>
>> re: keywords: it may be that the tc-adv package is not correctly
>> overriding the tc package. Sigh.
>>
>> try a opkg update; opkg remove tc-adv; opkg install tc-adv and see if
>> you get the keywords.
>
>
> Or I screwed up and the box is running the previous build?
>
> $ ssh root@mortar
> BusyBox v1.23.2 (2015-06-12 19:22:15 PDT) built-in shell (ash)
>
>
> I hope so because the alternative is debugging this-
>
> root@mortar:~# strace
> -ash: strace: not found
> root@mortar:~# echo $?
> 127
>
> (same happens when installing tc-adv)
>
> Alan



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Dave Taht]

probably easy to fix. Not that pimd is well maintained in the first place.

make[4]: Entering directory
`/build/cero3/src/ubnt/build_dir/target-mips_34kc_musl-1.1.10/pimd-2.1.8'
  CC      pim.o
pim.c: In function 'pim_read':
pim.c:128:5: error: implicit declaration of function 'sigblock'
[-Werror=implicit-function-declaration]
     omask = sigblock(sigmask(SIGALRM));
     ^
pim.c:128:5: error: implicit declaration of function 'sigmask'
[-Werror=implicit-function-declaration]
pim.c:136:5: error: implicit declaration of function 'sigsetmask'
[-Werror=implicit-function-declaration]
     (void)sigsetmask(omask);
     ^
cc1: all warnings being treated as errors
make[4]: *** [pim.o] Error 1
make[4]: Leaving directory
`/build/cero3/src/ubnt/build_dir/target-mips_34kc_musl-1.1.10/pimd-2.1.8'
make[3]: *** [/build/cero3/src/ubnt/build_dir/target-mips_34kc_musl-1.1.10/pimd-2.1.8/.built]
Error 2
make[3]: Leaving directory `/build/cero3/src/ubnt/feeds/cero/net/pimd'
make[2]: *** [package/feeds/cero/pimd/compile] Error 2
make[2]: Leaving directory `/build/cero3/src/ubnt'
make[1]: *** [/build/cero3/src/ubnt/staging_dir/target-mips_34kc_musl-1.1.10/stamp/.package_compile]
Error 2
make[1]: Leaving directory `/build/cero3/src/ubnt'
make: *** [world] Error 2

On Fri, Jun 19, 2015 at 1:51 PM, Dave Taht <dave.taht@gmail.com> wrote:
> hmm?
>
> opkg update; opkg install strace ?
>
> I have done several builds in this dir in the last few days, however,
> and that could be messing you up. I am trying to get a good build now,
> but it is blowing up on needing libssp for some reason.
>
> I really picked the wrong time to try and get something stable built.
> This was my last big chance to deploy a few routers in test before
> starting to travel extensively.
>
>
>
> On Fri, Jun 19, 2015 at 1:40 PM, Alan Jenkins
> <alan.christopher.jenkins@gmail.com> wrote:
>> On 19/06/15 21:12, Dave Taht wrote:
>>>
>>> re: keywords: it may be that the tc-adv package is not correctly
>>> overriding the tc package. Sigh.
>>>
>>> try a opkg update; opkg remove tc-adv; opkg install tc-adv and see if
>>> you get the keywords.
>>
>>
>> Or I screwed up and the box is running the previous build?
>>
>> $ ssh root@mortar
>> BusyBox v1.23.2 (2015-06-12 19:22:15 PDT) built-in shell (ash)
>>
>>
>> I hope so because the alternative is debugging this-
>>
>> root@mortar:~# strace
>> -ash: strace: not found
>> root@mortar:~# echo $?
>> 127
>>
>> (same happens when installing tc-adv)
>>
>> Alan
>
>
>
> --
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Alan Jenkins]

On 19/06/15 21:51, Dave Taht wrote:
> hmm?
>
> opkg update; opkg install strace ?

sorry.  Yes, that's what I did; I left that part out.  strace isn't 
installed by default, I tried to install it at some point, got "not found"

same error with tc command from tc-adv if I try to reinstall it


> I have done several builds in this dir in the last few days, however,
> and that could be messing you up.

Ok, that's it.

root@mortar:~# grep musl `which strace`
\x04/lib/ld-musl-mips-sf.so.1

Don't think that's going to work on my box somehow :)

root@mortar:~# grep -i uclibc `which which`
\x04/lib/ld-uClibc.so.0
__uClibc_main



>   I am trying to get a good build now,
> but it is blowing up on needing libssp for some reason.
>
> I really picked the wrong time to try and get something stable built.
> This was my last big chance to deploy a few routers in test before
> starting to travel extensively.
>
>
>
> On Fri, Jun 19, 2015 at 1:40 PM, Alan Jenkins
> <alan.christopher.jenkins@gmail.com> wrote:
>> On 19/06/15 21:12, Dave Taht wrote:
>>> re: keywords: it may be that the tc-adv package is not correctly
>>> overriding the tc package. Sigh.
>>>
>>> try a opkg update; opkg remove tc-adv; opkg install tc-adv and see if
>>> you get the keywords.
>>
>> Or I screwed up and the box is running the previous build?
>>
>> $ ssh root@mortar
>> BusyBox v1.23.2 (2015-06-12 19:22:15 PDT) built-in shell (ash)
>>
>>
>> I hope so because the alternative is debugging this-
>>
>> root@mortar:~# strace
>> -ash: strace: not found
>> root@mortar:~# echo $?
>> 127
>>
>> (same happens when installing tc-adv)
>>
>> Alan
>
>

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Dave Taht]

I am making one last effort to do a new build, please give me 10
minutes to see if it completes.

if it doesn't there will be much gnashing of teeth, and grumbling you
should probably be able to hear from your location.


On Fri, Jun 19, 2015 at 1:57 PM, Alan Jenkins
<alan.christopher.jenkins@gmail.com> wrote:
> On 19/06/15 21:51, Dave Taht wrote:
>>
>> hmm?
>>
>> opkg update; opkg install strace ?
>
>
> sorry.  Yes, that's what I did; I left that part out.  strace isn't
> installed by default, I tried to install it at some point, got "not found"
>
> same error with tc command from tc-adv if I try to reinstall it
>
>
>> I have done several builds in this dir in the last few days, however,
>> and that could be messing you up.
>
>
> Ok, that's it.
>
> root@mortar:~# grep musl `which strace`
> /lib/ld-musl-mips-sf.so.1
>
> Don't think that's going to work on my box somehow :)
>
> root@mortar:~# grep -i uclibc `which which`
> /lib/ld-uClibc.so.0
> __uClibc_main
>
>
>
>
>>   I am trying to get a good build now,
>> but it is blowing up on needing libssp for some reason.
>>
>> I really picked the wrong time to try and get something stable built.
>> This was my last big chance to deploy a few routers in test before
>> starting to travel extensively.
>>
>>
>>
>> On Fri, Jun 19, 2015 at 1:40 PM, Alan Jenkins
>> <alan.christopher.jenkins@gmail.com> wrote:
>>>
>>> On 19/06/15 21:12, Dave Taht wrote:
>>>>
>>>> re: keywords: it may be that the tc-adv package is not correctly
>>>> overriding the tc package. Sigh.
>>>>
>>>> try a opkg update; opkg remove tc-adv; opkg install tc-adv and see if
>>>> you get the keywords.
>>>
>>>
>>> Or I screwed up and the box is running the previous build?
>>>
>>> $ ssh root@mortar
>>> BusyBox v1.23.2 (2015-06-12 19:22:15 PDT) built-in shell (ash)
>>>
>>>
>>> I hope so because the alternative is debugging this-
>>>
>>> root@mortar:~# strace
>>> -ash: strace: not found
>>> root@mortar:~# echo $?
>>> 127
>>>
>>> (same happens when installing tc-adv)
>>>
>>> Alan
>>
>>
>>
>



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Dave Taht]

Fresh bits! Get your fresh bits here! (totally untested)

http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Luis E. Garcia]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]

Dave,
Just a  quick question:
Cake is not yet available through the LUCI interface, right?

Regards,
Luis

On Fri, Jun 19, 2015 at 4:24 PM, Dave Taht <dave.taht@gmail.com> wrote:

> Fresh bits! Get your fresh bits here! (totally untested)
>
> http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 1148 bytes --]

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Dave Taht]

On Fri, Jun 19, 2015 at 2:52 PM, Luis E. Garcia <luis@bitamins.net> wrote:
> Dave,
> Just a  quick question:
> Cake is not yet available through the LUCI interface, right?

There is no gui support as yet. Patches gladly accepted, as always.
>
> Regards,
> Luis
>
> On Fri, Jun 19, 2015 at 4:24 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> Fresh bits! Get your fresh bits here! (totally untested)
>>
>> http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>



-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 1354 bytes --]

Hi Luis,

it should be possible to select cake from the list of qdiscs. And then you should get a somewhat working cake setup. The most recent version also wired up a few more fields for cake including the advanced configuration string fields in the dangerous configuration options in the Queue Discipline tab. If you test it, please post your result (failure or success) here on the list....

Best Regards
       Sebastian

On June 19, 2015 11:52:43 PM GMT+02:00, "Luis E. Garcia" <luis@bitamins.net> wrote:
>Dave,
>Just a  quick question:
>Cake is not yet available through the LUCI interface, right?
>
>Regards,
>Luis
>
>On Fri, Jun 19, 2015 at 4:24 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
>> Fresh bits! Get your fresh bits here! (totally untested)
>>
>> http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Cerowrt-devel mailing list
>Cerowrt-devel@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/cerowrt-devel

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 2342 bytes --]

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Jim Reisert AD1C]

On Fri, Jun 19, 2015 at 3:24 PM, Dave Taht wrote:

> Fresh bits! Get your fresh bits here! (totally untested)
> http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/

I installed the factory image on a WNDR3800 and all came up as it
should.  WINS networking is working, both Dish Network and Sony
blu-ray players have wireless Internet access.

Are the "out-of-the-box" settings for SQM acceptable for Comcast (60
Mbs down/6 Mbs up)?
Should I check the "Enable" box on the SQM "Basic Settings" tab?
Should I change any settings on the "Queue Discipline" tab?

Thanks - Jim

-- 
Jim Reisert AD1C, <jjreisert@alum.mit.edu>, http://www.ad1c.us

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Sebastian Moeller]

Hi Jim,


On Jun 23, 2015, at 04:41 , Jim Reisert AD1C <jjreisert@alum.mit.edu> wrote:

> On Fri, Jun 19, 2015 at 3:24 PM, Dave Taht wrote:
> 
>> Fresh bits! Get your fresh bits here! (totally untested)
>> http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/
> 
> I installed the factory image on a WNDR3800 and all came up as it
> should.  WINS networking is working, both Dish Network and Sony
> blu-ray players have wireless Internet access.
> 
> Are the "out-of-the-box" settings for SQM acceptable for Comcast (60
> Mbs down/6 Mbs up)?

	Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm . Rich published a great set of instructions for setting up sqm-scripts under openwrt proper.

> Should I check the "Enable" box on the SQM "Basic Settings" tab?

	You certain should if you want it to be active, make sure though, to at least select the correct interface and put in rates for the shaper below your link rates, so probably 56Mbps for downlink and 5.7Mbps for uplink as first approximation.

> Should I change any settings on the "Queue Discipline" tab?

	You need to select the qos script of your choice, simple.qos if you want to have three priority tiers, simplest.qos if you do not.

Best Regards
	Sebastian

> 
> Thanks - Jim
> 
> -- 
> Jim Reisert AD1C, <jjreisert@alum.mit.edu>, http://www.ad1c.us
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

[Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Tue, 23 Jun 2015, Sebastian Moeller wrote:

> 	Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm . 
> Rich published a great set of instructions for setting up sqm-scripts 
> under openwrt proper.

I tried it on Linksys WRT1200AC with OpenWrt CC RC2. I configured sqm to 
have 800 megabit/s each direction, and ran iperf3 over IPv4 with NAT44 
from Linux box behind WRT1200AC to an OSX macbook connected on a switch on 
the same L2 subnet as the WAN port.

Linux <->WRT1200AC<->switch<->OSX

I get 765 megabit/s of throughput using single session, at sirq load of 
around 25%. If I lower the mss to 300 (to generate higher pps) I get 
around 560 megabit/s of throughput at 50% sirq. With 10 parallel TCP 
sessions, I get about the same. At MSS of 200 bytes, I get 400 megabit/s 
at 70% sirq.

If I turn off SQM completely, I get 600 megabit/s at 200 byte MSS single 
session at 80% sirq and 930 megabit/s at 26% sirq with default MSS.

So if you want high performing device that is OpenWRT compatible and still 
does forwarding using CPU so you can test queuing algorithms, the 
WRT1200AC and WRT1900ACv2 is the best I have been able to find currently 
(unless you go for x86 platform).

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

On Tue, Jun 23, 2015 at 5:55 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Tue, 23 Jun 2015, Sebastian Moeller wrote:
>
>>         Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm .
>> Rich published a great set of instructions for setting up sqm-scripts under
>> openwrt proper.
>
>
> I tried it on Linksys WRT1200AC with OpenWrt CC RC2. I configured sqm to
> have 800 megabit/s each direction, and ran iperf3 over IPv4 with NAT44 from
> Linux box behind WRT1200AC to an OSX macbook connected on a switch on the
> same L2 subnet as the WAN port.
>
> Linux <->WRT1200AC<->switch<->OSX
>
> I get 765 megabit/s of throughput using single session, at sirq load of
> around 25%. If I lower the mss to 300 (to generate higher pps) I get around
> 560 megabit/s of throughput at 50% sirq. With 10 parallel TCP sessions, I
> get about the same. At MSS of 200 bytes, I get 400 megabit/s at 70% sirq.
>
> If I turn off SQM completely, I get 600 megabit/s at 200 byte MSS single
> session at 80% sirq and 930 megabit/s at 26% sirq with default MSS.

Yer missing the more important figure. What is the induced latency in
all these cases? With the system being software limited, I would
imagine that other oddities arise. Running out of cpu, additional
oddities.

When going at hardware line rate, is fq_codel enabled? Does it ever
engage? (My limited testing showed that lacking BQL, delay accrued big
time in the drivers themselves on this platform)

Good idea on using a reduced MSS size!

I would really like to get to the point where cake ran on this
platform, but thus far we have not managed to get a working build, nor
push cake into mainline openwrt.... my observation is that a lot of
the overhead of sqm comes from tc filters, iptables rules and NAT.

> So if you want high performing device that is OpenWRT compatible and still
> does forwarding using CPU so you can test queuing algorithms, the WRT1200AC
> and WRT1900ACv2 is the best I have been able to find currently (unless you
> go for x86 platform).

yes, x86 is fastest. I have not tried the reduced mss idea on my
rangeley box, I will check that out!

> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

Hi Mikael,


On Jun 23, 2015, at 14:55 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Tue, 23 Jun 2015, Sebastian Moeller wrote:
> 
>> 	Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm . Rich published a great set of instructions for setting up sqm-scripts under openwrt proper.
> 
> I tried it on Linksys WRT1200AC with OpenWrt CC RC2. I configured sqm to have 800 megabit/s each direction, and ran iperf3 over IPv4 with NAT44 from Linux box behind WRT1200AC to an OSX macbook connected on a switch on the same L2 subnet as the WAN port.
> 
> Linux <->WRT1200AC<->switch<->OSX

	Thanks a lot, interesting data! Was this test stressing both directions at the same time? (My guess is if the test was UDP i don’t know, for a TCP test I am quite confident that it was uni-directional as the @full MTU data does not show enough loss to accommodate the roughly 2% reverse ACK traffic for the opposite direction).

> 
> I get 765 megabit/s of throughput using single session, at sirq load of around 25%. If I lower the mss to 300 (to generate higher pps) I get around 560 megabit/s of throughput at 50% sirq. With 10 parallel TCP sessions, I get about the same. At MSS of 200 bytes, I get 400 megabit/s at 70% sirq.

I assume iperf3 uses TCP or UDP streams and reports the payload rate, correct? Then we have a MSS of 1460 (with 20 bytes IPv4 header and 20 bytes for TCP or UDP).


@full MTU
MSS:
1500 - 20 - 20 = 1460 byte
Number of packets at 765 Mbps goodput:
(765 * 1000^2) / ((1500 - 20 - 20) * 8) = 65496.5753425 = 65K
On-the-wire packet size (OTWPS) assuming ethernet with FCS and no VLAN tag)s):
1500 + 14 + 4 = 1518 bytes
MSS to OTWPS ratio:
(1500 - 20 - 20) / (1500 + 14 + 4) = 0.961791831357
raw bandwidth consumed by 765 Mbps good put:
765 / ((1500 - 20 - 20) / (1500 + 14 + 4)) = ((765 * 1000^2) / ((1500 - 20 - 20) * 8)) * ((1500 + 14 + 4) * 8) = 795.390410959 Mbps
So basically (1 - (795.390410959/800))*100 = 0.58 % unexplained loss, not bad

@MSS 300
MSS:
300 byte
Number of packets at 560 Mbps goodput:
(560 * 1000^2) / ((300) * 8) = 233333.333333 = 233K
On-the-wire packet size (OTWPS) assuming ethernet with FCS:
300 + 20 + 20 + 14 + 4  = 358 bytes
MSS to OTWPS ratio:
(300) / (300 + 20 + 20 + 14 + 4) = 0.837988826816
raw bandwidth consumed by 560 Mbps good put:
560 / ((300) / (300 + 20 + 20 + 14 + 4 )) = ((560 * 1000^2) / ((300) * 8)) * ((300 + 20 + 20 + 14 + 4 ) * 8) = 668.266666667  Mbps
So basically (1 - (668.266666667/800))*100 = 16.4666666666 % unexplained loss, not pretty but bearable I guess

@MSS 200
MSS:
200 byte
Number of packets at 400 Mbps goodput:
(400 * 1000^2) / ((200) * 8) = 250000 = 250K
On-the-wire packet size (OTWPS) assuming ethernet with FCS:
200 + 20 + 20 + 14 + 4  = 258 bytes
MSS to OTWPS ratio:
(200) / (200 + 20 + 20 + 14 + 4) = 0.77519379845
raw bandwidth consumed by 400 Mbps good put:
400 / ((200) / (200 + 20 + 20 + 14 + 4 )) = ((400 * 1000^2) / ((200) * 8)) * ((200 + 20 + 20 + 14 + 4 ) * 8) = 516  Mbps
So basically (1 - (516/800))*100 = 35.5  % unexplained loss, that is sad. But the packet rate is still at 250K, I winder how this router does with 64 byte ethernet frames


> 
> If I turn off SQM completely, I get 600 megabit/s at 200 byte MSS single session at 80% sirq and 930 megabit/s at 26% sirq with default MSS.

	Since no shaper was used,  I think we need to include the inter-frame-gap and preamble to calculate the maximal payload rates for different packet sizes.

@1Gbps
MSS
(1500 - 20 - 20) = 1460 byte
Number of packets at 930 Mbps goodput:
(930 * 1000^2) / ((1500 - 20 - 20) * 8) = 79623.2876712 = 80K
To asses the maximum achievable at 1 GBE we need to take IFG and preamble into account I think
On-the-wire packet size (OTWPS) assuming ethernet with FCS plus inter frame gap and preamble:
1500 + 14 + 4 + 12 + 8  = 1538 bytes
MSS to OTWPS ratio:
(1500 - 20 - 20)  / (1500 + 14 + 4 + 12 + 8) = 0.949284785436
raw bandwidth consumed by 930 Mbps good put:
930 / ((1500 - 20 - 20) / (1500 + 14 + 4 + 12 + 8)) = ((930 * 1000^2) / ((1500 - 20 - 20) * 8)) * ((1500 + 14 + 4 + 12 + 8) * 8) = 979.684931507  Mbps
So basically (1 - (979.684931507/1000))*100 = 2.0315068493  % unexplained loss, not bad. 

@1Gbps
MSS
200 = 1460 byte
Number of packets at 600 Mbps goodput:
(600 * 1000^2) / ((200) * 8) = 375000 = 375K
On-the-wire packet size (OTWPS) assuming ethernet with FCS plus inter frame gap and preamble:
200 + 20 + 20 + 14 + 4 + 12 + 8  = 278 bytes
MSS to OTWPS ratio:
200  / (200 + 20 + 20 + 14 + 4 + 12 + 8) = 0.719424460432
raw bandwidth consumed by 600 Mbps good put:
600 / ((200) / (200 + 20 + 20 + 14 + 4 + 12 + 8)) = ((600 * 1000^2) / ((200) * 8)) * ((200 + 20 + 20 + 14 + 4 + 12 + 8) * 8) = 834  Mbps
So basically (1 - (834/1000))*100 = 16.6  % unexplained loss, not bad. 

As Dave said it would be nice see RRUL data from the same testbed. It would be so nice if flint had a way to send different sized TCP packets… (I guess this might be faked with MSS clamping in the router and relaying on path MTU discovery?)


> 
> So if you want high performing device that is OpenWRT compatible and still does forwarding using CPU so you can test queuing algorithms, the WRT1200AC and WRT1900ACv2 is the best I have been able to find currently (unless you go for x86 platform).

	The 1200AC retailed for around 200EUR in Germany the 1900ACv2 will be closer to 300EUR I guess, not too expensive but certainly above my impulse buy limit ;)


tack så mycket & Best Regards
	Sebastian

> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 99 bytes --]

Not so easy to find those in Finland, it seems, but I assume Amazon carry
them.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 138 bytes --]

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Tue, 23 Jun 2015, Jonathan Morton wrote:

> Not so easy to find those in Finland, it seems, but I assume Amazon 
> carry them.

www.webhallen.com is the only retailer in Sweden that currently have them 
in stock. They just now becoming available, so I would imagine in the next 
few weeks they'll be more widely available.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1662 bytes --]

On Tue, 23 Jun 2015, Sebastian Moeller wrote:

> 	Thanks a lot, interesting data! Was this test stressing both 
> directions at the same time? (My guess is if the test was UDP i don’t 
> know, for a TCP test I am quite confident that it was uni-directional as 
> the @full MTU data does not show enough loss to accommodate the roughly 
> 2% reverse ACK traffic for the opposite direction).

It was TCP using iperf3, so it was larger data packets one direction and 
ACKs going the other direction.

> So basically (1 - (795.390410959/800))*100 = 0.58 % unexplained loss, not bad

Oh, iperf3 can tell you packets lost, so I can re-run the tests and tell 
you exactly how many packets were lost and within what 1 second interval 
they were lost. My guess is that it would be better to run something else. 
My numbers were not meant to be as exact as you have used for calculation, 
I'd say I aimed for approximate number. I'll do a test run and give you 
more exact data.

> As Dave said it would be nice see RRUL data from the same testbed. It 
> would be so nice if flint had a way to send different sized TCP packets… 
> (I guess this might be faked with MSS clamping in the router and 
> relaying on path MTU discovery?)

That is my next thing to test.

> 	The 1200AC retailed for around 200EUR in Germany the 1900ACv2 will 
> be closer to 300EUR I guess, not too expensive but certainly above my 
> impulse buy limit ;)

Yes, I paid 195 EUR for it (1790 SEK), which includes 25% VAT. It's around 
150 USD (not including tax) so I'm hoping it'll drop 10-20% in price when 
it's more widely available in Europe.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 868 bytes --]

On Tue, 23 Jun 2015, Sebastian Moeller wrote:

> As Dave said it would be nice see RRUL data from the same testbed. It 
> would be so nice if flint had a way to send different sized TCP packets… 
> (I guess this might be faked with MSS clamping in the router and 
> relaying on path MTU discovery?)

What kind of tests should I run? I have rrul results now, both at the same 
time as running iperf3. I set iperf3 to run with 10 parallel streams, 
different MSS per test, and let it run for 30 seconds into the rrul test. 
So in all the tests, iperf3 session stops running half way into the rrul 
test.

I set sqm to 500M up and down on eth0, ECN up and down, and not to squash 
DSCP in any direction.

http://swm.pp.se/aqm/rrul-wrt1200ac-150624.pdf

Tell me if there is more information I can provide or tests to run.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

On Wed, Jun 24, 2015 at 4:31 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Tue, 23 Jun 2015, Sebastian Moeller wrote:
>
>> As Dave said it would be nice see RRUL data from the same testbed. It
>> would be so nice if flint had a way to send different sized TCP packets… (I
>> guess this might be faked with MSS clamping in the router and relaying on
>> path MTU discovery?)
>
>
> What kind of tests should I run? I have rrul results now, both at the same
> time as running iperf3. I set iperf3 to run with 10 parallel streams,
> different MSS per test, and let it run for 30 seconds into the rrul test. So
> in all the tests, iperf3 session stops running half way into the rrul test.
>
> I set sqm to 500M up and down on eth0, ECN up and down, and not to squash
> DSCP in any direction.
>
> http://swm.pp.se/aqm/rrul-wrt1200ac-150624.pdf

From what I see here you are rarely, if ever, engaging fq_codel
properly. Latencies are pretty high. In particular, I would suspect
you are hitting offloads hard, and the current (fixed in linux 4.1)
codel drop algorithm stops dropping below "maxpacket", which was meant
in the ns2 code to be a MTU, but in the linux code ended up being a
TSO sized (64k!) packet.

tc -s qdisc show # will show the maxpacket.

> Tell me if there is more information I can provide or tests to run.

0) I tend to have a script for everything... I can give you an example
script in another email.

1) tc -s qdisc show dev whatever # will show actuall drop/mark statistics

2) Please run your flent tests with -x --disable-log

-x collects more metadata, --disable-log disables the automatic log
scaling which is so hard to discern on these plots.

Use -t "title" to differentiate between variables under test.

3) I also tend to use flent's --remote-metadata=root@your_openwrtbox
to get the stats on that box into the metadata. You have to add your
local .ssh/id_rsa.pub key to
your_openwrtbox:/etc/dropbear/authorized_keys file to do this.

4) With all that in hand, sticking up a tarball of the results makes
for easy plotting of various other graphs, and using the flent-gui,
you can combine results from each run easily, also.

5) try disabling offloads on all interfaces on the router (or running cake)

My usual suite of tests is rrul, rrul_be, tcp_1up, tcp_1down, and tcp_2up_delay.

and rtt_fair (if you have more than one target server available)...
all without the iperf stuff....

Your mss changing idea is interesting, and I would like to do a mod to
tcp_2up_delay to sort of match your iperf combination + flent to
totally capture all data.

6) I am pretty interested as to what happens *without* sqm at the max
forwarding rate with fq_codel engaged on all these tests.

>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 4247 bytes --]

Here's a long-range (indoors) test result with the 1900AC.  I have about
9dB snr at the client (all the way across my house, which has plaster
walls):

http://www.dslreports.com/speedtest/737736

download buffering is the 1900ac, upload buffering is the mac laptop's
driver.  Both are trying too hard to deliver those packets, I think.

The driver has crazy over-buffering in the face of poor signal conditions.
OTOH, I'm amazed at the throughput numbers.  But this is an 80Mhz wide 5Ghz
channel, in a quiet environment, so it has LOTS of airspace to work in.

-Aaron

On Wed, Jun 24, 2015 at 9:32 AM, Dave Taht <dave.taht@gmail.com> wrote:

> On Wed, Jun 24, 2015 at 4:31 AM, Mikael Abrahamsson <swmike@swm.pp.se>
> wrote:
> > On Tue, 23 Jun 2015, Sebastian Moeller wrote:
> >
> >> As Dave said it would be nice see RRUL data from the same testbed. It
> >> would be so nice if flint had a way to send different sized TCP
> packets… (I
> >> guess this might be faked with MSS clamping in the router and relaying
> on
> >> path MTU discovery?)
> >
> >
> > What kind of tests should I run? I have rrul results now, both at the
> same
> > time as running iperf3. I set iperf3 to run with 10 parallel streams,
> > different MSS per test, and let it run for 30 seconds into the rrul
> test. So
> > in all the tests, iperf3 session stops running half way into the rrul
> test.
> >
> > I set sqm to 500M up and down on eth0, ECN up and down, and not to squash
> > DSCP in any direction.
> >
> > http://swm.pp.se/aqm/rrul-wrt1200ac-150624.pdf
>
> From what I see here you are rarely, if ever, engaging fq_codel
> properly. Latencies are pretty high. In particular, I would suspect
> you are hitting offloads hard, and the current (fixed in linux 4.1)
> codel drop algorithm stops dropping below "maxpacket", which was meant
> in the ns2 code to be a MTU, but in the linux code ended up being a
> TSO sized (64k!) packet.
>
> tc -s qdisc show # will show the maxpacket.
>
> > Tell me if there is more information I can provide or tests to run.
>
> 0) I tend to have a script for everything... I can give you an example
> script in another email.
>
> 1) tc -s qdisc show dev whatever # will show actuall drop/mark statistics
>
> 2) Please run your flent tests with -x --disable-log
>
> -x collects more metadata, --disable-log disables the automatic log
> scaling which is so hard to discern on these plots.
>
> Use -t "title" to differentiate between variables under test.
>
> 3) I also tend to use flent's --remote-metadata=root@your_openwrtbox
> to get the stats on that box into the metadata. You have to add your
> local .ssh/id_rsa.pub key to
> your_openwrtbox:/etc/dropbear/authorized_keys file to do this.
>
> 4) With all that in hand, sticking up a tarball of the results makes
> for easy plotting of various other graphs, and using the flent-gui,
> you can combine results from each run easily, also.
>
> 5) try disabling offloads on all interfaces on the router (or running cake)
>
> My usual suite of tests is rrul, rrul_be, tcp_1up, tcp_1down, and
> tcp_2up_delay.
>
> and rtt_fair (if you have more than one target server available)...
> all without the iperf stuff....
>
> Your mss changing idea is interesting, and I would like to do a mod to
> tcp_2up_delay to sort of match your iperf combination + flent to
> totally capture all data.
>
> 6) I am pretty interested as to what happens *without* sqm at the max
> forwarding rate with fq_codel engaged on all these tests.
>
> >
> > --
> > Mikael Abrahamsson    email: swmike@swm.pp.se
> >
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-devel@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >
>
>
>
> --
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 5862 bytes --]

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Wed, 24 Jun 2015, Aaron Wood wrote:

> Here's a long-range (indoors) test result with the 1900AC.  I have about
> 9dB snr at the client (all the way across my house, which has plaster
> walls):
>
> http://www.dslreports.com/speedtest/737736
>
> download buffering is the 1900ac, upload buffering is the mac laptop's
> driver.  Both are trying too hard to deliver those packets, I think.
>
> The driver has crazy over-buffering in the face of poor signal conditions.
> OTOH, I'm amazed at the throughput numbers.  But this is an 80Mhz wide 5Ghz
> channel, in a quiet environment, so it has LOTS of airspace to work in.

Is this over a 25/5 Internet connection? Because I wouldn't imagine that 
you would get any significant buffering on the wifi with that kind of 
Internet access speed. I regularily get hundreds of megabit/s over 
802.11ac wifi.

Also, I guess this is WRT1900ACv1? Running what OS?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 884 bytes --]

On Wed, Jun 24, 2015 at 8:07 PM, Mikael Abrahamsson <swmike@swm.pp.se>
wrote:

> On Wed, 24 Jun 2015, Aaron Wood wrote:
>
>  Here's a long-range (indoors) test result with the 1900AC.  I have about
>> 9dB snr at the client (all the way across my house, which has plaster
>> walls):
>>
>

> Is this over a 25/5 Internet connection? Because I wouldn't imagine that
> you would get any significant buffering on the wifi with that kind of
> Internet access speed. I regularily get hundreds of megabit/s over 802.11ac
> wifi.
>

Comcast cable, with ~150Mbps down and 12Mbps up.  here's a close-range
result for comparison:

http://www.dslreports.com/speedtest/675012



> Also, I guess this is WRT1900ACv1? Running what OS?


Yep, running OpenWRT CC RC1, with sqm-scripts installed, setup as in:

http://burntchrome.blogspot.com/2015/06/sqmscripts-before-and-after-at-160mbps.html

-Aaron

[-- Attachment #2: Type: text/html, Size: 2311 bytes --]

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Wed, 24 Jun 2015, Dave Taht wrote:

> From what I see here you are rarely, if ever, engaging fq_codel
> properly. Latencies are pretty high. In particular, I would suspect
> you are hitting offloads hard, and the current (fixed in linux 4.1)
> codel drop algorithm stops dropping below "maxpacket", which was meant
> in the ns2 code to be a MTU, but in the linux code ended up being a
> TSO sized (64k!) packet.
>
> tc -s qdisc show # will show the maxpacket.

http://swm.pp.se/aqm/qdisc-show.txt

I discovered also that I wasn't running ECN, I had set tcp_ecn = 2 on the 
linux box. All the tests done in 
http://swm.pp.se/aqm/flent-mikabr-150625-1.tar are now done with ECN 
working, without iperf3 running, and with and without SQM, but with 
default offload setting.

Also, now iperf3 says "0 packet lost" when I use that.

> 2) Please run your flent tests with -x --disable-log

Done.

> Use -t "title" to differentiate between variables under test.

Done.

> 3) I also tend to use flent's --remote-metadata=root@your_openwrtbox
> to get the stats on that box into the metadata. You have to add your
> local .ssh/id_rsa.pub key to
> your_openwrtbox:/etc/dropbear/authorized_keys file to do this.

Done.

> 4) With all that in hand, sticking up a tarball of the results makes
> for easy plotting of various other graphs, and using the flent-gui,
> you can combine results from each run easily, also.

See above.

> 5) try disabling offloads on all interfaces on the router (or running cake)

This is my next thing to test, I have some other things I need to try 
first.

> My usual suite of tests is rrul, rrul_be, tcp_1up, tcp_1down, and tcp_2up_delay.

Done.

> and rtt_fair (if you have more than one target server available)...
> all without the iperf stuff....

I do not have another machine readily available here at home.

> 6) I am pretty interested as to what happens *without* sqm at the max 
> forwarding rate with fq_codel engaged on all these tests.

This is included. Why did you want to do this test? Just to see if the 
WRT1200AC can do wirespeed forwarding with fq_codel on? Because with the 
setup I have, I don't see how there can be any buffering going on because 
it's a single gig port both in and out.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Thu, 25 Jun 2015, Mikael Abrahamsson wrote:

>> 5) try disabling offloads on all interfaces on the router (or running cake)
>
> This is my next thing to test, I have some other things I need to try first.

http://swm.pp.se/aqm/flent-mikabr-150625-2-sqm-tsogsogro-off.tar

It includes an ethtool -k eth0 and eth1 to show what the settings were.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

On Thu, Jun 25, 2015 at 2:12 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Wed, 24 Jun 2015, Dave Taht wrote:
>
>> From what I see here you are rarely, if ever, engaging fq_codel
>> properly. Latencies are pretty high. In particular, I would suspect
>> you are hitting offloads hard, and the current (fixed in linux 4.1)
>> codel drop algorithm stops dropping below "maxpacket", which was meant
>> in the ns2 code to be a MTU, but in the linux code ended up being a
>> TSO sized (64k!) packet.
>>
>> tc -s qdisc show # will show the maxpacket.
>
>
> http://swm.pp.se/aqm/qdisc-show.txt

Ugh. 64k maxpacket. I did not even know that was possible until I saw
it (GRO is usually for tcp acks, and peaks at 24k or so).

I will check to see if openwrt backported that crucial codel patch. Or
we can switch sqm "simplest" to use tbf, or we can do cake´s
peeling....

> I discovered also that I wasn't running ECN, I had set tcp_ecn = 2 on the
> linux box. All the tests done in
> http://swm.pp.se/aqm/flent-mikabr-150625-1.tar are now done with ECN
> working, without iperf3 running, and with and without SQM, but with default
> offload setting.
>
> Also, now iperf3 says "0 packet lost" when I use that.

cool. :) Your router has it off, too, and if you plan to use it for
other things than routing, you might want to turn it on (gleaned from
your metadata, thx!)

>> 2) Please run your flent tests with -x --disable-log
>
>
> Done.
>
>> Use -t "title" to differentiate between variables under test.
>
>
> Done.
>
>> 3) I also tend to use flent's --remote-metadata=root@your_openwrtbox
>> to get the stats on that box into the metadata. You have to add your
>> local .ssh/id_rsa.pub key to
>> your_openwrtbox:/etc/dropbear/authorized_keys file to do this.
>
>
> Done.

Unfortunately the core piece of metadata I wanted from the router was
the qdisc statistics. Didnt parse. Will file bug.

>
>> 4) With all that in hand, sticking up a tarball of the results makes
>> for easy plotting of various other graphs, and using the flent-gui,
>> you can combine results from each run easily, also.
>
>
> See above.
>
>> 5) try disabling offloads on all interfaces on the router (or running
>> cake)
>
>
> This is my next thing to test, I have some other things I need to try first.
>
>> My usual suite of tests is rrul, rrul_be, tcp_1up, tcp_1down, and
>> tcp_2up_delay.
>
>
> Done.
>
>> and rtt_fair (if you have more than one target server available)...
>> all without the iperf stuff....
>
>
> I do not have another machine readily available here at home.
>
>> 6) I am pretty interested as to what happens *without* sqm at the max
>> forwarding rate with fq_codel engaged on all these tests.
>
>
> This is included. Why did you want to do this test? Just to see if the
> WRT1200AC can do wirespeed forwarding with fq_codel on? Because with the
> setup I have, I don't see how there can be any buffering going on because
> it's a single gig port both in and out.

Well, it was good to see fq_codel actually engaging on a few of your
hardware queues. There are plenty of reasons why your egress might not
match your ingress periodically and vice versa.

To quote from the BQL commit: ( https://lwn.net/Articles/454378/ )

"Hardware queuing limits are typically specified in terms of a number
hardware descriptors, each of which has a variable size. The variability
of the size of individual queued items can have a very wide range. For
instance with the e1000 NIC the size could range from 64 bytes to 4K
(with TSO enabled). This variability makes it next to impossible to
choose a single queue limit that prevents starvation and provides lowest
possible latency.

The objective of byte queue limits is to set the limit to be the
minimum needed to prevent starvation between successive transmissions to
the hardware. The latency between two transmissions can be variable in a
system. It is dependent on interrupt frequency, NAPI polling latencies,
scheduling of the queuing discipline, lock contention, etc. Therefore we
propose that byte queue limits should be dynamic and change in
iaccordance with networking stack latencies a system encounters."

In particular, aggressive NAPI settings bug me, in routers. And I am
unfond of hardware multiqueue as presently implemented. Sometimes I
think it would be better to use them for QoS rather than fq with
birthday problems.

Another test idea for you would be to enable fq_codel on just the main
ethernet devices on the router, and not use hw mq on it at all. (tc
qdisc add dev each_ethernet_device_on_router root fq_codel)

2) You still have 15ms of delay at various rates. That is quite a lot
more than what I see on a rangeley (where we get below 5ms on sparse
traffic (partially due to local cpu scheduling delays), and 200*us* or
so when measured at the router with cake) I also see packet loss of
the measurement flows in most tests, which are possibly due to gro
forcing out smaller packets, or some other factor.

It is possible however, that the observed buffering here, is actually
on your host, server, or switch. (you have pfifo_fast on your host)
Try fq_codel on host and server (and/or sch_fq) and see what happens.
Disable tso/gro/gso on your server/host also. That leaves the switch
which I have no insight into. What switch chip is it? (see
/etc/config/network) - on the cerowrt project we got less buffering
out of the switch by enabling jumbo frames.

3) As for latency damage:

http://snapon.lab.bufferbloat.net/~d/gro_damage_to_latency.png # this
can also be somewhat due to ecn.

4) The diffserv marking behavior here was puzzling (losing BE
entirely) - my assumption is this was with also the simultaneous iperf
flows (280mbit on the uplink)?

http://snapon.lab.bufferbloat.net/~d/puzzling_diffserv_marking_behavior.png

5) Also on your hosts and servers, two sysctls seem to be helping
reduce the local bloat.

net.ipv4.tcp_limit_output_bytes = 8192 #  I have seen 4k
net.ipv4.tcp_notsent_lowat = 8192 # or 16k - I do not know the right
settings really for either of these

The default for the first was originally proposed to be 4k or so, but
that interacted badly with aggregating wifi drivers on single threaded
benchmarks. (sigh) More recently it was bumped to 256k to make the xen
people happier. (vms suck) On raw hardware, it seems like the lower
settings are pretty optimal in my range of tests....

The second is cheshire´s big change to osx and available in linux for
ages - It increases the number of context switches but keeps way more
traffic in the app, not the kernel. I think the latter can also be set
via a setsockopt.

Have fun exploring more variables! :)

6) I do sometimes find it hard to care about the last 15ms, given the
100s or 1000s of ms we are saving elsewhere.... and how rotten wifi
and wireless are... but I suppose in a world that is moving towards
2-4ms e2e latency along the ISP link on fiber, that 15ms is a lot. (I
am pretty sure everyone here is aware that my personal end-goal for
this work is to get to where I could play music with a drummer down
the street, and that that is about 2.7ms... and I have been waiting
for 20 years to get to be able to do that. I will probably be too old
and deaf and crippled by the time that happens to be able play
anything but chopsticks, and too poor to live near anyplace with
fiber... but...)

> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

Did you restart sqm after turning off offloads? maxpacket is
accumulative and wont change back otherwise.

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> Unfortunately the core piece of metadata I wanted from the router was
> the qdisc statistics. Didnt parse. Will file bug.

My guess is that in this case it's due to the openwrt box missing the tc
and ip binaries - those are not installed by default...

Or rather, sqm-scripts pulls in the tc binary, I think, but Flent uses
the ip binary to get the ip and route information to know from which
interfaces to pull the qdisc information from...

-Toke

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

On Thu, Jun 25, 2015 at 1:24 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> Dave Taht <dave.taht@gmail.com> writes:
>
>> Unfortunately the core piece of metadata I wanted from the router was
>> the qdisc statistics. Didnt parse. Will file bug.
>
> My guess is that in this case it's due to the openwrt box missing the tc
> and ip binaries - those are not installed by default...
>
> Or rather, sqm-scripts pulls in the tc binary, I think, but Flent uses
> the ip binary to get the ip and route information to know from which
> interfaces to pull the qdisc information from...
>
> -Toke

I keep wanting to have a way to have the metadata occupy a bottom
panel, and be pre-expanded, rather than the side. Metadata is becoming
increasingly useful...

In other news: Thank you for this commit! I still keep wanting a
rewrite in something faster than python, or to buy myself a faster pc
when I load up 100 datasets, or find some parallization
opportunities... but.... this was noticibly faster.

Author: Toke Høiland-Jørgensen <toke@toke.dk>
Date:   Tue Jun 9 13:04:58 2015 +0200

    Don't copy raw_values when creating ResultSet.

    ~3.5x speed-up of loading data files with raw values. Not bad for a
    two-line patch... :)



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 763 bytes --]

On Thu, 25 Jun 2015, Toke Høiland-Jørgensen wrote:

> Dave Taht <dave.taht@gmail.com> writes:
>
>> Unfortunately the core piece of metadata I wanted from the router was
>> the qdisc statistics. Didnt parse. Will file bug.
>
> My guess is that in this case it's due to the openwrt box missing the tc
> and ip binaries - those are not installed by default...

I did have tc, but I didn't have ip. Do you need "ip" or "ip-full"? I 
installed "ip".

> Or rather, sqm-scripts pulls in the tc binary, I think, but Flent uses 
> the ip binary to get the ip and route information to know from which 
> interfaces to pull the qdisc information from...

We'll see if my next test run includes the correct information then.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Thu, 25 Jun 2015, Dave Taht wrote:

> on your host, server, or switch. (you have pfifo_fast on your host) Try 
> fq_codel on host and server (and/or sch_fq) and see what happens. 
> Disable tso/gro/gso on your server/host also. That leaves the switch 
> which I have no insight into. What switch chip is it? (see 
> /etc/config/network) - on the cerowrt project we got less buffering out 
> of the switch by enabling jumbo frames.

So the complete physical setup is:

Linux laptop <cable> WRT1200AC <cable> TP-Link TL-SG3424 L2 switch <cable> <macbook pro thunderbolt port>

This means there are multiple places buffering can occur, that doesn't 
have fq_codel.

I can't tell what switch chip is being used, it's not on 
http://techinfodepot.info/wiki/Linksys_WRT1200AC and I don't know where to 
look to find it. In /etc/config/network there is just br-lan and eth0 and 
eth1.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

Hi Mikael,

thanks a lot.

On Jun 24, 2015, at 13:31 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Tue, 23 Jun 2015, Sebastian Moeller wrote:
> 
>> As Dave said it would be nice see RRUL data from the same testbed. It would be so nice if flint had a way to send different sized TCP packets… (I guess this might be faked with MSS clamping in the router and relaying on path MTU discovery?)
> 
> What kind of tests should I run? I have rrul results now, both at the same time as running iperf3.

	This is interesting. I also like the way iperf3 reports retransmits (BTW way how does it know this, I thought the kernel does the retransmits under the hood out of sight of applications). Since RRUL does not have such a nice report as iperf3 I will not be able to calculate the difference to the theoretical maximum transfer rates (also I lack insight on the ACK traffic). 

> I set iperf3 to run with 10 parallel streams, different MSS per test, and let it run for 30 seconds into the rrul test.

	The plots are interesting, even though I have a hard time with the logarithmic scale.


> So in all the tests, iperf3 session stops running half way into the rrul test.

	I like how the RRUL flows soak up the newly available bandwidth.

> 
> I set sqm to 500M up and down on eth0, ECN up and down, and not to squash DSCP in any direction.

	I will have a look at the flint files, but it looks like the 1200ac does 500Mbps bidirectionally with SQM, so this looks like a decent machine for the present and (near?) future. Now I just need to wait until prices come down a tad (I hope that the 1900AC v2 release should drop the 1200ac’s process a bit).

Thanks for the tests, now I know what router to try next (the edgerouterX, which I had eyed as a replacement for the shaper in the wndr3700 tops out at 130K packets per second and hence will not really work that well for a 100/40 Mbps link).

Best Regards
	Sebastian


> 
> http://swm.pp.se/aqm/rrul-wrt1200ac-150624.pdf
> 
> Tell me if there is more information I can provide or tests to run.
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Sebastian Moeller wrote:

> Thanks for the tests, now I know what router to try next (the 
> edgerouterX, which I had eyed as a replacement for the shaper in the 
> wndr3700 tops out at 130K packets per second and hence will not really 
> work that well for a 100/40 Mbps link).

I did some more tests and it seems with SQM at 500 megabit/s I start to 
lose packets at around 250-300k PPS. At these speeds, the rrul_be test is 
"only" 85k PPS at 500 megabit/s bidirectional with large packets.

Also, I have an Edgerouter ER-5, but as soon as it does CPU based 
forwarding it's really weak, easily under 100 megabit/s even with large 
packets. OpenVPN without encryption is less than 20 megabit/s.

Btw, the WRT1200AC is now becoming more widely available and it's 150 EUR 
incl 25% VAT and shipping here in Sweden now.

Btw, I tried WNDR3800 setting it to 100/100 SQM. It seems to max out 
around 25-30k PPS, but the difference is that when the CPU is full, it 
seems to delay/ECN-mark packets because there are no packets lost. When 
the WRT1200AC runs out of CPU it starts dropping packets. I always have 0 
packets lost with the WNDR3800 when doing iperf3 testing. I found this 
difference interesting, wonder where in the forwarding path the WRT1200AC 
loses packets?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

Hi Mikael,

On Jun 26, 2015, at 14:26 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Fri, 26 Jun 2015, Sebastian Moeller wrote:
> 
>> Thanks for the tests, now I know what router to try next (the edgerouterX, which I had eyed as a replacement for the shaper in the wndr3700 tops out at 130K packets per second and hence will not really work that well for a 100/40 Mbps link).
> 
> I did some more tests and it seems with SQM at 500 megabit/s I start to lose packets at around 250-300k PPS. At these speeds, the rrul_be test is "only" 85k PPS at 500 megabit/s bidirectional with large packets.

	Ah, that sounds like there 1200ac is a practical solution today; at 500Mbps there are 500^3/(64*8) = 244  Kpps at the smallest size (since the wire is at 1000 Mbps I think we should and can ignore inter frame gap and preamble, SQM certainly ignores them) so it looks like the router is really close to the theoretical maximum, and with larger packets things get way more relaxed (500^3/(1518*8) = 10 Kpps). Pretty decent for a router using no proprietary offload features.

> 
> Also, I have an Edgerouter ER-5, but as soon as it does CPU based forwarding it's really weak, easily under 100 megabit/s even with large packets. OpenVPN without encryption is less than 20 megabit/s.

	Ah, interesting, so neither the affordable edge router lite nor the (similar) ER-5 will cut it unless the offloads are used; I think the newer edgerouterX is rated for slightly higher speeds without offloads, but not nearly close to what your 1200ac does...

> 
> Btw, the WRT1200AC is now becoming more widely available and it's 150 EUR incl 25% VAT and shipping here in Sweden now.

	In Germany it still retails for around 200EUR with the cheapest offer at 180EUR (but that is an outlier); guess I should visit Sweden then ;)

> 
> Btw, I tried WNDR3800 setting it to 100/100 SQM.

	Yeah, not pretty, huh?

> It seems to max out around 25-30k PPS, but the difference is that when the CPU is full, it seems to delay/ECN-mark packets because there are no packets lost. When the WRT1200AC runs out of CPU it starts dropping packets. I always have 0 packets lost with the WNDR3800 when doing iperf3 testing. I found this difference interesting, wonder where in the forwarding path the WRT1200AC loses packets?

	Interesting observation, no idea, but intrigued ;)

Best Regards
	Sebastian

> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Mikael Abrahamsson wrote:

> Btw, I tried WNDR3800 setting it to 100/100 SQM. It seems to max out 
> around 25-30k PPS, but the difference is that when the CPU is full, it 
> seems to delay/ECN-mark packets because there are no packets lost. When 
> the WRT1200AC runs out of CPU it starts dropping packets. I always have 
> 0 packets lost with the WNDR3800 when doing iperf3 testing. I found this 
> difference interesting, wonder where in the forwarding path the 
> WRT1200AC loses packets?

I checked again, and my WDR4900 with same setup doesn't lose packets 
either. Even at 99% sirq, no packets are lost.

WRT1200AC starts to lose packets at 500 megabit/s SQM around MSS 300 and 
lower. If I turn off gso, tso and gro, I have to go to MSS 600 and above 
to avoid packet loss.

Does flent check for packet loss at all? Perhaps it's something to look 
into, because with ECN we really don't want to see any packets lost and 
this might be good to include test results for.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 200 bytes --]

Hypothesis: this might have to do with the receive path. Some devices might
have more capacity than others to buffer inbound packets until the CPU can
get around to servicing them.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 239 bytes --]

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Jonathan Morton wrote:

> Hypothesis: this might have to do with the receive path. Some devices might
> have more capacity than others to buffer inbound packets until the CPU can
> get around to servicing them.

Is there a way to tell? I am better at diagnosing Cisco CPU based routers 
than Linux ones. I looked in /sys/class/net/ethX/statistics but these 
drops are not recorded there.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 205 bytes --]

These would be hardware tail drops - there might not be a physical counter
recording them. But you could instrument three driver to see whether the
receive buffer is full when serviced.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 244 bytes --]

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

On Fri, Jun 26, 2015 at 9:35 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> These would be hardware tail drops - there might not be a physical counter
> recording them. But you could instrument three driver to see whether the
> receive buffer is full when serviced.

from drivers/net/ethernet/marvel/mvneta.c:

/* Max number of Rx descriptors */
#define MVNETA_MAX_RXD 128

this is probably too small, especially given the 64 it is willing to
wait for. At the same time, it is too large, as there are 8 hardware
queues in play here. So you get a huge burst from one flow, it gros it
all together.... aggghh...

/* Max number of Tx descriptors */
#define MVNETA_MAX_TXD 532

this realllllly needs BQL. Same problem(s). Only worse.


>
> - Jonathan Morton
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

Mikael, a simple test of the analysis I just did would be to use
ethtool to set your server to 100mbits (ethtool -s
your_ethernet_device advertise 0x008 and turn on fq_codel on both the
client and server.

if it is using classification for the hw mq, the rrul test from the
client will blow up half the queues. If it is doing hw fq instead,
the rrul_50_up test will blow up all 8 queues.

For giggles, try 0x002 (10mbit)

my own attempts to get a compile for this platform consistently blow
up here similarly for both musl and uclibc.

checking for arm-openwrt-linux-muslgnueabi-gcc...
/build/cero3/src/ac1900/build_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/gcc-linaro-4.8-2014.04-minimal/./gcc/xgcc
-B/build/cero3/src/ac1900/build_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/gcc-linaro-4.8-2014.04-minimal/./gcc/
-B/build/cero3/src/ac1900/staging_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/arm-openwrt-linux-muslgnueabi/bin/
-B/build/cero3/src/ac1900/staging_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/arm-openwrt-linux-muslgnueabi/lib/
-isystem /build/cero3/src/ac1900/staging_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/arm-openwrt-linux-muslgnueabi/include
-isystem /build/cero3/src/ac1900/staging_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/arm-openwrt-linux-muslgnueabi/sys-include
checking for suffix of object files... configure: error: in
`/build/cero3/src/ac1900/build_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/gcc-linaro-4.8-2014.04-minimal/arm-openwrt-linux-muslgnueabi/libgcc':
configure: error: cannot compute suffix of object files: cannot compile







On Fri, Jun 26, 2015 at 10:04 AM, Dave Taht <dave.taht@gmail.com> wrote:
> On Fri, Jun 26, 2015 at 9:35 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>> These would be hardware tail drops - there might not be a physical counter
>> recording them. But you could instrument three driver to see whether the
>> receive buffer is full when serviced.
>
> from drivers/net/ethernet/marvel/mvneta.c:
>
> /* Max number of Rx descriptors */
> #define MVNETA_MAX_RXD 128
>
> this is probably too small, especially given the 64 it is willing to
> wait for. At the same time, it is too large, as there are 8 hardware
> queues in play here. So you get a huge burst from one flow, it gros it
> all together.... aggghh...
>
> /* Max number of Tx descriptors */
> #define MVNETA_MAX_TXD 532
>
> this realllllly needs BQL. Same problem(s). Only worse.
>
>
>>
>> - Jonathan Morton
>>
>>
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>
>
>
> --
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Dave Taht wrote:

> Mikael, a simple test of the analysis I just did would be to use
> ethtool to set your server to 100mbits (ethtool -s
> your_ethernet_device advertise 0x008 and turn on fq_codel on both the
> client and server.

Hm what do you mean by "client" and "server"?

Where do you want the queueing to happen? Egress from the WRT1200AC 
towards the server? Then setting the WAN port of the WRT1200AC to 100 
megabit/s would work, yes.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

On Fri, Jun 26, 2015 at 11:38 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Fri, 26 Jun 2015, Dave Taht wrote:
>
>> Mikael, a simple test of the analysis I just did would be to use
>> ethtool to set your server to 100mbits (ethtool -s
>> your_ethernet_device advertise 0x008 and turn on fq_codel on both the
>> client and server.
>
>
> Hm what do you mean by "client" and "server"?
>
> Where do you want the queueing to happen? Egress from the WRT1200AC towards
> the server?

Yes.

> Then setting the WAN port of the WRT1200AC to 100 megabit/s
> would work, yes.

Yes, but I am unsure from looking at the driver that using ethtool on
the egress on the wrt1200ac will actually work, but
pretty sure it will work if you set it on the server. feel free to try both. :)

>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

On Fri, Jun 26, 2015 at 11:58 AM, Dave Taht <dave.taht@gmail.com> wrote:
> On Fri, Jun 26, 2015 at 11:38 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>> On Fri, 26 Jun 2015, Dave Taht wrote:
>>
>>> Mikael, a simple test of the analysis I just did would be to use
>>> ethtool to set your server to 100mbits (ethtool -s
>>> your_ethernet_device advertise 0x008 and turn on fq_codel on both the
>>> client and server.
>>
>>
>> Hm what do you mean by "client" and "server"?

your topology is:

client box - router - server

forcing the router - server link to 100mbit will push the egress
buffering into the router for the rrul_50_up test in particular.

>> Where do you want the queueing to happen? Egress from the WRT1200AC towards
>> the server?
>
> Yes.
>
>> Then setting the WAN port of the WRT1200AC to 100 megabit/s
>> would work, yes.
>
> Yes, but I am unsure from looking at the driver that using ethtool on
> the egress on the wrt1200ac will actually work, but
> pretty sure it will work if you set it on the server. feel free to try both. :)
>
>>
>> --
>> Mikael Abrahamsson    email: swmike@swm.pp.se
>
>
>
> --
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Dave Taht wrote:

> On Fri, Jun 26, 2015 at 11:58 AM, Dave Taht <dave.taht@gmail.com> wrote:
>> On Fri, Jun 26, 2015 at 11:38 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>> On Fri, 26 Jun 2015, Dave Taht wrote:
>>>
>>>> Mikael, a simple test of the analysis I just did would be to use
>>>> ethtool to set your server to 100mbits (ethtool -s
>>>> your_ethernet_device advertise 0x008 and turn on fq_codel on both the
>>>> client and server.
>>>
>>>
>>> Hm what do you mean by "client" and "server"?
>
> your topology is:
>
> client box - router - server
>
> forcing the router - server link to 100mbit will push the egress
> buffering into the router for the rrul_50_up test in particular.

No, my topology is <client - router - switch - server>, that is what made 
me confused.

So if I forced the eth0 router-switch link into 100M I will break the 
server->client direction (it'll be shallow buffer fifo) but at least 
client->server direction will exercise fq_codel on the router.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

On Fri, Jun 26, 2015 at 12:11 PM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Fri, 26 Jun 2015, Dave Taht wrote:
>
>> On Fri, Jun 26, 2015 at 11:58 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>> On Fri, Jun 26, 2015 at 11:38 AM, Mikael Abrahamsson <swmike@swm.pp.se>
>>> wrote:
>>>>
>>>> On Fri, 26 Jun 2015, Dave Taht wrote:
>>>>
>>>>> Mikael, a simple test of the analysis I just did would be to use
>>>>> ethtool to set your server to 100mbits (ethtool -s
>>>>> your_ethernet_device advertise 0x008 and turn on fq_codel on both the
>>>>> client and server.
>>>>
>>>>
>>>>
>>>> Hm what do you mean by "client" and "server"?
>>
>>
>> your topology is:
>>
>> client box - router - server
>>
>> forcing the router - server link to 100mbit will push the egress
>> buffering into the router for the rrul_50_up test in particular.
>
>
> No, my topology is <client - router - switch - server>, that is what made me
> confused.

> So if I forced the eth0 router-switch link into 100M I will break the
> server->client direction (it'll be shallow buffer fifo) but at least
> client->server direction will exercise fq_codel on the router.

well, maybe the driver will take ethtool on the mvneta. try it. :)

>
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Dave Taht wrote:

> Yes, but I am unsure from looking at the driver that using ethtool on 
> the egress on the wrt1200ac will actually work, but pretty sure it will 
> work if you set it on the server. feel free to try both. :)

I set speed 100 on my switch and did some new tests, I left SQM at 500M, 
don't know what happens then, thought it was worthwile to test? Remember, 
now I don't have fq_codel in the server->client direction, but there it's 
an low buffered switchport that is doing the rate adaptation.

Btw, now I am running a nightly build that I compiled for myself for the 
WRT1200AC, so if there is anything you would like me to try to change in 
the mvneta driver, I can certainly test that. I can do serial console 
access to the WRT1200AC if needed as well, I already "unbricked" it once.

Btw, disregard the title in the flent files, I just realised I forgot to 
change the title argument. This is without iperf3, with SQM set to 500M, 
but eth0 at 100megabit/full duplex.

http://swm.pp.se/aqm/wrt1200ac-150627-100m-sqm.tar

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

your results are showing basically tail drop behavior. Although I
would have expected intrinsic delay on the link to crack 100mbits on
the rrul test, not 20ms (which is still high), and you only hit 7 on
the single threaded tcp up test, based on what I saw in the driver.

turn off sqm, stay at 100mbit, let fq_codel ride, try the rrul_50_up
test. Also do a capture to see if CE was ever exerted.

I do not know why my linksys ac1200 build does not work. I generally
suspect it is because my big build server is getting ancient. Can you
send me your .config file for openwrt?

It would be VERY helpful if you pulled from ceropackages-3.14, and
added and built kmod-sched-cake and tc-adv and switched to using the
ceropackages feed also for luci-app-sqm and sqm-scripts. There are a
couple people here that would probably leap on that! :)

add to your feeds.conf

src-git https://github.com/dtaht/ceropackages-3.10.git

./scripts feeds update
./scripts feeds install kmod-sched-cake tc-adv kmod-sched-fq_pie
edit the .config file to make them modules or installed by default
make menuconfig then save
make

I went through hell trying to to figure out how to switch over to the
cero versions of these. If yer doing the builds, I could try hacking
the BQL. Maybe. In sweden.



On Fri, Jun 26, 2015 at 10:03 PM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Fri, 26 Jun 2015, Dave Taht wrote:
>
>> Yes, but I am unsure from looking at the driver that using ethtool on the
>> egress on the wrt1200ac will actually work, but pretty sure it will work if
>> you set it on the server. feel free to try both. :)
>
>
> I set speed 100 on my switch and did some new tests, I left SQM at 500M,
> don't know what happens then, thought it was worthwile to test? Remember,
> now I don't have fq_codel in the server->client direction, but there it's an
> low buffered switchport that is doing the rate adaptation.
>
> Btw, now I am running a nightly build that I compiled for myself for the
> WRT1200AC, so if there is anything you would like me to try to change in the
> mvneta driver, I can certainly test that. I can do serial console access to
> the WRT1200AC if needed as well, I already "unbricked" it once.
>
> Btw, disregard the title in the flent files, I just realised I forgot to
> change the title argument. This is without iperf3, with SQM set to 500M, but
> eth0 at 100megabit/full duplex.
>
> http://swm.pp.se/aqm/wrt1200ac-150627-100m-sqm.tar
>
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Dave Taht wrote:

> your results are showing basically tail drop behavior. Although I
> would have expected intrinsic delay on the link to crack 100mbits on
> the rrul test, not 20ms (which is still high), and you only hit 7 on
> the single threaded tcp up test, based on what I saw in the driver.
>
> turn off sqm, stay at 100mbit, let fq_codel ride, try the rrul_50_up
> test. Also do a capture to see if CE was ever exerted.

http://swm.pp.se/aqm/rrul_50_up-2015-06-27T072819.166452.iperf3_150626_9_100M.flent.gz

I have a capture of this as well, it's 1.5gigs. It has lots of mentions of 
ECT(0) but 0 mention of ECT(1). TOS should be 0x3 when the EC=1 ? I see no 
such packets. All the packets are 0x2.

http://swm.pp.se/aqm/rrul_50up-100M.pcap.bz2

I went back to gig and set SQM to 50 megabit/s bidirectional and looking 
at tc qdisc -s eth0 I see ecn_mark 10339 on a single queue, none of the 
others have non-zero ecn_mark.

> I do not know why my linksys ac1200 build does not work. I generally
> suspect it is because my big build server is getting ancient. Can you
> send me your .config file for openwrt?

http://swm.pp.se/aqm/wrt1200ac.config

This was my first try, I haven't fine tuned it yet, but it works (boots 
and moves packets anyway :P )

> It would be VERY helpful if you pulled from ceropackages-3.14, and
> added and built kmod-sched-cake and tc-adv and switched to using the
> ceropackages feed also for luci-app-sqm and sqm-scripts. There are a
> couple people here that would probably leap on that! :)

I will give it a go in the next few days.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

On Fri, Jun 26, 2015 at 10:50 PM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Fri, 26 Jun 2015, Dave Taht wrote:
>
>> your results are showing basically tail drop behavior. Although I
>> would have expected intrinsic delay on the link to crack 100mbits on
>> the rrul test, not 20ms (which is still high), and you only hit 7 on
>> the single threaded tcp up test, based on what I saw in the driver.
>>
>> turn off sqm, stay at 100mbit, let fq_codel ride, try the rrul_50_up
>> test. Also do a capture to see if CE was ever exerted.
>
>
> http://swm.pp.se/aqm/rrul_50_up-2015-06-27T072819.166452.iperf3_150626_9_100M.flent.gz
>
> I have a capture of this as well, it's 1.5gigs. It has lots of mentions of
> ECT(0) but 0 mention of ECT(1). TOS should be 0x3 when the EC=1 ? I see no
> such packets. All the packets are 0x2.
>
> http://swm.pp.se/aqm/rrul_50up-100M.pcap.bz2

Maybe you are dropping at the internal switch? A lot of manufacturers
ran everything through the switch, even the uplink, in recent years.
This will make things hard to fix.

> I went back to gig and set SQM to 50 megabit/s bidirectional and looking at
> tc qdisc -s eth0 I see ecn_mark 10339 on a single queue, none of the others
> have non-zero ecn_mark.

Yea, well, you need to use a diffserv enabled test to see marks or
drops in other queues.

Sort of my hope is that cake can run at 940mbit with software rate
limiting. But we probably need to find ways of parallizing the needed
softirq.

>
>> I do not know why my linksys ac1200 build does not work. I generally
>> suspect it is because my big build server is getting ancient. Can you
>> send me your .config file for openwrt?
>
>
> http://swm.pp.se/aqm/wrt1200ac.config
>
> This was my first try, I haven't fine tuned it yet, but it works (boots and
> moves packets anyway :P )

Must be nice to win on the first try.

>
>> It would be VERY helpful if you pulled from ceropackages-3.14, and
>> added and built kmod-sched-cake and tc-adv and switched to using the
>> ceropackages feed also for luci-app-sqm and sqm-scripts. There are a
>> couple people here that would probably leap on that! :)
>
>
> I will give it a go in the next few days.
>
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Sat, 27 Jun 2015, Dave Taht wrote:

> Maybe you are dropping at the internal switch? A lot of manufacturers
> ran everything through the switch, even the uplink, in recent years.
> This will make things hard to fix.

Looking at the vlans and pvids it looks like they've connected the SoC to 
port 5 and 6 on the switch, port 4 is the "Internet" port, and 0-3 is 
LAN1-4 port. Sigh, I had hoped at least eth0 (WAN/Internet) was a 
physically dedicated port.

root@OpenWrt:~# swconfig dev switch0 show
Global attributes:
         enable_vlan: 0
Port 0:
         mask: 0x004e: (0) 1 2 3 6
         qmode: 0
         status: link: up, speed: 1000 Mbps, duplex: full
         link: 1000
         pvid: 0
Port 1:
         mask: 0x004d: 0 (1) 2 3 6
         qmode: 0
         status: link: down
         link: 0
         pvid: 0
Port 2:
         mask: 0x004b: 0 1 (2) 3 6
         qmode: 0
         status: link: down
         link: 0
         pvid: 0
Port 3:
         mask: 0x0047: 0 1 2 (3) 6
         qmode: 0
         status: link: up, speed: 1000 Mbps, duplex: full
         link: 1000
         pvid: 0
Port 4:
         mask: 0x0020: (4) 5
         qmode: 0
         status: link: up, speed: 1000 Mbps, duplex: full
         link: 1000
         pvid: 0
Port 5:
         mask: 0x0010: 4 (5)
         qmode: 0
         status: link: up, speed: 1000 Mbps, duplex: full
         link: 1000
         pvid: 0
Port 6:
         mask: 0x000f: 0 1 2 3 (6)
         qmode: 0
         status: link: up, speed: 1000 Mbps, duplex: full
         link: 1000
         pvid: 0

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

Sigh.

It is possible, maybe, to build something using this chipset that does
not connect the wan port through the switch, built by someone else.
Maybe some other manufacturer did that. I had first evaluated this
chipset on the dual port mirabox, which as best as I recall had no
switch, two genuine ethernet ports (but it ran WAY too hot and at the
time, the kernel was ancient).

On Sat, Jun 27, 2015 at 11:23 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Sat, 27 Jun 2015, Dave Taht wrote:
>
>> Maybe you are dropping at the internal switch? A lot of manufacturers
>> ran everything through the switch, even the uplink, in recent years.
>> This will make things hard to fix.
>
>
> Looking at the vlans and pvids it looks like they've connected the SoC to
> port 5 and 6 on the switch, port 4 is the "Internet" port, and 0-3 is LAN1-4
> port. Sigh, I had hoped at least eth0 (WAN/Internet) was a physically
> dedicated port.
>
> root@OpenWrt:~# swconfig dev switch0 show
> Global attributes:
>         enable_vlan: 0
> Port 0:
>         mask: 0x004e: (0) 1 2 3 6
>         qmode: 0
>         status: link: up, speed: 1000 Mbps, duplex: full
>         link: 1000
>         pvid: 0
> Port 1:
>         mask: 0x004d: 0 (1) 2 3 6
>         qmode: 0
>         status: link: down
>         link: 0
>         pvid: 0
> Port 2:
>         mask: 0x004b: 0 1 (2) 3 6
>         qmode: 0
>         status: link: down
>         link: 0
>         pvid: 0
> Port 3:
>         mask: 0x0047: 0 1 2 (3) 6
>         qmode: 0
>         status: link: up, speed: 1000 Mbps, duplex: full
>         link: 1000
>         pvid: 0
> Port 4:
>         mask: 0x0020: (4) 5
>         qmode: 0
>         status: link: up, speed: 1000 Mbps, duplex: full
>         link: 1000
>         pvid: 0
> Port 5:
>         mask: 0x0010: 4 (5)
>         qmode: 0
>         status: link: up, speed: 1000 Mbps, duplex: full
>         link: 1000
>         pvid: 0
> Port 6:
>         mask: 0x000f: 0 1 2 3 (6)
>         qmode: 0
>         status: link: up, speed: 1000 Mbps, duplex: full
>         link: 1000
>         pvid: 0
>
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

Hi Dave, hi Mikael,

On Jun 27, 2015, at 19:59 , Dave Taht <dave.taht@gmail.com> wrote:

>> [...]
> 
> Yea, well, you need to use a diffserv enabled test to see marks or
> drops in other queues.
> 
> Sort of my hope is that cake can run at 940mbit with software rate
> limiting. But we probably need to find ways of parallizing the needed
> softirq.
[…]

	If we go that route we need to remember adding the 20 bytes worth of inter frame gap and preamble to the per packet overhead (in addition to the 18 bytes of “normal” ethernet header with frame check sequence), otherwise the shaper will not do what we expect it to at small packet sizes.

Best Regards
	Sebastian

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Dave Taht wrote:

> src-git https://github.com/dtaht/ceropackages-3.10.git
>
> ./scripts feeds update
> ./scripts feeds install kmod-sched-cake tc-adv kmod-sched-fq_pie
> edit the .config file to make them modules or installed by default
> make menuconfig then save
> make

I have now done this. I have sch_cake, and I have a tc I can add an qdisc 
"cake" with to for instance eth1 and see stats from it with "tc -s qdisc"

I however do not have anything "cake" in luci-app-sqm. I see luci-app-sqm 
in my "cero" feed, but it seems to install luci-app-sqm from regular 
OpenWrt instead.

So either if someone has any tips on how to fix this, or can just give me 
how to configure qdiscs manually, I will be able to do cake testing on the 
WRT1200AC.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]

Hi Mikael,


On Jun 28, 2015, at 09:06 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Fri, 26 Jun 2015, Dave Taht wrote:
> 
>> src-git https://github.com/dtaht/ceropackages-3.10.git
>> 
>> ./scripts feeds update
>> ./scripts feeds install kmod-sched-cake tc-adv kmod-sched-fq_pie
>> edit the .config file to make them modules or installed by default
>> make menuconfig then save
>> make
> 
> I have now done this. I have sch_cake, and I have a tc I can add an qdisc "cake" with to for instance eth1 and see stats from it with "tc -s qdisc"
> 
> I however do not have anything "cake" in luci-app-sqm. I see luci-app-sqm in my "cero" feed, but it seems to install luci-app-sqm from regular OpenWrt instead.

	Ah, I see the latest changes have not yet made it into the openwrt repository (due to lack of testing). As I do not have permissions/authority to push changes into the openwrt repository, there is nothing I can do to expedite the process. You could replace /usr/lib/lua/luci/model/cbi/sqm.lua on your router with the following (attached file)

[-- Attachment #2: sqm.lua --]
[-- Type: application/octet-stream, Size: 9571 bytes --]

--[[
LuCI - Lua Configuration Interface

Copyright 2014 Steven Barth <steven@midlink.org>
Copyright 2014 Dave Taht <dave.taht@bufferbloat.net>

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

$Id$
]]--

local wa = require "luci.tools.webadmin"
local fs = require "nixio.fs"
local net = require "luci.model.network".init()
local sys = require "luci.sys"
--local ifaces = net:get_interfaces()
local ifaces = sys.net:devices()
local path = "/usr/lib/sqm"

m = Map("sqm", translate("Smart Queue Management"),
	translate("With <abbr title=\"Smart Queue Management\">SQM</abbr> you " ..
		"can enable traffic shaping, better mixing (Fair Queueing)," ..
		" active queue length management (AQM) " ..
		" and prioritisation on one " ..
		"network interface."))

s = m:section(TypedSection, "queue", translate("Queues"))
s:tab("tab_basic", translate("Basic Settings"))
s:tab("tab_qdisc", translate("Queue Discipline"))
s:tab("tab_linklayer", translate("Link Layer Adaptation"))
s.addremove = true -- set to true to allow adding SQM instances in the GUI
s.anonymous = true

-- BASIC
e = s:taboption("tab_basic", Flag, "enabled", translate("Enable this SQM instance."))
e.rmempty = false

-- sm: following jow's advise, be helpful to the user and enable
--     sqm's init script if even a single sm instance/interface
--     is enabled; this is unexpected in that the init script gets
--     enabled as soon as at least one sqm instance is enabled
--     and that state is saved, so it does not require "Save & Apply"
--     to effect the init scripts.
--     the implementation was inpired/lifted from 
--     https://github.com/openwrt/luci/blob/master/applications/luci-app-minidlna/luasrc/model/cbi/minidlna.lua
function e.write(self, section, value)
    if value == "1" then
	luci.sys.init.enable("sqm")
	m.message = translate("The SQM GUI has just enabled the sqm initscript on your behalf. Remember to disable the sqm initscript manually under System Startup menu in case this change was not wished for.")
--	luci.sys.call("/etc/init.d/sqm start >/dev/null")
--    else
--	luci.sys.call("/etc/init.d/sqm stop >/dev/null")
--	luci.sys.init.disable("sqm")
    end
    return Flag.write(self, section, value)
end
-- TODO: inform the user what we just did...

n = s:taboption("tab_basic", ListValue, "interface", translate("Interface name"))
-- sm lifted from luci-app-wol, the original implementation failed to show pppoe-ge00 type interface names
for _, iface in ipairs(ifaces) do
--     if iface:is_up() then
--	n:value(iface:name())
--     end
	if iface ~= "lo" then 
		n:value(iface) 
	end
end
n.rmempty = false


dl = s:taboption("tab_basic", Value, "download", translate("Download speed (kbit/s) (ingress) set to 0 to selectively disable ingress shaping:"))
dl.datatype = "and(uinteger,min(0))"
dl.rmempty = false

ul = s:taboption("tab_basic", Value, "upload", translate("Upload speed (kbit/s) (egress) set to 0 to selectively disable egress shaping:"))
ul.datatype = "and(uinteger,min(0))"
ul.rmempty = false

-- QDISC

c = s:taboption("tab_qdisc", ListValue, "qdisc", translate("Queueing discipline"))
c:value("fq_codel", "fq_codel ("..translate("default")..")")
c:value("efq_codel")
c:value("nfq_codel")
c:value("sfq")
c:value("codel")
c:value("ns2_codel")
c:value("pie")
c:value("sfq")
c:value("cake")
c.default = "fq_codel"
c.rmempty = false

local qos_desc = ""
sc = s:taboption("tab_qdisc", ListValue, "script", translate("Queue setup script"))
for file in fs.dir(path) do
  if string.find(file, ".qos$") then
    sc:value(file)
  end
  if string.find(file, ".qos.help$") then
    fh = io.open(path .. "/" .. file, "r")
    qos_desc = qos_desc .. "<p><b>" .. file:gsub(".help$", "") .. ":</b><br />" .. fh:read("*a") .. "</p>"
  end
end
sc.default = "simple.qos"
sc.rmempty = false
sc.description = qos_desc

ad = s:taboption("tab_qdisc", Flag, "qdisc_advanced", translate("Show and Use Advanced Configuration. Advanced options will only be used as long as this box is checked."))
ad.default = false
ad.rmempty = true

squash_dscp  = s:taboption("tab_qdisc", ListValue, "squash_dscp", translate("Squash DSCP on inbound packets (ingress):"))
squash_dscp:value("1", "SQUASH")
squash_dscp:value("0", "DO NOT SQUASH")
squash_dscp.default = "1"
squash_dscp.rmempty = true
squash_dscp:depends("qdisc_advanced", "1")

squash_ingress = s:taboption("tab_qdisc", ListValue, "squash_ingress", translate("Ignore DSCP on ingress:"))
squash_ingress:value("1", "Ignore")
squash_ingress:value("0", "Allow")
squash_ingress.default = "1"
squash_ingress.rmempty = true
squash_ingress:depends("qdisc_advanced", "1")

iecn = s:taboption("tab_qdisc", ListValue, "ingress_ecn", translate("Explicit congestion notification (ECN) status on inbound packets (ingress):"))
iecn:value("ECN", "ECN ("..translate("default")..")")
iecn:value("NOECN")
iecn.default = "ECN"
iecn.rmempty = true
iecn:depends("qdisc_advanced", "1")

eecn = s:taboption("tab_qdisc", ListValue, "egress_ecn", translate("Explicit congestion notification (ECN) status on outbound packets (egress)."))
eecn:value("NOECN", "NOECN ("..translate("default")..")")
eecn:value("ECN")
eecn.default = "NOECN"
eecn.rmempty = true
eecn:depends("qdisc_advanced", "1")

ad2 = s:taboption("tab_qdisc", Flag, "qdisc_really_really_advanced", translate("Show and Use Dangerous Configuration. Dangerous options will only be used as long as this box is checked."))
ad2.default = false
ad2.rmempty = true
ad2:depends("qdisc_advanced", "1")

ilim = s:taboption("tab_qdisc", Value, "ilimit", translate("Hard limit on ingress queues; leave empty for default."))
-- ilim.default = 1000
ilim.isnumber = true
ilim.datatype = "and(uinteger,min(0))"
ilim.rmempty = true
ilim:depends("qdisc_really_really_advanced", "1")

elim = s:taboption("tab_qdisc", Value, "elimit", translate("Hard limit on egress queues; leave empty for default."))
-- elim.default = 1000
elim.datatype = "and(uinteger,min(0))"
elim.rmempty = true
elim:depends("qdisc_really_really_advanced", "1")


itarg = s:taboption("tab_qdisc", Value, "itarget", translate("Latency target for ingress, e.g 5ms [units: s, ms, or  us]; leave empty for automatic selection, put in the word default for the qdisc's default."))
itarg.datatype = "string"
itarg.rmempty = true
itarg:depends("qdisc_really_really_advanced", "1")

etarg = s:taboption("tab_qdisc", Value, "etarget", translate("Latency target for egress, e.g. 5ms [units: s, ms, or  us]; leave empty for automatic selection, put in the word default for the qdisc's default."))
etarg.datatype = "string"
etarg.rmempty = true
etarg:depends("qdisc_really_really_advanced", "1")



iqdisc_opts = s:taboption("tab_qdisc", Value, "iqdisc_opts", translate("Advanced option string to pass to the ingress queueing disciplines; no error checking, use very carefully."))
iqdisc_opts.rmempty = true
iqdisc_opts:depends("qdisc_really_really_advanced", "1")

eqdisc_opts = s:taboption("tab_qdisc", Value, "eqdisc_opts", translate("Advanced option string to pass to the egress queueing disciplines; no error checking, use very carefully."))
eqdisc_opts.rmempty = true
eqdisc_opts:depends("qdisc_really_really_advanced", "1")

-- LINKLAYER
ll = s:taboption("tab_linklayer", ListValue, "linklayer", translate("Which link layer to account for:"))
ll:value("none", "none ("..translate("default")..")")
ll:value("ethernet", "Ethernet with overhead: select for e.g. VDSL2.")
ll:value("atm", "ATM: select for e.g. ADSL1, ADSL2, ADSL2+.")
-- ll:value("adsl")	-- reduce the options
ll.default = "none"

po = s:taboption("tab_linklayer", Value, "overhead", translate("Per Packet Overhead (byte):"))
po.datatype = "and(integer,min(-1500))"
po.default = 0
po.isnumber = true
po.rmempty = true
po:depends("linklayer", "ethernet")
-- po:depends("linklayer", "adsl")
po:depends("linklayer", "atm")


adll = s:taboption("tab_linklayer", Flag, "linklayer_advanced", translate("Show Advanced Linklayer Options, (only needed if MTU > 1500). Advanced options will only be used as long as this box is checked."))
adll.rmempty = true
adll:depends("linklayer", "ethernet")
-- adll:depends("linklayer", "adsl")
adll:depends("linklayer", "atm")

smtu = s:taboption("tab_linklayer", Value, "tcMTU", translate("Maximal Size for size and rate calculations, tcMTU (byte); needs to be >= interface MTU + overhead:"))
smtu.datatype = "and(uinteger,min(0))"
smtu.default = 2047
smtu.isnumber = true
smtu.rmempty = true
smtu:depends("linklayer_advanced", "1")

stsize = s:taboption("tab_linklayer", Value, "tcTSIZE", translate("Number of entries in size/rate tables, TSIZE; for ATM choose TSIZE = (tcMTU + 1) / 16:"))
stsize.datatype = "and(uinteger,min(0))"
stsize.default = 128
stsize.isnumber = true
stsize.rmempty = true
stsize:depends("linklayer_advanced", "1")

smpu = s:taboption("tab_linklayer", Value, "tcMPU", translate("Minimal packet size, MPU (byte); needs to be > 0 for ethernet size tables:"))
smpu.datatype = "and(uinteger,min(0))"
smpu.default = 0
smpu.isnumber = true
smpu.rmempty = true
smpu:depends("linklayer_advanced", "1")

lla = s:taboption("tab_linklayer", ListValue, "linklayer_adaptation_mechanism", translate("Which linklayer adaptation mechanism to use; for testing only"))
lla:value("cake")
lla:value("htb_private")
lla:value("tc_stab", "tc_stab ("..translate("default")..")")
lla.default = "tc_stab"
lla.rmempty = true
lla:depends("linklayer_advanced", "1")

-- PRORITIES?

return m

[-- Attachment #3: Type: text/plain, Size: 1288 bytes --]

that should do the trick and get you to the most recent version that needs testers badly. The minimal change required is to add c:value(“cake”) to the following section of sqm.lua:

c = s:taboption("tab_qdisc", ListValue, "qdisc", translate("Queueing discipline"))
c:value("fq_codel", "fq_codel ("..translate("default")..")")
c:value("efq_codel")
c:value("nfq_codel")
c:value("sfq")
c:value("codel")
c:value("ns2_codel")
c:value("pie")
c:value("sfq”)
c:value(“cake")
c.default = "fq_codel"
c.rmempty = false

@Dave, which of these do we want to keep carrying and which can we safely retire? Especially in the openwrt context most of the experimental ones are not available anyways. (If anybody has a way to make tc or the kernel enumerate the qdiscs that are either compiled in or available as modules that would be much appreciated.)

Best Regards
	Sebastian


> 
> So either if someone has any tips on how to fix this, or can just give me how to configure qdiscs manually, I will be able to do cake testing on the WRT1200AC.
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Sun, 28 Jun 2015, Sebastian Moeller wrote:

> 	Ah, I see the latest changes have not yet made it into the openwrt 
> repository (due to lack of testing). As I do not have 
> permissions/authority to push changes into the openwrt repository, there 
> is nothing I can do to expedite the process. You could replace 
> /usr/lib/lua/luci/model/cbi/sqm.lua on your router with the following 
> (attached file)

Ok, I have done that, run a battery of tests both at 50/50 and 500/500 
with cake.

http://swm.pp.se/aqm/wrt1200ac-flent-cake-150628-1.tar

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

Hi Mikael,

On Jun 28, 2015, at 12:29 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Sun, 28 Jun 2015, Sebastian Moeller wrote:
> 
>> 	Ah, I see the latest changes have not yet made it into the openwrt repository (due to lack of testing). As I do not have permissions/authority to push changes into the openwrt repository, there is nothing I can do to expedite the process. You could replace /usr/lib/lua/luci/model/cbi/sqm.lua on your router with the following (attached file)
> 
> Ok, I have done that, run a battery of tests both at 50/50 and 500/500 with cake.
> 
> http://swm.pp.se/aqm/wrt1200ac-flent-cake-150628-1.tar

	This looks great, could you by any chance confirm that the GUI does allow to configure cake and that you can or can not set the overhead for cake in the link layer adjustments (LLA) tab? (select cake as link layer adjustment method, and put 42 into the overhead field and report the output of “tc -d qdisc” before and after selecting cake as LLA). @Toke: If that works, I think we can safely push these changes into the openwrt repositories...


Best Regards
	Sebastian

> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4028 bytes --]

On Sun, 28 Jun 2015, Sebastian Moeller wrote:

> 	This looks great, could you by any chance confirm that the GUI 
> does allow to configure cake and that you can or can not set the 
> overhead for cake in the link layer adjustments (LLA) tab? (select cake 
> as link layer adjustment method, and put 42 into the overhead field and 
> report the output of “tc -d qdisc” before and after selecting cake as 
> LLA). @Toke: If that works, I think we can safely push these changes 
> into the openwrt repositories...

Here is the output. What I don't see is both ingress and egress ECN 
markings even though I have selected this in the advanced configuration 
under Queue Discipline.

Before changing LLA:

root@OpenWrt:~# tc -d qdisc
qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat 
0 ver 3.17 direct_qlen 532
qdisc cake 110: dev eth0 parent 1:11 unlimited diffserv4 flows raw
qdisc cake 120: dev eth0 parent 1:12 unlimited diffserv4 flows raw
qdisc cake 130: dev eth0 parent 1:13 unlimited diffserv4 flows raw
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12 
direct_packets_stat 0 ver 3.17 direct_qlen 32
qdisc cake 110: dev ifb4eth0 parent 1:11 unlimited diffserv4 flows raw
qdisc cake 120: dev ifb4eth0 parent 1:12 unlimited diffserv4 flows raw
qdisc cake 130: dev ifb4eth0 parent 1:13 unlimited diffserv4 flows raw

After changing LLA:

root@OpenWrt:~# tc -d qdisc
qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat 
0 ver 3.17 direct_qlen 532
  linklayer ethernet overhead 42
qdisc cake 110: dev eth0 parent 1:11 unlimited diffserv4 flows raw
qdisc cake 120: dev eth0 parent 1:12 unlimited diffserv4 flows raw
qdisc cake 130: dev eth0 parent 1:13 unlimited diffserv4 flows raw
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12 
direct_packets_stat 0 ver 3.17 direct_qlen 32
  linklayer ethernet overhead 42
qdisc cake 110: dev ifb4eth0 parent 1:11 unlimited diffserv4 flows raw
qdisc cake 120: dev ifb4eth0 parent 1:12 unlimited diffserv4 flows raw
qdisc cake 130: dev ifb4eth0 parent 1:13 unlimited diffserv4 flows raw

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 86 bytes --]

To be honest, HTB + cake isn't really the preferred configuration.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 129 bytes --]

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

htb + cake is the wrong configuration. :)

cake has an integral bandwidth limiter and diffserv, there should be
no htb here and it looked like their was. I have only been using the
"simplest" qdisc, possibly the script for the simple qdisc is wrong?

On Sun, Jun 28, 2015 at 10:58 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> To be honest, HTB + cake isn't really the preferred configuration.
>
> - Jonathan Morton
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

Hi Dave, hi List,

On Jun 28, 2015, at 20:04 , Dave Taht <dave.taht@gmail.com> wrote:

> htb + cake is the wrong configuration. :)

	“Wrong” might be a bit hard, but certainly not the preferred solution. During my tests on your box simple.qos resulted in the following:
root@davedesk2:/usr/lib/sqm# tc -d qdisc
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn 
qdisc cake 8030: dev eth1 root refcnt 2 bandwidth 4500Kbit diffserv4 flows raw 
qdisc ingress ffff: dev eth1 parent ffff:fff1 ---------------- 
qdisc cake 8031: dev ifb4eth1 root refcnt 2 bandwidth 45Mbit besteffort flows atm overhead 0 
qdisc bfifo 800b: dev wlan1 root refcnt 5 limit 16Kb
qdisc cake 8008: dev wlan0 root refcnt 5 unlimited diffserv4 flows raw 

So either I mixed things up later or Mikael’s simple.qos somehow got stale. Anyway, there is work thee for me to do to fix this up properly.

Best Regards
	Sebastian



>  
> cake has an integral bandwidth limiter and diffserv, there should be
> no htb here and it looked like their was. I have only been using the
> "simplest" qdisc, possibly the script for the simple qdisc is wrong?
> 
> On Sun, Jun 28, 2015 at 10:58 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>> To be honest, HTB + cake isn't really the preferred configuration.
>> 
>> - Jonathan Morton
>> 
>> 
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
> 
> 
> 
> -- 
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 407 bytes --]

On Sun, 28 Jun 2015, Sebastian Moeller wrote:

> So either I mixed things up later or Mikael’s simple.qos somehow got 
> stale. Anyway, there is work thee for me to do to fix this up properly.

I believe this is simple.qos from whatever nightly OpenWRT build there is. 
It's quite likely it didn't pull any of the sqm-scripts from the CeroWrt 
feed at all.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 14 bytes --]

Hi Mikael,




[-- Attachment #2: sqm-scripts.zip --]
[-- Type: application/zip, Size: 28330 bytes --]

[-- Attachment #3: Type: text/plain, Size: 1214 bytes --]


On Jun 28, 2015, at 21:17 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Sun, 28 Jun 2015, Sebastian Moeller wrote:
> 
>> So either I mixed things up later or Mikael’s simple.qos somehow got stale. Anyway, there is work thee for me to do to fix this up properly.
> 
> I believe this is simple.qos from whatever nightly OpenWRT build there is. It's quite likely it didn't pull any of the sqm-scripts from the CeroWrt feed at all.


attached is the most recent version of sqm-scripts, just drop simple.qos, simples.qos and functions.sh from that into your router’s /usr/lib/sqm and you should be running to most recent version then. I would love to see the output of:
1) tc - d qdisc 					# before and after selecting cake in the GUI and choosing cake as the link layer adjustment mechanism filed (gated by the advanced options checkbox)
2) tc -d class show dev eth0
3) tc -d class show dev ifb4eth0


Maybe I screwed up the release numbers, maybe the older states have higher numbers and hence the changes do not appear in the openwrt repository (but not enough information, so conjecture on my part)

Best Regards
	Sebastian
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Sun, 28 Jun 2015, Sebastian Moeller wrote:

> Hi Mikael,

root@OpenWrt:~# tc -d qdisc
qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat 
0 ver 3.17 direct_qlen 532
qdisc fq_codel 110: dev eth0 parent 1:11 limit 1001p flows 1024 quantum 
300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 120: dev eth0 parent 1:12 limit 1001p flows 1024 quantum 
300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 130: dev eth0 parent 1:13 limit 1001p flows 1024 quantum 
300 target 5.0ms interval 100.0ms ecn
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12 
direct_packets_stat 0 ver 3.17 direct_qlen 32
qdisc fq_codel 110: dev ifb4eth0 parent 1:11 limit 1001p flows 1024 
quantum 500 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 120: dev ifb4eth0 parent 1:12 limit 1001p flows 1024 
quantum 1500 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 130: dev ifb4eth0 parent 1:13 limit 1001p flows 1024 
quantum 300 target 5.0ms interval 100.0ms ecn

then I change to cake and add 42 as overhead:

root@OpenWrt:~# tc -d qdisc
qdisc cake 8003: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows 
raw
  linklayer ethernet overhead 42
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 
quantum 300 target 5.0ms interval 100.0ms ecn

root@OpenWrt:~# tc -d class show dev eth0
class cake 8003:508 parent 8003:
class cake 8003:944 parent 8003:
root@OpenWrt:~# tc -d class show dev ifb4eth0
root@OpenWrt:~#

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

Hi Mikael,

On Jun 28, 2015, at 22:48 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Sun, 28 Jun 2015, Sebastian Moeller wrote:
> 
>> Hi Mikael,
> 
> root@OpenWrt:~# tc -d qdisc
> qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 532
> qdisc fq_codel 110: dev eth0 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 120: dev eth0 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 130: dev eth0 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 32
> qdisc fq_codel 110: dev ifb4eth0 parent 1:11 limit 1001p flows 1024 quantum 500 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 120: dev ifb4eth0 parent 1:12 limit 1001p flows 1024 quantum 1500 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 130: dev ifb4eth0 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> 
> then I change to cake and add 42 as overhead:
> 
> root@OpenWrt:~# tc -d qdisc
> qdisc cake 8003: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows raw

	Cake still did not take the overhead argument (or raw would have gone from the output)

> linklayer ethernet overhead 42

	This looks a bit like the output I would expect from using tic’s stab mechanism to define link layer adjustments and per packet overhead.

Could you post the result of:

cat /etc/config/sqm 

from the router, please? And the output of

/etc/init.d/sqm stop ; /etc/init.d/sqm start

this might give me a clue where to look...





> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn

	The ingress setup did not go through, which most likely means tc got unhappy about one of the options.

Thanks for your time in testing this, I fear sqm-scripts is not really in a good shape for cake testing yet.

Best Regars
	Sebastian

> 
> root@OpenWrt:~# tc -d class show dev eth0
> class cake 8003:508 parent 8003:
> class cake 8003:944 parent 8003:
> root@OpenWrt:~# tc -d class show dev ifb4eth0
> root@OpenWrt:~#
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Sun, 28 Jun 2015, Sebastian Moeller wrote:

> Could you post the result of:
>
> cat /etc/config/sqm
>
> from the router, please? And the output of
>
> /etc/init.d/sqm stop ; /etc/init.d/sqm start
>
> this might give me a clue where to look...

root@OpenWrt:~# cat /etc/config/sqm

config queue 'eth1'
         option script 'simple.qos'
         option interface 'eth0'
         option qdisc_advanced '1'
         option squash_dscp '0'
         option squash_ingress '0'
         option ingress_ecn 'ECN'
         option egress_ecn 'ECN'
         option qdisc_really_really_advanced '0'
         option enabled '1'
         option download '500000'
         option upload '500000'
         option qdisc 'cake'
         option linklayer 'ethernet'
         option overhead '42'

root@OpenWrt:~# /etc/init.d/sqm stop
SQM: Trying to start/stop SQM on all interfaces.
SQM: run.sh stop
SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0
SQM: ifb associated with interface eth0:
SQM: trying to create new IFB: ifb4eth0
RTNETLINK answers: File exists
SQM: /usr/lib/sqm/stop.sh: Stopping eth0
SQM: ifb associated with interface eth0:
SQM: /usr/lib/sqm/run.sh SQM qdiscs on eth0 removed
root@OpenWrt:~# /etc/init.d/sqm start
SQM: Trying to start/stop SQM on all interfaces.
SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos
SQM: ifb associated with interface eth0:
SQM: trying to create new IFB: ifb4eth0
RTNETLINK answers: File exists
SQM: cake
SQM: Keeping differentiated services code points (DSCP) from ingress.
SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet

This command never completes. I added -x to simple.qos and the RTNETLINK 
is from:

+ /usr/sbin/ip link add name ifb4eth0 type ifb
RTNETLINK answers: File exists

and the last commands that execute are:

SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
+ echo stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
+ get_cake_lla_string
+ STABSTRING=
+ [ tc_stab = cake -a ethernet != none ]
+ sqm_logger
+ logger -t SQM -s

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, Mikael Abrahamsson wrote:

> SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
> + echo stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
> + get_cake_lla_string
> + STABSTRING=
> + [ tc_stab = cake -a ethernet != none ]
> + sqm_logger
> + logger -t SQM -s

If I modify line 155 (the sqm_logger "${STABSTRING}" line) to add SQM: 
as well within the quotes so it can't get passed an empty string, the 
command completes.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

HI MIkael,


On Jun 29, 2015, at 07:11 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Mon, 29 Jun 2015, Mikael Abrahamsson wrote:
> 
>> SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
>> + echo stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
>> + get_cake_lla_string
>> + STABSTRING=
>> + [ tc_stab = cake -a ethernet != none ]
>> + sqm_logger
>> + logger -t SQM -s
> 
> If I modify line 155 (the sqm_logger "${STABSTRING}" line) to add SQM: as well within the quotes so it can't get passed an empty string, the command completes.

	Good work-around; not a real solution as sqm_logger() will also add "SQM:”. I think the solution most likely is to remove line 155 completely. But weirdly this seems to work with cerowrt…

Best Regards
	Sebastian

> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 610 bytes --]

On Mon, 29 Jun 2015, Sebastian Moeller wrote:

> 	Good work-around; not a real solution as sqm_logger() will also 
> add "SQM:”. I think the solution most likely is to remove line 155 
> completely. But weirdly this seems to work with cerowrt…

May I suggest a wrapper around the logger part that handles this case, ie 
if the string is empty? Because when the string is empty and it gets 
stuck, luci just waits and waits and waits. I mean, this specific case can 
be fixed, but for the future it would be more robust if this error case 
can't happen at all.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

HI Mikael,


On Jun 29, 2015, at 09:54 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Mon, 29 Jun 2015, Sebastian Moeller wrote:
> 
>> 	Good work-around; not a real solution as sqm_logger() will also add "SQM:”. I think the solution most likely is to remove line 155 completely. But weirdly this seems to work with cerowrt…
> 
> May I suggest a wrapper around the logger part that handles this case, ie if the string is empty? Because when the string is empty and it gets stuck, luci just waits and waits and waits. I mean, this specific case can be fixed, but for the future it would be more robust if this error case can't happen at all.

	Sounds reasonably defensive; what irk me is that in my testbed that specific error does not crop up; one more reason to code defensive I guess…

Best Regards
	Sebastian

> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

HI Mikael,


On Jun 29, 2015, at 09:54 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Mon, 29 Jun 2015, Sebastian Moeller wrote:
> 
>> 	Good work-around; not a real solution as sqm_logger() will also add "SQM:”. I think the solution most likely is to remove line 155 completely. But weirdly this seems to work with cerowrt…

	Correction, calling logger with an empty string works on cero (as well as on openwrt), but passing an empty string to sqm_logger results in the observed bug, so I can reproduce this locally.

> 
> May I suggest a wrapper around the logger part that handles this case, ie if the string is empty? Because when the string is empty and it gets stuck, luci just waits and waits and waits. I mean, this specific case can be fixed, but for the future it would be more robust if this error case can't happen at all.
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

I had a look at sqm_logger():
sqm_logger() {
    logger -t SQM -s ${1}
}

and hope that
sqm_logger() {
    logger -t SQM -s "${1}”
}

Fixes the issue for me and is more concise than trying to introduce real error handling in sqm_logger. Does this also work for you?

Best Regards
	Sebastian

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 459 bytes --]

On Mon, 29 Jun 2015, Sebastian Moeller wrote:

> and hope that
> sqm_logger() {
>    logger -t SQM -s "${1}”
> }
>
> Fixes the issue for me and is more concise than trying to introduce real error handling in sqm_logger. Does this also work for you?

Yes, I reverted my previous change, reproduced the hang, then changed the 
above, which does not hang. Seems like the most elegant way to solve this!

Thanks!

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

On Jun 29, 2015, at 10:17 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Mon, 29 Jun 2015, Sebastian Moeller wrote:
> 
>> and hope that
>> sqm_logger() {
>>   logger -t SQM -s "${1}”
>> }
>> 
>> Fixes the issue for me and is more concise than trying to introduce real error handling in sqm_logger. Does this also work for you?
> 
> Yes, I reverted my previous change, reproduced the hang, then changed the above, which does not hang. Seems like the most elegant way to solve this!

	Thanks for finding, reporting, debugging, fixing, and testing this. This change is committed to the ceropackages 3.10 repository.

Best Regards
	Sebastian 

> 
> Thanks!
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

Hi Mikael,


On Jun 29, 2015, at 06:58 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Sun, 28 Jun 2015, Sebastian Moeller wrote:
> 
>> Could you post the result of:
>> 
>> cat /etc/config/sqm
>> 
>> from the router, please? And the output of
>> 
>> /etc/init.d/sqm stop ; /etc/init.d/sqm start
>> 
>> this might give me a clue where to look...
> 
> root@OpenWrt:~# cat /etc/config/sqm
> 
> config queue 'eth1'
>        option script 'simple.qos'
>        option interface 'eth0'
>        option qdisc_advanced '1'
>        option squash_dscp '0'
>        option squash_ingress '0'
>        option ingress_ecn 'ECN'
>        option egress_ecn 'ECN'
>        option qdisc_really_really_advanced '0'
>        option enabled '1'
>        option download '500000'
>        option upload '500000'
>        option qdisc 'cake'
>        option linklayer 'ethernet'
>        option overhead ’42'

	Ah, I see, you are still using tc’s stab mechanism for account of per packet overhead and link layer adjustments instead of cake’s (you need to check the advanced options check box in the link layer adjustments tab, and then select “cake” as the lnk layer adjustment mechanism).

> 
> root@OpenWrt:~# /etc/init.d/sqm stop
> SQM: Trying to start/stop SQM on all interfaces.
> SQM: run.sh stop
> SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0
> SQM: ifb associated with interface eth0:
> SQM: trying to create new IFB: ifb4eth0
> RTNETLINK answers: File exists

	We can ignore this as it is cosmetic.

> SQM: /usr/lib/sqm/stop.sh: Stopping eth0
> SQM: ifb associated with interface eth0:
> SQM: /usr/lib/sqm/run.sh SQM qdiscs on eth0 removed
> root@OpenWrt:~# /etc/init.d/sqm start
> SQM: Trying to start/stop SQM on all interfaces.
> SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos
> SQM: ifb associated with interface eth0:
> SQM: trying to create new IFB: ifb4eth0
> RTNETLINK answers: File exists
> SQM: cake
> SQM: Keeping differentiated services code points (DSCP) from ingress.
> SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet

	Again, this is tc’s stab mechanism and not cake’s. But it should not break I guess...

> 
> This command never completes. I added -x to simple.qos and the RTNETLINK is from:
> 
> + /usr/sbin/ip link add name ifb4eth0 type ifb
> RTNETLINK answers: File exists
> 
> and the last commands that execute are:
> 
> SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
> + echo stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
> + get_cake_lla_string
> + STABSTRING=
> + [ tc_stab = cake -a ethernet != none ]
> + sqm_logger
> + logger -t SQM -s

	This is weird, under cerowrt this works; but that call to sqm_logger is not useful anyways, it is either repetitive or empty, so I will just delete it

Best Regards
	Sebastian


> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 12784 bytes --]

On Mon, 29 Jun 2015, Sebastian Moeller wrote:

> 	Ah, I see, you are still using tc’s stab mechanism for account of 
> per packet overhead and link layer adjustments instead of cake’s (you 
> need to check the advanced options check box in the link layer 
> adjustments tab, and then select “cake” as the lnk layer adjustment 
> mechanism).

Ok, so when I changed "stab" to cake, I get the following, but it still 
only shapes one way. If I change to discipline to "fq_codel" or "pie" I 
get bidirectonal shaping with otherwise same settings.

root@OpenWrt:~# cat /etc/config/sqm

config queue 'eth1'
 	option interface 'eth0'
 	option qdisc_advanced '1'
 	option squash_dscp '0'
 	option squash_ingress '0'
 	option ingress_ecn 'ECN'
 	option egress_ecn 'ECN'
 	option qdisc_really_really_advanced '0'
 	option download '50000'
 	option upload '50000'
 	option linklayer 'ethernet'
 	option overhead '42'
 	option linklayer_advanced '1'
 	option tcMTU '2047'
 	option tcTSIZE '128'
 	option tcMPU '0'
 	option linklayer_adaptation_mechanism 'cake'
 	option enabled '1'
 	option qdisc 'cake'
 	option script 'simple.qos'

root@OpenWrt:~# tc -d qdisc
qdisc cake 8002: dev eth0 root refcnt 9 bandwidth 50Mbit diffserv4 flows 
noatm overhead 42
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 
quantum 300 target 5.0ms interval 100.0ms ecn

root@OpenWrt:~# tc -d class show dev eth0
class cake 8002:4e1 parent 8002:
class cake 8002:b89 parent 8002:

root@OpenWrt:~# tc -d class show dev ifb4eth0
class fq_codel :12c parent none
class fq_codel :222 parent none

root@OpenWrt:~# /etc/init.d/sqm restart
SQM: Trying to start/stop SQM on all interfaces.
SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0
SQM: ifb associated with interface eth0:
SQM: trying to create new IFB: ifb4eth0
RTNETLINK answers: File exists
SQM: /usr/lib/sqm/stop.sh: Stopping eth0
SQM: ifb associated with interface eth0:
SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos
+ . /usr/lib/sqm/functions.sh
+ [ -z 50000 ]
+ [ -z 50000 ]
+ [ -z eth0 ]
+ [ -z cake ]
+ [ -z cake ]
+ [ -z ethernet ]
+ [ -z 42 ]
+ [ -z 2047 ]
+ [ -z 0 ]
+ [ -z 128 ]
+ [ -z  ]
+ AUTOFLOW=0
+ [ -z  ]
+ LIMIT=1001
+ [ -z  ]
+ ILIMIT=
+ [ -z  ]
+ ELIMIT=
+ [ -z  ]
+ ITARGET=
+ [ -z  ]
+ ETARGET=
+ [ -z ECN ]
+ [ -z ECN ]
+ [ -z 0 ]
+ [ -z 0 ]
+ [ -z  ]
+ IQDISC_OPTS=
+ [ -z  ]
+ EQDISC_OPTS=
+ [ -z  ]
+ which tc
+ TC=/usr/sbin/tc
+ [ -z  ]
+ which ip
+ IP=/usr/sbin/ip
+ [ -z  ]
+ which insmod
+ INSMOD=/usr/sbin/insmod
+ [ -z  ]
+ TARGET=5ms
+ [ -z  ]
+ IPT_MASK=0xff
+ [ -z  ]
+ IPT_MASK_STRING=/0xff
+ [ -z  ]
+ get_ifb_for_if eth0
+ CUR_IF=eth0
+ get_ifb_associated_with_if eth0
+ CUR_IF=eth0
+ tc+  -p filtergrep -o -e ifb[^)]\+
  show parent ffff: dev eth0
+ CUR_IFB=
+ sqm_logger ifb associated with interface eth0:
+ logger -t SQM -s ifb associated with interface eth0:
SQM: ifb associated with interface eth0:
+ echo
+ CUR_IFB=
+ [ -z  ]
+ create_new_ifb_for_if eth0
+ CUR_IF=eth0
+ MAX_IF_NAME_LENGTH=15
+ IFB_PREFIX=ifb4
+ NEW_IFB=ifb4eth0
+ IFB_NAME_LENGTH=8
+ [ 8 -gt 15 ]
+ sqm_logger trying to create new IFB: ifb4eth0
+ logger -t SQM -s trying to create new IFB: ifb4eth0
SQM: trying to create new IFB: ifb4eth0
+ /usr/sbin/ip link add name ifb4eth0 type ifb
RTNETLINK answers: File exists
+ echo ifb4eth0
+ CUR_IFB=ifb4eth0
+ [ -z ifb4eth0 ]
+ echo ifb4eth0
+ DEV=ifb4eth0
+ do_modules
+ insmod act_ipt
+ lsmod+ grep
  -q ^act_ipt
+ insmod sch_cake
+ + grep -q ^sch_cake
lsmod
+ insmod sch_ingress
+ + greplsmod
  -q ^sch_ingress
+ insmod act_mirred
+ + greplsmod -q ^act_mirred

+ insmod cls_fw
+ + grep -q ^cls_fw
lsmod
+ insmod sch_htb
+ + lsmodgrep -q ^sch_htb

+ ipt_setup
+ ipt -t mangle -N QOS_MARK_eth0
+ echo -t mangle -N QOS_MARK_eth0
+ sed s/-A/-D/g
+ d=-t mangle -N QOS_MARK_eth0
+ [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
+ echo -t mangle -N QOS_MARK_eth0
+ sed s/-I/-D/g
+ d=-t mangle -N QOS_MARK_eth0
+ [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
+ iptables -t mangle -N QOS_MARK_eth0
+ ip6tables -t mangle -N QOS_MARK_eth0
+ sqm_logger cake does all the diffserv work - no need for iptables rules
+ logger -t SQM -s cake
SQM: cake
+ [ 0 = 1 ]
+ sqm_logger Keeping differentiated services code points (DSCP) from 
ingress.
+ logger -t SQM -s Keeping differentiated services code points (DSCP) from 
ingress.
SQM: Keeping differentiated services code points (DSCP) from ingress.
+ CAKE_OPTS=
+ ipt -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ echo+ sed s/-A/-D/g
  -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
+ d=-t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ [ -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff 
-g QOS_MARK_eth0 ]
+ iptables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ ip6tables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ echo+ sed s/-I/-D/g
  -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
+ d=-t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ [ -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff 
-g QOS_MARK_eth0 ]
+ iptables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ ip6tables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ ipt -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ echo+ sed s/-A/-D/g
  -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ d=-t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ [ -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff 
-g QOS_MARK_eth0 ]
+ iptables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ ip6tables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ echo+ sed s/-I/-D/g
  -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ d=-t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ [ -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff 
-g QOS_MARK_eth0 ]
+ iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ ip6tables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g 
QOS_MARK_eth0
+ ipt -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
+ echo+ sed s/-A/-D/g
  -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
+ d=-t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
+ [ -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff != 
-t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
+ echo+ sed s/-I/-D/g
  -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
+ d=-t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
+ [ -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff != 
-t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
+ iptables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 
0x2/0xff
+ ip6tables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 
0x2/0xff
+ iptables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 
0x2/0xff
+ ip6tables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 
0x2/0xff
+ ipt -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP 
--set-dscp-class AF42
+ echo+ sed s/-A/-D/g
  -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP 
--set-dscp-class AF42
+ d=-t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP 
--set-dscp-class AF42
+ [ -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP 
--set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports 
123,53 -j DSCP --set-dscp-class AF42 ]
+ iptables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP 
--set-dscp-class AF42
+ ip6tables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP 
--set-dscp-class AF42
+ echo+ sed s/-I/-D/g
  -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP 
--set-dscp-class AF42
+ d=-t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP 
--set-dscp-class AF42
+ [ -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP 
--set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports 
123,53 -j DSCP --set-dscp-class AF42 ]
+ iptables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP 
--set-dscp-class AF42
+ ip6tables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP 
--set-dscp-class AF42
+ [ 50000 -ne 0 ]
+ egress
+ CEIL=50000
+ expr 50000 / 3
+ PRIO_RATE=16666
+ expr 50000 / 6
+ BE_RATE=8333
+ expr 50000 / 6
+ BK_RATE=8333
+ expr 50000 - 16
+ BE_CEIL=49984
+ get_mtu eth0 50000
+ BW=50000
+ cat /sys/class/net/eth0/mtu
+ F=1500
+ [ -z 1500 ]
+ [ 50000 -gt 20000 ]
+ F=3000
+ [ 50000 -gt 30000 ]
+ F=6000
+ [ 50000 -gt 40000 ]
+ F=12000
+ [ 50000 -gt 50000 ]
+ [ 50000 -gt 60000 ]
+ [ 50000 -gt 80000 ]
+ echo 12000
+ LQ=quantum 12000
+ /usr/sbin/tc qdisc del dev eth0 root
+ get_stab_string
+ STABSTRING=
+ [ cake = tc_stab -a ethernet != none ]
+ echo
+ get_cake_lla_string
+ STABSTRING=
+ [ cake = cake -a ethernet != none ]
+ [ ethernet = atm ]
+ STABSTRING= overhead 42
+ sqm_logger cake link layer adjustments:  overhead 42
+ logger -t SQM -s cake link layer adjustments: overhead 42
SQM: cake link layer adjustments: overhead 42
+ sqm_logger SQM:  overhead 42
+ logger -t SQM -s SQM: overhead 42
SQM: SQM: overhead 42
+ echo overhead 42
+ /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead 
42
+ sqm_logger egress shaping activated
+ logger -t SQM -s egress shaping activated
SQM: egress shaping activated
+ [ 50000 -ne 0 ]
+ ingress
+ CEIL=50000
+ expr 50000 / 3
+ PRIO_RATE=16666
+ expr 50000 / 6
+ BE_RATE=8333
+ expr 50000 / 6
+ BK_RATE=8333
+ expr 50000 - 16
+ BE_CEIL=49984
+ get_mtu eth0 50000
+ BW=50000
+ cat /sys/class/net/eth0/mtu
+ F=1500
+ [ -z 1500 ]
+ [ 50000 -gt 20000 ]
+ F=3000
+ [ 50000 -gt 30000 ]
+ F=6000
+ [ 50000 -gt 40000 ]
+ F=12000
+ e 50000 -gt 50000 ]
+ [ 50000 -gt 60000 ]
+ [ 50000 -gt 80000 ]
+ echo 12000
+ LQ=quantum 12000
+ /usr/sbin/tc qdisc del dev eth0 handle ffff: ingress
+ /usr/sbin/tc qdisc add dev eth0 handle ffff: ingress
+ /usr/sbin/tc qdisc del dev ifb4eth0 root
+ [ 0 = 1 ]
+ sqm_logger Perform DSCP based filtering on ingress. (3-tier 
classification)
+ logger -t SQM -s Perform DSCP based filtering on ingress. (3-tier 
classification)
SQM: Perform DSCP based filtering on ingress. (3-tier classification)
+ get_stab_string
+ STABSTRING=
+ [ cake = tc_stab -a ethernet != none ]
+ echo
+ get_cake_lla_string
+ STABSTRING=
+ [ cake = cake -a ethernet != none ]
+ [ ethernet = atm ]
+ STABSTRING= overhead 42
+ sqm_logger cake link layer adjustments:  overhead 42
+ logger -t SQM -s cake link layer adjustments: overhead 42
SQM: cake link layer adjustments: overhead 42
+ sqm_logger SQM:  overhead 42
+ logger -t SQM -s SQM: overhead 42
SQM: SQM: overhead 42
+ echo overhead 42
+ /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead 
42
RTNETLINK answers: File exists
+ ifconfig ifb4eth0 up
+ /usr/sbin/tc filter add dev eth0 parent ffff: protocol all prio 10 u32 
match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth0
+ sqm_logger ingress shaping activated
+ logger -t SQM -s ingress shaping activated
SQM: ingress shaping activated

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

HI Mikael,


On Jun 29, 2015, at 10:09 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Mon, 29 Jun 2015, Sebastian Moeller wrote:
> 
>> 	Ah, I see, you are still using tc’s stab mechanism for account of per packet overhead and link layer adjustments instead of cake’s (you need to check the advanced options check box in the link layer adjustments tab, and then select “cake” as the lnk layer adjustment mechanism).
> 
> Ok, so when I changed "stab" to cake, I get the following, but it still only shapes one way. If I change to discipline to "fq_codel" or "pie" I get bidirectonal shaping with otherwise same settings.

	Probabkr breakage I introduced...

> 
> root@OpenWrt:~# cat /etc/config/sqm
> 
> config queue 'eth1'
> 	option interface 'eth0'
> 	option qdisc_advanced '1'
> 	option squash_dscp '0'
> 	option squash_ingress '0'
> 	option ingress_ecn 'ECN'
> 	option egress_ecn 'ECN'
> 	option qdisc_really_really_advanced '0'
> 	option download '50000'
> 	option upload '50000'
> 	option linklayer 'ethernet'
> 	option overhead '42'
> 	option linklayer_advanced '1'
> 	option tcMTU '2047'
> 	option tcTSIZE '128'
> 	option tcMPU ‘0'

	we can ignore the last 3 savely.

> 	option linklayer_adaptation_mechanism 'cake'
> 	option enabled '1'
> 	option qdisc 'cake'
> 	option script 'simple.qos’

	Okay, it really is trying cake’s LLA now.

> 
> root@OpenWrt:~# tc -d qdisc
> qdisc cake 8002: dev eth0 root refcnt 9 bandwidth 50Mbit diffserv4 flows noatm overhead 42

	And that works on egress, excellent so bot sch_cake and tc are recent enough.

> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn

	Oops, ingress is not right, yet.

> 
> root@OpenWrt:~# tc -d class show dev eth0
> class cake 8002:4e1 parent 8002:
> class cake 8002:b89 parent 8002:
> 
> root@OpenWrt:~# tc -d class show dev ifb4eth0
> class fq_codel :12c parent none
> class fq_codel :222 parent none
> 
> root@OpenWrt:~# /etc/init.d/sqm restart
> SQM: Trying to start/stop SQM on all interfaces.
> SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0
> SQM: ifb associated with interface eth0:
> SQM: trying to create new IFB: ifb4eth0
> RTNETLINK answers: File exists
> SQM: /usr/lib/sqm/stop.sh: Stopping eth0
> SQM: ifb associated with interface eth0:
> SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos
> + . /usr/lib/sqm/functions.sh
> + [ -z 50000 ]
> + [ -z 50000 ]
> + [ -z eth0 ]
> + [ -z cake ]
> + [ -z cake ]
> + [ -z ethernet ]
> + [ -z 42 ]
> + [ -z 2047 ]
> + [ -z 0 ]
> + [ -z 128 ]
> + [ -z  ]
> + AUTOFLOW=0
> + [ -z  ]
> + LIMIT=1001
> + [ -z  ]
> + ILIMIT=
> + [ -z  ]
> + ELIMIT=
> + [ -z  ]
> + ITARGET=
> + [ -z  ]
> + ETARGET=
> + [ -z ECN ]
> + [ -z ECN ]
> + [ -z 0 ]
> + [ -z 0 ]
> + [ -z  ]
> + IQDISC_OPTS=
> + [ -z  ]
> + EQDISC_OPTS=
> + [ -z  ]
> + which tc
> + TC=/usr/sbin/tc
> + [ -z  ]
> + which ip
> + IP=/usr/sbin/ip
> + [ -z  ]
> + which insmod
> + INSMOD=/usr/sbin/insmod
> + [ -z  ]
> + TARGET=5ms
> + [ -z  ]
> + IPT_MASK=0xff
> + [ -z  ]
> + IPT_MASK_STRING=/0xff
> + [ -z  ]
> + get_ifb_for_if eth0
> + CUR_IF=eth0
> + get_ifb_associated_with_if eth0
> + CUR_IF=eth0
> + tc+  -p filtergrep -o -e ifb[^)]\+
> show parent ffff: dev eth0
> + CUR_IFB=
> + sqm_logger ifb associated with interface eth0:
> + logger -t SQM -s ifb associated with interface eth0:
> SQM: ifb associated with interface eth0:
> + echo
> + CUR_IFB=
> + [ -z  ]
> + create_new_ifb_for_if eth0
> + CUR_IF=eth0
> + MAX_IF_NAME_LENGTH=15
> + IFB_PREFIX=ifb4
> + NEW_IFB=ifb4eth0
> + IFB_NAME_LENGTH=8
> + [ 8 -gt 15 ]
> + sqm_logger trying to create new IFB: ifb4eth0
> + logger -t SQM -s trying to create new IFB: ifb4eth0
> SQM: trying to create new IFB: ifb4eth0
> + /usr/sbin/ip link add name ifb4eth0 type ifb
> RTNETLINK answers: File exists
> + echo ifb4eth0
> + CUR_IFB=ifb4eth0
> + [ -z ifb4eth0 ]
> + echo ifb4eth0
> + DEV=ifb4eth0
> + do_modules
> + insmod act_ipt
> + lsmod+ grep
> -q ^act_ipt
> + insmod sch_cake
> + + grep -q ^sch_cake
> lsmod
> + insmod sch_ingress
> + + greplsmod
> -q ^sch_ingress
> + insmod act_mirred
> + + greplsmod -q ^act_mirred
> 
> + insmod cls_fw
> + + grep -q ^cls_fw
> lsmod
> + insmod sch_htb
> + + lsmodgrep -q ^sch_htb
> 
> + ipt_setup
> + ipt -t mangle -N QOS_MARK_eth0
> + echo -t mangle -N QOS_MARK_eth0
> + sed s/-A/-D/g
> + d=-t mangle -N QOS_MARK_eth0
> + [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
> + echo -t mangle -N QOS_MARK_eth0
> + sed s/-I/-D/g
> + d=-t mangle -N QOS_MARK_eth0
> + [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
> + iptables -t mangle -N QOS_MARK_eth0
> + ip6tables -t mangle -N QOS_MARK_eth0
> + sqm_logger cake does all the diffserv work - no need for iptables rules
> + logger -t SQM -s cake
> SQM: cake
> + [ 0 = 1 ]
> + sqm_logger Keeping differentiated services code points (DSCP) from ingress.
> + logger -t SQM -s Keeping differentiated services code points (DSCP) from ingress.
> SQM: Keeping differentiated services code points (DSCP) from ingress.
> + CAKE_OPTS=
> + ipt -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-A/-D/g
> -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-I/-D/g
> -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ipt -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-A/-D/g
> -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-I/-D/g
> -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ipt -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + echo+ sed s/-A/-D/g
> -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + d=-t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + [ -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff != -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
> + echo+ sed s/-I/-D/g
> -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + d=-t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + [ -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff != -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
> + iptables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ip6tables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + iptables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ip6tables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ipt -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + echo+ sed s/-A/-D/g
> -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + d=-t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 ]
> + iptables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + ip6tables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + echo+ sed s/-I/-D/g
> -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + d=-t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 ]
> + iptables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + ip6tables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ 50000 -ne 0 ]
> + egress
> + CEIL=50000
> + expr 50000 / 3
> + PRIO_RATE=16666
> + expr 50000 / 6
> + BE_RATE=8333
> + expr 50000 / 6
> + BK_RATE=8333
> + expr 50000 - 16
> + BE_CEIL=49984
> + get_mtu eth0 50000
> + BW=50000
> + cat /sys/class/net/eth0/mtu
> + F=1500
> + [ -z 1500 ]
> + [ 50000 -gt 20000 ]
> + F=3000
> + [ 50000 -gt 30000 ]
> + F=6000
> + [ 50000 -gt 40000 ]
> + F=12000
> + [ 50000 -gt 50000 ]
> + [ 50000 -gt 60000 ]
> + [ 50000 -gt 80000 ]
> + echo 12000
> + LQ=quantum 12000
> + /usr/sbin/tc qdisc del dev eth0 root
> + get_stab_string
> + STABSTRING=
> + [ cake = tc_stab -a ethernet != none ]
> + echo
> + get_cake_lla_string
> + STABSTRING=
> + [ cake = cake -a ethernet != none ]
> + [ ethernet = atm ]
> + STABSTRING= overhead 42
> + sqm_logger cake link layer adjustments:  overhead 42
> + logger -t SQM -s cake link layer adjustments: overhead 42
> SQM: cake link layer adjustments: overhead 42
> + sqm_logger SQM:  overhead 42
> + logger -t SQM -s SQM: overhead 42
> SQM: SQM: overhead 42
> + echo overhead 42
> + /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead 42

	As seen above egress works.

> + sqm_logger egress shaping activated
> + logger -t SQM -s egress shaping activated
> SQM: egress shaping activated
> + [ 50000 -ne 0 ]
> + ingress
> + CEIL=50000
> + expr 50000 / 3
> + PRIO_RATE=16666
> + expr 50000 / 6
> + BE_RATE=8333
> + expr 50000 / 6
> + BK_RATE=8333
> + expr 50000 - 16
> + BE_CEIL=49984
> + get_mtu eth0 50000
> + BW=50000
> + cat /sys/class/net/eth0/mtu
> + F=1500
> + [ -z 1500 ]
> + [ 50000 -gt 20000 ]
> + F=3000
> + [ 50000 -gt 30000 ]
> + F=6000
> + [ 50000 -gt 40000 ]
> + F=12000
> + e 50000 -gt 50000 ]
> + [ 50000 -gt 60000 ]
> + [ 50000 -gt 80000 ]
> + echo 12000
> + LQ=quantum 12000
> + /usr/sbin/tc qdisc del dev eth0 handle ffff: ingress
> + /usr/sbin/tc qdisc add dev eth0 handle ffff: ingress
> + /usr/sbin/tc qdisc del dev ifb4eth0 root
> + [ 0 = 1 ]
> + sqm_logger Perform DSCP based filtering on ingress. (3-tier classification)
> + logger -t SQM -s Perform DSCP based filtering on ingress. (3-tier classification)
> SQM: Perform DSCP based filtering on ingress. (3-tier classification)
> + get_stab_string
> + STABSTRING=
> + [ cake = tc_stab -a ethernet != none ]
> + echo
> + get_cake_lla_string
> + STABSTRING=
> + [ cake = cake -a ethernet != none ]
> + [ ethernet = atm ]
> + STABSTRING= overhead 42
> + sqm_logger cake link layer adjustments:  overhead 42
> + logger -t SQM -s cake link layer adjustments: overhead 42
> SQM: cake link layer adjustments: overhead 42
> + sqm_logger SQM:  overhead 42
> + logger -t SQM -s SQM: overhead 42
> SQM: SQM: overhead 42
> + echo overhead 42
> + /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead 42

	As seen above egress works. What the F. There is an embarrassing copy and paste error in simple.qosline 180, instead of:
$TC qdisc add dev $IFACE root `get_stab_string` $QDISC bandwidth ${DOWNLINK}kbit `get_cake_lla_string` $CAKE_OPTS ${IQDISC_OPTS}

this line should obviously read:
$TC qdisc add dev $DEV root `get_stab_string` $QDISC bandwidth ${DOWNLINK}kbit `get_cake_lla_string` $CAKE_OPTS ${IQDISC_OPTS}

so simple.qos set-up egress twice, once with the egress parameters and once with the ingress parameters, I guess I did not test 3-tier ingress classification on Dave’s test machine.
Also I should rename a few variables. $DEV and $IFACE are not the best names. (Also cake needs its own .qos file(s) instead of simple and simplest becoming rather “complex”…) Thanks for keeping testing this.

Best Regards
	Sebastian

> RTNETLINK answers: File exists
> + ifconfig ifb4eth0 up
> + /usr/sbin/tc filter add dev eth0 parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth0
> + sqm_logger ingress shaping activated
> + logger -t SQM -s ingress shaping activated
> SQM: ingress shaping activated




> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

Hi Mikael,

since coke overhead seems to work, you can switch back to link layer none or set the overhead to 0 again. Thanks for testing this, which helps getting sqm-scripts into better shape, but will not be helpful fur your tests. In case you want to try fancy options for cake the advances qdisc option strings in the queue discipline tab should work (caveat no error checking so one typo and the qdisc will most likely not be set-up properly).

Best Regards
	Sebastian



On Jun 29, 2015, at 10:09 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Mon, 29 Jun 2015, Sebastian Moeller wrote:
> 
>> 	Ah, I see, you are still using tc’s stab mechanism for account of per packet overhead and link layer adjustments instead of cake’s (you need to check the advanced options check box in the link layer adjustments tab, and then select “cake” as the lnk layer adjustment mechanism).
> 
> Ok, so when I changed "stab" to cake, I get the following, but it still only shapes one way. If I change to discipline to "fq_codel" or "pie" I get bidirectonal shaping with otherwise same settings.
> 
> root@OpenWrt:~# cat /etc/config/sqm
> 
> config queue 'eth1'
> 	option interface 'eth0'
> 	option qdisc_advanced '1'
> 	option squash_dscp '0'
> 	option squash_ingress '0'
> 	option ingress_ecn 'ECN'
> 	option egress_ecn 'ECN'
> 	option qdisc_really_really_advanced '0'
> 	option download '50000'
> 	option upload '50000'
> 	option linklayer 'ethernet'
> 	option overhead '42'
> 	option linklayer_advanced '1'
> 	option tcMTU '2047'
> 	option tcTSIZE '128'
> 	option tcMPU '0'
> 	option linklayer_adaptation_mechanism 'cake'
> 	option enabled '1'
> 	option qdisc 'cake'
> 	option script 'simple.qos'
> 
> root@OpenWrt:~# tc -d qdisc
> qdisc cake 8002: dev eth0 root refcnt 9 bandwidth 50Mbit diffserv4 flows noatm overhead 42
> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> 
> root@OpenWrt:~# tc -d class show dev eth0
> class cake 8002:4e1 parent 8002:
> class cake 8002:b89 parent 8002:
> 
> root@OpenWrt:~# tc -d class show dev ifb4eth0
> class fq_codel :12c parent none
> class fq_codel :222 parent none
> 
> root@OpenWrt:~# /etc/init.d/sqm restart
> SQM: Trying to start/stop SQM on all interfaces.
> SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0
> SQM: ifb associated with interface eth0:
> SQM: trying to create new IFB: ifb4eth0
> RTNETLINK answers: File exists
> SQM: /usr/lib/sqm/stop.sh: Stopping eth0
> SQM: ifb associated with interface eth0:
> SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos
> + . /usr/lib/sqm/functions.sh
> + [ -z 50000 ]
> + [ -z 50000 ]
> + [ -z eth0 ]
> + [ -z cake ]
> + [ -z cake ]
> + [ -z ethernet ]
> + [ -z 42 ]
> + [ -z 2047 ]
> + [ -z 0 ]
> + [ -z 128 ]
> + [ -z  ]
> + AUTOFLOW=0
> + [ -z  ]
> + LIMIT=1001
> + [ -z  ]
> + ILIMIT=
> + [ -z  ]
> + ELIMIT=
> + [ -z  ]
> + ITARGET=
> + [ -z  ]
> + ETARGET=
> + [ -z ECN ]
> + [ -z ECN ]
> + [ -z 0 ]
> + [ -z 0 ]
> + [ -z  ]
> + IQDISC_OPTS=
> + [ -z  ]
> + EQDISC_OPTS=
> + [ -z  ]
> + which tc
> + TC=/usr/sbin/tc
> + [ -z  ]
> + which ip
> + IP=/usr/sbin/ip
> + [ -z  ]
> + which insmod
> + INSMOD=/usr/sbin/insmod
> + [ -z  ]
> + TARGET=5ms
> + [ -z  ]
> + IPT_MASK=0xff
> + [ -z  ]
> + IPT_MASK_STRING=/0xff
> + [ -z  ]
> + get_ifb_for_if eth0
> + CUR_IF=eth0
> + get_ifb_associated_with_if eth0
> + CUR_IF=eth0
> + tc+  -p filtergrep -o -e ifb[^)]\+
> show parent ffff: dev eth0
> + CUR_IFB=
> + sqm_logger ifb associated with interface eth0:
> + logger -t SQM -s ifb associated with interface eth0:
> SQM: ifb associated with interface eth0:
> + echo
> + CUR_IFB=
> + [ -z  ]
> + create_new_ifb_for_if eth0
> + CUR_IF=eth0
> + MAX_IF_NAME_LENGTH=15
> + IFB_PREFIX=ifb4
> + NEW_IFB=ifb4eth0
> + IFB_NAME_LENGTH=8
> + [ 8 -gt 15 ]
> + sqm_logger trying to create new IFB: ifb4eth0
> + logger -t SQM -s trying to create new IFB: ifb4eth0
> SQM: trying to create new IFB: ifb4eth0
> + /usr/sbin/ip link add name ifb4eth0 type ifb
> RTNETLINK answers: File exists
> + echo ifb4eth0
> + CUR_IFB=ifb4eth0
> + [ -z ifb4eth0 ]
> + echo ifb4eth0
> + DEV=ifb4eth0
> + do_modules
> + insmod act_ipt
> + lsmod+ grep
> -q ^act_ipt
> + insmod sch_cake
> + + grep -q ^sch_cake
> lsmod
> + insmod sch_ingress
> + + greplsmod
> -q ^sch_ingress
> + insmod act_mirred
> + + greplsmod -q ^act_mirred
> 
> + insmod cls_fw
> + + grep -q ^cls_fw
> lsmod
> + insmod sch_htb
> + + lsmodgrep -q ^sch_htb
> 
> + ipt_setup
> + ipt -t mangle -N QOS_MARK_eth0
> + echo -t mangle -N QOS_MARK_eth0
> + sed s/-A/-D/g
> + d=-t mangle -N QOS_MARK_eth0
> + [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
> + echo -t mangle -N QOS_MARK_eth0
> + sed s/-I/-D/g
> + d=-t mangle -N QOS_MARK_eth0
> + [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
> + iptables -t mangle -N QOS_MARK_eth0
> + ip6tables -t mangle -N QOS_MARK_eth0
> + sqm_logger cake does all the diffserv work - no need for iptables rules
> + logger -t SQM -s cake
> SQM: cake
> + [ 0 = 1 ]
> + sqm_logger Keeping differentiated services code points (DSCP) from ingress.
> + logger -t SQM -s Keeping differentiated services code points (DSCP) from ingress.
> SQM: Keeping differentiated services code points (DSCP) from ingress.
> + CAKE_OPTS=
> + ipt -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-A/-D/g
> -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-I/-D/g
> -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ipt -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-A/-D/g
> -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-I/-D/g
> -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ipt -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + echo+ sed s/-A/-D/g
> -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + d=-t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + [ -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff != -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
> + echo+ sed s/-I/-D/g
> -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + d=-t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + [ -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff != -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
> + iptables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ip6tables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + iptables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ip6tables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ipt -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + echo+ sed s/-A/-D/g
> -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + d=-t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 ]
> + iptables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + ip6tables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + echo+ sed s/-I/-D/g
> -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + d=-t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 ]
> + iptables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + ip6tables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ 50000 -ne 0 ]
> + egress
> + CEIL=50000
> + expr 50000 / 3
> + PRIO_RATE=16666
> + expr 50000 / 6
> + BE_RATE=8333
> + expr 50000 / 6
> + BK_RATE=8333
> + expr 50000 - 16
> + BE_CEIL=49984
> + get_mtu eth0 50000
> + BW=50000
> + cat /sys/class/net/eth0/mtu
> + F=1500
> + [ -z 1500 ]
> + [ 50000 -gt 20000 ]
> + F=3000
> + [ 50000 -gt 30000 ]
> + F=6000
> + [ 50000 -gt 40000 ]
> + F=12000
> + [ 50000 -gt 50000 ]
> + [ 50000 -gt 60000 ]
> + [ 50000 -gt 80000 ]
> + echo 12000
> + LQ=quantum 12000
> + /usr/sbin/tc qdisc del dev eth0 root
> + get_stab_string
> + STABSTRING=
> + [ cake = tc_stab -a ethernet != none ]
> + echo
> + get_cake_lla_string
> + STABSTRING=
> + [ cake = cake -a ethernet != none ]
> + [ ethernet = atm ]
> + STABSTRING= overhead 42
> + sqm_logger cake link layer adjustments:  overhead 42
> + logger -t SQM -s cake link layer adjustments: overhead 42
> SQM: cake link layer adjustments: overhead 42
> + sqm_logger SQM:  overhead 42
> + logger -t SQM -s SQM: overhead 42
> SQM: SQM: overhead 42
> + echo overhead 42
> + /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead 42
> + sqm_logger egress shaping activated
> + logger -t SQM -s egress shaping activated
> SQM: egress shaping activated
> + [ 50000 -ne 0 ]
> + ingress
> + CEIL=50000
> + expr 50000 / 3
> + PRIO_RATE=16666
> + expr 50000 / 6
> + BE_RATE=8333
> + expr 50000 / 6
> + BK_RATE=8333
> + expr 50000 - 16
> + BE_CEIL=49984
> + get_mtu eth0 50000
> + BW=50000
> + cat /sys/class/net/eth0/mtu
> + F=1500
> + [ -z 1500 ]
> + [ 50000 -gt 20000 ]
> + F=3000
> + [ 50000 -gt 30000 ]
> + F=6000
> + [ 50000 -gt 40000 ]
> + F=12000
> + e 50000 -gt 50000 ]
> + [ 50000 -gt 60000 ]
> + [ 50000 -gt 80000 ]
> + echo 12000
> + LQ=quantum 12000
> + /usr/sbin/tc qdisc del dev eth0 handle ffff: ingress
> + /usr/sbin/tc qdisc add dev eth0 handle ffff: ingress
> + /usr/sbin/tc qdisc del dev ifb4eth0 root
> + [ 0 = 1 ]
> + sqm_logger Perform DSCP based filtering on ingress. (3-tier classification)
> + logger -t SQM -s Perform DSCP based filtering on ingress. (3-tier classification)
> SQM: Perform DSCP based filtering on ingress. (3-tier classification)
> + get_stab_string
> + STABSTRING=
> + [ cake = tc_stab -a ethernet != none ]
> + echo
> + get_cake_lla_string
> + STABSTRING=
> + [ cake = cake -a ethernet != none ]
> + [ ethernet = atm ]
> + STABSTRING= overhead 42
> + sqm_logger cake link layer adjustments:  overhead 42
> + logger -t SQM -s cake link layer adjustments: overhead 42
> SQM: cake link layer adjustments: overhead 42
> + sqm_logger SQM:  overhead 42
> + logger -t SQM -s SQM: overhead 42
> SQM: SQM: overhead 42
> + echo overhead 42
> + /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead 42
> RTNETLINK answers: File exists
> + ifconfig ifb4eth0 up
> + /usr/sbin/tc filter add dev eth0 parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth0
> + sqm_logger ingress shaping activated
> + logger -t SQM -s ingress shaping activated
> SQM: ingress shaping activated
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, Sebastian Moeller wrote:

> Hi Mikael,
>
> since coke overhead seems to work, you can switch back to link layer 
> none or set the overhead to 0 again. Thanks for testing this, which 
> helps getting sqm-scripts into better shape, but will not be helpful fur 
> your tests. In case you want to try fancy options for cake the advances 
> qdisc option strings in the queue discipline tab should work (caveat no 
> error checking so one typo and the qdisc will most likely not be set-up 
> properly).

What I would like to get working is that it builds a fully working image 
out of the box. I have a problem with this, I believe it's because the 
ceropackages and openwrt standard packages have overlapping names:

#openwrt/feeds$ grep -i sqm-scripts *
grep: cero: Is a directory
cero.index:Depends: +libc +SSP_SUPPORT:libssp +USE_GLIBC:librt +USE_GLIBC:libpthread lua luci-base +sqm-scripts
cero.index:Source-Makefile: feeds/cero/net/sqm-scripts/Makefile
cero.index:Package: sqm-scripts
grep: cero.tmp: Is a directory
grep: luci: Is a directory
grep: luci.tmp: Is a directory
grep: management: Is a directory
grep: management.tmp: Is a directory
grep: packages: Is a directory
packages.index:Depends: +libc +SSP_SUPPORT:libssp +USE_GLIBC:librt +USE_GLIBC:libpthread lua luci-base +sqm-scripts
packages.index:Source-Makefile: feeds/packages/net/sqm-scripts/Makefile
packages.index:Package: sqm-scripts
grep: packages.tmp: Is a directory
grep: routing: Is a directory
grep: routing.tmp: Is a directory
grep: targets: Is a directory
grep: targets.tmp: Is a directory
grep: telephony: Is a directory
grep: telephony.tmp: Is a directory

How can I tell which one of these actually is included? Tried moving the 
feed statement so ceropackages was first but that doesn't seem to have 
helped, I still get OpenWrt regular sqm-scripts (no cake in luci-app-sqm).

#openwrt/feeds$ find . -name luci-app-sqm
./packages/net/luci-app-sqm
./cero/luci/luci-app-sqm

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:

> How can I tell which one of these actually is included? Tried moving the feed
> statement so ceropackages was first but that doesn't seem to have helped, I
> still get OpenWrt regular sqm-scripts (no cake in luci-app-sqm).

You need to have the cero feed first in feeds.conf, then do
./script/feeds uninstall sqm-scripts; ./script/feeds install sqm-scripts
-- that should pull it from the cero feed. If it doesn't work, try doing
./scripts/feeds update first

That seemed to work for me when I just tested it now, at least :)

-Toke

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 7338 bytes --]

On Mon, 29 Jun 2015, Toke Høiland-Jørgensen wrote:

> Mikael Abrahamsson <swmike@swm.pp.se> writes:
>
>> How can I tell which one of these actually is included? Tried moving the feed
>> statement so ceropackages was first but that doesn't seem to have helped, I
>> still get OpenWrt regular sqm-scripts (no cake in luci-app-sqm).
>
> You need to have the cero feed first in feeds.conf, then do
> ./script/feeds uninstall sqm-scripts; ./script/feeds install sqm-scripts
> -- that should pull it from the cero feed. If it doesn't work, try doing
> ./scripts/feeds update first
>
> That seemed to work for me when I just tested it now, at least :)

Hi,

Ok, yes, this worked, I must have forgotten do to update after I moved 
ceropackages to the top of the list before. Thanks!

So now I have a sysupgrade image for the wrt1200ac that out of the box 
comes with CeroPackages and working bidirectional shaping for cake (don't 
know why it didn't work before, it might have to do with my modifications. 
This time I didn't modify anything on-disk, this is purely from the 
CeroPackages feed).

I did try to get Kernel 4.1 to compile but that didn't work even though I 
removed some packages that didn't compile, I ended up with no .dts file 
and nothing to me obvious in scrollback to fix. So this is with 3.18.

Here are the cake 50M and 500M results and output from the router:

oot@OpenWrt:~# cat /etc/config/sqm

config queue 'eth1'
 	option interface 'eth0'
 	option qdisc_advanced '1'
 	option squash_dscp '0'
 	option squash_ingress '0'
 	option ingress_ecn 'ECN'
 	option egress_ecn 'ECN'
 	option qdisc_really_really_advanced '0'
 	option linklayer 'ethernet'
 	option overhead '42'
 	option linklayer_advanced '1'
 	option tcMTU '2047'
 	option tcTSIZE '128'
 	option tcMPU '0'
 	option enabled '1'
 	option script 'simple.qos'
 	option qdisc 'cake'
 	option linklayer_adaptation_mechanism 'cake'
 	option download '500000'
 	option upload '500000'

root@OpenWrt:~# tc -d qdisc
qdisc cake 8009: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows 
noatm overhead 42
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc cake 800a: dev ifb4eth0 root refcnt 2 bandwidth 500Mbit diffserv4 
flows noatm overhead 42

These are the results from 50M and 500M, also including 50up and 50down 
that I added to my test suite script.

http://swm.pp.se/aqm/rrul_150629-cake-4.tar

Then I re-did the test that Dave asked before, I set the wan port to 100 
megabit/s in my switch, and removed the SQM. It resulted in the following 
config:

root@OpenWrt:~# cat /etc/config/sqm

config queue 'eth1'
 	option interface 'eth0'
 	option qdisc_advanced '1'
 	option squash_dscp '0'
 	option squash_ingress '0'
 	option ingress_ecn 'ECN'
 	option egress_ecn 'ECN'
 	option qdisc_really_really_advanced '0'
 	option linklayer 'ethernet'
 	option overhead '42'
 	option linklayer_advanced '1'
 	option tcMTU '2047'
 	option tcTSIZE '128'
 	option tcMPU '0'
 	option script 'simple.qos'
 	option qdisc 'cake'
 	option linklayer_adaptation_mechanism 'cake'
 	option download '50000'
 	option upload '50000'
 	option enabled '0'

root@OpenWrt:~# tc -d qdisc
qdisc mq 0: dev eth0 root
qdisc fq_codel 0: dev eth0 parent :1 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :2 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :3 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :4 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :5 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :6 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :7 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :8 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc cake 800c: dev ifb4eth0 root refcnt 2 bandwidth 50Mbit diffserv4 
flows noatm overhead 42
root@OpenWrt:~# tc -d class show dev eth0
class mq :1 root
class mq :2 root
class mq :3 root
class mq :4 root
class mq :5 root
class mq :6 root
class mq :7 root
class mq :8 root

what worries me is this:

root@OpenWrt:~# ethtool eth0
Settings for eth0:
 	Supported ports: [ TP MII ]
 	Supported link modes:   1000baseT/Half 1000baseT/Full
 	Supported pause frame use: No
 	Supports auto-negotiation: Yes
 	Advertised link modes:  1000baseT/Half 1000baseT/Full
 	Advertised pause frame use: No
 	Advertised auto-negotiation: Yes
 	Link partner advertised link modes:  1000baseT/Full
 	Link partner advertised pause frame use: No
 	Link partner advertised auto-negotiation: No
 	Speed: 1000Mb/s
 	Duplex: Full
 	Port: MII
 	PHYAD: 0
 	Transceiver: external
 	Auto-negotiation: on
 	Link detected: yes

So basically even after the wan port went to 100/full, eth0 doesn't know 
about it (and it only supports gig speed (probably to the local switch) 
anyway. I am seeing dropped packets, so this would support this theory.

First I ran some tests with only that, then I set SQM to 90 megabit/s 
results here:

http://swm.pp.se/aqm/rrul_150629-cake-5.tar
http://swm.pp.se/aqm/rrul_150629-cake-6.tar

Next I am going to test wireless but it seems something has gone wrong 
because I can't get the wireless to enable properly, so that'll have to be 
for the next email after I fix that problem.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

Hi Mikael,

On Jun 29, 2015, at 15:00 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> [...]
> Hi,
> 
> Ok, yes, this worked, I must have forgotten do to update after I moved ceropackages to the top of the list before. Thanks!
> 
> So now I have a sysupgrade image for the wrt1200ac that out of the box comes with CeroPackages and working bidirectional shaping for cake (don't know why it didn't work before, it might have to do with my modifications. This time I didn't modify anything on-disk, this is purely from the CeroPackages feed).

	Erm, I committed the fix for the $DEV $IFACE confusion to the ceropackages repository, so you got the fixed version you helped fix ;) (that or you deselected ingress 3-tier classification).

> 
> I did try to get Kernel 4.1 to compile but that didn't work even though I removed some packages that didn't compile, I ended up with no .dts file and nothing to me obvious in scrollback to fix. So this is with 3.18.
> 
> Here are the cake 50M and 500M results and output from the router:
> 
> oot@OpenWrt:~# cat /etc/config/sqm
> 
> config queue 'eth1'
> 	option interface 'eth0'
> 	option qdisc_advanced '1'
> 	option squash_dscp '0'
> 	option squash_ingress '0'
> 	option ingress_ecn 'ECN'
> 	option egress_ecn 'ECN'
> 	option qdisc_really_really_advanced '0'
> 	option linklayer 'ethernet'
> 	option overhead '42'
> 	option linklayer_advanced '1'
> 	option tcMTU '2047'
> 	option tcTSIZE '128'
> 	option tcMPU '0'
> 	option enabled '1'
> 	option script 'simple.qos'
> 	option qdisc 'cake'
> 	option linklayer_adaptation_mechanism 'cake'
> 	option download '500000'
> 	option upload '500000'
> 
> root@OpenWrt:~# tc -d qdisc
> qdisc cake 8009: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows noatm overhead 42
> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc cake 800a: dev ifb4eth0 root refcnt 2 bandwidth 500Mbit diffserv4 flows noatm overhead 42

	I love that passing per packet overheads to cake works, but for your testing it is going to reduce the maximum achievable speed from:

TCP/IPv4 Payload at 500Mbps shaping:
1500 - 20 - 20 = 1460 Byte
payload to OTWS ratio
1460/1518 = 0.961791831357
Payload Bandwidth
500*1460/1518 = 480.90 Mbps

to:

TCP/IPv4 Payload at 500Mbps shaping: with overhead 42
MTU: 1500
MSS: 1500 - 20 - 20 = 1460 Byte
PerPacketOverhead: 42
configured On-The-Wire Size: 1500 + 42 = 1542 Byte
payload to OTWS ratio (due to the explicitly configured overhead42)
1460/1542 = 0.961791831357
Payload Bandwidth
500*1460/1542 = 473.411154345 Mbps

I really ned to get a new router to take part in all the fun…

Best Regards
	Sebastian



> 
> These are the results from 50M and 500M, also including 50up and 50down that I added to my test suite script.
> 
> http://swm.pp.se/aqm/rrul_150629-cake-4.tar
> 
> Then I re-did the test that Dave asked before, I set the wan port to 100 megabit/s in my switch, and removed the SQM. It resulted in the following config:
> 
> root@OpenWrt:~# cat /etc/config/sqm
> 
> config queue 'eth1'
> 	option interface 'eth0'
> 	option qdisc_advanced '1'
> 	option squash_dscp '0'
> 	option squash_ingress '0'
> 	option ingress_ecn 'ECN'
> 	option egress_ecn 'ECN'
> 	option qdisc_really_really_advanced '0'
> 	option linklayer 'ethernet'
> 	option overhead '42'
> 	option linklayer_advanced '1'
> 	option tcMTU '2047'
> 	option tcTSIZE '128'
> 	option tcMPU '0'
> 	option script 'simple.qos'
> 	option qdisc 'cake'
> 	option linklayer_adaptation_mechanism 'cake'
> 	option download '50000'
> 	option upload '50000'
> 	option enabled '0'
> 
> root@OpenWrt:~# tc -d qdisc
> qdisc mq 0: dev eth0 root
> qdisc fq_codel 0: dev eth0 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc cake 800c: dev ifb4eth0 root refcnt 2 bandwidth 50Mbit diffserv4 flows noatm overhead 42
> root@OpenWrt:~# tc -d class show dev eth0
> class mq :1 root
> class mq :2 root
> class mq :3 root
> class mq :4 root
> class mq :5 root
> class mq :6 root
> class mq :7 root
> class mq :8 root
> 
> what worries me is this:
> 
> root@OpenWrt:~# ethtool eth0
> Settings for eth0:
> 	Supported ports: [ TP MII ]
> 	Supported link modes:   1000baseT/Half 1000baseT/Full
> 	Supported pause frame use: No
> 	Supports auto-negotiation: Yes
> 	Advertised link modes:  1000baseT/Half 1000baseT/Full
> 	Advertised pause frame use: No
> 	Advertised auto-negotiation: Yes
> 	Link partner advertised link modes:  1000baseT/Full
> 	Link partner advertised pause frame use: No
> 	Link partner advertised auto-negotiation: No
> 	Speed: 1000Mb/s
> 	Duplex: Full
> 	Port: MII
> 	PHYAD: 0
> 	Transceiver: external
> 	Auto-negotiation: on
> 	Link detected: yes
> 
> So basically even after the wan port went to 100/full, eth0 doesn't know about it (and it only supports gig speed (probably to the local switch) anyway. I am seeing dropped packets, so this would support this theory.
> 
> First I ran some tests with only that, then I set SQM to 90 megabit/s results here:
> 
> http://swm.pp.se/aqm/rrul_150629-cake-5.tar
> http://swm.pp.se/aqm/rrul_150629-cake-6.tar
> 
> Next I am going to test wireless but it seems something has gone wrong because I can't get the wireless to enable properly, so that'll have to be for the next email after I fix that problem.
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se_______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 327 bytes --]


I would love to try out cake in my environment.  However, as a non-combatant, it would be nice to have an instruction sheet on how to set the latest version up, and what hardware it works best on (WRT1200AC?).  Obviously this is a work in progress, so that will change, but it would be nice to have a summarized wiki page.
 

[-- Attachment #2: Type: text/html, Size: 602 bytes --]

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 297 bytes --]

I'd also like to be able to try it out on CPE hardware. However, what I've
got is a Buffalo H300N, so I'll need build instructions (preferably
starting from an existing stock build) as well as setup.

The Buffalo isn't as powerful as some others, being based around a 34K core.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 368 bytes --]

[Cerowrt-devel] Build instructions for regular OpenWRT with Ceropackages

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, dpreed@reed.com wrote:

> I would love to try out cake in my environment.  However, as a 
> non-combatant, it would be nice to have an instruction sheet on how to 
> set the latest version up, and what hardware it works best on 
> (WRT1200AC?).  Obviously this is a work in progress, so that will 
> change, but it would be nice to have a summarized wiki page.

WRT1200AC seems to be the most powerful around, however it can't really be 
used for PHY 100M testing since both SoC ports goes to a switch that then 
terminate all the external ports. Therefore it's hard to test 
AQM-on-metal with it because the SoC links are never saturated, you always 
have to use an encompassing shaper (htb).

Here is a short writeup from what I learnt from the past days. I haven't 
verified every step, this is from memory.

Get ubuntu 14.04 LTS according to:

http://www.acme-dot.com/building-openwrt-14-07-barrier-breaker-on-ubuntu-and-os-x/

Check out either trunk ("git clone git://git.openwrt.org/openwrt.git") or 
Chaos Calmer RC ("clone git://git.openwrt.org/15.05/openwrt.git").

Copy feeds.conf.default to feeds.conf in the openwrt dir. Add first in 
file:

"src-git cero https://github.com/dtaht/ceropackages-3.10.git"

scripts/feeds update -a
scripts/feeds install luci luci-app-sqm sqm-scripts tc-adv ip ethtool kmod-sched-cake kmod-sched-fq_pie

make menuconfig

Now comes the hard part because you want to change * for everything that 
you want to install as default in the resulting image. M means it compiles 
the package but doesn't include it in the resulting image (for utilities). 
What the above does is only to make it available to "make menuconfig" as 
packages. I tend to choose traceroute, tcpdump and all the other nice to 
have utilities. You can download and use 
<http://swm.pp.se/aqm/wrt1200ac.config> (it's for trunk, ie the nightly, 
don't know if it works for CC RC2) and put as .config in the openwrt 
directory as a template.

I recommend to build on a fast machine with SSD, i/o is usually the 
limiting factor. I use an 3.5GHz core i5 dual core (4 with HT) and SSD, 
and with "make -j 10" it compiles in an hour or so. Subsequent compiles 
are quicker.

You need to find your platform etc. If you get an WRT1200AC you can use my 
builds if you want to. After this you need to go into the luci sqm scripts 
and set queueing algorithm etc.

some good commands to see what's going on:

tc -d qdisc
tc -s qdisc

Hope it helps.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] Build instructions for regular OpenWRT with Ceropackages

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 2934 bytes --]


What happens if the SoC ports aren't saturated, but the link is GigE?  That is, suppose this is an access link to a GigE home or office LAN with wired servers?


On Tuesday, June 30, 2015 9:58am, "Mikael Abrahamsson" <swmike@swm.pp.se> said:



> On Mon, 29 Jun 2015, dpreed@reed.com wrote:
> 
> > I would love to try out cake in my environment. However, as a
> > non-combatant, it would be nice to have an instruction sheet on how to
> > set the latest version up, and what hardware it works best on
> > (WRT1200AC?). Obviously this is a work in progress, so that will
> > change, but it would be nice to have a summarized wiki page.
> 
> WRT1200AC seems to be the most powerful around, however it can't really be
> used for PHY 100M testing since both SoC ports goes to a switch that then
> terminate all the external ports. Therefore it's hard to test
> AQM-on-metal with it because the SoC links are never saturated, you always
> have to use an encompassing shaper (htb).
> 
> Here is a short writeup from what I learnt from the past days. I haven't
> verified every step, this is from memory.
> 
> Get ubuntu 14.04 LTS according to:
> 
> http://www.acme-dot.com/building-openwrt-14-07-barrier-breaker-on-ubuntu-and-os-x/
> 
> Check out either trunk ("git clone git://git.openwrt.org/openwrt.git") or
> Chaos Calmer RC ("clone git://git.openwrt.org/15.05/openwrt.git").
> 
> Copy feeds.conf.default to feeds.conf in the openwrt dir. Add first in
> file:
> 
> "src-git cero https://github.com/dtaht/ceropackages-3.10.git"
> 
> scripts/feeds update -a
> scripts/feeds install luci luci-app-sqm sqm-scripts tc-adv ip ethtool
> kmod-sched-cake kmod-sched-fq_pie
> 
> make menuconfig
> 
> Now comes the hard part because you want to change * for everything that
> you want to install as default in the resulting image. M means it compiles
> the package but doesn't include it in the resulting image (for utilities).
> What the above does is only to make it available to "make menuconfig" as
> packages. I tend to choose traceroute, tcpdump and all the other nice to
> have utilities. You can download and use
> <http://swm.pp.se/aqm/wrt1200ac.config> (it's for trunk, ie the nightly,
> don't know if it works for CC RC2) and put as .config in the openwrt
> directory as a template.
> 
> I recommend to build on a fast machine with SSD, i/o is usually the
> limiting factor. I use an 3.5GHz core i5 dual core (4 with HT) and SSD,
> and with "make -j 10" it compiles in an hour or so. Subsequent compiles
> are quicker.
> 
> You need to find your platform etc. If you get an WRT1200AC you can use my
> builds if you want to. After this you need to go into the luci sqm scripts
> and set queueing algorithm etc.
> 
> some good commands to see what's going on:
> 
> tc -d qdisc
> tc -s qdisc
> 
> Hope it helps.
> 
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
> 

[-- Attachment #2: Type: text/html, Size: 3892 bytes --]

Re: [Cerowrt-devel] Build instructions for regular OpenWRT with Ceropackages

[Mikael Abrahamsson]

On Tue, 30 Jun 2015, dpreed@reed.com wrote:

> What happens if the SoC ports aren't saturated, but the link is GigE? 
> That is, suppose this is an access link to a GigE home or office LAN 
> with wired servers?

As far as I can tell, the device looks like this:

wifi2------
wifi1----\|
 	SOC2 6-|
 	SOC1 5-|
 	WAN  4-|
 	LAN1 3-| (switch)
 	LAN2 2-|
 	LAN3 1-|
 	LAN4 0-|

LAN1-4 and SOC2 is in one vlan, and SOC1 and WAN is in a second vlan. This 
basically means there is no way to get traffic into SOC1 that goes out 
SOC2 that will saturate either port, because they're both gige. Only way 
to saturate the SOC port would be if the SOC itself "created" traffic, for 
instance by being a fileserver, or if there is significant traffic on the 
wifi (which has PCI-E connectivity).

So it's impossible to congest SOC1 or SOC2 (egress) by running traffic 
LAN<->WAN alone.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] Build instructions for regular OpenWRT with Ceropackages

[David Lang]

On Tue, 30 Jun 2015, Mikael Abrahamsson wrote:

> On Tue, 30 Jun 2015, dpreed@reed.com wrote:
>
>> What happens if the SoC ports aren't saturated, but the link is GigE? That 
>> is, suppose this is an access link to a GigE home or office LAN with wired 
>> servers?
>
> As far as I can tell, the device looks like this:
>
> wifi2------
> wifi1----\|
> 	SOC2 6-|
> 	SOC1 5-|
> 	WAN  4-|
> 	LAN1 3-| (switch)
> 	LAN2 2-|
> 	LAN3 1-|
> 	LAN4 0-|
>
> LAN1-4 and SOC2 is in one vlan, and SOC1 and WAN is in a second vlan. This 
> basically means there is no way to get traffic into SOC1 that goes out SOC2 
> that will saturate either port, because they're both gige. Only way to 
> saturate the SOC port would be if the SOC itself "created" traffic, for 
> instance by being a fileserver, or if there is significant traffic on the 
> wifi (which has PCI-E connectivity).
>
> So it's impossible to congest SOC1 or SOC2 (egress) by running traffic 
> LAN<->WAN alone.

not true, the switch doesn't give any way for traffic to get from one vlan to 
the other one, so if you have gig-e connections on both sides, the traffic going 
from one to the other will have to go through the soc, so if there is more than 
1Gb of traffic in either direction the interface will be saturated.

The problem is if you have a slower connection, the bottleneck is in the switch 
not the soc. you may be able to set the soc-switch interface to 100Mb (make sure 
you have access through another interface in case you cut yourself off) and that 
would make the soc see the queue directly.

David Lang

Re: [Cerowrt-devel] Build instructions for regular OpenWRT with Ceropackages

[Mikael Abrahamsson]

On Wed, 1 Jul 2015, David Lang wrote:

> not true, the switch doesn't give any way for traffic to get from one vlan to 
> the other one, so if you have gig-e connections on both sides, the traffic 
> going from one to the other will have to go through the soc, so if there is 
> more than 1Gb of traffic in either direction the interface will be saturated.

I don't see how it can be. The switch has a 4 externally facing ports, 
these all go to a single SoC port that is GigE, so the SoC cannot ingest 
more than 1G of traffic from the 4 LAN ports. The L2 switch chip will do 
the egress dropping from LAN ports->SoC ports, so there is no AQM there.

> The problem is if you have a slower connection, the bottleneck is in the 
> switch not the soc. you may be able to set the soc-switch interface to 100Mb 
> (make sure you have access through another interface in case you cut yourself 
> off) and that would make the soc see the queue directly.

That is my point. There is no way by doing traffic LAN<->WAN to get egress 
congestion on the SoC ports, and it's on the SOC ports we can do AQM.

The SoC ports is gigabit ethernet only, no 10/100 available according to 
ethtool.

So the only way to generate congestion egress on the WAN SOC port is to 
add traffic locally from the SoC (iperf3 for instance), or adding traffic 
from Wifi.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] Build instructions for regular OpenWRT with Ceropackages

[Sebastian Moeller]

On Jul 1, 2015, at 12:32 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Wed, 1 Jul 2015, David Lang wrote:
> 
>> not true, the switch doesn't give any way for traffic to get from one vlan to the other one, so if you have gig-e connections on both sides, the traffic going from one to the other will have to go through the soc, so if there is more than 1Gb of traffic in either direction the interface will be saturated.
> 
> I don't see how it can be.

	If “saturated” means running at line rate, then I agree with David.

> The switch has a 4 externally facing ports, these all go to a single SoC port that is GigE, so the SoC cannot ingest more than 1G of traffic from the 4 LAN ports.

	But this is not SoC specific, no? If the router only had two ethernet ports and the LAN aggregation would happen at a dedicated switch between the clients and the router’s lan port the situation would be the same?

> The L2 switch chip will do the egress dropping from LAN ports->SoC ports, so there is no AQM there.

	Ingress/egress are always relative to something I guess. You can put AQM on any of the two SoC ports (or both) but that will not affect how the switch copes with overloads, unless the switch grows AQM (as Dave mentioned in earlier threads). I would hope that the LAN’s short RTTs and high bandwidth should make this bearable.

> 
>> The problem is if you have a slower connection, the bottleneck is in the switch not the soc. you may be able to set the soc-switch interface to 100Mb (make sure you have access through another interface in case you cut yourself off) and that would make the soc see the queue directly.
> 
> That is my point. There is no way by doing traffic LAN<->WAN to get egress congestion on the SoC ports, and it's on the SOC ports we can do AQM.

	Unless you reduce the egress rate to below the ingress rate. This really is why we need the soft shapers at all, otherwise the buffering moves from where we have control back into the DSLAMs/CMTSs/BRASs and CPE.

> 
> The SoC ports is gigabit ethernet only, no 10/100 available according to ethtool.
> 
> So the only way to generate congestion egress on the WAN SOC port is to add traffic locally from the SoC (iperf3 for instance), or adding traffic from Wifi.

	Well, that os shaping the other direction, so that more data can enter the router than leave. It is all a balancing game in the end…

Best Regards
	Sebastian


> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] Build instructions for regular OpenWRT with Ceropackages

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]


Mikael, very very helpful, thanks.
 
I now understand what you are trying to prove/test in your experiments, but there is definitely a need for cake when the dominant use is hi-bitrate WiFi (AC1900) talking to one or more 1 GigE wired paths.  And hi bitrate WiFi itself has significantly variable rate capability so it probably needs more feedback than cake might provide to deal with variability.
 
Since 100+ Mb/sec is supplied by many Internet Access Providers now,  it's timely to be able to process packets coming at those rates on the wireline side carried over 1 GigE (my provider, RCN, claims to offer 110+ here in Needham, but with an odd requirement that I buy their router if I get that service - I am trying to get to the bottom of what that is before I upgrade or switch to one of the two other providers, Comcast and Verizon. Maybe it is just that they want customers to not get screwed up if they have a router without 1GigE WAN adapter, complaining that they can't get 110.).
 
I'd like to see both whatever I get from the IAP, and also what my in-home NAS can provide, along with other services.


On Tuesday, June 30, 2015 3:58pm, "Mikael Abrahamsson" <swmike@swm.pp.se> said:



> On Tue, 30 Jun 2015, dpreed@reed.com wrote:
> 
> > What happens if the SoC ports aren't saturated, but the link is GigE?
> > That is, suppose this is an access link to a GigE home or office LAN
> > with wired servers?
> 
> As far as I can tell, the device looks like this:
> 
> wifi2------
> wifi1----\|
> SOC2 6-|
> SOC1 5-|
> WAN 4-|
> LAN1 3-| (switch)
> LAN2 2-|
> LAN3 1-|
> LAN4 0-|
> 
> LAN1-4 and SOC2 is in one vlan, and SOC1 and WAN is in a second vlan. This
> basically means there is no way to get traffic into SOC1 that goes out
> SOC2 that will saturate either port, because they're both gige. Only way
> to saturate the SOC port would be if the SOC itself "created" traffic, for
> instance by being a fileserver, or if there is significant traffic on the
> wifi (which has PCI-E connectivity).
> 
> So it's impossible to congest SOC1 or SOC2 (egress) by running traffic
> LAN<->WAN alone.
> 
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
> 

[-- Attachment #2: Type: text/html, Size: 3591 bytes --]

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

HI Mikael,, hi Jonathan,

> [...]
> 
> These are the results from 50M and 500M, also including 50up and 50down that I added to my test suite script.
> 
> http://swm.pp.se/aqm/rrul_150629-cake-4.tar
> 

Now both ingress and egress are up to roughly 455Mbps from roughly 360 with cake just playing leaf qdisc for HTB. This looks even better than before…

Best Regards
	Sebastian

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2399 bytes --]

On Mon, Jun 29, 2015 at 6:42 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> HI Mikael,, hi Jonathan,
>
>> [...]
>>
>> These are the results from 50M and 500M, also including 50up and 50down that I added to my test suite script.
>>
>> http://swm.pp.se/aqm/rrul_150629-cake-4.tar
>>
>
> Now both ingress and egress are up to roughly 455Mbps from roughly 360 with cake just playing leaf qdisc for HTB. This looks even better than before…

350 *usec* induced delay on the 50mbit rrul_be test. w00t!

Most of the tests compare well with the reference rangeley data now. I
would like a 900mbit soft shaped result.

1.2ms at 500mbit. Less of a w00t. Possible it is coming from elsewhere
on that path (fq or fq_codel on the server and client?)

cake currently peels at 1ms / flows (more or less)... NAPI is an
issue... hw mq an issue...

There are a half dozen things in the mvneta driver I would try to
reduce it's latency more. The easy ones:

reduce this to 16:

        netif_napi_add(dev, &pp->napi, mvneta_poll, NAPI_POLL_WEIGHT);

Reduce this to 24: (this will also reduce the max outstanding stuff in
the driver by  a LOT, but is still not BQL!)

/* Max number of allowed TCP segments for software TSO */
#define MVNETA_MAX_TSO_SEGS 100

Both of the will improve read side latency at the cost of more sirqs.

I do not know what reducing these will do, and would test both of the
above separately.

/* Coalescing */
#define MVNETA_TXDONE_COAL_PKTS         1
#define MVNETA_RX_COAL_PKTS             32
#define MVNETA_RX_COAL_USEC             100

As for cake itself, eric dumazet told us we dont need atomic ops in it,
and peeling at at even lower threshold has some appeal (to me, anyway)

attached is a patch for that, put it in your feeds/cero/kmod_sched_cake/patches
directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})

(bump up the makefile rel number also, if you want)





> Best Regards
>         Sebastian
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[-- Attachment #2: 0001-Rid-unneeded-atomic-ops-and-reduce-peeling-threshold.patch --]
[-- Type: text/x-patch, Size: 2510 bytes --]

From 46be609e95474e9db856b5e12756d4a7568adf42 Mon Sep 17 00:00:00 2001
From: Dave Taht <dave.taht@bufferbloat.net>
Date: Mon, 29 Jun 2015 09:38:00 -0700
Subject: [PATCH] Rid unneeded atomic ops and reduce peeling threshold

---
 sch_cake.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/sch_cake.c b/sch_cake.c
index 80e1cb2..9a358b9 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -121,7 +121,7 @@ struct cake_fqcd_sched_data {
 
     struct codel_params cparams;
     u32      drop_overlimit;
-    atomic_t flow_count;
+    u32 flow_count;
 
     struct list_head new_flows; /* list of new flows */
     struct list_head old_flows; /* list of old flows */
@@ -427,7 +427,7 @@ static int cake_enqueue(struct sk_buff *skb, struct Qdisc *sch)
 	 * Split GSO aggregates if they're likely to impair flow isolation
 	 * or if we need to know individual packet sizes for framing overhead.
 	 */
-	if(unlikely((len * max(atomic_read(&fqcd->flow_count), 1)) > q->peel_threshold && skb_is_gso(skb)))
+	if(unlikely((len * max(&fqcd->flow_count, 1)) > q->peel_threshold && skb_is_gso(skb)))
 	{
 		struct sk_buff *segs, *nskb;
 		netdev_features_t features = netif_skb_features(skb);
@@ -477,7 +477,7 @@ static int cake_enqueue(struct sk_buff *skb, struct Qdisc *sch)
 	/* flowchain */
 	if(list_empty(&flow->flowchain)) {
 		list_add_tail(&flow->flowchain, &fqcd->new_flows);
-		atomic_inc(&fqcd->flow_count);
+		fqcd->flow_count+=1;
 		flow->deficit = fqcd->quantum;
 		flow->dropped = 0;
 	}
@@ -615,7 +615,7 @@ retry:
 			list_move_tail(&flow->flowchain, &fqcd->old_flows);
 		} else {
 			list_del_init(&flow->flowchain);
-			atomic_dec(&fqcd->flow_count);
+			fqcd->flow_count-=1;
 		}
 		goto begin;
 	}
@@ -966,7 +966,7 @@ static void cake_reconfigure(struct Qdisc *sch)
 		if(q->buffer_limit < 65536)
 			q->buffer_limit = 65536;
 
-		q->peel_threshold = (q->rate_flags & CAKE_FLAG_ATM) ? 0 : min(65535U, q->rate_bps >> 10);
+		q->peel_threshold = (q->rate_flags & CAKE_FLAG_ATM) ? 0 : min(65535U, q->rate_bps >> 12);
 	} else {
 		q->buffer_limit = 1 << 20;
 		q->peel_threshold = 0;
@@ -1083,7 +1083,7 @@ static int cake_init(struct Qdisc *sch, struct nlattr *opt)
 		fqcd->perturbation = prandom_u32();
 		INIT_LIST_HEAD(&fqcd->new_flows);
 		INIT_LIST_HEAD(&fqcd->old_flows);
-		atomic_set(&fqcd->flow_count, 0);
+		fqcd->flow_count = 0;
 		/* codel_params_init(&fqcd->cparams); */
 
 		fqcd->flows    = cake_zalloc(fqcd->flows_cnt * sizeof(struct cake_fqcd_flow));
-- 
1.9.1

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 3875 bytes --]

Hi List,

On Jun 29, 2015, at 18:44 , Dave Taht <dave.taht@gmail.com> wrote:

> On Mon, Jun 29, 2015 at 6:42 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> HI Mikael,, hi Jonathan,
>> 
>>> [...]
>>> 
>>> These are the results from 50M and 500M, also including 50up and 50down that I added to my test suite script.
>>> 
>>> http://swm.pp.se/aqm/rrul_150629-cake-4.tar
>>> 
>> 
>> Now both ingress and egress are up to roughly 455Mbps from roughly 360 with cake just playing leaf qdisc for HTB. This looks even better than before…
> 
> 350 *usec* induced delay on the 50mbit rrul_be test. w00t!
> 
> Most of the tests compare well with the reference rangeley data now. I
> would like a 900mbit soft shaped result.

	Make sure to use the correct per packet overhead of 18 + 20, on gigabit ethernet inter-frame gap and preamble cost 20 Bytes worth of data. So with a MTU of 1500 thee is no issue (900 * (1538/1518) = 911.85770751 Mbps) but the smaller the packets get:

900 * (MTU+38/MTU+18) = 1000
(MTU+38/MTU+18) = (1000 / 900)
MTU+38 = (10 / 9) * (MTU + 18)
MTU + 38 = (10/9) * MTU + (10/9)*18
38 - (10/9)*18 = (10/9) * MTU - (9/9)MTU
38 - (10/9)*18 = (1/9) MTU
MTU = (38 - ((10/9)*18))*9  = 162
So for TCP/IPv4 MSS < 122 the shaper will not keep the ethernet hardware queues empty… 

On the other hand shaping at 
1000/(88/64) = 727.272727273 Mbps
should make sure that even at minimal packet size of 64byte shaping would still be keeping the ethernet queues “empty-ish”. If the 1200ac can shape at 900 I would rather specify the correct overhead though.


To make things a bit trickier, depending on the interface used the kernel will already account for the standards ethernet header without the frame check sequence, so I would guess in the 900Mbps soft shaper on ethN scenario one would need to add a per packet overhead of 24 bytes. If someone in the know could double check that reasoning I would be much obliged…


Best Regards
	Sebastian



> 
> 1.2ms at 500mbit. Less of a w00t. Possible it is coming from elsewhere
> on that path (fq or fq_codel on the server and client?)
> 
> cake currently peels at 1ms / flows (more or less)... NAPI is an
> issue... hw mq an issue...
> 
> There are a half dozen things in the mvneta driver I would try to
> reduce it's latency more. The easy ones:
> 
> reduce this to 16:
> 
>        netif_napi_add(dev, &pp->napi, mvneta_poll, NAPI_POLL_WEIGHT);
> 
> Reduce this to 24: (this will also reduce the max outstanding stuff in
> the driver by  a LOT, but is still not BQL!)
> 
> /* Max number of allowed TCP segments for software TSO */
> #define MVNETA_MAX_TSO_SEGS 100
> 
> Both of the will improve read side latency at the cost of more sirqs.
> 
> I do not know what reducing these will do, and would test both of the
> above separately.
> 
> /* Coalescing */
> #define MVNETA_TXDONE_COAL_PKTS         1
> #define MVNETA_RX_COAL_PKTS             32
> #define MVNETA_RX_COAL_USEC             100
> 
> As for cake itself, eric dumazet told us we dont need atomic ops in it,
> and peeling at at even lower threshold has some appeal (to me, anyway)
> 
> attached is a patch for that, put it in your feeds/cero/kmod_sched_cake/patches
> directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})
> 
> (bump up the makefile rel number also, if you want)
> 
> 
> 
> 
> 
>> Best Regards
>>        Sebastian
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> 
> 
> -- 
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Rid-unneeded-atomic-ops-and-reduce-peeling-threshold.patch --]
[-- Type: text/x-patch; name="0001-Rid-unneeded-atomic-ops-and-reduce-peeling-threshold.patch", Size: 2580 bytes --]

From 46be609e95474e9db856b5e12756d4a7568adf42 Mon Sep 17 00:00:00 2001
From: Dave Taht <dave.taht@bufferbloat.net>
Date: Mon, 29 Jun 2015 09:38:00 -0700
Subject: [PATCH] Rid unneeded atomic ops and reduce peeling threshold

---
 sch_cake.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/sch_cake.c b/sch_cake.c
index 80e1cb2..9a358b9 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -121,7 +121,7 @@ struct cake_fqcd_sched_data {
 
     struct codel_params cparams;
     u32      drop_overlimit;
-    atomic_t flow_count;
+    u32 flow_count;
 
     struct list_head new_flows; /* list of new flows */
     struct list_head old_flows; /* list of old flows */
@@ -427,7 +427,7 @@ static int cake_enqueue(struct sk_buff *skb, struct Qdisc *sch)
 	 * Split GSO aggregates if they're likely to impair flow isolation
 	 * or if we need to know individual packet sizes for framing overhead.
 	 */
-	if(unlikely((len * max(atomic_read(&fqcd->flow_count), 1)) > q->peel_threshold && skb_is_gso(skb)))
+	if(unlikely((len * max(&fqcd->flow_count, 1)) > q->peel_threshold && skb_is_gso(skb)))
 	{
 		struct sk_buff *segs, *nskb;
 		netdev_features_t features = netif_skb_features(skb);
@@ -477,7 +477,7 @@ static int cake_enqueue(struct sk_buff *skb, struct Qdisc *sch)
 	/* flowchain */
 	if(list_empty(&flow->flowchain)) {
 		list_add_tail(&flow->flowchain, &fqcd->new_flows);
-		atomic_inc(&fqcd->flow_count);
+		fqcd->flow_count+=1;
 		flow->deficit = fqcd->quantum;
 		flow->dropped = 0;
 	}
@@ -615,7 +615,7 @@ retry:
 			list_move_tail(&flow->flowchain, &fqcd->old_flows);
 		} else {
 			list_del_init(&flow->flowchain);
-			atomic_dec(&fqcd->flow_count);
+			fqcd->flow_count-=1;
 		}
 		goto begin;
 	}
@@ -966,7 +966,7 @@ static void cake_reconfigure(struct Qdisc *sch)
 		if(q->buffer_limit < 65536)
 			q->buffer_limit = 65536;
 
-		q->peel_threshold = (q->rate_flags & CAKE_FLAG_ATM) ? 0 : min(65535U, q->rate_bps >> 10);
+		q->peel_threshold = (q->rate_flags & CAKE_FLAG_ATM) ? 0 : min(65535U, q->rate_bps >> 12);
 	} else {
 		q->buffer_limit = 1 << 20;
 		q->peel_threshold = 0;
@@ -1083,7 +1083,7 @@ static int cake_init(struct Qdisc *sch, struct nlattr *opt)
 		fqcd->perturbation = prandom_u32();
 		INIT_LIST_HEAD(&fqcd->new_flows);
 		INIT_LIST_HEAD(&fqcd->old_flows);
-		atomic_set(&fqcd->flow_count, 0);
+		fqcd->flow_count = 0;
 		/* codel_params_init(&fqcd->cparams); */
 
 		fqcd->flows    = cake_zalloc(fqcd->flows_cnt * sizeof(struct cake_fqcd_flow));
-- 
1.9.1


[-- Attachment #3: Type: text/plain, Size: 4 bytes --]

> 

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, Dave Taht wrote:

> Most of the tests compare well with the reference rangeley data now. I
> would like a 900mbit soft shaped result.

http://swm.pp.se/aqm/rrul_150629-cake-9.tar

Above is with cake as link layer adaptation and pie, fq_codel and cake as 
900 meg bidirectional shaped.

Then I removed the ll adaptation and changed to simplest.qos and ran 
another test with cake:

http://swm.pp.se/aqm/rrul_150629-cake-10.tar

Now I'm going to try out if my new version with your cake patch will boot 
and work.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, Dave Taht wrote:

> attached is a patch for that, put it in your feeds/cero/kmod_sched_cake/patches
> directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})

Test results: http://swm.pp.se/aqm/rrul_150629-cake-11.tar

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, Dave Taht wrote:

> attached is a patch for that, put it in your feeds/cero/kmod_sched_cake/patches
> directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})

I compiled openwrt trunk with linux kernel v4.0 and this patch, and the 
results are here <http://swm.pp.se/aqm/rrul_150630-cake-l4.0-1.tar>.

As far as I can tell sirq load is higher rather than lower so it doesn't 
seem like kernel 4.0 has any significant performance benefits, rather the 
opposite.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Tue, 30 Jun 2015, Mikael Abrahamsson wrote:

> On Mon, 29 Jun 2015, Dave Taht wrote:
>
>> attached is a patch for that, put it in your 
>> feeds/cero/kmod_sched_cake/patches
>> directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})
>
> I compiled openwrt trunk with linux kernel v4.0 and this patch, and the 
> results are here <http://swm.pp.se/aqm/rrul_150630-cake-l4.0-1.tar>.
>
> As far as I can tell sirq load is higher rather than lower so it doesn't seem 
> like kernel 4.0 has any significant performance benefits, rather the 
> opposite.

So it seems I was mistaken. I now did some tests, with 4.0 with the 
cakepatch, with 3.18 with the cakepatch (that Dave sent the other day), 
and then 3.18 with regular cake as it is in the ceropackages repo 
yesterday.

The columns are mss size, -R or not means reverse, so with -R main packet 
sizes are going in the server->client direction, ie flowing into eth0 
which hosts the htb. Then it's the megabit/s as measured by iperf and then 
the sirq load as seen by top. This jumps around a bit so don't read too 
much into it. However, it looks like 4.0 is actually a slight improvement 
especially for smaller packet sizes and in the server-client direction.

4.0 cakepatch

-M 200 -R   167 M 94%
-M 300 -R   188 M 71%
-M 600 -R   362 M 77%
        -R   861 M 88%
-M 200      350 M 88%
-M 300      380 M 80%
-M 600      680 M 63%
             860 M 55%

3.18 - cakepatch

-M 200 -R   140M 83%
-M 300 -R   167M 72%
-M 600 -R   308M 69%
        -R   750M 82%
-M 200      289M 74%
-M 300      406M 73%
-M 600      780M 80%
             860M 57%

3.18 vanilla

-M 200 -R  150M 90%
-M 300 -R  166M 72%
-M 600 -R  305M 68%
-M     -R  740M 82%
-M 200     304M 80%
-M 300     440M 80%
-M 600     800M 81%
            863M 56%

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

Hi,

for the sake of spreading this caveat, it does seem like the open source 
driver for wifi in the WRT1200AC isn't in very good shape. It works, but 
it's very slow, speeds vary depending on what kind of client I connect 
with, and the latency is horrible even with fq_codel or cake enabled on 
it.

So if you were considering buying it and using it as a "daily driver" for 
wifi in your home, wait until I have better news regarding the wifi 
driver. I've already ping:ed Marvell about it and also spoken to one 
developer working for Belkin/Linksys, and they're aware of the problem.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:

> So if you were considering buying it and using it as a "daily driver"
> for wifi in your home, wait until I have better news regarding the
> wifi driver. I've already ping:ed Marvell about it and also spoken to
> one developer working for Belkin/Linksys, and they're aware of the
> problem.

Do keep us up to date on this. I have one on my desk ready to be flashed
once I get a bit of free time. Don't have anyone else to annoy with
possible poor performance, though, so can live with experimental
drivers :)

Oh, and of course I volunteer to test any such new drivers. Hmm, maybe I
should try to get my hands on a 802.11ac *client* as well... :P

Do you have a link to your .config for your builds somewhere?

-Toke

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 658 bytes --]

On Thu, 2 Jul 2015, Toke Høiland-Jørgensen wrote:

> Do you have a link to your .config for your builds somewhere?

http://swm.pp.se/aqm/wrt1200ac.config

BUT! I have had problems getting WPA2 to work properly with this .config. 
I must have missed something that is needed that has to do with the 
password/crypto handling.

There already is a profile for the WRT1200AC (caiman) in Chaos 
Calmer RC and trunk, so it's actually not that hard to get working. The 
biggest problem is finding all those utilities one wants and making sure 
they're compiled into the image so one doesn't have to add them later.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:

>> Do you have a link to your .config for your builds somewhere?
>
> http://swm.pp.se/aqm/wrt1200ac.config

Cool, thanks!

> BUT! I have had problems getting WPA2 to work properly with this
> .config. I must have missed something that is needed that has to do
> with the password/crypto handling.
>
> There already is a profile for the WRT1200AC (caiman) in Chaos Calmer
> RC and trunk, so it's actually not that hard to get working. The
> biggest problem is finding all those utilities one wants and making
> sure they're compiled into the image so one doesn't have to add them
> later.

Yeah, realise that. Still have my old .config from when I used to build
cerowrt for the WNDR lying around somewhere, so will take a look at
that and make sure everything is in there :)

-Toke

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 1579 bytes --]


Having not bought a 1200ac yet, I was wondering if I should splurge for the 1900ac v2 (which has lots of memory unlike the 1900ac v1).
 
Any thoughts on the compatibility of this with the 1200ac?
 
Current plans are to deploy Supermicro Mini ITX A1SRI-2558F-O Quad Core (Rangely) as my externally facing router and "services" platform, and either one of the above as my experimental wireless solution.



On Thursday, July 2, 2015 11:47am, "Toke Høiland-Jørgensen" <toke@toke.dk> said:



> Mikael Abrahamsson <swmike@swm.pp.se> writes:
> 
> >> Do you have a link to your .config for your builds somewhere?
> >
> > http://swm.pp.se/aqm/wrt1200ac.config
> 
> Cool, thanks!
> 
> > BUT! I have had problems getting WPA2 to work properly with this
> > .config. I must have missed something that is needed that has to do
> > with the password/crypto handling.
> >
> > There already is a profile for the WRT1200AC (caiman) in Chaos Calmer
> > RC and trunk, so it's actually not that hard to get working. The
> > biggest problem is finding all those utilities one wants and making
> > sure they're compiled into the image so one doesn't have to add them
> > later.
> 
> Yeah, realise that. Still have my old .config from when I used to build
> cerowrt for the WNDR lying around somewhere, so will take a look at
> that and make sure everything is in there :)
> 
> -Toke
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 2039 bytes --]

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Thu, 2 Jul 2015, dpreed@reed.com wrote:

> Having not bought a 1200ac yet, I was wondering if I should splurge for 
> the 1900ac v2 (which has lots of memory unlike the 1900ac v1).

From what I can tell, the only thing that differs from the WRT1200AC is 
the radio. It still uses marvell radio but with more "everything", apart 
from RAM that (for some reason) the WRT1200AC has 512MB RAM but according 
to http://wiki.openwrt.org/toh/linksys/wrt1900ac the WRT1900ACv2 only has 
256MB RAM.

I don't have any information on the performance of the radio in the 
WRT1900ACv2.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[David Lang]

[-- Attachment #1: Type: TEXT/Plain, Size: 1882 bytes --]

It looks like the 1900v2 and the 1200 have the same chipset, i saw that the 
1900v2 got more memory and a faster cpu to bring it up to match the 1200.

My understanding is that the only difference between the two is 2x2 vs 3x3 and 
the cost.

David Lang


  On Thu, 2 Jul 2015, dpreed@reed.com wrote:

> Having not bought a 1200ac yet, I was wondering if I should splurge for the 1900ac v2 (which has lots of memory unlike the 1900ac v1).
> 
> Any thoughts on the compatibility of this with the 1200ac?
> 
> Current plans are to deploy Supermicro Mini ITX A1SRI-2558F-O Quad Core (Rangely) as my externally facing router and "services" platform, and either one of the above as my experimental wireless solution.
>
>
>
> On Thursday, July 2, 2015 11:47am, "Toke Høiland-Jørgensen" <toke@toke.dk> said:
>
>
>
>> Mikael Abrahamsson <swmike@swm.pp.se> writes:
>> 
>> >> Do you have a link to your .config for your builds somewhere?
>> >
>> > http://swm.pp.se/aqm/wrt1200ac.config
>> 
>> Cool, thanks!
>> 
>> > BUT! I have had problems getting WPA2 to work properly with this
>> > .config. I must have missed something that is needed that has to do
>> > with the password/crypto handling.
>> >
>> > There already is a profile for the WRT1200AC (caiman) in Chaos Calmer
>> > RC and trunk, so it's actually not that hard to get working. The
>> > biggest problem is finding all those utilities one wants and making
>> > sure they're compiled into the image so one doesn't have to add them
>> > later.
>> 
>> Yeah, realise that. Still have my old .config from when I used to build
>> cerowrt for the WNDR lying around somewhere, so will take a look at
>> that and make sure everything is in there :)
>> 
>> -Toke
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Rich Brown]

Folks,

>> ... The biggest problem is finding all those utilities one wants and making
>> sure they're compiled into the image so one doesn't have to add them
>> later.
> 
> Yeah, realise that. Still have my old .config from when I used to build
> cerowrt for the WNDR lying around somewhere, so will take a look at
> that and make sure everything is in there :)

The CeroWrtScripts repo also has config-cerowrt.sh, a shell script that I run immediately after flashing to configure a flock of things.

The second step (after configuring and bringing up ge00) is to pull in all the packages I want to use. This might save a bunch of wrasslin' with the build scripts/configuration. Check the script at:  https://github.com/richb-hanover/CeroWrtScripts#config-cerowrtsh

Best,

Rich

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Toke Høiland-Jørgensen]

Rich Brown <richb.hanover@gmail.com> writes:

> The CeroWrtScripts repo also has config-cerowrt.sh, a shell script
> that I run immediately after flashing to configure a flock of things.

Well I tend to build the config into the image, i.e. build an image
specifically for the box I want to install. This has the nice property
of allowing me to (remotely) reflash my main gateway and have it come up
fully configured.

Except when I mess up, of course. But that never happens! :P

-Toke

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Thu, 2 Jul 2015, Mikael Abrahamsson wrote:

> for the sake of spreading this caveat, it does seem like the open source 
> driver for wifi in the WRT1200AC isn't in very good shape. It works, but 
> it's very slow, speeds vary depending on what kind of client I connect 
> with, and the latency is horrible even with fq_codel or cake enabled on 
> it.

I have now done some more testing, and it seems it was my compiles that 
must have screwed something up. With the OpenWrt CC RC2 kernel, I get 100+ 
of megabits/s of throughput even with my USB3 AC1300 dongle. Notice, I am 
not sure the port I have on my laptop is USB3, so it might very well be 
limited by the USB(2) bus.

Here is my usual test suite run on the OpenWrt CC RC2 kernel, fq_codel at 
ubuntu 15.04 client and on the WRT1200AC.

http://swm.pp.se/aqm/rrul_usb1300-150703.tar

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Sun, 28 Jun 2015, Sebastian Moeller wrote:

After the modification, the installed policy looks better, I am now not 
getting any drops in iperf3 (=ECN is working). Please see link to test 
results. From my bw monitoring script, it looks like I am receiving more 
than 500 megabit/s though, even though sending is spot on at 500 
megabit/s. Yeah, the receiving limiter isn't working at all, I see now in 
rrul_be that I am getting full gige speed download.

root@OpenWrt:~# tc -d qdisc
qdisc cake 8005: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows 
raw
  linklayer ethernet overhead 42
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 
quantum 300 target 5.0ms interval 100.0ms ecn
root@OpenWrt:~# tc -d class show dev eth0
class cake 8005:6 parent 8005:
class cake 8005:44b parent 8005:
class cake 8005:50c parent 8005:
class cake 8005:5fc parent 8005:
class cake 8005:716 parent 8005:
class cake 8005:7b3 parent 8005:
class cake 8005:b43 parent 8005:
class cake 8005:c7f parent 8005:
class cake 8005:c87 parent 8005:
class cake 8005:d8d parent 8005:
root@OpenWrt:~# tc -d class show dev ifb4eth0
class fq_codel :32c parent none

I have put the results at http://swm.pp.se/aqm/rrul_150629-cake-1.tar

I also have a small shell script that prints out pps and bw once per 
second that I run on my ubuntu machine, just to see approx how much 
traffic is being used. Might be useful for someone.

http://swm.pp.se/aqm/measurebw.sh

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

Hi Mikael,

thanks a lot.

On Jun 28, 2015, at 19:32 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Sun, 28 Jun 2015, Sebastian Moeller wrote:
> 
>> 	This looks great, could you by any chance confirm that the GUI does allow to configure cake and that you can or can not set the overhead for cake in the link layer adjustments (LLA) tab? (select cake as link layer adjustment method, and put 42 into the overhead field and report the output of “tc -d qdisc” before and after selecting cake as LLA). @Toke: If that works, I think we can safely push these changes into the openwrt repositories...
> 
> Here is the output. What I don't see is both ingress and egress ECN markings even though I have selected this in the advanced configuration under Queue Discipline.

	Good point, I have not connected these fields with cake yet, will do in the near future. I believe cake defaults to ECN, but if no packets are marked during a test, but rather dropped, something is fishy...

> 
> Before changing LLA:
> 
> root@OpenWrt:~# tc -d qdisc
> qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 532
> qdisc cake 110: dev eth0 parent 1:11 unlimited diffserv4 flows raw
> qdisc cake 120: dev eth0 parent 1:12 unlimited diffserv4 flows raw
> qdisc cake 130: dev eth0 parent 1:13 unlimited diffserv4 flows raw
> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 32
> qdisc cake 110: dev ifb4eth0 parent 1:11 unlimited diffserv4 flows raw
> qdisc cake 120: dev ifb4eth0 parent 1:12 unlimited diffserv4 flows raw
> qdisc cake 130: dev ifb4eth0 parent 1:13 unlimited diffserv4 flows raw
> 
> After changing LLA:
> 
> root@OpenWrt:~# tc -d qdisc
> qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 532
> linklayer ethernet overhead 42
> qdisc cake 110: dev eth0 parent 1:11 unlimited diffserv4 flows raw
> qdisc cake 120: dev eth0 parent 1:12 unlimited diffserv4 flows raw
> qdisc cake 130: dev eth0 parent 1:13 unlimited diffserv4 flows raw
> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 32
> linklayer ethernet overhead 42
> qdisc cake 110: dev ifb4eth0 parent 1:11 unlimited diffserv4 flows raw
> qdisc cake 120: dev ifb4eth0 parent 1:12 unlimited diffserv4 flows raw
> qdisc cake 130: dev ifb4eth0 parent 1:13 unlimited diffserv4 flows raw

	This is showing tha the cake setup went wrong. It is using HTB as shaper instead of cake. There should be no cake line. Could you post your /usr/lib/sqm/simple.qos file please.

Best Regards
	Sebastian

> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

On Fri, Jun 26, 2015 at 9:18 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> Hypothesis: this might have to do with the receive path. Some devices might
> have more capacity than others to buffer inbound packets until the CPU can
> get around to servicing them.

*Good* hypothesis. I am certain I have seen this on multiple occasions
on other hardware. Hard to confirm.

Wet paint... so I finally got off my arse and looked at the driver this morning.

given that this is a multi-core box, I would lean towards a smaller
napi_poll_weight, which unfortunately is a constant (64) in the code.
4 cores can take interrupts faster. (and I hate napi on routers) I
have sometimes longed for an IQL (ingress queue limits) to also handle
differences in packet size, dynamically changing the poll weight based
on load - increasing it for loads with lots of small packets,
decreasing it for lots of big packets.

Furthermore this thing is doing software gro (up to 64 packets at a
time) which is a LOT of processing at this layer in the stack.

Its a two line patch to cut the weight to 16, but I have never managed
to get a working build for this platform.

> - Jonathan Morton
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

HI Mikael,


On Jun 26, 2015, at 16:49 , Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Fri, 26 Jun 2015, Mikael Abrahamsson wrote:
> 
>> Btw, I tried WNDR3800 setting it to 100/100 SQM. It seems to max out around 25-30k PPS, but the difference is that when the CPU is full, it seems to delay/ECN-mark packets because there are no packets lost. When the WRT1200AC runs out of CPU it starts dropping packets. I always have 0 packets lost with the WNDR3800 when doing iperf3 testing. I found this difference interesting, wonder where in the forwarding path the WRT1200AC loses packets?
> 
> I checked again, and my WDR4900 with same setup doesn't lose packets either. Even at 99% sirq, no packets are lost.

	Tangent: What is the shaper rate the wdr4900 can push with sqm-scripts? (Before your 1200ac results the ppc-soc in the wdr4900 looked like the finest little router platform in the last years, too bad it was ignored by the mass market...)

> 
> WRT1200AC starts to lose packets at 500 megabit/s SQM around MSS 300 and lower. If I turn off gso, tso and gro, I have to go to MSS 600 and above to avoid packet loss.

	So the offloads buy roughly a doubling of the achievable packet rate, not bad, but as your results show also knot essential.


Best Regards
	Sebastian

> 
> Does flent check for packet loss at all? Perhaps it's something to look into, because with ECN we really don't want to see any packets lost and this might be good to include test results for.
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Sebastian Moeller wrote:

> 	Tangent: What is the shaper rate the wdr4900 can push with 
> sqm-scripts? (Before your 1200ac results the ppc-soc in the wdr4900 
> looked like the finest little router platform in the last years, too bad 
> it was ignored by the mass market...)

Well, if I set SQM to 500 megabit/s and MSS to 300 I am able to get 70 
megabit/s of iperf3 through it at 42k PPS. At default MSS I get 150 
megabit/s at 25k PPS through it. This is at 100% sirq.

Does this help, or do you want me to do any other tests. I have the 
WDR4900 powered up and on my desk right now.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Dave Taht]

On Fri, Jun 26, 2015 at 9:36 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Fri, 26 Jun 2015, Sebastian Moeller wrote:
>
>>         Tangent: What is the shaper rate the wdr4900 can push with
>> sqm-scripts? (Before your 1200ac results the ppc-soc in the wdr4900 looked
>> like the finest little router platform in the last years, too bad it was
>> ignored by the mass market...)
>
>
> Well, if I set SQM to 500 megabit/s and MSS to 300 I am able to get 70
> megabit/s of iperf3 through it at 42k PPS. At default MSS I get 150
> megabit/s at 25k PPS through it. This is at 100% sirq.
>
> Does this help, or do you want me to do any other tests. I have the WDR4900
> powered up and on my desk right now.

I am never allergic to somene running a comprehensive flent suite
through something, and sticking the results up somewhere.

:)

There are always more surprises and more things we seem to have to wackamole.

>
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Dave Taht wrote:

> I am never allergic to somene running a comprehensive flent suite
> through something, and sticking the results up somewhere.

http://swm.pp.se/aqm/wdr4900-150626-9.tar

Happy no sneezing!

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Jim Reisert AD1C]

On Tue, Jun 23, 2015 at 1:20 AM, Sebastian Moeller wrote:

> Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm.
> Rich published a great set of instructions for setting up sqm-scripts under OpenWRT proper.

Thanks, Sebastian.  I did as you suggested.

It was eye-opening looking at the results from
http://www.dslreports.com/speedtest both before and after enabling
SQM.  Wow!

- Jim

-- 
Jim Reisert AD1C, <jjreisert@alum.mit.edu>, http://www.ad1c.us

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Dave Taht]

On Tue, Jun 23, 2015 at 7:35 AM, Jim Reisert AD1C
<jjreisert@alum.mit.edu> wrote:
> On Tue, Jun 23, 2015 at 1:20 AM, Sebastian Moeller wrote:
>
>> Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm.
>> Rich published a great set of instructions for setting up sqm-scripts under OpenWRT proper.
>
> Thanks, Sebastian.  I did as you suggested.
>
> It was eye-opening looking at the results from
> http://www.dslreports.com/speedtest both before and after enabling
> SQM.  Wow!

I think one of the most effective graphs we ever did was the "you are
here. You could be here." one.

I would like to add it to this:

http://www.dslreports.com/speedtest/results/bufferbloat?up=1

>
> - Jim
>
> --
> Jim Reisert AD1C, <jjreisert@alum.mit.edu>, http://www.ad1c.us



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Sebastian Moeller]

Hi Alan,

On Jun 19, 2015, at 19:35 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:

> On 19/06/2015, Sebastian Moeller <moeller0@gmx.de> wrote:
>> Hi Alan,
>> 
>> excellent, thanks a million.
>> 
>> On Jun 19, 2015, at 16:44 , Alan Jenkins
>> <alan.christopher.jenkins@gmail.com> wrote:
>> 
>>> Hi
>>> 
>>> I guess I've done the complementary half to Seb's test :).  Basically
>>> "cake overhead 40" didn't work, but that's the fault of cake in this
>>> build.  Or tc, as Johnathan suggested.  (The "cake atm" part seems to
>>> work, as per my previous test).
>> 
>> 	Great!
>> 
>>> 
>>> "tc qdisc" says "cake overhead 0", as Sebastian noticed.  And the test
>>> results show "cake overhead 40" does not give a measurable
>>> improvement.  But "tc stab overhead 40" does.
>>> 
>>> I ran this test with the updated sqm-scripts and I think they're doing
>>> the right thing.
>> 
>> 	Thanks for testing this, especially as I can not due to a lack of an
>> ADSL-link (and lack of cake actually, last I looked all I could find was
>> cookies in my browser and a promise of pie in my router)
>> 
>>> 
>>> 
>>> Method:
>>> 
>>> I used the updated files from sqm-scripts,
>>> 
>>> (once I remembered to mark them executable.  Lacking that causes a
>>> failure with no error messages, because sqm-scripts checks before
>>> running them :)
>>> 
>>> but didn't bother updating & using luci-app-sqm.
>> 
>> 	Ah, okay, I guess I did test this part with Dave’s help, so this should
>> work with the most recent sqm.lua.
>> 
>>> 
>>> The test was to compare netperf-runner results - ping during combined
>>> upload & download - for "overhead 40" and "overhead 0".  I tested both
>>> values of linklayer_adaptation_mechanism.
>>> 
>>> I had to repeat 6 times (60s per run for each overhead) because of
>>> random variation in the range of 3-4ms.  I alternated "overhead 40"
>>> and "overhead 0" to try and exclude longer-term variation effects.
>>> 
>>> With "stab overhead 40", median latency was better by about 3-4ms.
>>> With "cake overhead 40", there is no such effect.
>> 
>> 	Intersting, when I still had a 6M/1M ADSL link, I saw much larger latency
>> under load increases when setting the per packet overhead to small, but I
>> had my egress shaper running at 100% of line rate, so the system was rigged
>> for maximum effect that way. How are your shapers typically set?
> 
> For this test I try to push it, today I used 95%.  I started trying
> 100%, which is still much better than unshaped.  I was scared off by
> the random variation, I think it was higher at 100%.

	Fair enough, but only at 100% you will notice if the overhead is off by a single byte… You are running sqm on the pppoe interface, correct?

> 
> For long term use I reduce it, because I've seen the line rate vary
> slightly.  (1020k up today, 912 a while back.  Currently it reports a
> "max" figure I don't understand, it's about 1100 despite being
> rebooted daily.  16390k down).

	A pity that your line is that flaky, but it only differs between reboots and does not change during the day? As far as I know only SRA would allow rate changes while the sync stays up. So you either have unscheduled resyncs during the day which drag in a rate change or between days. Anyway aiming with the shaper rate at below the best case sync seems wise unless you want to fiddle with modem and router daily. (It would be sweet if one could query the actual sync speed from XDSL-modems from the LAN side in a standardized fashion, so sqm could learn to set the shaper automatically).

Best Regards
	Sebastian

> 
> Alan