[Cerowrt-devel] Status of SQM-Scripts integration into OpenWRT Barrier Breaker?

[Cerowrt-devel] Status of SQM-Scripts integration into OpenWRT Barrier Breaker?

[Frits Riep]

[-- Attachment #1: Type: text/plain, Size: 8735 bytes --]

From earlier notes in the list, and from the CeroWRT Wiki, I believe the
intent is to back port SQM into OpenWRT.

 

As there are a wide variety of hardware platforms supported under OpenWRT,
if SQM scripts, and a Luci-App-SQM, then many installed base routers, and
readily available routers would greatly benefit.

 

I may be misunderstanding, but I thought this was a goal.  Am I incorrect,
or have the plans changed?

 

Also, I do have some questions.

 

*         I believe fq-codel is integrated into the latest available
versions of Linux.  If so, other than setting upload and download limits, do
we already have buffer control handled with all newer OpenWRT releases?

*         If so, do we still need to download and configure the packages,
QOS-Scripts, and Luci-App-Qos?  If so, do these packages do anything besides
allow the configuration of Upload and Download limits?

*         If we can, in fact, control bufferbloat with OpenWRT and the
latest Qos-scripts and Luci-App-QOS, then would the integration of SQM into
OpenWRT provide much benefit in controlling bloat?  If so, are there still
plans to make that happen.

 

I am very happy with the configuration of my home router running CeroWRT on
a Netgear WNDR-3800 (running close to the latest build), on a Verizon FIOS
connection at 75 Mbs Down / and 75 Mbs Up, and on running pings with
Speedtest notice a definate difference in latency (very tight control) vs up
to 100 ms latency.

 

Thanks to all for the good work, and I hope that these are useful questions.

Frits

 

 

 

 

-----Original Message-----
From: cerowrt-devel-bounces@lists.bufferbloat.net
[mailto:cerowrt-devel-bounces@lists.bufferbloat.net] On Behalf Of
cerowrt-devel-request@lists.bufferbloat.net
Sent: Sunday, August 24, 2014 2:55 PM
To: cerowrt-devel@lists.bufferbloat.net
Subject: Cerowrt-devel Digest, Vol 33, Issue 23

 

Send Cerowrt-devel mailing list submissions to

                 <mailto:cerowrt-devel@lists.bufferbloat.net>
cerowrt-devel@lists.bufferbloat.net

 

To subscribe or unsubscribe via the World Wide Web, visit

                 <https://lists.bufferbloat.net/listinfo/cerowrt-devel>
https://lists.bufferbloat.net/listinfo/cerowrt-devel

or, via email, send a message with subject or body 'help' to

                 <mailto:cerowrt-devel-request@lists.bufferbloat.net>
cerowrt-devel-request@lists.bufferbloat.net

 

You can reach the person managing the list at

                 <mailto:cerowrt-devel-owner@lists.bufferbloat.net>
cerowrt-devel-owner@lists.bufferbloat.net

 

When replying, please edit your Subject line so it is more specific than
"Re: Contents of Cerowrt-devel digest..."

 

 

Today's Topics:

 

   1. SQM vs DSL (Sebastian Moeller)

 

 

----------------------------------------------------------------------

 

Message: 1

Date: Sun, 24 Aug 2014 20:54:49 +0200

From: Sebastian Moeller < <mailto:moeller0@gmx.de> moeller0@gmx.de>

To: Dave T?ht < <mailto:dave.taht@gmail.com> dave.taht@gmail.com>,
cerowrt-devel

                < <mailto:cerowrt-devel@lists.bufferbloat.net>
cerowrt-devel@lists.bufferbloat.net>

Subject: [Cerowrt-devel] SQM vs DSL

Message-ID: < <mailto:7E74D1C2-5686-40C4-813B-23BE96D736E5@gmx.de>
7E74D1C2-5686-40C4-813B-23BE96D736E5@gmx.de>

Content-Type: text/plain; charset="windows-1252"

 

Hi Dave, hi List,

 

to avoid real work I embarked on a (quick) test of my DSL line?s bloat
(again). Since I just realized that my ISP has increased my per-packet
overhead from 40 bytes to 44 bytes (probably by switching onto a platform
that uses VLAN tags between DSL-modem and DSLAM/BRAS, but I digress) my
recent netperf-wrapper tests all had been invalidated, since most likely the
link layer encapsulation was wrong. So I set out to figure out how close I
can shape my DSL line ad still keep decent behavior under load. I used
Toke?s great netperf-wrapper (the RRUL test to be precise) against
netperf-eu.bufferbloat.net and tested a number of different up- and
downstream shaping values (on an ATM based adsl2+ line with a line rate of
17613Kbps down and 2604 Kbps up, for all tests the proper link layer
encapsulation (ATM with overhead 44) has been used) from a macbook attached
via the 5GHz radio. 

                The first interesting result is that for my line shaping the
upstream to 2000Kbps (76%) actually shows no better ICMP-CDF than shaping to
2575Kbps (98.9%) or even 2604 (100%) independent of the shaping of the
downstream.  Actually, the slower the uplink gets the larger the average
latency gets, just as the upstream target value increases to counter the
longer transfer times ("tc -d qdisc? on cerowrt will tell the actual
effective target values).

                Downstream shaping results however are more interesting and
less clear.

                Attached are three plots of a number of experiments with
different downstream-shaping values from 8000 Kbps (8000KU2604K in the
legend) up to the line-rate at 17613Kbps (17613KU2604K) to no shaping
(0KU0K, that is SQM configured with 0 for up- and downstream). It is quite
obvious that no shaping has the highest good-put but at a terrible latency
cost (and the DSL-modem is not terribly over-buffered with 85% of the ICMP
CDF still below 100ms, but several excursions into the >1000ms range). All
downstream shaping values at this overview scale show a massive improvement
of latency under load.

 

 

                The second plot (*ping_box_scaled.png) shows once more that
no shaping is bad for latency under load (from left to right UDP BE, UDP BK,
ICMP, UDP EF, averages). Also quite obvious, shaping the downstream just to
line rate (gray box: 17613, 100%) still does not give too good a result with
lots of variability (but way better than no shaping). Decreasing the
downstream further improves the averages for all latency probes but
relatively mild compared to the no-shaping and 100% line-rate shaping
conditions.        After the tests I converged on 15852 (90%) and 2575 (99%)
which on average shows a increase of latency under load of 15ms pretty close
to the theoretical expected value of 10ms (5ms codel target in each
direction), which given the access via WLAN is quite decent. (Also a wired
test afterwards showed an average increase go 11ms close enough to theory to
call it a day.)

 

 

                I guess the 2 interesting points are: 

1)            With a non-congested DSLAM, DSL upstream shaping can be pushed
to the extreme of 100% of line rate without having to pay a severe price in
latency growth under load. (I remember that we had a discussion that
DSL-modems should learn fq_codel so we could regain the lost upstream
bandwidth; I guess with proper handling of the encapsulation there is no
bandwidth left on the table ;) Still it would be so much nicer if all modems
would do this by themselves). 

2)            On the downstream side of shaping, it really seems to be a
question of figuring out which trade-off between bandwidth and latency under
load one is willing to accept. (And finally note to self, perform future
tests with a wired connection until wifi is fast ;) )

 

                P.S.: I hope the images are not too large and still
readable.

Best Regards

                Sebastian

 

-------------- next part --------------

An HTML attachment was scrubbed...

URL: <
<https://lists.bufferbloat.net/pipermail/cerowrt-devel/attachments/20140824/
63b73396/attachment.html>
https://lists.bufferbloat.net/pipermail/cerowrt-devel/attachments/20140824/6
3b73396/attachment.html>

-------------- next part --------------

A non-text attachment was scrubbed...

Name: dsl_shaping_140824_totals.png

Type: image/png

Size: 58371 bytes

Desc: not available

URL: <
<https://lists.bufferbloat.net/pipermail/cerowrt-devel/attachments/20140824/
63b73396/attachment.png>
https://lists.bufferbloat.net/pipermail/cerowrt-devel/attachments/20140824/6
3b73396/attachment.png>

-------------- next part --------------

A non-text attachment was scrubbed...

Name: dsl_shaping_140824_ping_box_scaled.png

Type: image/png

Size: 88710 bytes

Desc: not available

URL: <
<https://lists.bufferbloat.net/pipermail/cerowrt-devel/attachments/20140824/
63b73396/attachment-0001.png>
https://lists.bufferbloat.net/pipermail/cerowrt-devel/attachments/20140824/6
3b73396/attachment-0001.png>

 

------------------------------

 

_______________________________________________

Cerowrt-devel mailing list

 <mailto:Cerowrt-devel@lists.bufferbloat.net>
Cerowrt-devel@lists.bufferbloat.net

 <https://lists.bufferbloat.net/listinfo/cerowrt-devel>
https://lists.bufferbloat.net/listinfo/cerowrt-devel

 

 

End of Cerowrt-devel Digest, Vol 33, Issue 23

*********************************************


[-- Attachment #2: Type: text/html, Size: 19143 bytes --]

Re: [Cerowrt-devel] Status of SQM-Scripts integration into OpenWRT Barrier Breaker?

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 2583 bytes --]

"Frits Riep" <riep@riepnet.com> writes:

> I may be misunderstanding, but I thought this was a goal. Am I
> incorrect, or have the plans changed?

Don't think the plans have changed per se. More of a case of needing
more testing to be included in Openwrt.

There's some discussion of this here:

https://github.com/dtaht/ceropackages-3.10/issues/8

Testing is welcome!

> · I believe fq-codel is integrated into the latest available versions
> of Linux. If so, other than setting upload and download limits, do we
> already have buffer control handled with all newer OpenWRT releases?

Yes, fq_codel is included in the Linux kernel, and in Openwrt it is even
the default. So if your router is actually at the bottleneck
(physically), things should pretty much just work (in theory). However,
most routers are *not* at the bottleneck; there tends to be a modem of
some sort (whether cable, DSL or FIOS) in-between. Which is where
software rate-limiting (which is what SQM-scripts do) comes into play.

> · If so, do we still need to download and configure the packages,
> QOS-Scripts, and Luci-App-Qos? If so, do these packages do anything
> besides allow the configuration of Upload and Download limits?

Well, the QoS-script packages have some downsides and some upsides. The
main upside is configurability, I think. Including things like manually
classifying traffic etc. The downside is that it doesn't do IPv6 right,
and that SQM-scripts does link layer adaptation right, which I don't
think QoS-scripts do. Also, QoS-scripts is more complex and harder to
configure (I think).

> · If we can, in fact, control bufferbloat with OpenWRT and the latest
> Qos-scripts and Luci-App-QOS, then would the integration of SQM into
> OpenWRT provide much benefit in controlling bloat? If so, are there
> still plans to make that happen.

Well, I do believe there would be some merit to having SQM included (at
least as an optional package) in openwrt. More testing is the main thing
required, I think. And if we aim at completely replacing QoS-scripts,
(some of) the configurability needs to be added to SQM.

> I am very happy with the configuration of my home router running
> CeroWRT on a Netgear WNDR-3800 (running close to the latest build), on
> a Verizon FIOS connection at 75 Mbs Down / and 75 Mbs Up, and on
> running pings with Speedtest notice a definate difference in latency
> (very tight control) vs up to 100 ms latency.

Is this with SQM enabled? That usually tops out at around ~50Mbps due to
CPU limits...

-Toke

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: [Cerowrt-devel] Status of SQM-Scripts integration into OpenWRT Barrier Breaker?

[Sebastian Moeller]

Hi Toke hi Frits,

just a few small things to add to Toke’s...

On Aug 25, 2014, at 14:53 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> "Frits Riep" <riep@riepnet.com> writes:
> 
>> I may be misunderstanding, but I thought this was a goal. Am I
>> incorrect, or have the plans changed?
> 
> Don't think the plans have changed per se. More of a case of needing
> more testing to be included in Openwrt.
> 
> There's some discussion of this here:
> 
> https://github.com/dtaht/ceropackages-3.10/issues/8
> 
> Testing is welcome!
> 
>> · I believe fq-codel is integrated into the latest available versions
>> of Linux. If so, other than setting upload and download limits, do we
>> already have buffer control handled with all newer OpenWRT releases?
> 
> Yes, fq_codel is included in the Linux kernel, and in Openwrt it is even
> the default. So if your router is actually at the bottleneck
> (physically), things should pretty much just work (in theory). However,
> most routers are *not* at the bottleneck; there tends to be a modem of
> some sort (whether cable, DSL or FIOS) in-between. Which is where
> software rate-limiting (which is what SQM-scripts do) comes into play.

	I do not think that all ethernet drivers for OpenWRT capable devices support byte queue limits (BQL) yet. Devices without BQL probably also need bandwidth shaping to work well.

> 
>> · If so, do we still need to download and configure the packages,
>> QOS-Scripts, and Luci-App-Qos? If so, do these packages do anything
>> besides allow the configuration of Upload and Download limits?
> 
> Well, the QoS-script packages have some downsides and some upsides. The
> main upside is configurability, I think. Including things like manually
> classifying traffic etc.

	All correct, but note though that the l7-filters (for filtering on a per application basis) seem to be on the way out, I think that openwrt CC will not support them anymore. (And there seems to be doubts about how well the current l7 filtering still works (it seems the filter rules have not been updated in a long time); and for encrypted traffic they do not work at all. For these Reasons I think Gargoyle is actively recommending to avoid l7).

> The downside is that it doesn't do IPv6 right,
> and that SQM-scripts does link layer adaptation right, which I don't
> think QoS-scripts do.

	Note SQM’s link layer adjustments need to be checked for dual-stack IPv6 and IPv4 traffic on the wan port. (Which I would do except IPv6 does not work for me…)

> Also, QoS-scripts is more complex and harder to
> configure (I think).
> 
>> · If we can, in fact, control bufferbloat with OpenWRT and the latest
>> Qos-scripts and Luci-App-QOS, then would the integration of SQM into
>> OpenWRT provide much benefit in controlling bloat? If so, are there
>> still plans to make that happen.
> 
> Well, I do believe there would be some merit to having SQM included (at
> least as an optional package) in openwrt. More testing is the main thing
> required, I think. And if we aim at completely replacing QoS-scripts,
> (some of) the configurability needs to be added to SQM.
> 
>> I am very happy with the configuration of my home router running
>> CeroWRT on a Netgear WNDR-3800 (running close to the latest build), on
>> a Verizon FIOS connection at 75 Mbs Down / and 75 Mbs Up, and on
>> running pings with Speedtest notice a definate difference in latency
>> (very tight control) vs up to 100 ms latency.
> 
> Is this with SQM enabled? That usually tops out at around ~50Mbps due to
> CPU limits…

	Curious as well…
Best Regards
	Sebastian

> 
> -Toke
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] Status of SQM-Scripts integration into OpenWRT Barrier Breaker?

[Dave Taht]

On Mon, Aug 25, 2014 at 5:53 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> "Frits Riep" <riep@riepnet.com> writes:
>
>> I may be misunderstanding, but I thought this was a goal. Am I
>> incorrect, or have the plans changed?

No the plan is and has always been to push the SQM system upstream not
just to openwrt
but also to a debian and fedora installable package.

I would have settled for improving the qos-scripts to handle ipv6
correctly, however.

> Don't think the plans have changed per se. More of a case of needing
> more testing to be included in Openwrt.

I note that the past month has been the first time in several years
that I didn't build
a version of cerowrt even weekly. The last 8 months have been
something of a "death
march" for me on getting it stable (see  bug 442, seemingly fixed in
3.10.50-1), and it has been a real relief
to put it down for a while and catch up on other things (like sleep
and paying work).

I do plan to resync with barrier breaker RC3 or RC4, but I don't mind
at all taking a few more weeks away from it to breathe and plan on the
next major phase of it all, which is to make-wifi-fast. There is also
a need to do a "bql-on-everything" project, identifying and fixing
(adding 4-8 lines of code to) all the ethernet chips found in openwrt
platforms...

...and I'd also like to clearly identify at least one openwrt board
that worked well at line rate on USA DSL.

As for SQM...

There are a couple things still missing from SQM that I would like to
have in there.

1) Detection of actually available AQM and packet scheduling kernel
modules so that the user can't choose something that is unavailable
either at the command line or the gui
2) better control of prioritization as per qos-scripts
3) debian apt, fedora rpm, and arch pacman packaging
4) A rewrite to make it smaller and cleaner

Moving further forward:

There is a feature I'd longed to have which is to have the three tier
"simplest" setup work at line rate rather than through htb. This would
probably involve writing a new qdisc entirely or switching to a DRR +
fq_codel based system more like what free.fr uses. Despite multiple
efforts we never managed to beat that...

Performance on low end hardware is a real problem, we peak out at
about 50mbits of shaping on the (processor designed in 1989!) wndr3800
we presently use, and seem to have hit a similar (if slightly higher)
limit on the edge router lite hardware. So coming up with a faster
thing than htb would be nice also, as well as some way to warn the
user they are seeking an impossible to achieve setting.

Gargoyle (and stream boost) have an automatic bandwidth sensing system
which is worth looking at.

Better onboard tests for determining a "good" rate would also be nice.
netperf, as currently used is a bit heavyweight, and although we have
ramped up to a half dozen donated end servers.

None of the above are really barriers to getting sqm-scripts and gui
into some more mainline openwrt package repository, it had been my
full intent to get them into openwrt barrier breaker but I was
juggling far too many other things at the time of the first freeze to
accomplish that.

Certainly more testing on more platforms is needed ASAP, and I have
been appreciating the efforts of those doing so, while taking a bit of
a break from it all.

> There's some discussion of this here:
>
> https://github.com/dtaht/ceropackages-3.10/issues/8
>
> Testing is welcome!
>
>> · I believe fq-codel is integrated into the latest available versions
>> of Linux. If so, other than setting upload and download limits, do we
>> already have buffer control handled with all newer OpenWRT releases?
>
> Yes, fq_codel is included in the Linux kernel, and in Openwrt it is even
> the default.

Just as a note, it is not only the default, but modified to support a
default quantum of 300, rather than an MTU. This is quite reasonable
for an AP or STA, less so for a box running at 1000s of megabits.

And it's not clear that elsewhere in openwrt (say, on x86) certain
offloads are not turned off like TSO, GSO, GRO,
that can affect performance. This is done by the overlarge and quite
bloated "debloat" script in cerowrt and not in
mainline openwrt.

Lastly, fq_codel with the default 5 tuple hash is not the right thing
for a wifi AP, something that sorts on a per-sta basis would be better
- and there is much other work to be done in the wifi stack to improve
it.

> So if your router is actually at the bottleneck
> (physically), things should pretty much just work (in theory). However,
> most routers are *not* at the bottleneck; there tends to be a modem of
> some sort (whether cable, DSL or FIOS) in-between. Which is where

Finding devices that have good device drivers for each of these ISP
technologies would be nice.

> software rate-limiting (which is what SQM-scripts do) comes into play.
>
>> · If so, do we still need to download and configure the packages,
>> QOS-Scripts, and Luci-App-Qos? If so, do these packages do anything
>> besides allow the configuration of Upload and Download limits?
>
> Well, the QoS-script packages have some downsides and some upsides. The
> main upside is configurability, I think. Including things like manually
> classifying traffic etc. The downside is that it doesn't do IPv6 right,
> and that SQM-scripts does link layer adaptation right, which I don't
> think QoS-scripts do. Also, QoS-scripts is more complex and harder to
> configure (I think).
>
>> · If we can, in fact, control bufferbloat with OpenWRT and the latest
>> Qos-scripts and Luci-App-QOS, then would the integration of SQM into
>> OpenWRT provide much benefit in controlling bloat? If so, are there
>> still plans to make that happen.
>
> Well, I do believe there would be some merit to having SQM included (at
> least as an optional package) in openwrt. More testing is the main thing
> required, I think. And if we aim at completely replacing QoS-scripts,
> (some of) the configurability needs to be added to SQM.

I have longed to find a sane way to benchmark the hfsc approach qos-scripts
uses vs the htb scripts sqm uses vs the DRR approach free.fr uses.

And to find a faster rate limiter than any of them. I keep thinking something
at the BQL level would work...


>> I am very happy with the configuration of my home router running
>> CeroWRT on a Netgear WNDR-3800 (running close to the latest build), on
>> a Verizon FIOS connection at 75 Mbs Down / and 75 Mbs Up, and on
>> running pings with Speedtest notice a definate difference in latency
>> (very tight control) vs up to 100 ms latency.

I had found on the FIOS connection that ESR has that only downstream
shaping seemed needed.

> Is this with SQM enabled? That usually tops out at around ~50Mbps due to
> CPU limits...


>
> -Toke
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article