Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Dave Taht]

On Wed, Oct 01, 2014 at 12:10:46PM -0400, Weedy wrote:
> On 30/03/14 06:29 PM, Dave Taht wrote:
> > On Sun, Mar 30, 2014 at 02:24:44PM -0400, Weedy wrote:
> >> On Sat, Mar 29, 2014 at 2:56 PM, Dave Täht <dave.taht@bufferbloat.net>wrote:
> >>
> >>> From: Dave Taht <dave.taht@bufferbloat.net>
> >>>
> >>> This adds support for the bufferbloat project's "Smart Queue Management"
> >>> (SQM) system, which improves over openwrt's qos-scripts in the following
> >>> ways
> >>>
> >>> + Uses HTB with two models for managing traffic
> >>>   a simplest one that merely uses fq_codel, and a three tier one that does
> >>>   some basic and tunable packet prioritization.
> >>>
> >>> + Works with ipv6 and ipv4 correctly (unlike qos-scripts)
> >>> + extensive support for fixing ADSL and PPOe framing problems
> >>> + Partial support for key diffserv markings
> >>> + highly tuned fq_codel implementation especially for low bandwidths
> >>> + Tested heavily on cable modems and on dsl devices
> >>>
> >>> It is a disimprovement in that:
> >>>
> >>> - There are no built-in tricks for doing l7 classification,
> >>> or other forms of packet inspection.
> >>>
> >>> - We haven't explored hfsc all that much, prefering to rely
> >>> on the predictable behavior of htb + fq_codel for everything
> >>>
> >>> - And there is support for a few qdiscs that are not in the linux
> >>> kernel mainline that remain experimental.
> >>> ---
> >>>  net/sqm-scripts/Makefile                           |   48 +++
> >>>  net/sqm-scripts/files/etc/config/sqm               |   11 +
> >>>  net/sqm-scripts/files/etc/init.d/sqm               |   23 ++
> >>>  net/sqm-scripts/files/usr/lib/sqm/functions.sh     |  335
> >>> ++++++++++++++++++++
> >>>  net/sqm-scripts/files/usr/lib/sqm/run.sh           |   67 ++++
> >>>  net/sqm-scripts/files/usr/lib/sqm/simple.qos       |  187 +++++++++++
> >>>  net/sqm-scripts/files/usr/lib/sqm/simple.qos.help  |    1 +
> >>>  net/sqm-scripts/files/usr/lib/sqm/simplest.qos     |   84 +++++
> >>>  .../files/usr/lib/sqm/simplest.qos.help            |    1 +
> >>>  net/sqm-scripts/files/usr/lib/sqm/stop.sh          |   22 ++
> >>>  10 files changed, 779 insertions(+)
> >>>  create mode 100644 net/sqm-scripts/Makefile
> >>>  create mode 100644 net/sqm-scripts/files/etc/config/sqm
> >>>  create mode 100755 net/sqm-scripts/files/etc/init.d/sqm
> >>>  create mode 100644 net/sqm-scripts/files/usr/lib/sqm/functions.sh
> >>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/run.sh
> >>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/simple.qos
> >>>  create mode 100644 net/sqm-scripts/files/usr/lib/sqm/simple.qos.help
> >>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/simplest.qos
> >>>  create mode 100644 net/sqm-scripts/files/usr/lib/sqm/simplest.qos.help
> >>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/stop.sh
> >>>
> >>> diff --git a/net/sqm-scripts/files/etc/config/sqm
> >>> b/net/sqm-scripts/files/etc/config/sqm
> >>> new file mode 100644
> >>> index 0000000..547d321
> >>> --- /dev/null
> >>> +++ b/net/sqm-scripts/files/etc/config/sqm
> >>> @@ -0,0 +1,11 @@
> >>> +
> >>> +config queue 'ge00'
> >>> +        option enabled '0'
> >>> +        option interface 'ge00'
> >>> +        option download '20000'
> >>> +        option upload '4000'
> >>> +        option qdisc 'fq_codel'
> >>> +        option script 'simple.qos'
> >>> +        option qdisc_advanced '0'
> >>> +        option linklayer 'none'
> >>> +
> >>>
> >>
> >> How hard is this to config from the command line/vim?
> > 
> > There are a few more options than this (for DSL compensation, ecn
> > and advanced configuration), the above would work if you changed
> > enabled to '1' and the device from ge00 to your wan device. (not
> > the "wan" firewall rule, presently. )
> > 
> > It does help to have a sane long term and realistic measurement of your
> > network using something like the rrul test rather than the oft-gamed speedtest.
> > 
> > http://www.bufferbloat.net/projects/cerowrt/wiki/Setting_up_SQM_for_CeroWrt_310
> > 
> > You are right, we should fully document all the variables in this file.
> > Until recently they were kind of in flux.
> > 
> >> I've never needed or really wanted luci on my box, I just use vim. Going by
> >> this patch, there is either nothing to config or no examples. I would think
> >> shipping a roughly equivalent config to what ships in qos-scripts would be
> >> a good start to get people testing.
> > 
> > /etc/init.d/sqm start,stop etc work as expected.
> > 
> >> IE: highest priority: small ARP, DNS, and SSH packets
> > 
> > The SQM default is local DNS and ntp. fq_codel automagically optimizes
> > for other sparse flows like arp, ssh, mosh, tcp syn, synack, etc,
> > no need to do that via classification.
> > 
> >> normal: HTTP, HTTPS
> > 
> > So you want to deprioritize vpn, smtp, rsync, dropbox, http on odd ports,
> > caching servers, etc, all in favor of the web gods?
> > 
> > we just toss all that into the normal (best effort bin) and let fq_codel
> > sort it out.
> > 
> >> bulk: everything else.
> > 
> > We do respect the diffserv marking of CS1 (background) and toss it
> > into the background queue.
> > 
> >> Side note: Will SQM do bandwidth slicing? Or at least handle "hostile"
> >> environments better? I say "hostile" as in roommates or maybe teenage kids.
> >> Multiple people/devices thinking they are entitled to the entire WAN
> >> bandwidth at all times.
> > 
> > fq_codel does fair (well, we call it "flow") queuing.
> > 
> > https://tools.ietf.org/html/draft-hoeiland-joergensen-aqm-fq-codel-00
> > 
> > And manages the depth of flows via codel.
> > 
> > http://tools.ietf.org/html/draft-nichols-tsvwg-codel-02
> > 
> > In other words, it will be fair to all fat flows generated by everyone,
> > and slice flows down to the defined quantum and turn them back into
> > packets.
> > 
> > The "simplest.qos" model in SQM works remarkably well without trying to 
> > classify anything at all. I encourage people to try merely that and have
> > their preconceptions altered. 
> > 
> > The three-tier model (simple.qos), is more like what people think they want,
> > but the default is set to the bare minimum of what worked well in testing.
> > 
> > Example: a lot of flows are marked CS1 that shouldn't be, and starving
> > that queue to like 5% rather than it's current 30% turned out badly. 
> > 
> > In terms of identifying and "punishing" abusers, well, the only thing
> > that stresses this code out even the slightest is dozens of torrent flows.
> > 
> > Give it a shot. :)
> > 
> I feel like this died.

It didn't die. 

*I* died. 

I'd been on a death march for the last 8 months trying to
get the last bugs out of openwrt/cerowrt, and when the last big one
got fixed (bug 442 in the cerowrt database, multiple other trackers)

I put out a release of 3.10.50-1 pre BBrc1 and went to sleep.

When I woke up, about a week ago, everybody had nearly 2 months
uptime, good throughput, and a bunch of minor nits here and there.

Hooray!

The prospect of resyncing with BBrcX intimidates me, and I have
had a ton of other things that slid to take care of, so I've
been catching up on those. Sebastian has been taking
care of SQM nits...

https://github.com/dtaht/ceropackages-3.10/issues/8

And Jonathon morton has been pouring it all into
pure C - with an integral bandwidth shaper that we
hope will be faster and more efficient than htb.

See an early result:

http://pastebin.com/zz06WhJr

It takes much of the heavy lifting out of the existing
sqm scripts. 

tc qdisc add dev eth1 root cake bandwidth 80mbit


So I don't know where to go. Certainly I'd like to
see the battle hardened sqm scripts (which are more
flexible than the C code above) get more widely used
and in BB.

openwrt users can do that today by adding the ceropackages repo to their build system.
or just installing the sqm-scripts and luci-app-sqm.

or we can clean it up further for openwrt mainline.

But I haven't seen one core openwrt dev say, yes, we want this mainlined, here's what you need to fix, so I'm inclined to go back to my cave, get more sleep, and work on the successor.

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Alpha Sparc]

[-- Attachment #1: Type: text/plain, Size: 9082 bytes --]

How good is the throughput on CeroWrt compared to OpenWrt ?
On Oct 2, 2014 9:55 AM, "Dave Taht" <dave.taht@bufferbloat.net> wrote:

> On Wed, Oct 01, 2014 at 12:10:46PM -0400, Weedy wrote:
> > On 30/03/14 06:29 PM, Dave Taht wrote:
> > > On Sun, Mar 30, 2014 at 02:24:44PM -0400, Weedy wrote:
> > >> On Sat, Mar 29, 2014 at 2:56 PM, Dave Täht <dave.taht@bufferbloat.net
> >wrote:
> > >>
> > >>> From: Dave Taht <dave.taht@bufferbloat.net>
> > >>>
> > >>> This adds support for the bufferbloat project's "Smart Queue
> Management"
> > >>> (SQM) system, which improves over openwrt's qos-scripts in the
> following
> > >>> ways
> > >>>
> > >>> + Uses HTB with two models for managing traffic
> > >>>   a simplest one that merely uses fq_codel, and a three tier one
> that does
> > >>>   some basic and tunable packet prioritization.
> > >>>
> > >>> + Works with ipv6 and ipv4 correctly (unlike qos-scripts)
> > >>> + extensive support for fixing ADSL and PPOe framing problems
> > >>> + Partial support for key diffserv markings
> > >>> + highly tuned fq_codel implementation especially for low bandwidths
> > >>> + Tested heavily on cable modems and on dsl devices
> > >>>
> > >>> It is a disimprovement in that:
> > >>>
> > >>> - There are no built-in tricks for doing l7 classification,
> > >>> or other forms of packet inspection.
> > >>>
> > >>> - We haven't explored hfsc all that much, prefering to rely
> > >>> on the predictable behavior of htb + fq_codel for everything
> > >>>
> > >>> - And there is support for a few qdiscs that are not in the linux
> > >>> kernel mainline that remain experimental.
> > >>> ---
> > >>>  net/sqm-scripts/Makefile                           |   48 +++
> > >>>  net/sqm-scripts/files/etc/config/sqm               |   11 +
> > >>>  net/sqm-scripts/files/etc/init.d/sqm               |   23 ++
> > >>>  net/sqm-scripts/files/usr/lib/sqm/functions.sh     |  335
> > >>> ++++++++++++++++++++
> > >>>  net/sqm-scripts/files/usr/lib/sqm/run.sh           |   67 ++++
> > >>>  net/sqm-scripts/files/usr/lib/sqm/simple.qos       |  187
> +++++++++++
> > >>>  net/sqm-scripts/files/usr/lib/sqm/simple.qos.help  |    1 +
> > >>>  net/sqm-scripts/files/usr/lib/sqm/simplest.qos     |   84 +++++
> > >>>  .../files/usr/lib/sqm/simplest.qos.help            |    1 +
> > >>>  net/sqm-scripts/files/usr/lib/sqm/stop.sh          |   22 ++
> > >>>  10 files changed, 779 insertions(+)
> > >>>  create mode 100644 net/sqm-scripts/Makefile
> > >>>  create mode 100644 net/sqm-scripts/files/etc/config/sqm
> > >>>  create mode 100755 net/sqm-scripts/files/etc/init.d/sqm
> > >>>  create mode 100644 net/sqm-scripts/files/usr/lib/sqm/functions.sh
> > >>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/run.sh
> > >>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/simple.qos
> > >>>  create mode 100644 net/sqm-scripts/files/usr/lib/sqm/simple.qos.help
> > >>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/simplest.qos
> > >>>  create mode 100644
> net/sqm-scripts/files/usr/lib/sqm/simplest.qos.help
> > >>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/stop.sh
> > >>>
> > >>> diff --git a/net/sqm-scripts/files/etc/config/sqm
> > >>> b/net/sqm-scripts/files/etc/config/sqm
> > >>> new file mode 100644
> > >>> index 0000000..547d321
> > >>> --- /dev/null
> > >>> +++ b/net/sqm-scripts/files/etc/config/sqm
> > >>> @@ -0,0 +1,11 @@
> > >>> +
> > >>> +config queue 'ge00'
> > >>> +        option enabled '0'
> > >>> +        option interface 'ge00'
> > >>> +        option download '20000'
> > >>> +        option upload '4000'
> > >>> +        option qdisc 'fq_codel'
> > >>> +        option script 'simple.qos'
> > >>> +        option qdisc_advanced '0'
> > >>> +        option linklayer 'none'
> > >>> +
> > >>>
> > >>
> > >> How hard is this to config from the command line/vim?
> > >
> > > There are a few more options than this (for DSL compensation, ecn
> > > and advanced configuration), the above would work if you changed
> > > enabled to '1' and the device from ge00 to your wan device. (not
> > > the "wan" firewall rule, presently. )
> > >
> > > It does help to have a sane long term and realistic measurement of your
> > > network using something like the rrul test rather than the oft-gamed
> speedtest.
> > >
> > >
> http://www.bufferbloat.net/projects/cerowrt/wiki/Setting_up_SQM_for_CeroWrt_310
> > >
> > > You are right, we should fully document all the variables in this file.
> > > Until recently they were kind of in flux.
> > >
> > >> I've never needed or really wanted luci on my box, I just use vim.
> Going by
> > >> this patch, there is either nothing to config or no examples. I would
> think
> > >> shipping a roughly equivalent config to what ships in qos-scripts
> would be
> > >> a good start to get people testing.
> > >
> > > /etc/init.d/sqm start,stop etc work as expected.
> > >
> > >> IE: highest priority: small ARP, DNS, and SSH packets
> > >
> > > The SQM default is local DNS and ntp. fq_codel automagically optimizes
> > > for other sparse flows like arp, ssh, mosh, tcp syn, synack, etc,
> > > no need to do that via classification.
> > >
> > >> normal: HTTP, HTTPS
> > >
> > > So you want to deprioritize vpn, smtp, rsync, dropbox, http on odd
> ports,
> > > caching servers, etc, all in favor of the web gods?
> > >
> > > we just toss all that into the normal (best effort bin) and let
> fq_codel
> > > sort it out.
> > >
> > >> bulk: everything else.
> > >
> > > We do respect the diffserv marking of CS1 (background) and toss it
> > > into the background queue.
> > >
> > >> Side note: Will SQM do bandwidth slicing? Or at least handle "hostile"
> > >> environments better? I say "hostile" as in roommates or maybe teenage
> kids.
> > >> Multiple people/devices thinking they are entitled to the entire WAN
> > >> bandwidth at all times.
> > >
> > > fq_codel does fair (well, we call it "flow") queuing.
> > >
> > > https://tools.ietf.org/html/draft-hoeiland-joergensen-aqm-fq-codel-00
> > >
> > > And manages the depth of flows via codel.
> > >
> > > http://tools.ietf.org/html/draft-nichols-tsvwg-codel-02
> > >
> > > In other words, it will be fair to all fat flows generated by everyone,
> > > and slice flows down to the defined quantum and turn them back into
> > > packets.
> > >
> > > The "simplest.qos" model in SQM works remarkably well without trying to
> > > classify anything at all. I encourage people to try merely that and
> have
> > > their preconceptions altered.
> > >
> > > The three-tier model (simple.qos), is more like what people think they
> want,
> > > but the default is set to the bare minimum of what worked well in
> testing.
> > >
> > > Example: a lot of flows are marked CS1 that shouldn't be, and starving
> > > that queue to like 5% rather than it's current 30% turned out badly.
> > >
> > > In terms of identifying and "punishing" abusers, well, the only thing
> > > that stresses this code out even the slightest is dozens of torrent
> flows.
> > >
> > > Give it a shot. :)
> > >
> > I feel like this died.
>
> It didn't die.
>
> *I* died.
>
> I'd been on a death march for the last 8 months trying to
> get the last bugs out of openwrt/cerowrt, and when the last big one
> got fixed (bug 442 in the cerowrt database, multiple other trackers)
>
> I put out a release of 3.10.50-1 pre BBrc1 and went to sleep.
>
> When I woke up, about a week ago, everybody had nearly 2 months
> uptime, good throughput, and a bunch of minor nits here and there.
>
> Hooray!
>
> The prospect of resyncing with BBrcX intimidates me, and I have
> had a ton of other things that slid to take care of, so I've
> been catching up on those. Sebastian has been taking
> care of SQM nits...
>
> https://github.com/dtaht/ceropackages-3.10/issues/8
>
> And Jonathon morton has been pouring it all into
> pure C - with an integral bandwidth shaper that we
> hope will be faster and more efficient than htb.
>
> See an early result:
>
> http://pastebin.com/zz06WhJr
>
> It takes much of the heavy lifting out of the existing
> sqm scripts.
>
> tc qdisc add dev eth1 root cake bandwidth 80mbit
>
>
> So I don't know where to go. Certainly I'd like to
> see the battle hardened sqm scripts (which are more
> flexible than the C code above) get more widely used
> and in BB.
>
> openwrt users can do that today by adding the ceropackages repo to their
> build system.
> or just installing the sqm-scripts and luci-app-sqm.
>
> or we can clean it up further for openwrt mainline.
>
> But I haven't seen one core openwrt dev say, yes, we want this mainlined,
> here's what you need to fix, so I'm inclined to go back to my cave, get
> more sleep, and work on the successor.
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>

[-- Attachment #2: Type: text/html, Size: 12053 bytes --]

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[David Lang]

[-- Attachment #1: Type: TEXT/Plain, Size: 9693 bytes --]

On Thu, 2 Oct 2014, Alpha Sparc wrote:

> How good is the throughput on CeroWrt compared to OpenWrt ?

The focus of CeroWrt is on reducing latency, not increasing throughput. If you 
run into really badd bufferbloat problems without these scrips, then these 
scripts can result more more 'goodput' (useable data as opposed to 'throughput' 
bits on the wire) getting through, but in the usual case there will be a 
(slight) reduction in the peak throughput.

This is especially so on the inbound side of things because the router is having 
to work indirectly to throttle the senders so that they don't overload the 
router at the other end of the connection.

I beleive that on the WNDR3800, it's able to work up to about 50Mb with the 
existing configurations. A faster CPU would do better, a slower one worse. The 
re-write that Dave is talking about is hoting to improve this. From the pastebin 
link Dave listed below, they have it up to ~80Mb now

David Lang

> On Oct 2, 2014 9:55 AM, "Dave Taht" <dave.taht@bufferbloat.net> wrote:
>
>> On Wed, Oct 01, 2014 at 12:10:46PM -0400, Weedy wrote:
>>> On 30/03/14 06:29 PM, Dave Taht wrote:
>>>> On Sun, Mar 30, 2014 at 02:24:44PM -0400, Weedy wrote:
>>>>> On Sat, Mar 29, 2014 at 2:56 PM, Dave Täht <dave.taht@bufferbloat.net
>>> wrote:
>>>>>
>>>>>> From: Dave Taht <dave.taht@bufferbloat.net>
>>>>>>
>>>>>> This adds support for the bufferbloat project's "Smart Queue
>> Management"
>>>>>> (SQM) system, which improves over openwrt's qos-scripts in the
>> following
>>>>>> ways
>>>>>>
>>>>>> + Uses HTB with two models for managing traffic
>>>>>>   a simplest one that merely uses fq_codel, and a three tier one
>> that does
>>>>>>   some basic and tunable packet prioritization.
>>>>>>
>>>>>> + Works with ipv6 and ipv4 correctly (unlike qos-scripts)
>>>>>> + extensive support for fixing ADSL and PPOe framing problems
>>>>>> + Partial support for key diffserv markings
>>>>>> + highly tuned fq_codel implementation especially for low bandwidths
>>>>>> + Tested heavily on cable modems and on dsl devices
>>>>>>
>>>>>> It is a disimprovement in that:
>>>>>>
>>>>>> - There are no built-in tricks for doing l7 classification,
>>>>>> or other forms of packet inspection.
>>>>>>
>>>>>> - We haven't explored hfsc all that much, prefering to rely
>>>>>> on the predictable behavior of htb + fq_codel for everything
>>>>>>
>>>>>> - And there is support for a few qdiscs that are not in the linux
>>>>>> kernel mainline that remain experimental.
>>>>>> ---
>>>>>>  net/sqm-scripts/Makefile                           |   48 +++
>>>>>>  net/sqm-scripts/files/etc/config/sqm               |   11 +
>>>>>>  net/sqm-scripts/files/etc/init.d/sqm               |   23 ++
>>>>>>  net/sqm-scripts/files/usr/lib/sqm/functions.sh     |  335
>>>>>> ++++++++++++++++++++
>>>>>>  net/sqm-scripts/files/usr/lib/sqm/run.sh           |   67 ++++
>>>>>>  net/sqm-scripts/files/usr/lib/sqm/simple.qos       |  187
>> +++++++++++
>>>>>>  net/sqm-scripts/files/usr/lib/sqm/simple.qos.help  |    1 +
>>>>>>  net/sqm-scripts/files/usr/lib/sqm/simplest.qos     |   84 +++++
>>>>>>  .../files/usr/lib/sqm/simplest.qos.help            |    1 +
>>>>>>  net/sqm-scripts/files/usr/lib/sqm/stop.sh          |   22 ++
>>>>>>  10 files changed, 779 insertions(+)
>>>>>>  create mode 100644 net/sqm-scripts/Makefile
>>>>>>  create mode 100644 net/sqm-scripts/files/etc/config/sqm
>>>>>>  create mode 100755 net/sqm-scripts/files/etc/init.d/sqm
>>>>>>  create mode 100644 net/sqm-scripts/files/usr/lib/sqm/functions.sh
>>>>>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/run.sh
>>>>>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/simple.qos
>>>>>>  create mode 100644 net/sqm-scripts/files/usr/lib/sqm/simple.qos.help
>>>>>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/simplest.qos
>>>>>>  create mode 100644
>> net/sqm-scripts/files/usr/lib/sqm/simplest.qos.help
>>>>>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/stop.sh
>>>>>>
>>>>>> diff --git a/net/sqm-scripts/files/etc/config/sqm
>>>>>> b/net/sqm-scripts/files/etc/config/sqm
>>>>>> new file mode 100644
>>>>>> index 0000000..547d321
>>>>>> --- /dev/null
>>>>>> +++ b/net/sqm-scripts/files/etc/config/sqm
>>>>>> @@ -0,0 +1,11 @@
>>>>>> +
>>>>>> +config queue 'ge00'
>>>>>> +        option enabled '0'
>>>>>> +        option interface 'ge00'
>>>>>> +        option download '20000'
>>>>>> +        option upload '4000'
>>>>>> +        option qdisc 'fq_codel'
>>>>>> +        option script 'simple.qos'
>>>>>> +        option qdisc_advanced '0'
>>>>>> +        option linklayer 'none'
>>>>>> +
>>>>>>
>>>>>
>>>>> How hard is this to config from the command line/vim?
>>>>
>>>> There are a few more options than this (for DSL compensation, ecn
>>>> and advanced configuration), the above would work if you changed
>>>> enabled to '1' and the device from ge00 to your wan device. (not
>>>> the "wan" firewall rule, presently. )
>>>>
>>>> It does help to have a sane long term and realistic measurement of your
>>>> network using something like the rrul test rather than the oft-gamed
>> speedtest.
>>>>
>>>>
>> http://www.bufferbloat.net/projects/cerowrt/wiki/Setting_up_SQM_for_CeroWrt_310
>>>>
>>>> You are right, we should fully document all the variables in this file.
>>>> Until recently they were kind of in flux.
>>>>
>>>>> I've never needed or really wanted luci on my box, I just use vim.
>> Going by
>>>>> this patch, there is either nothing to config or no examples. I would
>> think
>>>>> shipping a roughly equivalent config to what ships in qos-scripts
>> would be
>>>>> a good start to get people testing.
>>>>
>>>> /etc/init.d/sqm start,stop etc work as expected.
>>>>
>>>>> IE: highest priority: small ARP, DNS, and SSH packets
>>>>
>>>> The SQM default is local DNS and ntp. fq_codel automagically optimizes
>>>> for other sparse flows like arp, ssh, mosh, tcp syn, synack, etc,
>>>> no need to do that via classification.
>>>>
>>>>> normal: HTTP, HTTPS
>>>>
>>>> So you want to deprioritize vpn, smtp, rsync, dropbox, http on odd
>> ports,
>>>> caching servers, etc, all in favor of the web gods?
>>>>
>>>> we just toss all that into the normal (best effort bin) and let
>> fq_codel
>>>> sort it out.
>>>>
>>>>> bulk: everything else.
>>>>
>>>> We do respect the diffserv marking of CS1 (background) and toss it
>>>> into the background queue.
>>>>
>>>>> Side note: Will SQM do bandwidth slicing? Or at least handle "hostile"
>>>>> environments better? I say "hostile" as in roommates or maybe teenage
>> kids.
>>>>> Multiple people/devices thinking they are entitled to the entire WAN
>>>>> bandwidth at all times.
>>>>
>>>> fq_codel does fair (well, we call it "flow") queuing.
>>>>
>>>> https://tools.ietf.org/html/draft-hoeiland-joergensen-aqm-fq-codel-00
>>>>
>>>> And manages the depth of flows via codel.
>>>>
>>>> http://tools.ietf.org/html/draft-nichols-tsvwg-codel-02
>>>>
>>>> In other words, it will be fair to all fat flows generated by everyone,
>>>> and slice flows down to the defined quantum and turn them back into
>>>> packets.
>>>>
>>>> The "simplest.qos" model in SQM works remarkably well without trying to
>>>> classify anything at all. I encourage people to try merely that and
>> have
>>>> their preconceptions altered.
>>>>
>>>> The three-tier model (simple.qos), is more like what people think they
>> want,
>>>> but the default is set to the bare minimum of what worked well in
>> testing.
>>>>
>>>> Example: a lot of flows are marked CS1 that shouldn't be, and starving
>>>> that queue to like 5% rather than it's current 30% turned out badly.
>>>>
>>>> In terms of identifying and "punishing" abusers, well, the only thing
>>>> that stresses this code out even the slightest is dozens of torrent
>> flows.
>>>>
>>>> Give it a shot. :)
>>>>
>>> I feel like this died.
>>
>> It didn't die.
>>
>> *I* died.
>>
>> I'd been on a death march for the last 8 months trying to
>> get the last bugs out of openwrt/cerowrt, and when the last big one
>> got fixed (bug 442 in the cerowrt database, multiple other trackers)
>>
>> I put out a release of 3.10.50-1 pre BBrc1 and went to sleep.
>>
>> When I woke up, about a week ago, everybody had nearly 2 months
>> uptime, good throughput, and a bunch of minor nits here and there.
>>
>> Hooray!
>>
>> The prospect of resyncing with BBrcX intimidates me, and I have
>> had a ton of other things that slid to take care of, so I've
>> been catching up on those. Sebastian has been taking
>> care of SQM nits...
>>
>> https://github.com/dtaht/ceropackages-3.10/issues/8
>>
>> And Jonathon morton has been pouring it all into
>> pure C - with an integral bandwidth shaper that we
>> hope will be faster and more efficient than htb.
>>
>> See an early result:
>>
>> http://pastebin.com/zz06WhJr
>>
>> It takes much of the heavy lifting out of the existing
>> sqm scripts.
>>
>> tc qdisc add dev eth1 root cake bandwidth 80mbit
>>
>>
>> So I don't know where to go. Certainly I'd like to
>> see the battle hardened sqm scripts (which are more
>> flexible than the C code above) get more widely used
>> and in BB.
>>
>> openwrt users can do that today by adding the ceropackages repo to their
>> build system.
>> or just installing the sqm-scripts and luci-app-sqm.
>>
>> or we can clean it up further for openwrt mainline.
>>
>> But I haven't seen one core openwrt dev say, yes, we want this mainlined,
>> here's what you need to fix, so I'm inclined to go back to my cave, get
>> more sleep, and work on the successor.
>> _______________________________________________
>> openwrt-devel mailing list
>> openwrt-devel@lists.openwrt.org
>> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>>
>

[-- Attachment #2: Type: TEXT/PLAIN, Size: 172 bytes --]

_______________________________________________
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Sebastian Moeller]

Hi Weedy,

On Oct 2, 2014, at 05:46 , Alpha Sparc <alphasparc@gmail.com> wrote:

> How good is the throughput on CeroWrt compared to OpenWrt ?

	I assume you are talking about the pure routing performance with no firewall/NAT and traffic-shaping involved? I think they pretty much are equal (pretty much the same kernel and most of the cerowrt guts are from openwrt bb trunk). But I have not tested that (I have only one cerowrt/openwrt capable router and that pretty much is my main router).
	If you are talking about comparing QOS-scripts with SQM-scripts, they also seem to top out at roughly 50-60 Mbps (down- and uplink combined), it seems hfsc (qos-scripts) and HTB (sam-scripts) are equally expensive on MIPS.
	Now if you are setup to do tests yourself I would love to hear the results. I would be happy to help you getting SQM-scripts to work (so far all people interested disappeared before or just after sharing initial test results).

Best Regards
	Sebastian


> 
> On Oct 2, 2014 9:55 AM, "Dave Taht" <dave.taht@bufferbloat.net> wrote:
> On Wed, Oct 01, 2014 at 12:10:46PM -0400, Weedy wrote:
> > On 30/03/14 06:29 PM, Dave Taht wrote:
> > > On Sun, Mar 30, 2014 at 02:24:44PM -0400, Weedy wrote:
> > >> On Sat, Mar 29, 2014 at 2:56 PM, Dave Täht <dave.taht@bufferbloat.net>wrote:
> > >>
> > >>> From: Dave Taht <dave.taht@bufferbloat.net>
> > >>>
> > >>> This adds support for the bufferbloat project's "Smart Queue Management"
> > >>> (SQM) system, which improves over openwrt's qos-scripts in the following
> > >>> ways
> > >>>
> > >>> + Uses HTB with two models for managing traffic
> > >>>   a simplest one that merely uses fq_codel, and a three tier one that does
> > >>>   some basic and tunable packet prioritization.
> > >>>
> > >>> + Works with ipv6 and ipv4 correctly (unlike qos-scripts)
> > >>> + extensive support for fixing ADSL and PPOe framing problems
> > >>> + Partial support for key diffserv markings
> > >>> + highly tuned fq_codel implementation especially for low bandwidths
> > >>> + Tested heavily on cable modems and on dsl devices
> > >>>
> > >>> It is a disimprovement in that:
> > >>>
> > >>> - There are no built-in tricks for doing l7 classification,
> > >>> or other forms of packet inspection.
> > >>>
> > >>> - We haven't explored hfsc all that much, prefering to rely
> > >>> on the predictable behavior of htb + fq_codel for everything
> > >>>
> > >>> - And there is support for a few qdiscs that are not in the linux
> > >>> kernel mainline that remain experimental.
> > >>> ---
> > >>>  net/sqm-scripts/Makefile                           |   48 +++
> > >>>  net/sqm-scripts/files/etc/config/sqm               |   11 +
> > >>>  net/sqm-scripts/files/etc/init.d/sqm               |   23 ++
> > >>>  net/sqm-scripts/files/usr/lib/sqm/functions.sh     |  335
> > >>> ++++++++++++++++++++
> > >>>  net/sqm-scripts/files/usr/lib/sqm/run.sh           |   67 ++++
> > >>>  net/sqm-scripts/files/usr/lib/sqm/simple.qos       |  187 +++++++++++
> > >>>  net/sqm-scripts/files/usr/lib/sqm/simple.qos.help  |    1 +
> > >>>  net/sqm-scripts/files/usr/lib/sqm/simplest.qos     |   84 +++++
> > >>>  .../files/usr/lib/sqm/simplest.qos.help            |    1 +
> > >>>  net/sqm-scripts/files/usr/lib/sqm/stop.sh          |   22 ++
> > >>>  10 files changed, 779 insertions(+)
> > >>>  create mode 100644 net/sqm-scripts/Makefile
> > >>>  create mode 100644 net/sqm-scripts/files/etc/config/sqm
> > >>>  create mode 100755 net/sqm-scripts/files/etc/init.d/sqm
> > >>>  create mode 100644 net/sqm-scripts/files/usr/lib/sqm/functions.sh
> > >>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/run.sh
> > >>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/simple.qos
> > >>>  create mode 100644 net/sqm-scripts/files/usr/lib/sqm/simple.qos.help
> > >>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/simplest.qos
> > >>>  create mode 100644 net/sqm-scripts/files/usr/lib/sqm/simplest.qos.help
> > >>>  create mode 100755 net/sqm-scripts/files/usr/lib/sqm/stop.sh
> > >>>
> > >>> diff --git a/net/sqm-scripts/files/etc/config/sqm
> > >>> b/net/sqm-scripts/files/etc/config/sqm
> > >>> new file mode 100644
> > >>> index 0000000..547d321
> > >>> --- /dev/null
> > >>> +++ b/net/sqm-scripts/files/etc/config/sqm
> > >>> @@ -0,0 +1,11 @@
> > >>> +
> > >>> +config queue 'ge00'
> > >>> +        option enabled '0'
> > >>> +        option interface 'ge00'
> > >>> +        option download '20000'
> > >>> +        option upload '4000'
> > >>> +        option qdisc 'fq_codel'
> > >>> +        option script 'simple.qos'
> > >>> +        option qdisc_advanced '0'
> > >>> +        option linklayer 'none'
> > >>> +
> > >>>
> > >>
> > >> How hard is this to config from the command line/vim?
> > >
> > > There are a few more options than this (for DSL compensation, ecn
> > > and advanced configuration), the above would work if you changed
> > > enabled to '1' and the device from ge00 to your wan device. (not
> > > the "wan" firewall rule, presently. )
> > >
> > > It does help to have a sane long term and realistic measurement of your
> > > network using something like the rrul test rather than the oft-gamed speedtest.
> > >
> > > http://www.bufferbloat.net/projects/cerowrt/wiki/Setting_up_SQM_for_CeroWrt_310
> > >
> > > You are right, we should fully document all the variables in this file.
> > > Until recently they were kind of in flux.
> > >
> > >> I've never needed or really wanted luci on my box, I just use vim. Going by
> > >> this patch, there is either nothing to config or no examples. I would think
> > >> shipping a roughly equivalent config to what ships in qos-scripts would be
> > >> a good start to get people testing.
> > >
> > > /etc/init.d/sqm start,stop etc work as expected.
> > >
> > >> IE: highest priority: small ARP, DNS, and SSH packets
> > >
> > > The SQM default is local DNS and ntp. fq_codel automagically optimizes
> > > for other sparse flows like arp, ssh, mosh, tcp syn, synack, etc,
> > > no need to do that via classification.
> > >
> > >> normal: HTTP, HTTPS
> > >
> > > So you want to deprioritize vpn, smtp, rsync, dropbox, http on odd ports,
> > > caching servers, etc, all in favor of the web gods?
> > >
> > > we just toss all that into the normal (best effort bin) and let fq_codel
> > > sort it out.
> > >
> > >> bulk: everything else.
> > >
> > > We do respect the diffserv marking of CS1 (background) and toss it
> > > into the background queue.
> > >
> > >> Side note: Will SQM do bandwidth slicing? Or at least handle "hostile"
> > >> environments better? I say "hostile" as in roommates or maybe teenage kids.
> > >> Multiple people/devices thinking they are entitled to the entire WAN
> > >> bandwidth at all times.
> > >
> > > fq_codel does fair (well, we call it "flow") queuing.
> > >
> > > https://tools.ietf.org/html/draft-hoeiland-joergensen-aqm-fq-codel-00
> > >
> > > And manages the depth of flows via codel.
> > >
> > > http://tools.ietf.org/html/draft-nichols-tsvwg-codel-02
> > >
> > > In other words, it will be fair to all fat flows generated by everyone,
> > > and slice flows down to the defined quantum and turn them back into
> > > packets.
> > >
> > > The "simplest.qos" model in SQM works remarkably well without trying to
> > > classify anything at all. I encourage people to try merely that and have
> > > their preconceptions altered.
> > >
> > > The three-tier model (simple.qos), is more like what people think they want,
> > > but the default is set to the bare minimum of what worked well in testing.
> > >
> > > Example: a lot of flows are marked CS1 that shouldn't be, and starving
> > > that queue to like 5% rather than it's current 30% turned out badly.
> > >
> > > In terms of identifying and "punishing" abusers, well, the only thing
> > > that stresses this code out even the slightest is dozens of torrent flows.
> > >
> > > Give it a shot. :)
> > >
> > I feel like this died.
> 
> It didn't die.
> 
> *I* died.
> 
> I'd been on a death march for the last 8 months trying to
> get the last bugs out of openwrt/cerowrt, and when the last big one
> got fixed (bug 442 in the cerowrt database, multiple other trackers)
> 
> I put out a release of 3.10.50-1 pre BBrc1 and went to sleep.
> 
> When I woke up, about a week ago, everybody had nearly 2 months
> uptime, good throughput, and a bunch of minor nits here and there.
> 
> Hooray!
> 
> The prospect of resyncing with BBrcX intimidates me, and I have
> had a ton of other things that slid to take care of, so I've
> been catching up on those. Sebastian has been taking
> care of SQM nits...
> 
> https://github.com/dtaht/ceropackages-3.10/issues/8
> 
> And Jonathon morton has been pouring it all into
> pure C - with an integral bandwidth shaper that we
> hope will be faster and more efficient than htb.
> 
> See an early result:
> 
> http://pastebin.com/zz06WhJr
> 
> It takes much of the heavy lifting out of the existing
> sqm scripts.
> 
> tc qdisc add dev eth1 root cake bandwidth 80mbit
> 
> 
> So I don't know where to go. Certainly I'd like to
> see the battle hardened sqm scripts (which are more
> flexible than the C code above) get more widely used
> and in BB.
> 
> openwrt users can do that today by adding the ceropackages repo to their build system.
> or just installing the sqm-scripts and luci-app-sqm.
> 
> or we can clean it up further for openwrt mainline.
> 
> But I haven't seen one core openwrt dev say, yes, we want this mainlined, here's what you need to fix, so I'm inclined to go back to my cave, get more sleep, and work on the successor.
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Richard A. Smith]

On 10/02/2014 10:05 AM, Sebastian Moeller wrote:

> 	I assume you are talking about the pure routing performance with no firewall/NAT and traffic-shaping involved? I think they pretty much are equal (pretty much the same kernel and most of the cerowrt guts are from openwrt bb trunk). But I have not tested that (I have only one cerowrt/openwrt capable router and that pretty much is my main router).
> 	If you are talking about comparing QOS-scripts with SQM-scripts, they also seem to top out at roughly 50-60 Mbps (down- and uplink combined), it seems hfsc (qos-scripts) and HTB (sam-scripts) are equally expensive on MIPS.
> 	Now if you are setup to do tests yourself I would love to hear the results. I would be happy to help you getting SQM-scripts to work (so far all people interested disappeared before or just after sharing initial test results).

Do you still need testers?  I have a bit of an interest here.

I have spare routers that I can run OpenWRT or CeroWRt on and I'm setup 
to test with netperf, netperf-wrapper on my local network
  ( desktop -> router -> laptop )  it's Gbit so I can easily saturate 
the router.

What I don't have is a lot of time but I can do a few runs in the 
evenings or on weekends.  I also am very out of touch with the latest 
and greatest QoS vs SQM development and configuration so you will have 
to feed me test recipes.

-- 
Richard A. Smith  <richard@laptop.org>
Former One Laptop per Child

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Sebastian Moeller]

Hi Richard,


On Oct 6, 2014, at 18:02 , Richard A. Smith <richard@laptop.org> wrote:

> On 10/02/2014 10:05 AM, Sebastian Moeller wrote:
> 
>> 	I assume you are talking about the pure routing performance with no firewall/NAT and traffic-shaping involved? I think they pretty much are equal (pretty much the same kernel and most of the cerowrt guts are from openwrt bb trunk). But I have not tested that (I have only one cerowrt/openwrt capable router and that pretty much is my main router).
>> 	If you are talking about comparing QOS-scripts with SQM-scripts, they also seem to top out at roughly 50-60 Mbps (down- and uplink combined), it seems hfsc (qos-scripts) and HTB (sam-scripts) are equally expensive on MIPS.
>> 	Now if you are setup to do tests yourself I would love to hear the results. I would be happy to help you getting SQM-scripts to work (so far all people interested disappeared before or just after sharing initial test results).
> 
> Do you still need testers?  I have a bit of an interest here.

	Oh, sure every little bit of testing is helpful (especially on openwrt, as I am not setup to test openwrt at all). I might be that SQM-scripts will explode spectacularly, but I hope that there is only a little “impedance-mismatch” ;)

> 
> I have spare routers that I can run OpenWRT or CeroWRt on and I'm setup to test with netperf, netperf-wrapper on my local network
> ( desktop -> router -> laptop )  it's Gbit so I can easily saturate the router.

	That sounds great. I think the first test should be to run SQM under cerowrt, so you get a feel of how things should look. I typically run netperf-wrapper rrul tests (for ipv4 and if available for ipv6) through cerowrt with different settings for SQM. A second step then is to instal SQM-scripts under openwrt and check whether the same settings produce the same results ;)

> 
> What I don't have is a lot of time but I can do a few runs in the evenings or on weekends.  

	I think all that is needed is testing a few relevant shaping bandwidth combinations (always Downlink/Uplink, 3000Kbps/512Kbps, 16000Kbps/1000Kbps, 50000Kbps/10000Kbps, 100000Kbps/40000Kbps, and 0/50000Kbps, with a setting of 0 disabling shaping in a particular direction, 0/50000Kbps with the ethernet interface set 100Mbps, and with SQM disabled: pick the test set most relevant to your planned deployment to save time) And finally it would be great to test whether the ATM link layer encapsulation also works…
	Basically that is the set of things to test, but the most important is just testing one of them to see hw SQM-scripts (and luci-app-sqm works under stock openwrt)


> I also am very out of touch with the latest and greatest QoS vs SQM development and configuration so you will have to feed me test recipes.

	Oh, happy to help out with this, thanks to Toke’s netperf-wrapper my tests typically look like (for say 3000Kbps/512Kbps):
date ; ping -c 10 netperf-eu.bufferbloat.net ; ./netperf-wrapper --ipv4 -l 300 -s 0.4 -H netperf-eu.bufferbloat.net rrul -p all_scaled --disable-log -t IPv4_test_D3000Kbps-U512Kbps

followed by:
date ; ping -c 10 netperf-eu.bufferbloat.net ; ./netperf-wrapper --ipv6 -l 300 -s 0.4 -H netperf-eu.bufferbloat.net rrul -p all_scaled --disable-log -t IPv6_test_D3000Kbps-U512Kbps

for links faster than ~5Mbps or so you can skip the -s 0.4 line and for links slower than 2Mbps you probably need rouse -s 0.8 or even -s 1.0, running for 5 minutes (-l 300) makes sure you can also judge the robustness/stability of the shaping...
And then simply load the resulting output files in netperf-wrapper’s GUI to simplify comparison between the different SQM settings (and since these tests work independent of the used shaper you could also run the relevant shaper settings for your own situation through QOS. So you can compare how these two stack up against each other…)
	It might be a good idea to capture the output of:
logread
and:
tc -d qdisc
and
tc class show dev ge00 (for cerowrt, “tc class show dev wan” for openwrt, I believe)
before each test run on the router (this allows to confirm whether the selected shaper settings actually were applied properly)

I also, very unscientifically, ssh into the router while the tests are running and start “top -d 1” and visually monitor the %idle and %sirq (softinterrupts), the first goes to 0% and the second to >90% once you reached the your router’s shaping limit.

So just let me know what you are willing/ready to test and we will take it from there okay? (I would already be a happy camper if you could just install the current SQM-scripts on openwrt and just send me the output of “logread” after installing and activating SQM, as well as the output from “tc -d qdisc” before and after enabling SQM, and finally the output of running “/etc/init.d/sqm stop ; /etc/init.d/sqm start” on the router’s console; that hopefully works or at least gives some indication what might be off. If you could throw in a quick netperf-wrapper RRUL test through the router I will be most delighted ;))

Best Regards
	Sebastian


> 
> -- 
> Richard A. Smith  <richard@laptop.org>
> Former One Laptop per Child

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Dave Taht]

On Wed, Oct 08, 2014 at 11:10:46PM +0300, Hannu Nyman wrote:
> Dave Taht wrote on Thu Oct 2 03:49:15 CEST 2014:
> > So I don't know where to go. Certainly I'd like to see the battle
> hardened sqm scripts (which are more flexible than the C code above)
> get more widely used and in BB.
> 
> SQM seems to work ok with the current Chaos Calmer trunk.

Well, for now... I'd expect the delta to break eventually...

> 
> I have included your SQM in my trunk WNDR3700/3800 community build (
> https://forum.openwrt.org/viewtopic.php?id=28392 ) and it works ok.
> Arokh has also picked it up for his build (
> https://forum.openwrt.org/viewtopic.php?id=50914 ). So you might get
> more feedback sooner or later, if users of our builds really do try
> it.

Excellent. I'd *really* like to get some testers doing ingress shaping
at above 60-80mbits, which seems to be a brick wall we've hit on
the ar71xx and octeon, on other platforms like the arm and x86.

A tool we use a lot is netperf-wrapper.

> I feel that the best way to get wider testing and real-life usage
> for SQM would be to create a pull request in the new packages feed
> in Github, to get both the SQM itself and the Luci frontend included
> there. Being available in the packages feed would create more
> interest for the package. If it proves to work ok, the devs might
> then import it into the core Openwrt repo (where the old qos-scripts
> is).

I went to sleep pre BBrc1. I woke up, and everything was in github.

It's still not clear to me how I'd build cero from frozen BB sources,
rather than the evolving github packages.

> 
> 
> Ps. For my own build I made a slight modification to the Luci menu> item, as pure "SQM" does not say much. I changed it to "SQM QoS".
> 

Yes, SQM by itself doesn't mean enough.

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Dave Taht]

On Thu, Oct 09, 2014 at 01:01:48AM +0200, Stephan Günther wrote:
> Hi,
> 
> On Wed, Oct 8, 2014 at 10:10 PM, Hannu Nyman <hannu.nyman@iki.fi> wrote:
> > Dave Taht wrote on Thu Oct 2 03:49:15 CEST 2014:
> >> So I don't know where to go. Certainly I'd like to see the battle hardened
> >> sqm scripts (which are more flexible than the C code above) get more widely
> >> used and in BB.
> >
> > SQM seems to work ok with the current Chaos Calmer trunk.
> 
> Works for mee too, and performs much better than the old luci-app-qos.
> I would love to see this as part of OpenWrt.

OK. I don't see it feasible to retire qos-scripts as that has less dependencies
than sqm does - sqm needs "ip" and tc to function.

But I'd certainly like to see it available in the main openwrt repo by default.

And: I'd like the next version to do what sqm does, in pure c,
at line rate OR software rate limited, faster, better,
smaller... 
 
> I did some RRUL test using netperf-wrapper on my ADSL 15/1Mbps PPPoE
> link and it looks good in the graphs. I also have an 6in4 tunnel

I always love it when people post their results and the .json.gz files
for the various netperf-wrapper tests somewhere. It has been good to 
build an ever increasing database of valid tests and valid setups, given
that things like speedtest.net are so lame.

Examples:

http://burntchrome.blogspot.com/2014_05_01_archive.html

http://snapon.lab.bufferbloat.net/~cero2/jimreisert/results.html

While I'm at it there were a couple manefestos along the way.

This explains exactly where and why wondershaper was flawed:

http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die

And this talks to the need for fq + aqm on *everything*

http://gettys.wordpress.com/2013/07/10/low-latency-requires-smart-queuing-traditional-aqm-is-not-enough/

(I was unhappy qos-scripts just disabled ecn entirely. ECN is looking
 safely deployable in a fq'd system IMHO).

Last manefesto above does not go into the (slight) remaining need for a few levels
of classification, one reason is here:

http://perso.telecom-paristech.fr/~drossi/paper/rossi13tma-b.pdf

It is my hope that by explaining the "why" of sqm we could 
come up with something better before making it available
at larger scale.

> inside PPPoE and IIRC fq_codel should detect these ipv6 flows. RRUL

fq_codel dissects the headers for ipip, ipv6, gre and another protocol
I forget, correctly, and fq's them correctly. 

However my head explodes as to what happens or which
device should be used when that is further encapsulated.

> looks good at IPv6. Had this running at home for some days now, with
> moderate traffic and no issues so far.

Well loading it up is the only way to tell if you're winning.

> But I was wondering which interface to select luci-app-sqm, as no
> tunnel intefaces are shown here. So i used the ethernet interface
> instead of the PPPoE link. Is this fine? Minutes ago, I did a quick
> test and applied SQM to the PPPoE link by fixing luci-base to return
> also the virtual interfaces in net:get_interfaces(). But i didn't
> notice any difference or my test was too sloppy.

Well, sebastian just made a few SQM changes also in the ceropackages
repo and PPPoe over atm makes my head hurt. See cerowrt-devel
for more list. 

I'm a huge believer in measurements, and netperf-wrapper has been
the closest thing the Internet has ever had to one that accurately
measures latency under load. Recently it was proven to scale all
the way to 40GigE. 

Things like speedtest are increasingly inaccurate above 20mbits,
and doesn't measure induced latency at all...

and netalyzer, being written in java, doesn't get past 20mbits
either.

> 
> --
> Stephan
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Richard Smith]

On 10/06/2014 03:41 PM, Sebastian Moeller wrote:

>>
>> I have spare routers that I can run OpenWRT or CeroWRt on and I'm
>> setup to test with netperf, netperf-wrapper on my local network (
>> desktop -> router -> laptop )  it's Gbit so I can easily saturate
>> the router.
>
> That sounds great. I think the first test should be to run SQM under
> cerowrt, so you get a feel of how things should look. I typically run
> netperf-wrapper rrul tests (for ipv4 and if available for ipv6)
> through cerowrt with different settings for SQM. A second step then
> is to instal SQM-scripts under openwrt and check whether the same
> settings produce the same results ;)

Ok.  I'll get my spare WNDR3700v2 up running the latest cerowrt.  Should 
be in the next few days.  I'm a bit strapped for time right now.

> So just let me know what you are willing/ready to test and we will
> take it from there okay? (I would already be a happy camper if you
> could just install the current SQM-scripts on openwrt and just send
> me the output of “logread” after installing and activating SQM, as
> well as the output from “tc -d qdisc” before and after enabling SQM,
> and finally the output of running “/etc/init.d/sqm stop ;
> /etc/init.d/sqm start” on the router’s console; that hopefully works
> or at least gives some indication what might be off. If you could
> throw in a quick netperf-wrapper RRUL test through the router I will
> be most delighted ;))

Ok. That sounds like a good starter step.  First question: Where are the 
current SQM scripts located and what's the process for installing them? 
  Are they packages or stuff that I scp over and manually install?

I'll let you know when I've taken a stab at getting that working.

 > Richard A. Smith  <richard@laptop.org>
 > Former One Laptop per Child

Just FYI: I'm slowly switching my list subs over from my previous 
@laptop.org address to my personal address.  Just switched this one over 
thus the email address change.

-- 
Richard A. Smith

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Richard Smith]

On 10/08/2014 11:01 PM, Dave Taht wrote:

> Excellent. I'd *really* like to get some testers doing ingress shaping
> at above 60-80mbits, which seems to be a brick wall we've hit on
> the ar71xx and octeon, on other platforms like the arm and x86.

I can do ingress shaping tests at above 80mbits at work since we now 
have a 100mbit downlink, but I don't have cerowrt hardware that runs 
fast enough to keep up.  Also I would only be able to do them on the 
weekend or late at night when no one else is in the office when I can 
swap out the router.  I can't leave experimental setups alone running 
our office link.

FYI: I've purchased a Nighthawk x6 for some testing.  I did x6 even 
though its CPU speed is slower than the x4 because I want to see if the 
twin 5Ghz radios and the promise of automatic load balancing helps on 
the wireless side (we have a lot of 5Ghz wireless clients)  If it 
doesn't hold up as a router we may still use it as just an AP.

Like you mentioned the stock firmware is not the greatest and there's no 
actual bandwidth control settings.  Hopefully openwrt support will show 
up soon.

My initial tests with Gbit pipes on either side are certainly impressive 
as it can route at over 900 Mbs while keeping latency at 12ms or lower. 
  Tonight I'm planning on testing it with my home Comcast cable link and 
see if the low-latency holds.

-- 
Richard A. Smith

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Sebastian Moeller]

Hi Richard,

On Oct 9, 2014, at 18:42 , Richard Smith <smithbone@gmail.com> wrote:

> On 10/06/2014 03:41 PM, Sebastian Moeller wrote:
> 
>>> 
>>> I have spare routers that I can run OpenWRT or CeroWRt on and I'm
>>> setup to test with netperf, netperf-wrapper on my local network (
>>> desktop -> router -> laptop )  it's Gbit so I can easily saturate
>>> the router.
>> 
>> That sounds great. I think the first test should be to run SQM under
>> cerowrt, so you get a feel of how things should look. I typically run
>> netperf-wrapper rrul tests (for ipv4 and if available for ipv6)
>> through cerowrt with different settings for SQM. A second step then
>> is to instal SQM-scripts under openwrt and check whether the same
>> settings produce the same results ;)
> 
> Ok.  I'll get my spare WNDR3700v2 up running the latest cerowrt.  Should be in the next few days.  I'm a bit strapped for time right now.

	No hurry, I really appreciate your help and there is no time pressure (rather the other way around, I do this for a hobby, and real life leaves almost no time for  that ;) )

> 
>> So just let me know what you are willing/ready to test and we will
>> take it from there okay? (I would already be a happy camper if you
>> could just install the current SQM-scripts on openwrt and just send
>> me the output of “logread” after installing and activating SQM, as
>> well as the output from “tc -d qdisc” before and after enabling SQM,
>> and finally the output of running “/etc/init.d/sqm stop ;
>> /etc/init.d/sqm start” on the router’s console; that hopefully works
>> or at least gives some indication what might be off. If you could
>> throw in a quick netperf-wrapper RRUL test through the router I will
>> be most delighted ;))
> 
> Ok. That sounds like a good starter step.  First question: Where are the current SQM scripts located and what's the process for installing them?  Are they packages or stuff that I scp over and manually install?

	So the repository lives on:
https://github.com/dtaht/ceropackages-3.10
 I just followed Dave’s advice and used:
git clone git@github.com:dtaht/ceropackages-3.3.git
to get my own version to play around with. Oh, SQM basically is the work of Dave (who wrote the scripts doing the actual AQM&QOS work) and Toke you packaged all this nicely and created the GUI for it (all I ever did was shuffling the GUI around a bit). I just started hacking around the files from sqm-scripts and luck-app-sqm on my cerowrt (so I could immediately do some testing), so I never got around building actual packages (I assume you need to be setup to build whole openwrt/cerowrt images and that is outside the scope of my recreational coding), But that said if you copy the files from ceropackages-3.10/net/sqm-scripts/files to the matching directory on your router you should have a working sqm-script. Or just use a cerowrt built as SQM is standard and potentially hnyman’s build ( https://forum.openwrt.org/viewtopic.php?id=28392 ) that also includes SQM. 

Best Regards
	Sebastian

> 
> I'll let you know when I've taken a stab at getting that working.
> 
> > Richard A. Smith  <richard@laptop.org>
> > Former One Laptop per Child
> 
> Just FYI: I'm slowly switching my list subs over from my previous @laptop.org address to my personal address.  Just switched this one over thus the email address change.
> 
> -- 
> Richard A. Smith

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Luis E. Garcia]

[-- Attachment #1: Type: text/plain, Size: 4120 bytes --]

Please excuse my dumb question or just point me to the right documentation.
But is there an easy way to run AQM/Codel on BB 14.07 ?? I'm running BB on
a MyNETN750 that looks like it meets the HW requirements.

Regards,
Luis Garcia

On Thu, Oct 9, 2014 at 12:57 PM, Sebastian Moeller <moeller0@gmx.de> wrote:

> Hi Richard,
>
> On Oct 9, 2014, at 18:42 , Richard Smith <smithbone@gmail.com> wrote:
>
> > On 10/06/2014 03:41 PM, Sebastian Moeller wrote:
> >
> >>>
> >>> I have spare routers that I can run OpenWRT or CeroWRt on and I'm
> >>> setup to test with netperf, netperf-wrapper on my local network (
> >>> desktop -> router -> laptop )  it's Gbit so I can easily saturate
> >>> the router.
> >>
> >> That sounds great. I think the first test should be to run SQM under
> >> cerowrt, so you get a feel of how things should look. I typically run
> >> netperf-wrapper rrul tests (for ipv4 and if available for ipv6)
> >> through cerowrt with different settings for SQM. A second step then
> >> is to instal SQM-scripts under openwrt and check whether the same
> >> settings produce the same results ;)
> >
> > Ok.  I'll get my spare WNDR3700v2 up running the latest cerowrt.  Should
> be in the next few days.  I'm a bit strapped for time right now.
>
>         No hurry, I really appreciate your help and there is no time
> pressure (rather the other way around, I do this for a hobby, and real life
> leaves almost no time for  that ;) )
>
> >
> >> So just let me know what you are willing/ready to test and we will
> >> take it from there okay? (I would already be a happy camper if you
> >> could just install the current SQM-scripts on openwrt and just send
> >> me the output of “logread” after installing and activating SQM, as
> >> well as the output from “tc -d qdisc” before and after enabling SQM,
> >> and finally the output of running “/etc/init.d/sqm stop ;
> >> /etc/init.d/sqm start” on the router’s console; that hopefully works
> >> or at least gives some indication what might be off. If you could
> >> throw in a quick netperf-wrapper RRUL test through the router I will
> >> be most delighted ;))
> >
> > Ok. That sounds like a good starter step.  First question: Where are the
> current SQM scripts located and what's the process for installing them?
> Are they packages or stuff that I scp over and manually install?
>
>         So the repository lives on:
> https://github.com/dtaht/ceropackages-3.10
>  I just followed Dave’s advice and used:
> git clone git@github.com:dtaht/ceropackages-3.3.git
> to get my own version to play around with. Oh, SQM basically is the work
> of Dave (who wrote the scripts doing the actual AQM&QOS work) and Toke you
> packaged all this nicely and created the GUI for it (all I ever did was
> shuffling the GUI around a bit). I just started hacking around the files
> from sqm-scripts and luck-app-sqm on my cerowrt (so I could immediately do
> some testing), so I never got around building actual packages (I assume you
> need to be setup to build whole openwrt/cerowrt images and that is outside
> the scope of my recreational coding), But that said if you copy the files
> from ceropackages-3.10/net/sqm-scripts/files to the matching directory on
> your router you should have a working sqm-script. Or just use a cerowrt
> built as SQM is standard and potentially hnyman’s build (
> https://forum.openwrt.org/viewtopic.php?id=28392 ) that also includes SQM.
>
> Best Regards
>         Sebastian
>
> >
> > I'll let you know when I've taken a stab at getting that working.
> >
> > > Richard A. Smith  <richard@laptop.org>
> > > Former One Laptop per Child
> >
> > Just FYI: I'm slowly switching my list subs over from my previous @
> laptop.org address to my personal address.  Just switched this one over
> thus the email address change.
> >
> > --
> > Richard A. Smith
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>

[-- Attachment #2: Type: text/html, Size: 5228 bytes --]

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Dave Taht]

3.3 is obsolete, use 3.10.

On Thu, Oct 9, 2014 at 10:57 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Richard,
>
> On Oct 9, 2014, at 18:42 , Richard Smith <smithbone@gmail.com> wrote:
>
>> On 10/06/2014 03:41 PM, Sebastian Moeller wrote:
>>
>>>>
>>>> I have spare routers that I can run OpenWRT or CeroWRt on and I'm
>>>> setup to test with netperf, netperf-wrapper on my local network (
>>>> desktop -> router -> laptop )  it's Gbit so I can easily saturate
>>>> the router.
>>>
>>> That sounds great. I think the first test should be to run SQM under
>>> cerowrt, so you get a feel of how things should look. I typically run
>>> netperf-wrapper rrul tests (for ipv4 and if available for ipv6)
>>> through cerowrt with different settings for SQM. A second step then
>>> is to instal SQM-scripts under openwrt and check whether the same
>>> settings produce the same results ;)
>>
>> Ok.  I'll get my spare WNDR3700v2 up running the latest cerowrt.  Should be in the next few days.  I'm a bit strapped for time right now.
>
>         No hurry, I really appreciate your help and there is no time pressure (rather the other way around, I do this for a hobby, and real life leaves almost no time for  that ;) )
>
>>
>>> So just let me know what you are willing/ready to test and we will
>>> take it from there okay? (I would already be a happy camper if you
>>> could just install the current SQM-scripts on openwrt and just send
>>> me the output of “logread” after installing and activating SQM, as
>>> well as the output from “tc -d qdisc” before and after enabling SQM,
>>> and finally the output of running “/etc/init.d/sqm stop ;
>>> /etc/init.d/sqm start” on the router’s console; that hopefully works
>>> or at least gives some indication what might be off. If you could
>>> throw in a quick netperf-wrapper RRUL test through the router I will
>>> be most delighted ;))
>>
>> Ok. That sounds like a good starter step.  First question: Where are the current SQM scripts located and what's the process for installing them?  Are they packages or stuff that I scp over and manually install?
>
>         So the repository lives on:
> https://github.com/dtaht/ceropackages-3.10
>  I just followed Dave’s advice and used:
> git clone git@github.com:dtaht/ceropackages-3.3.git
> to get my own version to play around with. Oh, SQM basically is the work of Dave (who wrote the scripts doing the actual AQM&QOS work) and Toke you packaged all this nicely and created the GUI for it (all I ever did was shuffling the GUI around a bit). I just started hacking around the files from sqm-scripts and luck-app-sqm on my cerowrt (so I could immediately do some testing), so I never got around building actual packages (I assume you need to be setup to build whole openwrt/cerowrt images and that is outside the scope of my recreational coding), But that said if you copy the files from ceropackages-3.10/net/sqm-scripts/files to the matching directory on your router you should have a working sqm-script. Or just use a cerowrt built as SQM is standard and potentially hnyman’s build ( https://forum.openwrt.org/viewtopic.php?id=28392 ) that also includes SQM.
>
> Best Regards
>         Sebastian
>
>>
>> I'll let you know when I've taken a stab at getting that working.
>>
>> > Richard A. Smith  <richard@laptop.org>
>> > Former One Laptop per Child
>>
>> Just FYI: I'm slowly switching my list subs over from my previous @laptop.org address to my personal address.  Just switched this one over thus the email address change.
>>
>> --
>> Richard A. Smith
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht

https://www.bufferbloat.net/projects/make-wifi-fast

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Sebastian Moeller]

Ooops,


On Oct 9, 2014, at 20:13 , Dave Taht <dave.taht@gmail.com> wrote:

> 3.3 is obsolete, use 3.10.

What Dave says, I should have actually looked at the copied link…

Sorry
	Sebastian


> 
> On Thu, Oct 9, 2014 at 10:57 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> Hi Richard,
>> 
>> On Oct 9, 2014, at 18:42 , Richard Smith <smithbone@gmail.com> wrote:
>> 
>>> On 10/06/2014 03:41 PM, Sebastian Moeller wrote:
>>> 
>>>>> 
>>>>> I have spare routers that I can run OpenWRT or CeroWRt on and I'm
>>>>> setup to test with netperf, netperf-wrapper on my local network (
>>>>> desktop -> router -> laptop )  it's Gbit so I can easily saturate
>>>>> the router.
>>>> 
>>>> That sounds great. I think the first test should be to run SQM under
>>>> cerowrt, so you get a feel of how things should look. I typically run
>>>> netperf-wrapper rrul tests (for ipv4 and if available for ipv6)
>>>> through cerowrt with different settings for SQM. A second step then
>>>> is to instal SQM-scripts under openwrt and check whether the same
>>>> settings produce the same results ;)
>>> 
>>> Ok.  I'll get my spare WNDR3700v2 up running the latest cerowrt.  Should be in the next few days.  I'm a bit strapped for time right now.
>> 
>>        No hurry, I really appreciate your help and there is no time pressure (rather the other way around, I do this for a hobby, and real life leaves almost no time for  that ;) )
>> 
>>> 
>>>> So just let me know what you are willing/ready to test and we will
>>>> take it from there okay? (I would already be a happy camper if you
>>>> could just install the current SQM-scripts on openwrt and just send
>>>> me the output of “logread” after installing and activating SQM, as
>>>> well as the output from “tc -d qdisc” before and after enabling SQM,
>>>> and finally the output of running “/etc/init.d/sqm stop ;
>>>> /etc/init.d/sqm start” on the router’s console; that hopefully works
>>>> or at least gives some indication what might be off. If you could
>>>> throw in a quick netperf-wrapper RRUL test through the router I will
>>>> be most delighted ;))
>>> 
>>> Ok. That sounds like a good starter step.  First question: Where are the current SQM scripts located and what's the process for installing them?  Are they packages or stuff that I scp over and manually install?
>> 
>>        So the repository lives on:
>> https://github.com/dtaht/ceropackages-3.10
>> I just followed Dave’s advice and used:
>> git clone git@github.com:dtaht/ceropackages-3.3.git
>> to get my own version to play around with. Oh, SQM basically is the work of Dave (who wrote the scripts doing the actual AQM&QOS work) and Toke you packaged all this nicely and created the GUI for it (all I ever did was shuffling the GUI around a bit). I just started hacking around the files from sqm-scripts and luck-app-sqm on my cerowrt (so I could immediately do some testing), so I never got around building actual packages (I assume you need to be setup to build whole openwrt/cerowrt images and that is outside the scope of my recreational coding), But that said if you copy the files from ceropackages-3.10/net/sqm-scripts/files to the matching directory on your router you should have a working sqm-script. Or just use a cerowrt built as SQM is standard and potentially hnyman’s build ( https://forum.openwrt.org/viewtopic.php?id=28392 ) that also includes SQM.
>> 
>> Best Regards
>>        Sebastian
>> 
>>> 
>>> I'll let you know when I've taken a stab at getting that working.
>>> 
>>>> Richard A. Smith  <richard@laptop.org>
>>>> Former One Laptop per Child
>>> 
>>> Just FYI: I'm slowly switching my list subs over from my previous @laptop.org address to my personal address.  Just switched this one over thus the email address change.
>>> 
>>> --
>>> Richard A. Smith
>> 
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> 
> 
> -- 
> Dave Täht
> 
> https://www.bufferbloat.net/projects/make-wifi-fast

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Sebastian Moeller]

Hi Luis,


On Oct 9, 2014, at 20:05 , Luis E. Garcia <luis@bitamins.net> wrote:

> Please excuse my dumb question or just point me to the right documentation.
> But is there an easy way to run AQM/Codel on BB 14.07 ?? I'm running BB on a MyNETN750 that looks like it meets the HW requirements.

	There probably is, I assume you need to install ip-full from the BB repository, and then copy http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.50-1/packages/luci-app-sqm_3-1_all.ipk and http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.50-1/packages/sqm-scripts_6-1_all.ipk onto your router and try:
opkg install sqm-scripts_6-1_all.ipk
and:
opkg install luci-app-sqm_3-1_all.ipk

You might need to play with opkg options, but in theory it should work… There probably also is a nicer way like adding a stanza similar to:
src/gz toronto http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.50-1/packages
to /etc/opkg.conf
but I do not know the exact incantation of that, maybe hnyman, or Dave know the correct answer to this..

Best Regards
	Sebastian

> 
> Regards,
> Luis Garcia
> 
> On Thu, Oct 9, 2014 at 12:57 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Richard,
> 
> On Oct 9, 2014, at 18:42 , Richard Smith <smithbone@gmail.com> wrote:
> 
> > On 10/06/2014 03:41 PM, Sebastian Moeller wrote:
> >
> >>>
> >>> I have spare routers that I can run OpenWRT or CeroWRt on and I'm
> >>> setup to test with netperf, netperf-wrapper on my local network (
> >>> desktop -> router -> laptop )  it's Gbit so I can easily saturate
> >>> the router.
> >>
> >> That sounds great. I think the first test should be to run SQM under
> >> cerowrt, so you get a feel of how things should look. I typically run
> >> netperf-wrapper rrul tests (for ipv4 and if available for ipv6)
> >> through cerowrt with different settings for SQM. A second step then
> >> is to instal SQM-scripts under openwrt and check whether the same
> >> settings produce the same results ;)
> >
> > Ok.  I'll get my spare WNDR3700v2 up running the latest cerowrt.  Should be in the next few days.  I'm a bit strapped for time right now.
> 
>         No hurry, I really appreciate your help and there is no time pressure (rather the other way around, I do this for a hobby, and real life leaves almost no time for  that ;) )
> 
> >
> >> So just let me know what you are willing/ready to test and we will
> >> take it from there okay? (I would already be a happy camper if you
> >> could just install the current SQM-scripts on openwrt and just send
> >> me the output of “logread” after installing and activating SQM, as
> >> well as the output from “tc -d qdisc” before and after enabling SQM,
> >> and finally the output of running “/etc/init.d/sqm stop ;
> >> /etc/init.d/sqm start” on the router’s console; that hopefully works
> >> or at least gives some indication what might be off. If you could
> >> throw in a quick netperf-wrapper RRUL test through the router I will
> >> be most delighted ;))
> >
> > Ok. That sounds like a good starter step.  First question: Where are the current SQM scripts located and what's the process for installing them?  Are they packages or stuff that I scp over and manually install?
> 
>         So the repository lives on:
> https://github.com/dtaht/ceropackages-3.10
>  I just followed Dave’s advice and used:
> git clone git@github.com:dtaht/ceropackages-3.3.git
> to get my own version to play around with. Oh, SQM basically is the work of Dave (who wrote the scripts doing the actual AQM&QOS work) and Toke you packaged all this nicely and created the GUI for it (all I ever did was shuffling the GUI around a bit). I just started hacking around the files from sqm-scripts and luck-app-sqm on my cerowrt (so I could immediately do some testing), so I never got around building actual packages (I assume you need to be setup to build whole openwrt/cerowrt images and that is outside the scope of my recreational coding), But that said if you copy the files from ceropackages-3.10/net/sqm-scripts/files to the matching directory on your router you should have a working sqm-script. Or just use a cerowrt built as SQM is standard and potentially hnyman’s build ( https://forum.openwrt.org/viewtopic.php?id=28392 ) that also includes SQM.
> 
> Best Regards
>         Sebastian
> 
> >
> > I'll let you know when I've taken a stab at getting that working.
> >
> > > Richard A. Smith  <richard@laptop.org>
> > > Former One Laptop per Child
> >
> > Just FYI: I'm slowly switching my list subs over from my previous @laptop.org address to my personal address.  Just switched this one over thus the email address change.
> >
> > --
> > Richard A. Smith
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
> 
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Luis E. Garcia]

[-- Attachment #1: Type: text/plain, Size: 6284 bytes --]

Sebastian , Dave,
Thanks for these instructions.
I just finished a round of 3 months testing this setup on a couple of MyNet
N750 ( AR71xx based ) in several locations.
The feedback from users is very positive and encouraging - Most of them say
that they feel the networks behave a lot snappier than before [ 7.5Mbps Up/
1Mbps Down].

All I had to do was install the 3 packages over BB 14.07 :
ip-full from the OpenWRT BB repository
and luci-app-sqm + sam-scripts from the CeroWRT repository.

Will the later two packages ever make it to the main OpenWRT package list?
or there other packages that on the main OpenWRT distribution that can be
used to provide this functionality? ( the 2 CeroWRT packages above make
really simple to setup ).

Regards,
Luis Garcia

On Thu, Oct 9, 2014 at 2:59 PM, Sebastian Moeller <moeller0@gmx.de> wrote:

> Hi Luis,
>
>
> On Oct 9, 2014, at 20:05 , Luis E. Garcia <luis@bitamins.net> wrote:
>
> > Please excuse my dumb question or just point me to the right
> documentation.
> > But is there an easy way to run AQM/Codel on BB 14.07 ?? I'm running BB
> on a MyNETN750 that looks like it meets the HW requirements.
>
>         There probably is, I assume you need to install ip-full from the
> BB repository, and then copy
> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.50-1/packages/luci-app-sqm_3-1_all.ipk
> and
> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.50-1/packages/sqm-scripts_6-1_all.ipk
> onto your router and try:
> opkg install sqm-scripts_6-1_all.ipk
> and:
> opkg install luci-app-sqm_3-1_all.ipk
>
> You might need to play with opkg options, but in theory it should work…
> There probably also is a nicer way like adding a stanza similar to:
> src/gz toronto
> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.50-1/packages
> to /etc/opkg.conf
> but I do not know the exact incantation of that, maybe hnyman, or Dave
> know the correct answer to this..
>
> Best Regards
>         Sebastian
>
> >
> > Regards,
> > Luis Garcia
> >
> > On Thu, Oct 9, 2014 at 12:57 PM, Sebastian Moeller <moeller0@gmx.de>
> wrote:
> > Hi Richard,
> >
> > On Oct 9, 2014, at 18:42 , Richard Smith <smithbone@gmail.com> wrote:
> >
> > > On 10/06/2014 03:41 PM, Sebastian Moeller wrote:
> > >
> > >>>
> > >>> I have spare routers that I can run OpenWRT or CeroWRt on and I'm
> > >>> setup to test with netperf, netperf-wrapper on my local network (
> > >>> desktop -> router -> laptop )  it's Gbit so I can easily saturate
> > >>> the router.
> > >>
> > >> That sounds great. I think the first test should be to run SQM under
> > >> cerowrt, so you get a feel of how things should look. I typically run
> > >> netperf-wrapper rrul tests (for ipv4 and if available for ipv6)
> > >> through cerowrt with different settings for SQM. A second step then
> > >> is to instal SQM-scripts under openwrt and check whether the same
> > >> settings produce the same results ;)
> > >
> > > Ok.  I'll get my spare WNDR3700v2 up running the latest cerowrt.
> Should be in the next few days.  I'm a bit strapped for time right now.
> >
> >         No hurry, I really appreciate your help and there is no time
> pressure (rather the other way around, I do this for a hobby, and real life
> leaves almost no time for  that ;) )
> >
> > >
> > >> So just let me know what you are willing/ready to test and we will
> > >> take it from there okay? (I would already be a happy camper if you
> > >> could just install the current SQM-scripts on openwrt and just send
> > >> me the output of “logread” after installing and activating SQM, as
> > >> well as the output from “tc -d qdisc” before and after enabling SQM,
> > >> and finally the output of running “/etc/init.d/sqm stop ;
> > >> /etc/init.d/sqm start” on the router’s console; that hopefully works
> > >> or at least gives some indication what might be off. If you could
> > >> throw in a quick netperf-wrapper RRUL test through the router I will
> > >> be most delighted ;))
> > >
> > > Ok. That sounds like a good starter step.  First question: Where are
> the current SQM scripts located and what's the process for installing
> them?  Are they packages or stuff that I scp over and manually install?
> >
> >         So the repository lives on:
> > https://github.com/dtaht/ceropackages-3.10
> >  I just followed Dave’s advice and used:
> > git clone git@github.com:dtaht/ceropackages-3.3.git
> > to get my own version to play around with. Oh, SQM basically is the work
> of Dave (who wrote the scripts doing the actual AQM&QOS work) and Toke you
> packaged all this nicely and created the GUI for it (all I ever did was
> shuffling the GUI around a bit). I just started hacking around the files
> from sqm-scripts and luck-app-sqm on my cerowrt (so I could immediately do
> some testing), so I never got around building actual packages (I assume you
> need to be setup to build whole openwrt/cerowrt images and that is outside
> the scope of my recreational coding), But that said if you copy the files
> from ceropackages-3.10/net/sqm-scripts/files to the matching directory on
> your router you should have a working sqm-script. Or just use a cerowrt
> built as SQM is standard and potentially hnyman’s build (
> https://forum.openwrt.org/viewtopic.php?id=28392 ) that also includes SQM.
> >
> > Best Regards
> >         Sebastian
> >
> > >
> > > I'll let you know when I've taken a stab at getting that working.
> > >
> > > > Richard A. Smith  <richard@laptop.org>
> > > > Former One Laptop per Child
> > >
> > > Just FYI: I'm slowly switching my list subs over from my previous @
> laptop.org address to my personal address.  Just switched this one over
> thus the email address change.
> > >
> > > --
> > > Richard A. Smith
> > _______________________________________________
> > openwrt-devel mailing list
> > openwrt-devel@lists.openwrt.org
> > https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
> >
> > _______________________________________________
> > openwrt-devel mailing list
> > openwrt-devel@lists.openwrt.org
> > https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>
>

[-- Attachment #2: Type: text/html, Size: 8447 bytes --]

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Dave Taht]

sqm-scripts entered the openwrt package repos for chaos calmer quite
some time back.

On Thu, Feb 12, 2015 at 6:02 PM, Luis E. Garcia <luis@bitamins.net> wrote:
> Sebastian , Dave,
> Thanks for these instructions.
> I just finished a round of 3 months testing this setup on a couple of MyNet
> N750 ( AR71xx based ) in several locations.
> The feedback from users is very positive and encouraging - Most of them say
> that they feel the networks behave a lot snappier than before [ 7.5Mbps Up/
> 1Mbps Down].
>
> All I had to do was install the 3 packages over BB 14.07 :
> ip-full from the OpenWRT BB repository
> and luci-app-sqm + sam-scripts from the CeroWRT repository.
>
> Will the later two packages ever make it to the main OpenWRT package list?
> or there other packages that on the main OpenWRT distribution that can be
> used to provide this functionality? ( the 2 CeroWRT packages above make
> really simple to setup ).
>
> Regards,
> Luis Garcia
>
> On Thu, Oct 9, 2014 at 2:59 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> Hi Luis,
>>
>>
>> On Oct 9, 2014, at 20:05 , Luis E. Garcia <luis@bitamins.net> wrote:
>>
>> > Please excuse my dumb question or just point me to the right
>> > documentation.
>> > But is there an easy way to run AQM/Codel on BB 14.07 ?? I'm running BB
>> > on a MyNETN750 that looks like it meets the HW requirements.
>>
>>         There probably is, I assume you need to install ip-full from the
>> BB repository, and then copy
>> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.50-1/packages/luci-app-sqm_3-1_all.ipk
>> and
>> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.50-1/packages/sqm-scripts_6-1_all.ipk
>> onto your router and try:
>> opkg install sqm-scripts_6-1_all.ipk
>> and:
>> opkg install luci-app-sqm_3-1_all.ipk
>>
>> You might need to play with opkg options, but in theory it should work…
>> There probably also is a nicer way like adding a stanza similar to:
>> src/gz toronto
>> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.50-1/packages
>> to /etc/opkg.conf
>> but I do not know the exact incantation of that, maybe hnyman, or Dave
>> know the correct answer to this..
>>
>> Best Regards
>>         Sebastian
>>
>> >
>> > Regards,
>> > Luis Garcia
>> >
>> > On Thu, Oct 9, 2014 at 12:57 PM, Sebastian Moeller <moeller0@gmx.de>
>> > wrote:
>> > Hi Richard,
>> >
>> > On Oct 9, 2014, at 18:42 , Richard Smith <smithbone@gmail.com> wrote:
>> >
>> > > On 10/06/2014 03:41 PM, Sebastian Moeller wrote:
>> > >
>> > >>>
>> > >>> I have spare routers that I can run OpenWRT or CeroWRt on and I'm
>> > >>> setup to test with netperf, netperf-wrapper on my local network (
>> > >>> desktop -> router -> laptop )  it's Gbit so I can easily saturate
>> > >>> the router.
>> > >>
>> > >> That sounds great. I think the first test should be to run SQM under
>> > >> cerowrt, so you get a feel of how things should look. I typically run
>> > >> netperf-wrapper rrul tests (for ipv4 and if available for ipv6)
>> > >> through cerowrt with different settings for SQM. A second step then
>> > >> is to instal SQM-scripts under openwrt and check whether the same
>> > >> settings produce the same results ;)
>> > >
>> > > Ok.  I'll get my spare WNDR3700v2 up running the latest cerowrt.
>> > > Should be in the next few days.  I'm a bit strapped for time right now.
>> >
>> >         No hurry, I really appreciate your help and there is no time
>> > pressure (rather the other way around, I do this for a hobby, and real life
>> > leaves almost no time for  that ;) )
>> >
>> > >
>> > >> So just let me know what you are willing/ready to test and we will
>> > >> take it from there okay? (I would already be a happy camper if you
>> > >> could just install the current SQM-scripts on openwrt and just send
>> > >> me the output of “logread” after installing and activating SQM, as
>> > >> well as the output from “tc -d qdisc” before and after enabling SQM,
>> > >> and finally the output of running “/etc/init.d/sqm stop ;
>> > >> /etc/init.d/sqm start” on the router’s console; that hopefully works
>> > >> or at least gives some indication what might be off. If you could
>> > >> throw in a quick netperf-wrapper RRUL test through the router I will
>> > >> be most delighted ;))
>> > >
>> > > Ok. That sounds like a good starter step.  First question: Where are
>> > > the current SQM scripts located and what's the process for installing them?
>> > > Are they packages or stuff that I scp over and manually install?
>> >
>> >         So the repository lives on:
>> > https://github.com/dtaht/ceropackages-3.10
>> >  I just followed Dave’s advice and used:
>> > git clone git@github.com:dtaht/ceropackages-3.3.git
>> > to get my own version to play around with. Oh, SQM basically is the work
>> > of Dave (who wrote the scripts doing the actual AQM&QOS work) and Toke you
>> > packaged all this nicely and created the GUI for it (all I ever did was
>> > shuffling the GUI around a bit). I just started hacking around the files
>> > from sqm-scripts and luck-app-sqm on my cerowrt (so I could immediately do
>> > some testing), so I never got around building actual packages (I assume you
>> > need to be setup to build whole openwrt/cerowrt images and that is outside
>> > the scope of my recreational coding), But that said if you copy the files
>> > from ceropackages-3.10/net/sqm-scripts/files to the matching directory on
>> > your router you should have a working sqm-script. Or just use a cerowrt
>> > built as SQM is standard and potentially hnyman’s build (
>> > https://forum.openwrt.org/viewtopic.php?id=28392 ) that also includes SQM.
>> >
>> > Best Regards
>> >         Sebastian
>> >
>> > >
>> > > I'll let you know when I've taken a stab at getting that working.
>> > >
>> > > > Richard A. Smith  <richard@laptop.org>
>> > > > Former One Laptop per Child
>> > >
>> > > Just FYI: I'm slowly switching my list subs over from my previous
>> > > @laptop.org address to my personal address.  Just switched this one over
>> > > thus the email address change.
>> > >
>> > > --
>> > > Richard A. Smith
>> > _______________________________________________
>> > openwrt-devel mailing list
>> > openwrt-devel@lists.openwrt.org
>> > https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>> >
>> > _______________________________________________
>> > openwrt-devel mailing list
>> > openwrt-devel@lists.openwrt.org
>> > https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>>
>



-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] [OpenWrt-Devel] [RFC PATCH] packages: Smart Queue Management for AQM Packet Scheduling and Qos from CeroWrt

[Luis E. Garcia]

[-- Attachment #1: Type: text/plain, Size: 7500 bytes --]

That's great news.
Thanks again Dave - I found both SQM packages on the Chaos Calmer files.
I'll run some more test with users.

Regards,
Luis

On Thursday, February 12, 2015, Dave Taht <dave.taht@gmail.com> wrote:

> sqm-scripts entered the openwrt package repos for chaos calmer quite
> some time back.
>
> On Thu, Feb 12, 2015 at 6:02 PM, Luis E. Garcia <luis@bitamins.net
> <javascript:;>> wrote:
> > Sebastian , Dave,
> > Thanks for these instructions.
> > I just finished a round of 3 months testing this setup on a couple of
> MyNet
> > N750 ( AR71xx based ) in several locations.
> > The feedback from users is very positive and encouraging - Most of them
> say
> > that they feel the networks behave a lot snappier than before [ 7.5Mbps
> Up/
> > 1Mbps Down].
> >
> > All I had to do was install the 3 packages over BB 14.07 :
> > ip-full from the OpenWRT BB repository
> > and luci-app-sqm + sam-scripts from the CeroWRT repository.
> >
> > Will the later two packages ever make it to the main OpenWRT package
> list?
> > or there other packages that on the main OpenWRT distribution that can be
> > used to provide this functionality? ( the 2 CeroWRT packages above make
> > really simple to setup ).
> >
> > Regards,
> > Luis Garcia
> >
> > On Thu, Oct 9, 2014 at 2:59 PM, Sebastian Moeller <moeller0@gmx.de
> <javascript:;>> wrote:
> >>
> >> Hi Luis,
> >>
> >>
> >> On Oct 9, 2014, at 20:05 , Luis E. Garcia <luis@bitamins.net
> <javascript:;>> wrote:
> >>
> >> > Please excuse my dumb question or just point me to the right
> >> > documentation.
> >> > But is there an easy way to run AQM/Codel on BB 14.07 ?? I'm running
> BB
> >> > on a MyNETN750 that looks like it meets the HW requirements.
> >>
> >>         There probably is, I assume you need to install ip-full from the
> >> BB repository, and then copy
> >>
> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.50-1/packages/luci-app-sqm_3-1_all.ipk
> >> and
> >>
> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.50-1/packages/sqm-scripts_6-1_all.ipk
> >> onto your router and try:
> >> opkg install sqm-scripts_6-1_all.ipk
> >> and:
> >> opkg install luci-app-sqm_3-1_all.ipk
> >>
> >> You might need to play with opkg options, but in theory it should work…
> >> There probably also is a nicer way like adding a stanza similar to:
> >> src/gz toronto
> >>
> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.50-1/packages
> >> to /etc/opkg.conf
> >> but I do not know the exact incantation of that, maybe hnyman, or Dave
> >> know the correct answer to this..
> >>
> >> Best Regards
> >>         Sebastian
> >>
> >> >
> >> > Regards,
> >> > Luis Garcia
> >> >
> >> > On Thu, Oct 9, 2014 at 12:57 PM, Sebastian Moeller <moeller0@gmx.de
> <javascript:;>>
> >> > wrote:
> >> > Hi Richard,
> >> >
> >> > On Oct 9, 2014, at 18:42 , Richard Smith <smithbone@gmail.com
> <javascript:;>> wrote:
> >> >
> >> > > On 10/06/2014 03:41 PM, Sebastian Moeller wrote:
> >> > >
> >> > >>>
> >> > >>> I have spare routers that I can run OpenWRT or CeroWRt on and I'm
> >> > >>> setup to test with netperf, netperf-wrapper on my local network (
> >> > >>> desktop -> router -> laptop )  it's Gbit so I can easily saturate
> >> > >>> the router.
> >> > >>
> >> > >> That sounds great. I think the first test should be to run SQM
> under
> >> > >> cerowrt, so you get a feel of how things should look. I typically
> run
> >> > >> netperf-wrapper rrul tests (for ipv4 and if available for ipv6)
> >> > >> through cerowrt with different settings for SQM. A second step then
> >> > >> is to instal SQM-scripts under openwrt and check whether the same
> >> > >> settings produce the same results ;)
> >> > >
> >> > > Ok.  I'll get my spare WNDR3700v2 up running the latest cerowrt.
> >> > > Should be in the next few days.  I'm a bit strapped for time right
> now.
> >> >
> >> >         No hurry, I really appreciate your help and there is no time
> >> > pressure (rather the other way around, I do this for a hobby, and
> real life
> >> > leaves almost no time for  that ;) )
> >> >
> >> > >
> >> > >> So just let me know what you are willing/ready to test and we will
> >> > >> take it from there okay? (I would already be a happy camper if you
> >> > >> could just install the current SQM-scripts on openwrt and just send
> >> > >> me the output of “logread” after installing and activating SQM, as
> >> > >> well as the output from “tc -d qdisc” before and after enabling
> SQM,
> >> > >> and finally the output of running “/etc/init.d/sqm stop ;
> >> > >> /etc/init.d/sqm start” on the router’s console; that hopefully
> works
> >> > >> or at least gives some indication what might be off. If you could
> >> > >> throw in a quick netperf-wrapper RRUL test through the router I
> will
> >> > >> be most delighted ;))
> >> > >
> >> > > Ok. That sounds like a good starter step.  First question: Where are
> >> > > the current SQM scripts located and what's the process for
> installing them?
> >> > > Are they packages or stuff that I scp over and manually install?
> >> >
> >> >         So the repository lives on:
> >> > https://github.com/dtaht/ceropackages-3.10
> >> >  I just followed Dave’s advice and used:
> >> > git clone git@github.com:dtaht/ceropackages-3.3.git
> >> > to get my own version to play around with. Oh, SQM basically is the
> work
> >> > of Dave (who wrote the scripts doing the actual AQM&QOS work) and
> Toke you
> >> > packaged all this nicely and created the GUI for it (all I ever did
> was
> >> > shuffling the GUI around a bit). I just started hacking around the
> files
> >> > from sqm-scripts and luck-app-sqm on my cerowrt (so I could
> immediately do
> >> > some testing), so I never got around building actual packages (I
> assume you
> >> > need to be setup to build whole openwrt/cerowrt images and that is
> outside
> >> > the scope of my recreational coding), But that said if you copy the
> files
> >> > from ceropackages-3.10/net/sqm-scripts/files to the matching
> directory on
> >> > your router you should have a working sqm-script. Or just use a
> cerowrt
> >> > built as SQM is standard and potentially hnyman’s build (
> >> > https://forum.openwrt.org/viewtopic.php?id=28392 ) that also
> includes SQM.
> >> >
> >> > Best Regards
> >> >         Sebastian
> >> >
> >> > >
> >> > > I'll let you know when I've taken a stab at getting that working.
> >> > >
> >> > > > Richard A. Smith  <richard@laptop.org <javascript:;>>
> >> > > > Former One Laptop per Child
> >> > >
> >> > > Just FYI: I'm slowly switching my list subs over from my previous
> >> > > @laptop.org address to my personal address.  Just switched this
> one over
> >> > > thus the email address change.
> >> > >
> >> > > --
> >> > > Richard A. Smith
> >> > _______________________________________________
> >> > openwrt-devel mailing list
> >> > openwrt-devel@lists.openwrt.org <javascript:;>
> >> > https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
> >> >
> >> > _______________________________________________
> >> > openwrt-devel mailing list
> >> > openwrt-devel@lists.openwrt.org <javascript:;>
> >> > https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
> >>
> >
>
>
>
> --
> Dave Täht
>
> thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks
>

[-- Attachment #2: Type: text/html, Size: 10834 bytes --]