[Cerowrt-devel] Cero 3.10.24-5 no longer supports multiple AQMs?

Mon Dec 23 14:00:28 EST 2013

Just a pair of quick comments on something you said below. I'll look
over your scripts later.

There is PLENTY of sense in shaping inbound traffic. Inbound is the
bulk of the problem in many cases - verizon has 300ms of inbound
buffering in their 25/25mbit service, and comcast often well over a
second on their 25Mbit/4. (and often over a second on outbound but the
new modem I've been testing is only about 200ms. But the bulk of the
delay is on inbound, by far)

comcast shaped only on inbound:

http://snapon.lab.bufferbloat.net/~cero2/armory.com/3.10.24.5/oneway/149.20.63.30.rrul-ethernet-ecn.svg

comcast unshaped entirely exhibits almost 2 seconds of delay before it
starts dropping packets.

http://snapon.lab.bufferbloat.net/~cero2/armory.com/unshaped/149.20.63.30.rrul-comcast_unshaped.svg

UGH! This is the kind of performance cable users have to deal with! I
hope the CMTS folk get their act together soon.

And the normal glorious post
whatever-the-heck-we're-going-to-call-aqm-scripts-ceroshaper result:

http://snapon.lab.bufferbloat.net/~cero2/armory.com/3.10.24.5/149.20.63.30.rrul_noclassification-ethernet-ecn.svg

So... a lot of people keep insisting that "shaping inbound doesn't
work" on the client device, and it just aint true. I had hoped to just
be able to fix the cable modems in docsis 3.1, but that isn't going to
be what happens, sadly.

Sure: a primitive use of a policer doesn't work well (see wondershaper
result below), but attaching htb + fq_codel on ingress works fine.
Perhaps we need to collect a wide swath of results from tools like
netanalyzer, too? with inbound and outbound enabled/disabled in
combination?

What might have caused confusion? is that I generally enable ECN on
inbound so that ECN compliant streams don't get their packets dropped,
but marked, when it's time to slow down. (Very little traffic
is ECN marked.)

Anyway, I recently went through a round of tests of 2.10.24, realizing
only later that the instruction trap error was skewing the data on
some tests. There are some new results.

This is wondershaper on a 25Mbit/4Mbit comcast service. The inbound
policer drops far too many packets to get even half the allocated
bandwidth. (Wondershaper has many other problems. It needs to die!)

http://snapon.lab.bufferbloat.net/~cero2/armory.com/3.10.24.5/149.20.63.30.rrul-wondershaper.svg

I do not know to what extent DPI is used to mess with torrents, but I
got a good 50 client download going of ubuntu and still had very good
performance for normal tcp, and web pages are pretty good, too.

http://snapon.lab.bufferbloat.net/~cero2/armory.com/with_torrent/ipv4-2.svg

(as always these dirs contain far more data than just what I'm cherry
picking above, notably a bunch of simpler tcp up/down/bidir plots.
feel free to move around)

On Mon, Dec 23, 2013 at 5:33 AM, Richard O <rocon46 at hotmail.com> wrote:
> Dave Taht <dave.taht <at> gmail.com> writes:
>
>>
>> On Sun, Dec 22, 2013 at 11:22 AM, Sebastian Moeller <moeller0 <at> gmx.de>
> wrote:
>> > Hi Richard,
>> >
>> >
>> > On Dec 22, 2013, at 09:38 , Richard O <rocon46 <at> hotmail.com> wrote:
>> >
>> >> Heya,
>> >>
>> >> I'll try to keep this short, but I'm simply just an end-user of Cero
> who has
>> >> using it for a few months now. It's been great! So great that I've been
>> >> using it as my main router after the first few weeks.
>> >>
>> >> Anyway, I've just recently upgraded from 3.10.11-2 to the latest build
>> >> (3.10.42-5), and I noticed that you can only set one AQM at any given time.
>> >> Restoring my old settings restores my old set of AQMs - all of which
> startup
>> >> nicely - but only two out four have their "ingress" start up: ifb0 and ifb1
>> >> while ifb2 and ifb3 remain down. Their entries still appear in the firewall
>> >> and I'm unable to bring them up using ifconfig.
>> >>
>> >
>> >         If I recall correctly, we just disabled the ability to setup AQM
> on multiple interfaces, as to our
>> knowledge it was not clear whether it would work at all and whether
> someone had actually tested it. It
>> should be relatively easy to reenable it, if it works. Question, since
> most people are happy with just
>> running AQM on the wan link, why do you run it on 4 interfaces?
>> >
>> >
>> > Best Regards
>> >         Sebastian
>
>
> In short, I run it on four interfaces (ge00, sw00, sw10, and gw00) because I
> need to: police the data usage of certain users, place a limit on the guest
> network, deprioritize torrents, prioritize game traffic, and to keep file
> sharing from tearing apart the wifi.
>
> Everything is running off wifi and before Cero, wifi-to-wifi file sharing
> used to shut the whole thing down for everyone else. Since then fq_codel has
> done a decent job at keeping the whole thing usable w/o me having to do
> anything to it. It'd probably be better if I DID specify a class for it but
> we don't really do it that often, so I never got around to it.
>
>
>> >
>> >> As you can probably tell, I don't know very much Linux and simply use
>> >> rc.local to modify the classes and qdiscs for each interface to suit my
> needs.
>> >>
>> >> I'm just curious if this is going to be a permanent thing or not. Other
> then
>> >> that, keep up the great work guys. Cero's been good to me, and I'll
> probably
>> >> just revert to the old build if newer builds are going to be running with a
>> >> single aqm. I never really did encountered any problems with it.
>> >>
>> >> _______________________________________________
>> >> Cerowrt-devel mailing list
>> >> Cerowrt-devel <at> lists.bufferbloat.net
>> >> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> >
>> > _______________________________________________
>> > Cerowrt-devel mailing list
>> > Cerowrt-devel <at> lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>> A) it is cool that someone is using this functionality and I am
>> curious as to what it was being used for... what devices did you use
>> it on, and why? (paste your config file?)
>
> Just a PC, two laptops, a game console, and a whole bunch of smartphones.
> See Sebastian^
>
> This is probably gonna get long and verbose but... we mostly web browse with
> nearly all computers and devices running on the 2.4Ghz frequency and have
> the game console play online games on the 5Ghz. We also torrent and stream
> some files across the network occasionally.
>
> Our ADSL connection isn't very fast (972kbits ↑ /8100kbits ↓) so I need to
> make sure torrents don't kill general web browsing and interrupt online
> games. I've also placed caps on the guest network for guest who come around
> and on one user on sw00 who is download happy. The game console is also
> prioritized.
>
> Anyway, I run simple.qos/fq_codel on ge00, sw00 and sw10 and for the guest
> network I run simplest.qos/fq. I then use rc.local to make some changes that
> I feel would help.
>
> Hrm, I'm not quite sure how to post my Cero config as I don't see any way to
> attach anything here, but I guess I could post what I have in my rc.local
> file via a huge text dump.
>
> I haven't done a very good job at documentation and there are A LOT of lines
> which are commented out 'cause of experimentation — I'll try to remove most
> of those. Other then that, everything will probably look like an amateur's
> attempt at traffic shaping... but here you go.
>
>
>
>
> ---------------------------------------------
> # Put your custom commands here that should be executed once
> # the system init finished. By default this file does nothing.
> #------------------
>
> #ifb- (0-1-2)
> tc qdisc del dev ifb0 parent 1:12 handle 120:
> tc qdisc del dev ifb0 parent 1:13 handle 130:
> tc qdisc del dev ifb1 parent 1:12 handle 120:
> tc qdisc del dev ifb1 parent 1:13 handle 130:
>
> tc qdisc add dev ifb0 parent 1:12 handle 120: fq_codel limit 1000 flows 1024
> quantum 300 target 5000 interval 100000 ecn
> tc qdisc add dev ifb0 parent 1:13 handle 130: fq_codel limit 1000 flows 1024
> quantum 300 target 5000 interval 100000 ecn
> tc qdisc add dev ifb1 parent 1:12 handle 120: fq_codel limit 1000 flows 1024
> quantum 300 target 5000 interval 100000 ecn
> tc qdisc add dev ifb1 parent 1:13 handle 130: fq_codel limit 1000 flows 1024
> quantum 300 target 5000 interval 100000 ecn
> #---------------
>
> #XBL
> iptables -t mangle -A FORWARD -s 172.30.42.102 -p udp -m udp --sport 3074 -j
> DSCP --set-dscp-class EF
> iptables -t mangle -A FORWARD -d 172.30.42.102 -p udp -m udp --dport 3074 -j
> DSCP --set-dscp-class EF
> iptables -t mangle -D QOS_MARK_ge00 -p 0 -m dscp --dscp-class EF -j MARK
> --set-mark 0x1
> iptables -t mangle -D QOS_MARK_sw10 -p 0 -m dscp --dscp-class EF -j MARK
> --set-mark 0x1
>
>
> iptables -t mangle -A QOS_MARK_ge00 -p udp -m dscp --dscp-class EF -j MARK
> --set-mark 0xa
> iptables -t mangle -A QOS_MARK_sw10 -p udp -m dscp --dscp-class EF -j MARK
> --set-mark 0xa
>
> #Torrent
> iptables -t mangle -A FORWARD -p udp -m udp --source-port 22223:22226 -j
> DSCP --set-dscp-class AF13
> iptables -t mangle -A FORWARD -p tcp -m tcp --source-port 22223:22226 -j
> DSCP --set-dscp-class AF13
> iptables -t mangle -A FORWARD -p udp -m udp --dport 22223:22226 -j DSCP
> --set-dscp-class AF13
> iptables -t mangle -A FORWARD -p tcp -m tcp --dport 22223:22226 -j DSCP
> --set-dscp-class AF13
>
> iptables -t mangle -A QOS_MARK_ge00 -p 0 -m dscp --dscp-class AF13 -j MARK
> --set-mark 0x3
> iptables -t mangle -A QOS_MARK_sw00 -p 0 -m dscp --dscp-class AF13 -j MARK
> --set-mark 0x3
>
>
> ## XBL
> tc qdisc add dev ge00 parent 1:10 handle 100: nfq_codel quantum 300 limit
> 1000 noecn target 100000
> tc filter add dev ge00 parent 1: prio 1 protocol ip handle 0xa fw flowid 1:10
>
> tc qdisc add dev sw10 parent 1:10 handle 100: nfq_codel quantum 300 limit
> 1000 noecn
> tc filter add dev sw10 parent 1: prio 1 protocol ip handle 0xa fw flowid 1:10
>
>
> #Inbound
> #Move ::800Flow
> tc filter add dev ifb0 parent 1: handle 801::808 protocol ip prio 1 u32
> match u32 0x00000000 0x00fc0000 flowid 1:12
> tc filter add dev ifb1 parent 1: handle 801::809 protocol ip prio 1 u32
> match u32 0x00000000 0x00fc0000 flowid 1:12
> tc filter add dev ifb2 parent 1: handle 801::803 protocol ip prio 1 u32
> match u32 0x00000000 0x00fc0000 flowid 1:12
> tc filter del dev ifb0 parent 1: handle 801::800 prio 1 u32
> tc filter del dev ifb1 parent 1: handle 801::800 prio 1 u32
> tc filter del dev ifb2 parent 1: handle 801::800 prio 1 u32
>
> #LIVE
> tc qdisc add dev ifb0 parent 1:10 handle 100: nfq_codel quantum 500 limit
> 1000 ecn
> #tc filter add dev ifb0 parent 1: handle 801::800 prio 1 protocol ip u32
> match ip dst 172.30.42.102 match ip protocol 0x11 0xff match ip dport 3074
> 0xffff flowid 1:10 ## Doesn't work..?
> tc filter add dev ifb0 parent 1: protocol ip prio 1 handle 801::800 u32
> match ip protocol 0x11 0xff match ip dport 3074 0xffff flowid 1:10
> tc filter add dev ifb0 parent 1: protocol ip prio 1 handle 801::801 u32
> match ip protocol 0x11 0xff match ip sport 3074 0xffff flowid 1:10
>
> tc qdisc add dev ifb2 parent 1:10 handle 100: nfq_codel quantum 300 limit
> 1000 ecn
> tc filter add dev ifb2 parent 1: handle 801::800 prio 1 protocol ip u32
> match ip src 172.30.42.102 match ip protocol 0x11 0xff match ip sport 3074
> 0xffff flowid 1:10
> tc filter add dev ifb2 parent 1: prio 1 protocol ip handle 801::801 u32
> match ip protocol 0x11 0xff match ip sport 3074 0xffff flowid 1:10
>
> #------------------
>
> ## Torrent
> tc class replace dev ge00 parent 1: classid 1:13 htb rate 10000 ceil 320000
> prio 3 quantum 1478
> tc class replace dev ifb1 parent 1: classid 1:13 htb rate 10000 ceil 350000
> prio 3 quantum 1478
>
> tc filter add dev ifb0 parent 1: handle 801::803 protocol ip prio 1 u32
> match ip tos 0x38 0xff flowid 1:13
> tc filter add dev ifb0 parent 1: handle 801::804 prio 1 protocol ip u32
> match ip dport 22223 0xffff flowid 1:13
> tc filter add dev ifb0 parent 1: handle 801::805 prio 1 protocol ip u32
> match ip dport 22225 0xffff flowid 1:13
> tc filter add dev ifb0 parent 1: handle 801::806 prio 1 protocol ip u32
> match ip dport 22227 0xffff flowid 1:13
>
> tc filter add dev ifb1 parent 1: handle 801::803 protocol ip prio 1 u32
> match ip tos 0x38 0xff flowid 1:13 ##Mark testing - doesn't work
> tc filter add dev ifb1 parent 1: handle 801::804 prio 1 protocol ip u32
> match ip sport 22223 0xffff flowid 1:13
> tc filter add dev ifb1 parent 1: handle 801::805 prio 1 protocol ip u32
> match ip sport 22224 0xffff flowid 1:13
> tc filter add dev ifb1 parent 1: handle 801::806 prio 1 protocol ip u32
> match ip sport 22225 0xffff flowid 1:13
> tc filter add dev ifb1 parent 1: handle 801::807 prio 1 protocol ip u32
> match ip sport 22226 0xffff flowid 1:13
> tc filter add dev ifb1 parent 1: handle 801::808 prio 1 protocol ip u32
> match ip sport 22227 0xffff flowid 1:13
>
>
> ## Billy's Laptop - Speed Cap
> iptables -t mangle -A FORWARD -s 172.30.42.73 -j DSCP --set-dscp-class AF12
> iptables -t mangle -A FORWARD -d 172.30.42.73 -j DSCP --set-dscp-class AF12
> iptables -t mangle -A FORWARD -s 172.30.42.74 -j DSCP --set-dscp-class AF12
> iptables -t mangle -A FORWARD -d 172.30.42.74 -j DSCP --set-dscp-class AF12
>
> iptables -t mangle -A QOS_MARK_ge00 -p 0 -m dscp --dscp-class AF12 -j MARK
> --set-mark 0x4
> iptables -t mangle -A QOS_MARK_sw00 -p 0 -m dscp --dscp-class AF12 -j MARK
> --set-mark 0x4
>
> ##Unable to cap via ifb0. sw00 works nicely, though.
> tc class add dev ge00 parent 1:1 classid 1:14 htb rate 50000bit ceil
> 200000bit prio 4 quantum 1478
> tc class add dev sw00 parent 1:1 classid 1:14 htb rate 200Kbit ceil 800Kbit
> prio 4 quantum 1478
> tc qdisc add dev ge00 parent 1:14 handle 140: fq_codel limit 600 flows 1024
> quantum 300 target 5000 interval 100000 noecn
> tc qdisc add dev sw00 parent 1:14 handle 140: fq_codel limit 600 flows 1024
> quantum 300 target 5000 interval 100000 noecn
> tc filter add dev ge00 parent 1: prio 3 protocol ip handle 0x4 fw flowid 1:14
> tc filter add dev sw00 parent 1: prio 3 protocol ip handle 0x4 fw flowid 1:14
>
> #tc filter add dev ifb0 parent 1: handle 801::802 prio 1 protocol ip u32
> match ip dst 172.30.42.73 flowid 1:13 ##Doesn't work
> tc class add dev ifb1 parent 1:1 classid 1:14 htb rate 50000bit ceil
> 200000bit prio 4 quantum 1478
> tc qdisc add dev ifb1 parent 1:14 handle 140: fq_codel limit 600 flows 1024
> quantum 300 target 5000 interval 100000 ecn
> tc filter add dev ifb1 parent 1: handle 801::802 prio 1 protocol ip u32
> match ip src 172.30.42.73 flowid 1:14
>
> #ICMP (ip protocol 1) in the interactive class
> #tc filter add dev ifb0 parent 1: handle 801::802 protocol ip prio 1 u32
> match ip protocol 1 0xff flowid 1:11
> #tc filter add dev ifb1 parent 1: handle 801::801 protocol ip prio 1 u32
> match ip protocol 1 0xff flowid 1:11
> #tc filter add dev ifb2 parent 1: handle 801::802 protocol ip prio 1 u32
> match ip protocol 1 0xff flowid 1:11
> #iptables -t mangle -A QOS_MARK_ge00 -p ICMP -j MARK --set-mark 0x1
> #iptables -t mangle -A QOS_MARK_sw00 -p ICMP -j MARK --set-mark 0x1
> #iptables -t mangle -A QOS_MARK_sw10 -p ICMP -j MARK --set-mark 0x1
>
> #Guest Network 2.4G
> iptables -t mangle -A FORWARD -s 172.30.42.129/27 -j DSCP --set-dscp-class AF12
> iptables -t mangle -A FORWARD -d 172.30.42.129/27 -j DSCP --set-dscp-class AF12
>
> #22223 0xffff flowid (...)
> #tc filter add (...) u32 match ip [s-d]port 22224 0xfffe flowid (...)
> #tc filter add (...) u32 match ip [s-d]port 22226 0xffff flowid (...)
>
>
> /etc/fixdaemons &
> exit 0
>
> ---------------------------------------------
>
> A few notes:
> Before you ask, yes, I've heard time and time again while lurking around
> here that there's no sense in shaping inbound traffic. It's already past the
> bottleneck so you might as well just take the data and prevent retransmits
> etc. I thought so too, but doing this for torrents seems to make everything
> feel a lot more responsive while torrenting then not doing it at all. It
> could be a just a placebo effect but, eh. I feel it helps.
>
> The game console is placed to hog all the bandwidth if it needs it 'cause
> someone wants to host games sometimes. Apparently, you need as much data as
> you can get to host games so I can't just throw it in the interactive class
> (1:11). It only uses about half of what we have, anyway, so I don't mind.
>
> The rest is just traffic shaping for 'Billy' and the guest network.
>
>
>>
>> B) I think the problem is not the script but the ifb module getting
>> inserted with only two ifbs allowed.
>>
>> try a:
>>
>> /etc/init.d/aqm stop
>> rmmod ifb
>> insmod ifb numifbs=8
>> /etc/init.d/aqm start
>>
>
> This worked beautifully. Thank you.
>
>
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html