[Cerowrt-devel] Cero 3.10.24-5 no longer supports multiple AQMs?

[Richard O]

Dave Taht <dave.taht <at> gmail.com> writes:

> 
> On Sun, Dec 22, 2013 at 11:22 AM, Sebastian Moeller <moeller0 <at> gmx.de>
wrote:
> > Hi Richard,
> >
> >
> > On Dec 22, 2013, at 09:38 , Richard O <rocon46 <at> hotmail.com> wrote:
> >
> >> Heya,
> >>
> >> I'll try to keep this short, but I'm simply just an end-user of Cero
who has
> >> using it for a few months now. It's been great! So great that I've been
> >> using it as my main router after the first few weeks.
> >>
> >> Anyway, I've just recently upgraded from 3.10.11-2 to the latest build
> >> (3.10.42-5), and I noticed that you can only set one AQM at any given time.
> >> Restoring my old settings restores my old set of AQMs - all of which
startup
> >> nicely - but only two out four have their "ingress" start up: ifb0 and ifb1
> >> while ifb2 and ifb3 remain down. Their entries still appear in the firewall
> >> and I'm unable to bring them up using ifconfig.
> >>
> >
> >         If I recall correctly, we just disabled the ability to setup AQM
on multiple interfaces, as to our
> knowledge it was not clear whether it would work at all and whether
someone had actually tested it. It
> should be relatively easy to reenable it, if it works. Question, since
most people are happy with just
> running AQM on the wan link, why do you run it on 4 interfaces?
> >
> >
> > Best Regards
> >         Sebastian


In short, I run it on four interfaces (ge00, sw00, sw10, and gw00) because I
need to: police the data usage of certain users, place a limit on the guest
network, deprioritize torrents, prioritize game traffic, and to keep file
sharing from tearing apart the wifi.

Everything is running off wifi and before Cero, wifi-to-wifi file sharing
used to shut the whole thing down for everyone else. Since then fq_codel has
done a decent job at keeping the whole thing usable w/o me having to do
anything to it. It'd probably be better if I DID specify a class for it but
we don't really do it that often, so I never got around to it.


> >
> >> As you can probably tell, I don't know very much Linux and simply use
> >> rc.local to modify the classes and qdiscs for each interface to suit my
needs.
> >>
> >> I'm just curious if this is going to be a permanent thing or not. Other
then
> >> that, keep up the great work guys. Cero's been good to me, and I'll
probably
> >> just revert to the old build if newer builds are going to be running with a
> >> single aqm. I never really did encountered any problems with it.
> >>
> >> _______________________________________________
> >> Cerowrt-devel mailing list
> >> Cerowrt-devel <at> lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-devel <at> lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> A) it is cool that someone is using this functionality and I am
> curious as to what it was being used for... what devices did you use
> it on, and why? (paste your config file?)

Just a PC, two laptops, a game console, and a whole bunch of smartphones.
See Sebastian^

This is probably gonna get long and verbose but... we mostly web browse with
nearly all computers and devices running on the 2.4Ghz frequency and have
the game console play online games on the 5Ghz. We also torrent and stream
some files across the network occasionally.

Our ADSL connection isn't very fast (972kbits ↑ /8100kbits ↓) so I need to
make sure torrents don't kill general web browsing and interrupt online
games. I've also placed caps on the guest network for guest who come around
and on one user on sw00 who is download happy. The game console is also
prioritized.

Anyway, I run simple.qos/fq_codel on ge00, sw00 and sw10 and for the guest
network I run simplest.qos/fq. I then use rc.local to make some changes that
I feel would help.

Hrm, I'm not quite sure how to post my Cero config as I don't see any way to
attach anything here, but I guess I could post what I have in my rc.local
file via a huge text dump.

I haven't done a very good job at documentation and there are A LOT of lines
which are commented out 'cause of experimentation — I'll try to remove most
of those. Other then that, everything will probably look like an amateur's
attempt at traffic shaping... but here you go.




---------------------------------------------
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.
#------------------

#ifb- (0-1-2)
tc qdisc del dev ifb0 parent 1:12 handle 120:
tc qdisc del dev ifb0 parent 1:13 handle 130:
tc qdisc del dev ifb1 parent 1:12 handle 120:
tc qdisc del dev ifb1 parent 1:13 handle 130:

tc qdisc add dev ifb0 parent 1:12 handle 120: fq_codel limit 1000 flows 1024
quantum 300 target 5000 interval 100000 ecn
tc qdisc add dev ifb0 parent 1:13 handle 130: fq_codel limit 1000 flows 1024
quantum 300 target 5000 interval 100000 ecn
tc qdisc add dev ifb1 parent 1:12 handle 120: fq_codel limit 1000 flows 1024
quantum 300 target 5000 interval 100000 ecn
tc qdisc add dev ifb1 parent 1:13 handle 130: fq_codel limit 1000 flows 1024
quantum 300 target 5000 interval 100000 ecn
#---------------

#XBL
iptables -t mangle -A FORWARD -s 172.30.42.102 -p udp -m udp --sport 3074 -j
DSCP --set-dscp-class EF
iptables -t mangle -A FORWARD -d 172.30.42.102 -p udp -m udp --dport 3074 -j
DSCP --set-dscp-class EF
iptables -t mangle -D QOS_MARK_ge00 -p 0 -m dscp --dscp-class EF -j MARK
--set-mark 0x1
iptables -t mangle -D QOS_MARK_sw10 -p 0 -m dscp --dscp-class EF -j MARK
--set-mark 0x1


iptables -t mangle -A QOS_MARK_ge00 -p udp -m dscp --dscp-class EF -j MARK
--set-mark 0xa
iptables -t mangle -A QOS_MARK_sw10 -p udp -m dscp --dscp-class EF -j MARK
--set-mark 0xa

#Torrent
iptables -t mangle -A FORWARD -p udp -m udp --source-port 22223:22226 -j
DSCP --set-dscp-class AF13
iptables -t mangle -A FORWARD -p tcp -m tcp --source-port 22223:22226 -j
DSCP --set-dscp-class AF13
iptables -t mangle -A FORWARD -p udp -m udp --dport 22223:22226 -j DSCP
--set-dscp-class AF13
iptables -t mangle -A FORWARD -p tcp -m tcp --dport 22223:22226 -j DSCP
--set-dscp-class AF13

iptables -t mangle -A QOS_MARK_ge00 -p 0 -m dscp --dscp-class AF13 -j MARK
--set-mark 0x3
iptables -t mangle -A QOS_MARK_sw00 -p 0 -m dscp --dscp-class AF13 -j MARK
--set-mark 0x3


## XBL
tc qdisc add dev ge00 parent 1:10 handle 100: nfq_codel quantum 300 limit
1000 noecn target 100000
tc filter add dev ge00 parent 1: prio 1 protocol ip handle 0xa fw flowid 1:10

tc qdisc add dev sw10 parent 1:10 handle 100: nfq_codel quantum 300 limit
1000 noecn
tc filter add dev sw10 parent 1: prio 1 protocol ip handle 0xa fw flowid 1:10


#Inbound
#Move ::800Flow
tc filter add dev ifb0 parent 1: handle 801::808 protocol ip prio 1 u32
match u32 0x00000000 0x00fc0000 flowid 1:12
tc filter add dev ifb1 parent 1: handle 801::809 protocol ip prio 1 u32
match u32 0x00000000 0x00fc0000 flowid 1:12
tc filter add dev ifb2 parent 1: handle 801::803 protocol ip prio 1 u32
match u32 0x00000000 0x00fc0000 flowid 1:12
tc filter del dev ifb0 parent 1: handle 801::800 prio 1 u32
tc filter del dev ifb1 parent 1: handle 801::800 prio 1 u32
tc filter del dev ifb2 parent 1: handle 801::800 prio 1 u32

#LIVE
tc qdisc add dev ifb0 parent 1:10 handle 100: nfq_codel quantum 500 limit
1000 ecn
#tc filter add dev ifb0 parent 1: handle 801::800 prio 1 protocol ip u32
match ip dst 172.30.42.102 match ip protocol 0x11 0xff match ip dport 3074
0xffff flowid 1:10 ## Doesn't work..?
tc filter add dev ifb0 parent 1: protocol ip prio 1 handle 801::800 u32
match ip protocol 0x11 0xff match ip dport 3074 0xffff flowid 1:10
tc filter add dev ifb0 parent 1: protocol ip prio 1 handle 801::801 u32
match ip protocol 0x11 0xff match ip sport 3074 0xffff flowid 1:10

tc qdisc add dev ifb2 parent 1:10 handle 100: nfq_codel quantum 300 limit
1000 ecn
tc filter add dev ifb2 parent 1: handle 801::800 prio 1 protocol ip u32
match ip src 172.30.42.102 match ip protocol 0x11 0xff match ip sport 3074
0xffff flowid 1:10
tc filter add dev ifb2 parent 1: prio 1 protocol ip handle 801::801 u32
match ip protocol 0x11 0xff match ip sport 3074 0xffff flowid 1:10

#------------------

## Torrent
tc class replace dev ge00 parent 1: classid 1:13 htb rate 10000 ceil 320000
prio 3 quantum 1478
tc class replace dev ifb1 parent 1: classid 1:13 htb rate 10000 ceil 350000
prio 3 quantum 1478

tc filter add dev ifb0 parent 1: handle 801::803 protocol ip prio 1 u32
match ip tos 0x38 0xff flowid 1:13
tc filter add dev ifb0 parent 1: handle 801::804 prio 1 protocol ip u32
match ip dport 22223 0xffff flowid 1:13
tc filter add dev ifb0 parent 1: handle 801::805 prio 1 protocol ip u32
match ip dport 22225 0xffff flowid 1:13
tc filter add dev ifb0 parent 1: handle 801::806 prio 1 protocol ip u32
match ip dport 22227 0xffff flowid 1:13

tc filter add dev ifb1 parent 1: handle 801::803 protocol ip prio 1 u32
match ip tos 0x38 0xff flowid 1:13 ##Mark testing - doesn't work
tc filter add dev ifb1 parent 1: handle 801::804 prio 1 protocol ip u32
match ip sport 22223 0xffff flowid 1:13
tc filter add dev ifb1 parent 1: handle 801::805 prio 1 protocol ip u32
match ip sport 22224 0xffff flowid 1:13
tc filter add dev ifb1 parent 1: handle 801::806 prio 1 protocol ip u32
match ip sport 22225 0xffff flowid 1:13
tc filter add dev ifb1 parent 1: handle 801::807 prio 1 protocol ip u32
match ip sport 22226 0xffff flowid 1:13
tc filter add dev ifb1 parent 1: handle 801::808 prio 1 protocol ip u32
match ip sport 22227 0xffff flowid 1:13


## Billy's Laptop - Speed Cap
iptables -t mangle -A FORWARD -s 172.30.42.73 -j DSCP --set-dscp-class AF12
iptables -t mangle -A FORWARD -d 172.30.42.73 -j DSCP --set-dscp-class AF12
iptables -t mangle -A FORWARD -s 172.30.42.74 -j DSCP --set-dscp-class AF12
iptables -t mangle -A FORWARD -d 172.30.42.74 -j DSCP --set-dscp-class AF12

iptables -t mangle -A QOS_MARK_ge00 -p 0 -m dscp --dscp-class AF12 -j MARK
--set-mark 0x4
iptables -t mangle -A QOS_MARK_sw00 -p 0 -m dscp --dscp-class AF12 -j MARK
--set-mark 0x4

##Unable to cap via ifb0. sw00 works nicely, though.
tc class add dev ge00 parent 1:1 classid 1:14 htb rate 50000bit ceil
200000bit prio 4 quantum 1478
tc class add dev sw00 parent 1:1 classid 1:14 htb rate 200Kbit ceil 800Kbit
prio 4 quantum 1478 
tc qdisc add dev ge00 parent 1:14 handle 140: fq_codel limit 600 flows 1024
quantum 300 target 5000 interval 100000 noecn
tc qdisc add dev sw00 parent 1:14 handle 140: fq_codel limit 600 flows 1024
quantum 300 target 5000 interval 100000 noecn
tc filter add dev ge00 parent 1: prio 3 protocol ip handle 0x4 fw flowid 1:14
tc filter add dev sw00 parent 1: prio 3 protocol ip handle 0x4 fw flowid 1:14

#tc filter add dev ifb0 parent 1: handle 801::802 prio 1 protocol ip u32
match ip dst 172.30.42.73 flowid 1:13 ##Doesn't work
tc class add dev ifb1 parent 1:1 classid 1:14 htb rate 50000bit ceil
200000bit prio 4 quantum 1478
tc qdisc add dev ifb1 parent 1:14 handle 140: fq_codel limit 600 flows 1024
quantum 300 target 5000 interval 100000 ecn
tc filter add dev ifb1 parent 1: handle 801::802 prio 1 protocol ip u32
match ip src 172.30.42.73 flowid 1:14

#ICMP (ip protocol 1) in the interactive class
#tc filter add dev ifb0 parent 1: handle 801::802 protocol ip prio 1 u32
match ip protocol 1 0xff flowid 1:11
#tc filter add dev ifb1 parent 1: handle 801::801 protocol ip prio 1 u32
match ip protocol 1 0xff flowid 1:11
#tc filter add dev ifb2 parent 1: handle 801::802 protocol ip prio 1 u32
match ip protocol 1 0xff flowid 1:11
#iptables -t mangle -A QOS_MARK_ge00 -p ICMP -j MARK --set-mark 0x1
#iptables -t mangle -A QOS_MARK_sw00 -p ICMP -j MARK --set-mark 0x1
#iptables -t mangle -A QOS_MARK_sw10 -p ICMP -j MARK --set-mark 0x1

#Guest Network 2.4G
iptables -t mangle -A FORWARD -s 172.30.42.129/27 -j DSCP --set-dscp-class AF12
iptables -t mangle -A FORWARD -d 172.30.42.129/27 -j DSCP --set-dscp-class AF12

#22223 0xffff flowid (...)
#tc filter add (...) u32 match ip [s-d]port 22224 0xfffe flowid (...)
#tc filter add (...) u32 match ip [s-d]port 22226 0xffff flowid (...)


/etc/fixdaemons &
exit 0

---------------------------------------------

A few notes:
Before you ask, yes, I've heard time and time again while lurking around
here that there's no sense in shaping inbound traffic. It's already past the
bottleneck so you might as well just take the data and prevent retransmits
etc. I thought so too, but doing this for torrents seems to make everything
feel a lot more responsive while torrenting then not doing it at all. It
could be a just a placebo effect but, eh. I feel it helps.

The game console is placed to hog all the bandwidth if it needs it 'cause
someone wants to host games sometimes. Apparently, you need as much data as
you can get to host games so I can't just throw it in the interactive class
(1:11). It only uses about half of what we have, anyway, so I don't mind.

The rest is just traffic shaping for 'Billy' and the guest network.
 

> 
> B) I think the problem is not the script but the ifb module getting
> inserted with only two ifbs allowed.
> 
> try a:
> 
> /etc/init.d/aqm stop
> rmmod ifb
> insmod ifb numifbs=8
> /etc/init.d/aqm start
>

This worked beautifully. Thank you.