[Cerowrt-devel] [Bloat] applying fq_codel on a multiple wan router
Carlos R. Pasqualini
pasqualinic at fcal.uner.edu.ar
Fri Nov 7 18:35:24 EST 2014
El jue, 06-11-2014 a las 07:29 -0800, Dave Taht escribió:
> a better list for this sort of question is the cerowrt-devel list.
>
> Regrettably the load balancing part of the question requires more
> brain cells than I have available at the moment. The SQM scripts could
> be hacked up to do what you want, I think, but I have to think about
> it (and this week is ietf which I am trying to put to bed)
Thank you dave
> On Fri, Oct 31, 2014 at 4:23 AM, Carlos R. Pasqualini
> <pasqualinic at fcal.uner.edu.ar> wrote:
> > Hi all
> >
> > I opt to write this lengthy mail to document all the process, i hope these
> > not be a problem to the list's members.
> >
> >
> > In our campus we have a Debian box managing the bulk internet access of our
> > users.
> >
> > For the sake of simplicity, until everything works as expected, i have put
> > in separated machines the firewall rules, intervlan rules and every other
> > complexity (found in the FWLAN Box), from the load balance and QoS which is
> > the machine i will explain; this machine i called 'BALANCER'.
> >
> >
> > On the LAN side i have a real NIC, eth0, with IP 10.110.84.1/24
> >
> > On the WAN side i have one NIC, eth1, with N ADSLs (3 at the moment but, i
> > need to be ready to increase that number so i prefer to build a script with
> > a N variable and not a fixed number) each one in his own VLAN:
> >
> >
> > |-- eth1.1001 -- 10.12.1.244/24 gw 10.12.1.2
> > |Local VLANS 172.24/16| -- [FWLAN] 10.110.84.2 -- eth0 -- [BALANCER]
> > eth1--|-- eth1.1002 -- 10.12.2.244/24 gw 10.12.2.2
> >
> > |-- eth1.1003 -- 10.12.3.244/24 gw 10.12.3.2
> >
> > Everything works (almost) as expected without QoS (suffering bufferbloat,
> > but that is a queue problem, not a router problem)
> >
> >
> > So yesterday i was working on adding some QoS (HTB/fq_codel) to this ADSLs.
> >
> > As documented, to be effective, fq_codel needs to be in the bottleneck so i
> > need to shape the traffic of each ADSL line to numbers under those
> > advertised by our ISP (Thinking to using HTB for it)
> >
> > Reading somewhere, i found that i cannot apply a queue to a VLAN interface,
> > only to a real one, so the suggested solution was to put a bridge on each
> > VLAN and apply the queue to each bridge which can apply those queues
> >
> > well i have made that modification:
> >
> > |-- eth1.1001 -- br1 -- 10.12.1.244/24 gw 10.12.1.2
> > eth0 -- BALANCER eth1--|-- eth1.1002 -- br2 -- 10.12.2.244/24 gw 10.12.2.2
> > |-- eth1.1003 -- br3 -- 10.12.3.244/24 gw 10.12.3.2
> >
> > and with some little tweaking i was able to get it to work (still without
> > queues or shaper)
> >
> > The script that seems to be working is (or the interesting part of it):
> >
> > firewall_balancer_bridges.sh
> > ------------------------
> > $IP route del default
> > $IP rule show|grep ADSL|awk '{gsub($1,"ip rule delete",$0);print $0}'|bash
> >
> > $IPT -t mangle -A PREROUTING -j CONNMARK --restore-mark
> > $IPT -t mangle -A PREROUTING -m comment --comment "this stream is already
> > marked; escape early" -m mark ! --mark 0 -j ACCEPT
> >
> > ROUTERS="route add default scope global "
> > for LINK in `links`; do
> > TABLE=`link_table $LINK`
> > FWMARK=`link_fwmark $LINK`
> > LINKIP=`if_address $LINK`
> > LINKPEER=`if_peer $LINK`
> >
> > $IP route add $LINKPEER dev $LINK src $LINKIP table $TABLE
> > $IP route add default via $LINKPEER table $TABLE
> > $IP route add 172.23.0.0/16 via 10.110.84.2 dev eth0 table $TABLE
> > $IP route add 172.24.0.0/16 via 10.110.84.2 dev eth0 table $TABLE
> > $IP route add 172.25.0.0/16 via 10.110.84.2 dev eth0 table $TABLE
> > $IP route add 10.110.84.0/24 dev eth0 table $TABLE
> >
> > $IP route add $LINKPEER dev $LINK src $LINKIP
> > $IP rule add from $LINKIP table $TABLE
> > $IP rule add iif $LINK table $TABLE
> > $IP rule add oif $LINK table $TABLE
> > $IP rule add fwmark $FWMARK table $TABLE
> >
> > $IPT -t mangle -N $TABLE
> > $IPT -t mangle -A $TABLE -m comment --comment "send via $TABLE" -j
> > MARK --set-mark $FWMARK
> > $IPT -t mangle -A $TABLE -j CONNMARK --save-mark
> > $IPT -t mangle -A $TABLE -j RETURN
> >
> > $IPT -t mangle -A PREROUTING -m comment --comment "prevent
> > asynchronous routing $LINK" -i $LINK -m conntrack --ctstate NEW -j $TABLE
> >
> > $IPT -t mangle -A OUTPUT -o $LINK -m conntrack --ctstate NEW -j
> > $TABLE
> > $IPT -t mangle -A POSTROUTING -o $LINK -m conntrack --ctstate NEW -j
> > $TABLE
> >
> > $IPT -t nat -A POSTROUTING -o $LINK -j SNAT --to-source $LINKIP
> > if [[ `ping -c $TCANT -I $LINK 8.8.8.8|grep -c '64 bytes'` -eq
> > $TCANT ]]; then
> > ROUTERS="$ROUTERS nexthop via $LINKPEER dev $LINK weight 1"
> > else
> > echo "La ruta $TABLE/$LINK presenta pérdida de paquetes"
> > fi
> > done
> > $IP $ROUTERS
> >
> >
> > echo 1 > /proc/sys/net/ipv4/ip_forward
> > ----------------------------
> >
> > (In Argentina we do not have IPv6 yet on the ISPs, so i want it working IPv4
> > only right now and add IPv6 later)
> >
> >
> > I found a script which seems to be OK in:
> >
> > http://wiki.gentoo.org/wiki/Traffic_shaping
> >
> >
> > And started to adapt it to suit my needs, and here is where everything broke
> > up:
> >
> >
> > firewall_balancer_bridges_shaper.sh
> > -----------------------------------------
> > ...
> > # Ensure to have enough IFBs
> > modprobe ifb numifbs=8
> > ...
> > ROUTERS="route add default scope global "
> > for LINK in `links`; do
> > TABLE=`link_table $LINK`
> > FWMARK=`link_fwmark $LINK`
> > LINKIP=`if_address $LINK`
> > LINKPEER=`if_peer $LINK`
> > IFVDEV=`link_ifb $LINK`
> >
> >
> > ############################################################################
> > # Traffic Shaper
> >
> > # Clear old queuing disciplines (qdisc) on the interfaces
> > $TC qdisc del dev $LINK root
> > $TC qdisc del dev $LINK ingress
> > $TC qdisc del dev $IFVDEV root
> > $TC qdisc del dev $IFVDEV ingress
> >
> > # Ingress
> >
> > # Create ingress on external interface
> > $TC qdisc add dev $LINK handle ffff: ingress
> >
> > $IFC $IFVDEV up # if the interace is not up bad things happen
> >
> > # Forward all ingress traffic to the IFB device
> > $TC filter add dev $LINK parent ffff: protocol all u32 match u32 0 0
> > action mirred egress redirect dev $IFVDEV
> >
> > # Create an EGRESS filter on the IFB device
> > $TC qdisc add dev $IFVDEV root handle 1: htb default 11
> >
> > # Add root class HTB with rate limiting
> >
> > $TC class add dev $IFVDEV parent 1: classid 1:1 htb rate $BWDOWN
> > $TC class add dev $IFVDEV parent 1:1 classid 1:11 htb rate $BWDOWN
> > prio 0 quantum $HTB_Q
> >
> >
> > # Add FQ_CODEL qdisc with ECN support (if you want ecn)
> > $TC qdisc add dev $IFVDEV parent 1:11 fq_codel quantum $CODEL_Q ecn
> >
> > # Egress
> > # Add FQ_CODEL to EGRESS on external interface
> > $TC qdisc add dev $LINK root handle 1: htb default 11
> >
> > # Add root class HTB with rate limiting
> > $TC class add dev $LINK parent 1: classid 1:1 htb rate $BWUP
> > $TC class add dev $LINK parent 1:1 classid 1:11 htb rate $BWUP prio
> > 0 quantum $HTB_Q
> >
> > # Note: You can apply a packet limit here and on ingress if you are
> > memory constrained - e.g
> > # for low bandwidths and machines with < 64MB of ram, limit 1000 is
> > good, otherwise no point
> >
> > # Add FQ_CODEL qdisc without ECN support - on egress it's generally
> > better to just drop the packet
> > # but feel free to enable it if you want.
> >
> > $TC qdisc add dev $LINK parent 1:11 fq_codel quantum $CODEL_Q noecn
> >
> >
> >
> > ############################################################################
> >
> > $IP route add $LINKPEER dev $LINK src $LINKIP table $TABLE
> > $IP route add default via $LINKPEER table $TABLE
> > ...
> > --------------------------
> >
> >
> > but it stopped to work... i even was unable to ping each ADSL's router IP
> > (10.12.1.2)
> > I reverted to the previous script (firewall_balancer_bridges.sh) and it
> > worked but without fq_codel :(
> >
> >
> > I feel stuck on this, can anyone give a hint on what to change/test and/or a
> > different way of doing this?
> >
> >
> > I don't feel sane to build a bridge for every vlan for applying a shaper (on
> > the FWLAN box i have 34 VLANs), this shall be done using tc only but didn't
> > found an example of it, but this is another story.
> >
> >
> >
> > Best Regards
> >
This email that Dave posted to this list was written a week ago.
As i can see my ascii art is very odd, so i have made a simple diagram
for clarifying (at least part of) the setup [1]
In between i was trying to re-think the issue because there is one thing
i don't like: the need for a bridge for each VLAN.
The need for them comes from the (in)ability to throttle each VLAN
interface to the speed of each ADSL Line but later i found [2] that i
can use 'tc filter' to assign traffic to different HTB buckets based on
the VLAN Tag, and limit the speed for each bucket based on the speed of
each ADSL line and then apply fq_codel for the last bucket of each class
[1] http://archivo.fcal.uner.edu.ar/files/pasqualinic/Balancer.png
[2] http://brownian.org.ua/?p=1000&langswitch_lang=en
Best Regards
Carlos Pasqualini
More information about the Cerowrt-devel
mailing list