[Bloat] applying fq_codel on a multiple wan router
Carlos R. Pasqualini
pasqualinic at fcal.uner.edu.ar
Fri Oct 31 07:23:18 EDT 2014
Hi all
I opt to write this lengthy mail to document all the process, i hope
these not be a problem to the list's members.
In our campus we have a Debian box managing the bulk internet access of
our users.
For the sake of simplicity, until everything works as expected, i have
put in separated machines the firewall rules, intervlan rules and every
other complexity (found in the FWLAN Box), from the load balance and QoS
which is the machine i will explain; this machine i called 'BALANCER'.
On the LAN side i have a real NIC, eth0, with IP 10.110.84.1/24
On the WAN side i have one NIC, eth1, with N ADSLs (3 at the moment but,
i need to be ready to increase that number so i prefer to build a script
with a N variable and not a fixed number) each one in his own VLAN:
|-- eth1.1001 -- 10.12.1.244/24 gw 10.12.1.2
|Local VLANS 172.24/16| -- [FWLAN] 10.110.84.2 -- eth0 -- [BALANCER]
eth1--|-- eth1.1002 -- 10.12.2.244/24 gw 10.12.2.2
|-- eth1.1003 -- 10.12.3.244/24 gw 10.12.3.2
Everything works (almost) as expected without QoS (suffering
bufferbloat, but that is a queue problem, not a router problem)
So yesterday i was working on adding some QoS (HTB/fq_codel) to this
ADSLs.
As documented, to be effective, fq_codel needs to be in the bottleneck
so i need to shape the traffic of each ADSL line to numbers under those
advertised by our ISP (Thinking to using HTB for it)
Reading somewhere, i found that i cannot apply a queue to a VLAN
interface, only to a real one, so the suggested solution was to put a
bridge on each VLAN and apply the queue to each bridge which can apply
those queues
well i have made that modification:
|-- eth1.1001 -- br1 -- 10.12.1.244/24 gw
10.12.1.2
eth0 -- BALANCER eth1--|-- eth1.1002 -- br2 -- 10.12.2.244/24 gw
10.12.2.2
|-- eth1.1003 -- br3 -- 10.12.3.244/24 gw
10.12.3.2
and with some little tweaking i was able to get it to work (still
without queues or shaper)
The script that seems to be working is (or the interesting part of it):
firewall_balancer_bridges.sh
------------------------
$IP route del default
$IP rule show|grep ADSL|awk '{gsub($1,"ip rule delete",$0);print
$0}'|bash
$IPT -t mangle -A PREROUTING -j CONNMARK --restore-mark
$IPT -t mangle -A PREROUTING -m comment --comment "this stream is
already marked; escape early" -m mark ! --mark 0 -j ACCEPT
ROUTERS="route add default scope global "
for LINK in `links`; do
TABLE=`link_table $LINK`
FWMARK=`link_fwmark $LINK`
LINKIP=`if_address $LINK`
LINKPEER=`if_peer $LINK`
$IP route add $LINKPEER dev $LINK src $LINKIP table $TABLE
$IP route add default via $LINKPEER table $TABLE
$IP route add 172.23.0.0/16 via 10.110.84.2 dev eth0 table
$TABLE
$IP route add 172.24.0.0/16 via 10.110.84.2 dev eth0 table
$TABLE
$IP route add 172.25.0.0/16 via 10.110.84.2 dev eth0 table
$TABLE
$IP route add 10.110.84.0/24 dev eth0 table $TABLE
$IP route add $LINKPEER dev $LINK src $LINKIP
$IP rule add from $LINKIP table $TABLE
$IP rule add iif $LINK table $TABLE
$IP rule add oif $LINK table $TABLE
$IP rule add fwmark $FWMARK table $TABLE
$IPT -t mangle -N $TABLE
$IPT -t mangle -A $TABLE -m comment --comment "send via $TABLE"
-j MARK --set-mark $FWMARK
$IPT -t mangle -A $TABLE -j CONNMARK --save-mark
$IPT -t mangle -A $TABLE -j RETURN
$IPT -t mangle -A PREROUTING -m comment --comment "prevent
asynchronous routing $LINK" -i $LINK -m conntrack --ctstate NEW -j
$TABLE
$IPT -t mangle -A OUTPUT -o $LINK -m conntrack --ctstate NEW -j
$TABLE
$IPT -t mangle -A POSTROUTING -o $LINK -m conntrack --ctstate
NEW -j $TABLE
$IPT -t nat -A POSTROUTING -o $LINK -j SNAT --to-source $LINKIP
if [[ `ping -c $TCANT -I $LINK 8.8.8.8|grep -c '64 bytes'` -eq
$TCANT ]]; then
ROUTERS="$ROUTERS nexthop via $LINKPEER dev $LINK weight
1"
else
echo "La ruta $TABLE/$LINK presenta pérdida de paquetes"
fi
done
$IP $ROUTERS
echo 1 > /proc/sys/net/ipv4/ip_forward
----------------------------
(In Argentina we do not have IPv6 yet on the ISPs, so i want it working
IPv4 only right now and add IPv6 later)
I found a script which seems to be OK in:
http://wiki.gentoo.org/wiki/Traffic_shaping
And started to adapt it to suit my needs, and here is where everything
broke up:
firewall_balancer_bridges_shaper.sh
-----------------------------------------
...
# Ensure to have enough IFBs
modprobe ifb numifbs=8
...
ROUTERS="route add default scope global "
for LINK in `links`; do
TABLE=`link_table $LINK`
FWMARK=`link_fwmark $LINK`
LINKIP=`if_address $LINK`
LINKPEER=`if_peer $LINK`
IFVDEV=`link_ifb $LINK`
############################################################################
# Traffic Shaper
# Clear old queuing disciplines (qdisc) on the interfaces
$TC qdisc del dev $LINK root
$TC qdisc del dev $LINK ingress
$TC qdisc del dev $IFVDEV root
$TC qdisc del dev $IFVDEV ingress
# Ingress
# Create ingress on external interface
$TC qdisc add dev $LINK handle ffff: ingress
$IFC $IFVDEV up # if the interace is not up bad things happen
# Forward all ingress traffic to the IFB device
$TC filter add dev $LINK parent ffff: protocol all u32 match u32
0 0 action mirred egress redirect dev $IFVDEV
# Create an EGRESS filter on the IFB device
$TC qdisc add dev $IFVDEV root handle 1: htb default 11
# Add root class HTB with rate limiting
$TC class add dev $IFVDEV parent 1: classid 1:1 htb rate $BWDOWN
$TC class add dev $IFVDEV parent 1:1 classid 1:11 htb rate
$BWDOWN prio 0 quantum $HTB_Q
# Add FQ_CODEL qdisc with ECN support (if you want ecn)
$TC qdisc add dev $IFVDEV parent 1:11 fq_codel quantum $CODEL_Q
ecn
# Egress
# Add FQ_CODEL to EGRESS on external interface
$TC qdisc add dev $LINK root handle 1: htb default 11
# Add root class HTB with rate limiting
$TC class add dev $LINK parent 1: classid 1:1 htb rate $BWUP
$TC class add dev $LINK parent 1:1 classid 1:11 htb rate $BWUP
prio 0 quantum $HTB_Q
# Note: You can apply a packet limit here and on ingress if you
are memory constrained - e.g
# for low bandwidths and machines with < 64MB of ram, limit 1000
is good, otherwise no point
# Add FQ_CODEL qdisc without ECN support - on egress it's
generally better to just drop the packet
# but feel free to enable it if you want.
$TC qdisc add dev $LINK parent 1:11 fq_codel quantum $CODEL_Q
noecn
############################################################################
$IP route add $LINKPEER dev $LINK src $LINKIP table $TABLE
$IP route add default via $LINKPEER table $TABLE
...
--------------------------
but it stopped to work... i even was unable to ping each ADSL's router
IP (10.12.1.2)
I reverted to the previous script (firewall_balancer_bridges.sh) and it
worked but without fq_codel :(
I feel stuck on this, can anyone give a hint on what to change/test
and/or a different way of doing this?
I don't feel sane to build a bridge for every vlan for applying a shaper
(on the FWLAN box i have 34 VLANs), this shall be done using tc only but
didn't found an example of it, but this is another story.
Best Regards
--
Carlos R. Pasqualini
Administración de Redes
Facultad de Ciencias de la Alimentación
Universidad Nacional de Entre Ríos
---------------------------------------------------------
Recuerde que ningun Administrador de red le solicitará
el envío de una contraseña por correo electrónico.
Si recibe un correo en el que se solicita una contraseña
seguramente se trata de un fraude; avise del mismo a sus
administradores de red a la brevedad.
---------------------------------------------------------
Facultad de Ciencias de la Alimentación
Universidad Nacional de Entre Ríos
http://www.fcal.uner.edu.ar/
More information about the Bloat
mailing list