[Cerowrt-devel] [Bloat] applying fq_codel on a multiple wan router
Dave Taht
dave.taht at gmail.com
Thu Nov 6 10:29:17 EST 2014
a better list for this sort of question is the cerowrt-devel list.
Regrettably the load balancing part of the question requires more
brain cells than I have available at the moment. The SQM scripts could
be hacked up to do what you want, I think, but I have to think about
it (and this week is ietf which I am trying to put to bed)
On Fri, Oct 31, 2014 at 4:23 AM, Carlos R. Pasqualini
<pasqualinic at fcal.uner.edu.ar> wrote:
> Hi all
>
> I opt to write this lengthy mail to document all the process, i hope these
> not be a problem to the list's members.
>
>
> In our campus we have a Debian box managing the bulk internet access of our
> users.
>
> For the sake of simplicity, until everything works as expected, i have put
> in separated machines the firewall rules, intervlan rules and every other
> complexity (found in the FWLAN Box), from the load balance and QoS which is
> the machine i will explain; this machine i called 'BALANCER'.
>
>
> On the LAN side i have a real NIC, eth0, with IP 10.110.84.1/24
>
> On the WAN side i have one NIC, eth1, with N ADSLs (3 at the moment but, i
> need to be ready to increase that number so i prefer to build a script with
> a N variable and not a fixed number) each one in his own VLAN:
>
>
> |-- eth1.1001 -- 10.12.1.244/24 gw 10.12.1.2
> |Local VLANS 172.24/16| -- [FWLAN] 10.110.84.2 -- eth0 -- [BALANCER]
> eth1--|-- eth1.1002 -- 10.12.2.244/24 gw 10.12.2.2
>
> |-- eth1.1003 -- 10.12.3.244/24 gw 10.12.3.2
>
> Everything works (almost) as expected without QoS (suffering bufferbloat,
> but that is a queue problem, not a router problem)
>
>
> So yesterday i was working on adding some QoS (HTB/fq_codel) to this ADSLs.
>
> As documented, to be effective, fq_codel needs to be in the bottleneck so i
> need to shape the traffic of each ADSL line to numbers under those
> advertised by our ISP (Thinking to using HTB for it)
>
> Reading somewhere, i found that i cannot apply a queue to a VLAN interface,
> only to a real one, so the suggested solution was to put a bridge on each
> VLAN and apply the queue to each bridge which can apply those queues
>
> well i have made that modification:
>
> |-- eth1.1001 -- br1 -- 10.12.1.244/24 gw 10.12.1.2
> eth0 -- BALANCER eth1--|-- eth1.1002 -- br2 -- 10.12.2.244/24 gw 10.12.2.2
> |-- eth1.1003 -- br3 -- 10.12.3.244/24 gw 10.12.3.2
>
> and with some little tweaking i was able to get it to work (still without
> queues or shaper)
>
> The script that seems to be working is (or the interesting part of it):
>
> firewall_balancer_bridges.sh
> ------------------------
> $IP route del default
> $IP rule show|grep ADSL|awk '{gsub($1,"ip rule delete",$0);print $0}'|bash
>
> $IPT -t mangle -A PREROUTING -j CONNMARK --restore-mark
> $IPT -t mangle -A PREROUTING -m comment --comment "this stream is already
> marked; escape early" -m mark ! --mark 0 -j ACCEPT
>
> ROUTERS="route add default scope global "
> for LINK in `links`; do
> TABLE=`link_table $LINK`
> FWMARK=`link_fwmark $LINK`
> LINKIP=`if_address $LINK`
> LINKPEER=`if_peer $LINK`
>
> $IP route add $LINKPEER dev $LINK src $LINKIP table $TABLE
> $IP route add default via $LINKPEER table $TABLE
> $IP route add 172.23.0.0/16 via 10.110.84.2 dev eth0 table $TABLE
> $IP route add 172.24.0.0/16 via 10.110.84.2 dev eth0 table $TABLE
> $IP route add 172.25.0.0/16 via 10.110.84.2 dev eth0 table $TABLE
> $IP route add 10.110.84.0/24 dev eth0 table $TABLE
>
> $IP route add $LINKPEER dev $LINK src $LINKIP
> $IP rule add from $LINKIP table $TABLE
> $IP rule add iif $LINK table $TABLE
> $IP rule add oif $LINK table $TABLE
> $IP rule add fwmark $FWMARK table $TABLE
>
> $IPT -t mangle -N $TABLE
> $IPT -t mangle -A $TABLE -m comment --comment "send via $TABLE" -j
> MARK --set-mark $FWMARK
> $IPT -t mangle -A $TABLE -j CONNMARK --save-mark
> $IPT -t mangle -A $TABLE -j RETURN
>
> $IPT -t mangle -A PREROUTING -m comment --comment "prevent
> asynchronous routing $LINK" -i $LINK -m conntrack --ctstate NEW -j $TABLE
>
> $IPT -t mangle -A OUTPUT -o $LINK -m conntrack --ctstate NEW -j
> $TABLE
> $IPT -t mangle -A POSTROUTING -o $LINK -m conntrack --ctstate NEW -j
> $TABLE
>
> $IPT -t nat -A POSTROUTING -o $LINK -j SNAT --to-source $LINKIP
> if [[ `ping -c $TCANT -I $LINK 8.8.8.8|grep -c '64 bytes'` -eq
> $TCANT ]]; then
> ROUTERS="$ROUTERS nexthop via $LINKPEER dev $LINK weight 1"
> else
> echo "La ruta $TABLE/$LINK presenta pérdida de paquetes"
> fi
> done
> $IP $ROUTERS
>
>
> echo 1 > /proc/sys/net/ipv4/ip_forward
> ----------------------------
>
> (In Argentina we do not have IPv6 yet on the ISPs, so i want it working IPv4
> only right now and add IPv6 later)
>
>
> I found a script which seems to be OK in:
>
> http://wiki.gentoo.org/wiki/Traffic_shaping
>
>
> And started to adapt it to suit my needs, and here is where everything broke
> up:
>
>
> firewall_balancer_bridges_shaper.sh
> -----------------------------------------
> ...
> # Ensure to have enough IFBs
> modprobe ifb numifbs=8
> ...
> ROUTERS="route add default scope global "
> for LINK in `links`; do
> TABLE=`link_table $LINK`
> FWMARK=`link_fwmark $LINK`
> LINKIP=`if_address $LINK`
> LINKPEER=`if_peer $LINK`
> IFVDEV=`link_ifb $LINK`
>
>
> ############################################################################
> # Traffic Shaper
>
> # Clear old queuing disciplines (qdisc) on the interfaces
> $TC qdisc del dev $LINK root
> $TC qdisc del dev $LINK ingress
> $TC qdisc del dev $IFVDEV root
> $TC qdisc del dev $IFVDEV ingress
>
> # Ingress
>
> # Create ingress on external interface
> $TC qdisc add dev $LINK handle ffff: ingress
>
> $IFC $IFVDEV up # if the interace is not up bad things happen
>
> # Forward all ingress traffic to the IFB device
> $TC filter add dev $LINK parent ffff: protocol all u32 match u32 0 0
> action mirred egress redirect dev $IFVDEV
>
> # Create an EGRESS filter on the IFB device
> $TC qdisc add dev $IFVDEV root handle 1: htb default 11
>
> # Add root class HTB with rate limiting
>
> $TC class add dev $IFVDEV parent 1: classid 1:1 htb rate $BWDOWN
> $TC class add dev $IFVDEV parent 1:1 classid 1:11 htb rate $BWDOWN
> prio 0 quantum $HTB_Q
>
>
> # Add FQ_CODEL qdisc with ECN support (if you want ecn)
> $TC qdisc add dev $IFVDEV parent 1:11 fq_codel quantum $CODEL_Q ecn
>
> # Egress
> # Add FQ_CODEL to EGRESS on external interface
> $TC qdisc add dev $LINK root handle 1: htb default 11
>
> # Add root class HTB with rate limiting
> $TC class add dev $LINK parent 1: classid 1:1 htb rate $BWUP
> $TC class add dev $LINK parent 1:1 classid 1:11 htb rate $BWUP prio
> 0 quantum $HTB_Q
>
> # Note: You can apply a packet limit here and on ingress if you are
> memory constrained - e.g
> # for low bandwidths and machines with < 64MB of ram, limit 1000 is
> good, otherwise no point
>
> # Add FQ_CODEL qdisc without ECN support - on egress it's generally
> better to just drop the packet
> # but feel free to enable it if you want.
>
> $TC qdisc add dev $LINK parent 1:11 fq_codel quantum $CODEL_Q noecn
>
>
>
> ############################################################################
>
> $IP route add $LINKPEER dev $LINK src $LINKIP table $TABLE
> $IP route add default via $LINKPEER table $TABLE
> ...
> --------------------------
>
>
> but it stopped to work... i even was unable to ping each ADSL's router IP
> (10.12.1.2)
> I reverted to the previous script (firewall_balancer_bridges.sh) and it
> worked but without fq_codel :(
>
>
> I feel stuck on this, can anyone give a hint on what to change/test and/or a
> different way of doing this?
>
>
> I don't feel sane to build a bridge for every vlan for applying a shaper (on
> the FWLAN box i have 34 VLANs), this shall be done using tc only but didn't
> found an example of it, but this is another story.
>
>
>
> Best Regards
>
> --
>
>
> Carlos R. Pasqualini
> Administración de Redes
> Facultad de Ciencias de la Alimentación
> Universidad Nacional de Entre Ríos
>
> ---------------------------------------------------------
> Recuerde que ningun Administrador de red le solicitará
> el envío de una contraseña por correo electrónico.
> Si recibe un correo en el que se solicita una contraseña
> seguramente se trata de un fraude; avise del mismo a sus
> administradores de red a la brevedad.
> ---------------------------------------------------------
> Facultad de Ciencias de la Alimentación
> Universidad Nacional de Entre Ríos
> http://www.fcal.uner.edu.ar/
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
--
Dave Täht
thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks
More information about the Cerowrt-devel
mailing list