From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.fcal.uner.edu.ar (mx1.fcal.uner.edu.ar [200.61.249.197]) by huchra.bufferbloat.net (Postfix) with ESMTP id 4EE7221F5AE for ; Fri, 31 Oct 2014 04:23:46 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mx1.fcal.uner.edu.ar (Postfix) with ESMTP id 261B72460002 for ; Fri, 31 Oct 2014 08:23:44 -0300 (ART) X-Virus-Scanned: Debian amavisd-new at mx1.fcal.uner.edu.ar Received: from mx1.fcal.uner.edu.ar ([127.0.0.1]) by localhost (mx1.fcal.uner.edu.ar [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EPeuYr172aLj for ; Fri, 31 Oct 2014 08:23:24 -0300 (ART) Received: from webmail.fcal.uner.edu.ar (unknown [172.25.1.38]) by mx1.fcal.uner.edu.ar (Postfix) with ESMTP id 52830E7A2D5 for ; Fri, 31 Oct 2014 08:23:18 -0300 (ART) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Fri, 31 Oct 2014 08:23:18 -0300 From: "Carlos R. Pasqualini" To: bloat@lists.bufferbloat.net Organization: =?UTF-8?Q?Adm=2E_Redes_Facultad_de_Ciencias_de_la_Alimentac?= =?UTF-8?Q?i=C3=B3n?= Message-ID: <42f8ba65327eec7da91debecd9f69843@fcal.uner.edu.ar> X-Sender: pasqualinic@fcal.uner.edu.ar User-Agent: FCAL Webmail/0.9.2 Subject: [Bloat] =?utf-8?q?applying_fq=5Fcodel_on_a_multiple_wan_router?= X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Oct 2014 11:24:19 -0000 Hi all I opt to write this lengthy mail to document all the process, i hope these not be a problem to the list's members. In our campus we have a Debian box managing the bulk internet access of our users. For the sake of simplicity, until everything works as expected, i have put in separated machines the firewall rules, intervlan rules and every other complexity (found in the FWLAN Box), from the load balance and QoS which is the machine i will explain; this machine i called 'BALANCER'. On the LAN side i have a real NIC, eth0, with IP 10.110.84.1/24 On the WAN side i have one NIC, eth1, with N ADSLs (3 at the moment but, i need to be ready to increase that number so i prefer to build a script with a N variable and not a fixed number) each one in his own VLAN: |-- eth1.1001 -- 10.12.1.244/24 gw 10.12.1.2 |Local VLANS 172.24/16| -- [FWLAN] 10.110.84.2 -- eth0 -- [BALANCER] eth1--|-- eth1.1002 -- 10.12.2.244/24 gw 10.12.2.2 |-- eth1.1003 -- 10.12.3.244/24 gw 10.12.3.2 Everything works (almost) as expected without QoS (suffering bufferbloat, but that is a queue problem, not a router problem) So yesterday i was working on adding some QoS (HTB/fq_codel) to this ADSLs. As documented, to be effective, fq_codel needs to be in the bottleneck so i need to shape the traffic of each ADSL line to numbers under those advertised by our ISP (Thinking to using HTB for it) Reading somewhere, i found that i cannot apply a queue to a VLAN interface, only to a real one, so the suggested solution was to put a bridge on each VLAN and apply the queue to each bridge which can apply those queues well i have made that modification: |-- eth1.1001 -- br1 -- 10.12.1.244/24 gw 10.12.1.2 eth0 -- BALANCER eth1--|-- eth1.1002 -- br2 -- 10.12.2.244/24 gw 10.12.2.2 |-- eth1.1003 -- br3 -- 10.12.3.244/24 gw 10.12.3.2 and with some little tweaking i was able to get it to work (still without queues or shaper) The script that seems to be working is (or the interesting part of it): firewall_balancer_bridges.sh ------------------------ $IP route del default $IP rule show|grep ADSL|awk '{gsub($1,"ip rule delete",$0);print $0}'|bash $IPT -t mangle -A PREROUTING -j CONNMARK --restore-mark $IPT -t mangle -A PREROUTING -m comment --comment "this stream is already marked; escape early" -m mark ! --mark 0 -j ACCEPT ROUTERS="route add default scope global " for LINK in `links`; do TABLE=`link_table $LINK` FWMARK=`link_fwmark $LINK` LINKIP=`if_address $LINK` LINKPEER=`if_peer $LINK` $IP route add $LINKPEER dev $LINK src $LINKIP table $TABLE $IP route add default via $LINKPEER table $TABLE $IP route add 172.23.0.0/16 via 10.110.84.2 dev eth0 table $TABLE $IP route add 172.24.0.0/16 via 10.110.84.2 dev eth0 table $TABLE $IP route add 172.25.0.0/16 via 10.110.84.2 dev eth0 table $TABLE $IP route add 10.110.84.0/24 dev eth0 table $TABLE $IP route add $LINKPEER dev $LINK src $LINKIP $IP rule add from $LINKIP table $TABLE $IP rule add iif $LINK table $TABLE $IP rule add oif $LINK table $TABLE $IP rule add fwmark $FWMARK table $TABLE $IPT -t mangle -N $TABLE $IPT -t mangle -A $TABLE -m comment --comment "send via $TABLE" -j MARK --set-mark $FWMARK $IPT -t mangle -A $TABLE -j CONNMARK --save-mark $IPT -t mangle -A $TABLE -j RETURN $IPT -t mangle -A PREROUTING -m comment --comment "prevent asynchronous routing $LINK" -i $LINK -m conntrack --ctstate NEW -j $TABLE $IPT -t mangle -A OUTPUT -o $LINK -m conntrack --ctstate NEW -j $TABLE $IPT -t mangle -A POSTROUTING -o $LINK -m conntrack --ctstate NEW -j $TABLE $IPT -t nat -A POSTROUTING -o $LINK -j SNAT --to-source $LINKIP if [[ `ping -c $TCANT -I $LINK 8.8.8.8|grep -c '64 bytes'` -eq $TCANT ]]; then ROUTERS="$ROUTERS nexthop via $LINKPEER dev $LINK weight 1" else echo "La ruta $TABLE/$LINK presenta pérdida de paquetes" fi done $IP $ROUTERS echo 1 > /proc/sys/net/ipv4/ip_forward ---------------------------- (In Argentina we do not have IPv6 yet on the ISPs, so i want it working IPv4 only right now and add IPv6 later) I found a script which seems to be OK in: http://wiki.gentoo.org/wiki/Traffic_shaping And started to adapt it to suit my needs, and here is where everything broke up: firewall_balancer_bridges_shaper.sh ----------------------------------------- ... # Ensure to have enough IFBs modprobe ifb numifbs=8 ... ROUTERS="route add default scope global " for LINK in `links`; do TABLE=`link_table $LINK` FWMARK=`link_fwmark $LINK` LINKIP=`if_address $LINK` LINKPEER=`if_peer $LINK` IFVDEV=`link_ifb $LINK` ############################################################################ # Traffic Shaper # Clear old queuing disciplines (qdisc) on the interfaces $TC qdisc del dev $LINK root $TC qdisc del dev $LINK ingress $TC qdisc del dev $IFVDEV root $TC qdisc del dev $IFVDEV ingress # Ingress # Create ingress on external interface $TC qdisc add dev $LINK handle ffff: ingress $IFC $IFVDEV up # if the interace is not up bad things happen # Forward all ingress traffic to the IFB device $TC filter add dev $LINK parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev $IFVDEV # Create an EGRESS filter on the IFB device $TC qdisc add dev $IFVDEV root handle 1: htb default 11 # Add root class HTB with rate limiting $TC class add dev $IFVDEV parent 1: classid 1:1 htb rate $BWDOWN $TC class add dev $IFVDEV parent 1:1 classid 1:11 htb rate $BWDOWN prio 0 quantum $HTB_Q # Add FQ_CODEL qdisc with ECN support (if you want ecn) $TC qdisc add dev $IFVDEV parent 1:11 fq_codel quantum $CODEL_Q ecn # Egress # Add FQ_CODEL to EGRESS on external interface $TC qdisc add dev $LINK root handle 1: htb default 11 # Add root class HTB with rate limiting $TC class add dev $LINK parent 1: classid 1:1 htb rate $BWUP $TC class add dev $LINK parent 1:1 classid 1:11 htb rate $BWUP prio 0 quantum $HTB_Q # Note: You can apply a packet limit here and on ingress if you are memory constrained - e.g # for low bandwidths and machines with < 64MB of ram, limit 1000 is good, otherwise no point # Add FQ_CODEL qdisc without ECN support - on egress it's generally better to just drop the packet # but feel free to enable it if you want. $TC qdisc add dev $LINK parent 1:11 fq_codel quantum $CODEL_Q noecn ############################################################################ $IP route add $LINKPEER dev $LINK src $LINKIP table $TABLE $IP route add default via $LINKPEER table $TABLE ... -------------------------- but it stopped to work... i even was unable to ping each ADSL's router IP (10.12.1.2) I reverted to the previous script (firewall_balancer_bridges.sh) and it worked but without fq_codel :( I feel stuck on this, can anyone give a hint on what to change/test and/or a different way of doing this? I don't feel sane to build a bridge for every vlan for applying a shaper (on the FWLAN box i have 34 VLANs), this shall be done using tc only but didn't found an example of it, but this is another story. Best Regards -- Carlos R. Pasqualini Administración de Redes Facultad de Ciencias de la Alimentación Universidad Nacional de Entre Ríos --------------------------------------------------------- Recuerde que ningun Administrador de red le solicitará el envío de una contraseña por correo electrónico. Si recibe un correo en el que se solicita una contraseña seguramente se trata de un fraude; avise del mismo a sus administradores de red a la brevedad. --------------------------------------------------------- Facultad de Ciencias de la Alimentación Universidad Nacional de Entre Ríos http://www.fcal.uner.edu.ar/