From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ob0-x233.google.com (mail-ob0-x233.google.com [IPv6:2607:f8b0:4003:c01::233]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id E2ABC21F30E for ; Thu, 6 Nov 2014 07:29:23 -0800 (PST) Received: by mail-ob0-f179.google.com with SMTP id m8so1012395obr.10 for ; Thu, 06 Nov 2014 07:29:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=6wLML4DHDikfzv/cJiZm3Z2lrqocCqwPvVztJEaJBPk=; b=CnpHNOgp9zGmBy7FXMZq8DwkgNZW63G+r1+mh7v9bsrcb71seygllPe2xLkZs3vdIT 36NtqoZxZ0JozJFxdlrzr9JQg0dRvhfAVlfp2hN3J1lCk711b8kgyFW3/vRe5MyW1ak0 pJPGCtTY2HXWSmQCYEEhl33WbIFiI5zGQ9VVtDFvTH555QAEX7FUyzycXXcu9dU7krwF 6P1zL0NH9M89memZ4+3rheINNsWudlB24W52o09qbU8FiCV2El5kP+mtUkK9mvjBEW5l lkaoKuHNM7uEr/RCTGHigc8VmxOG9v/uo+ibTRXe3XUtDJoO4/6G9MF2V0oLhv3Xz9Tw oszQ== MIME-Version: 1.0 X-Received: by 10.202.104.141 with SMTP id o13mr3926177oik.65.1415287757396; Thu, 06 Nov 2014 07:29:17 -0800 (PST) Received: by 10.202.227.211 with HTTP; Thu, 6 Nov 2014 07:29:17 -0800 (PST) In-Reply-To: <42f8ba65327eec7da91debecd9f69843@fcal.uner.edu.ar> References: <42f8ba65327eec7da91debecd9f69843@fcal.uner.edu.ar> Date: Thu, 6 Nov 2014 07:29:17 -0800 Message-ID: From: Dave Taht To: "Carlos R. Pasqualini" , "cerowrt-devel@lists.bufferbloat.net" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Cerowrt-devel] [Bloat] applying fq_codel on a multiple wan router X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Nov 2014 15:29:52 -0000 a better list for this sort of question is the cerowrt-devel list. Regrettably the load balancing part of the question requires more brain cells than I have available at the moment. The SQM scripts could be hacked up to do what you want, I think, but I have to think about it (and this week is ietf which I am trying to put to bed) On Fri, Oct 31, 2014 at 4:23 AM, Carlos R. Pasqualini wrote: > Hi all > > I opt to write this lengthy mail to document all the process, i hope thes= e > not be a problem to the list's members. > > > In our campus we have a Debian box managing the bulk internet access of o= ur > users. > > For the sake of simplicity, until everything works as expected, i have pu= t > in separated machines the firewall rules, intervlan rules and every other > complexity (found in the FWLAN Box), from the load balance and QoS which = is > the machine i will explain; this machine i called 'BALANCER'. > > > On the LAN side i have a real NIC, eth0, with IP 10.110.84.1/24 > > On the WAN side i have one NIC, eth1, with N ADSLs (3 at the moment but, = i > need to be ready to increase that number so i prefer to build a script wi= th > a N variable and not a fixed number) each one in his own VLAN: > > > |-- eth1.1001 -- 10.12.1.244/24 gw 10.12.1.2 > |Local VLANS 172.24/16| -- [FWLAN] 10.110.84.2 -- eth0 -- [BALANCER] > eth1--|-- eth1.1002 -- 10.12.2.244/24 gw 10.12.2.2 > > |-- eth1.1003 -- 10.12.3.244/24 gw 10.12.3.2 > > Everything works (almost) as expected without QoS (suffering bufferbloat, > but that is a queue problem, not a router problem) > > > So yesterday i was working on adding some QoS (HTB/fq_codel) to this ADSL= s. > > As documented, to be effective, fq_codel needs to be in the bottleneck so= i > need to shape the traffic of each ADSL line to numbers under those > advertised by our ISP (Thinking to using HTB for it) > > Reading somewhere, i found that i cannot apply a queue to a VLAN interfac= e, > only to a real one, so the suggested solution was to put a bridge on each > VLAN and apply the queue to each bridge which can apply those queues > > well i have made that modification: > > |-- eth1.1001 -- br1 -- 10.12.1.244/24 gw 10.12.1= .2 > eth0 -- BALANCER eth1--|-- eth1.1002 -- br2 -- 10.12.2.244/24 gw 10.12.2= .2 > |-- eth1.1003 -- br3 -- 10.12.3.244/24 gw 10.12.3= .2 > > and with some little tweaking i was able to get it to work (still without > queues or shaper) > > The script that seems to be working is (or the interesting part of it): > > firewall_balancer_bridges.sh > ------------------------ > $IP route del default > $IP rule show|grep ADSL|awk '{gsub($1,"ip rule delete",$0);print $0}'|bas= h > > $IPT -t mangle -A PREROUTING -j CONNMARK --restore-mark > $IPT -t mangle -A PREROUTING -m comment --comment "this stream is already > marked; escape early" -m mark ! --mark 0 -j ACCEPT > > ROUTERS=3D"route add default scope global " > for LINK in `links`; do > TABLE=3D`link_table $LINK` > FWMARK=3D`link_fwmark $LINK` > LINKIP=3D`if_address $LINK` > LINKPEER=3D`if_peer $LINK` > > $IP route add $LINKPEER dev $LINK src $LINKIP table $TABLE > $IP route add default via $LINKPEER table $TABLE > $IP route add 172.23.0.0/16 via 10.110.84.2 dev eth0 table $TABLE > $IP route add 172.24.0.0/16 via 10.110.84.2 dev eth0 table $TABLE > $IP route add 172.25.0.0/16 via 10.110.84.2 dev eth0 table $TABLE > $IP route add 10.110.84.0/24 dev eth0 table $TABLE > > $IP route add $LINKPEER dev $LINK src $LINKIP > $IP rule add from $LINKIP table $TABLE > $IP rule add iif $LINK table $TABLE > $IP rule add oif $LINK table $TABLE > $IP rule add fwmark $FWMARK table $TABLE > > $IPT -t mangle -N $TABLE > $IPT -t mangle -A $TABLE -m comment --comment "send via $TABLE" -= j > MARK --set-mark $FWMARK > $IPT -t mangle -A $TABLE -j CONNMARK --save-mark > $IPT -t mangle -A $TABLE -j RETURN > > $IPT -t mangle -A PREROUTING -m comment --comment "prevent > asynchronous routing $LINK" -i $LINK -m conntrack --ctstate NEW -j $TABLE > > $IPT -t mangle -A OUTPUT -o $LINK -m conntrack --ctstate NEW -j > $TABLE > $IPT -t mangle -A POSTROUTING -o $LINK -m conntrack --ctstate NEW= -j > $TABLE > > $IPT -t nat -A POSTROUTING -o $LINK -j SNAT --to-source $LINKIP > if [[ `ping -c $TCANT -I $LINK 8.8.8.8|grep -c '64 bytes'` -eq > $TCANT ]]; then > ROUTERS=3D"$ROUTERS nexthop via $LINKPEER dev $LINK weigh= t 1" > else > echo "La ruta $TABLE/$LINK presenta p=C3=A9rdida de paque= tes" > fi > done > $IP $ROUTERS > > > echo 1 > /proc/sys/net/ipv4/ip_forward > ---------------------------- > > (In Argentina we do not have IPv6 yet on the ISPs, so i want it working I= Pv4 > only right now and add IPv6 later) > > > I found a script which seems to be OK in: > > http://wiki.gentoo.org/wiki/Traffic_shaping > > > And started to adapt it to suit my needs, and here is where everything br= oke > up: > > > firewall_balancer_bridges_shaper.sh > ----------------------------------------- > ... > # Ensure to have enough IFBs > modprobe ifb numifbs=3D8 > ... > ROUTERS=3D"route add default scope global " > for LINK in `links`; do > TABLE=3D`link_table $LINK` > FWMARK=3D`link_fwmark $LINK` > LINKIP=3D`if_address $LINK` > LINKPEER=3D`if_peer $LINK` > IFVDEV=3D`link_ifb $LINK` > > > #########################################################################= ### > # Traffic Shaper > > # Clear old queuing disciplines (qdisc) on the interfaces > $TC qdisc del dev $LINK root > $TC qdisc del dev $LINK ingress > $TC qdisc del dev $IFVDEV root > $TC qdisc del dev $IFVDEV ingress > > # Ingress > > # Create ingress on external interface > $TC qdisc add dev $LINK handle ffff: ingress > > $IFC $IFVDEV up # if the interace is not up bad things happen > > # Forward all ingress traffic to the IFB device > $TC filter add dev $LINK parent ffff: protocol all u32 match u32 = 0 0 > action mirred egress redirect dev $IFVDEV > > # Create an EGRESS filter on the IFB device > $TC qdisc add dev $IFVDEV root handle 1: htb default 11 > > # Add root class HTB with rate limiting > > $TC class add dev $IFVDEV parent 1: classid 1:1 htb rate $BWDOWN > $TC class add dev $IFVDEV parent 1:1 classid 1:11 htb rate $BWDOW= N > prio 0 quantum $HTB_Q > > > # Add FQ_CODEL qdisc with ECN support (if you want ecn) > $TC qdisc add dev $IFVDEV parent 1:11 fq_codel quantum $CODEL_Q e= cn > > # Egress > # Add FQ_CODEL to EGRESS on external interface > $TC qdisc add dev $LINK root handle 1: htb default 11 > > # Add root class HTB with rate limiting > $TC class add dev $LINK parent 1: classid 1:1 htb rate $BWUP > $TC class add dev $LINK parent 1:1 classid 1:11 htb rate $BWUP pr= io > 0 quantum $HTB_Q > > # Note: You can apply a packet limit here and on ingress if you a= re > memory constrained - e.g > # for low bandwidths and machines with < 64MB of ram, limit 1000 = is > good, otherwise no point > > # Add FQ_CODEL qdisc without ECN support - on egress it's general= ly > better to just drop the packet > # but feel free to enable it if you want. > > $TC qdisc add dev $LINK parent 1:11 fq_codel quantum $CODEL_Q noe= cn > > > > #########################################################################= ### > > $IP route add $LINKPEER dev $LINK src $LINKIP table $TABLE > $IP route add default via $LINKPEER table $TABLE > ... > -------------------------- > > > but it stopped to work... i even was unable to ping each ADSL's router IP > (10.12.1.2) > I reverted to the previous script (firewall_balancer_bridges.sh) and it > worked but without fq_codel :( > > > I feel stuck on this, can anyone give a hint on what to change/test and/o= r a > different way of doing this? > > > I don't feel sane to build a bridge for every vlan for applying a shaper = (on > the FWLAN box i have 34 VLANs), this shall be done using tc only but didn= 't > found an example of it, but this is another story. > > > > Best Regards > > -- > > > Carlos R. Pasqualini > Administraci=C3=B3n de Redes > Facultad de Ciencias de la Alimentaci=C3=B3n > Universidad Nacional de Entre R=C3=ADos > > --------------------------------------------------------- > Recuerde que ningun Administrador de red le solicitar=C3=A1 > el env=C3=ADo de una contrase=C3=B1a por correo electr=C3=B3nico. > Si recibe un correo en el que se solicita una contrase=C3=B1a > seguramente se trata de un fraude; avise del mismo a sus > administradores de red a la brevedad. > --------------------------------------------------------- > Facultad de Ciencias de la Alimentaci=C3=B3n > Universidad Nacional de Entre R=C3=ADos > http://www.fcal.uner.edu.ar/ > > > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat --=20 Dave T=C3=A4ht thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks