From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lb0-x235.google.com (mail-lb0-x235.google.com [IPv6:2a00:1450:4010:c04::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id CD2663B2A2 for ; Mon, 28 Mar 2016 15:21:06 -0400 (EDT) Received: by mail-lb0-x235.google.com with SMTP id qe11so87221615lbc.3 for ; Mon, 28 Mar 2016 12:21:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=gyWDNR+kBdBFBruIiEC2/PutbmVFLWQkot+gStX0CZ0=; b=jaKJoXL2bTs2HhhynIsImKO1aaEI7sLuPMoWomNu53+yb/gDL6KegLpAFk6kygiBmw 0+3Yf9mMNZiJrQmjm4a2/1odDtiYNgxBpjGiH1AK7BZSBuI/jz91JfIYdSMFNncFwRwW M7893CZUutmGwn04P1dXl4tsiXyNKZsV28gJy5CRpY756/T27gXmvpJ0OWui47klYygz h4oO4o41X0hcFz7tqu/SjgkhTbkWM68G0XmfxIEI6cjgjj+AeeV0ap1hzcMh1W+Ai0n1 B5rfQYOWqq8flPn1Mr04xS2euXd7w5kQbEeJtBximTu++xb1llqdbp+N6fRVb7ylQ8um M82Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=gyWDNR+kBdBFBruIiEC2/PutbmVFLWQkot+gStX0CZ0=; b=aRxT3d8ddVFVLckBDG9UShE6wI/QYPkqOvtKQm+XOacU0WbaYQSDPcKxmS+dvTBBYr RwGnTWuR9oOKaT2Ny1OgPXCFQ2x2s29fDQOpNeWHKCZPM5cccGyIBA+uEzZUeYAiXa3L Bo90XTsnJsmPyd98AtBjx1RB8V6w5cYDdFjX0yHjv3/v7rMOOnjCb/yh7MbKwbHh98/D 8WaowvmGyIh0xGEKqc8WJCbHmnutwk/OcXTSMRIP3kY8VdZ000iT/Mtxc8DQRB0twsAW v9vi7Nugg40lOaZTacNrmGh0a/LXuRRVlduuOXLW4SQJVDzVi6CNLFw/l0zJtliOiXs0 1W/g== X-Gm-Message-State: AD7BkJIjzDALfcGDU+XJrlO5UOCA7u0vd5sj12DtijL6RuJqmfCH2JtksjFuZ/iXMchZFg== X-Received: by 10.112.235.5 with SMTP id ui5mr11068741lbc.111.1459192865568; Mon, 28 Mar 2016 12:21:05 -0700 (PDT) Received: from bass.home.chromatix.fi (37-33-67-252.bb.dnainternet.fi. [37.33.67.252]) by smtp.gmail.com with ESMTPSA id h188sm4657080lfd.14.2016.03.28.12.21.04 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 28 Mar 2016 12:21:04 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) From: Jonathan Morton In-Reply-To: Date: Mon, 28 Mar 2016 22:20:58 +0300 Cc: moeller0 , cake@lists.bufferbloat.net Content-Transfer-Encoding: quoted-printable Message-Id: <198CD4FC-6BD4-40DA-BE2F-22BBBA1E204D@gmail.com> References: <34953EAA-4C76-456E-9B9E-3A73D0DACCDE@gmail.com> <855E3354-30E6-4658-AF38-A0C1E92085CE@gmail.com> <66A00804-E571-4A44-BE3E-422F78C1F1F7@gmail.com> To: Allan Pinto X-Mailer: Apple Mail (2.3112) Subject: Re: [Cake] cake separate qos for lan X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 19:21:07 -0000 > On 28 Mar, 2016, at 15:25, Allan Pinto wrote: >=20 > I should have made this more clear please see below topology with = added comments. the customers connecting to the linux router can be in = range from 100 to 2000, so shaping on the switch is not really a option. = I am right now testing on a i3 machine, but for actual live testing am = planning to test with i7 or a xeon. >=20 > Cache-Server [ connected to internet gateway , = traffic can be sent to it via wccp or policy based routing ] > | > internet---->internet Gateway =E2=80=94> L2 switch [ MEN network on = fiber ] --> LInux router with cake[ includes a pppoe server which = authenticates with radius ] - - [ pppoe connection over a fiber men = network ] --> customer [ customers can be 100 to 2000 ] I see - so you are doing something like FTTP to a block of flats, where = each flat is potentially a separate customer. That also means you have = lots of virtual interfaces (each carrying one PPPoE session) over one = physical interface. > getting a illegal filter id for these two commands,=20 >=20 > >tc filter replace dev ppp0 protocol ip prio 1 handle 11 u32 match ip = src $CACHE_IP/32 > >tc filter replace dev ppp0 protocol ip prio 2 handle 12 u32 action = mirred egress redirect dev ifb0 Hmm. Apparently filters need to be attached to a classful qdisc as a = parent, which cake is not - I had hoped that the root class would act as = a surrogate. This filter system has a lot of under-documented and = counter-intuitive behaviour. Did you try the IMQ alternative? I don=E2=80=99t see any reason why it = shouldn=E2=80=99t work, as long as you can build a kernel with IMQ = support. But since you have lots of customers per router, there may still be a = sensible way to do it without IMQ. Here we step back from the idea of = attaching cake instances directly to the virtual interfaces, and instead = construct a complete shaping system using a classful structure on the = physical device. Each customer gets a class (and thus a cake instance = which you can set the bandwidth of individually), and the cache gets its = own separate class (and a cake instance). This reduces the number of = cake instances required considerably. I believe cake has been tested as capable of handling 10Gb/sec traffic = on commodity hardware. Certainly I have personally had a very modest = machine (an AMD E-450) shaping accurately at essentially GigE line rate, = that being the fastest network hardware I own. I don=E2=80=99t think = this will be materially affected by the number of cake instances = present, but as always, hard data from real experiments trumps theory. However, by the time traffic hits the physical egress interface, it has = already been encapsulated in PPPoE, making it much harder to write a = filter to identify which customer=E2=80=99s traffic it is, or whether it = came from the cache. So we need to do the shaping on an IFB device, = with the traffic redirected from the internal ingress device, where it = has not yet been encapsulated. For sanity=E2=80=99s sake, we should = also put a basic cake instance on the physical egress port. The following is thus a combination of ingress filtering and a hashed = filter, to handle up to 1022 customers with consecutive IPv4 addresses. = Adjust as required. Beware of the leopard. tc qdisc replace dev $EGRESS root handle 1: cake bandwidth $LOTS = besteffort flows ip link set ifb0 up tc qdisc replace dev $INGRESS handle ffff: ingress tc filter replace dev $INGRESS parent ffff: protocol all u32 action = mirred egress redirect dev ifb0 tc filter replace dev ifb0 parent 2:0 prio 5 protocol ip u32 tc filter add dev ifb0 parent 2:0 prio 6 handle 3: protocol ip u32 match = ip src $CACHEIP flowid 2:3ff tc qdisc replace dev ifb0 parent 2:3ff handle 5:3ff cake bandwidth = $PLENTY besteffort triple-isolate tc filter add dev ifb0 parent 2:0 prio 5 handle 4: protocol ip u32 = divisor 1024 for each customer, with $HEXID in [1..3fe]: tc filter add dev ifb0 protocol ip parent 2:0 prio 5 u32 ht = 4:${HEXID}: match ip dst $CUSTIP flowid 2:${HEXID} tc qdisc replace dev ifb0 parent 2:${HEXID} handle 5:${HEXID} = cake bandwidth $CUSTRATE triple-isolate tc filter add dev ifb0 protocol ip parent 2:0 prio 5 u32 ht 800:: match = ip dst ${CUSTNET}/22 hashkey mask 0x000003ff at 16 link 4: Honestly, I think the IMQ version is simpler and easier to understand, = as well as catching all customer traffic. The above mess doesn=E2=80=99t = even handle IPv6... - Jonathan Morton