From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id B7CB33B29E for ; Wed, 9 Jan 2019 01:13:40 -0500 (EST) Received: by mail-wr1-x443.google.com with SMTP id x10so6374883wrs.8 for ; Tue, 08 Jan 2019 22:13:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=C1w486I9F+xsgSCip5HP2aR4mqxScD4feOYx2dJJj8M=; b=chY04GCb7RCDrIsN8/MLWIrX1XAYXTwWGihxZKdUlx/vN+d3T2o3At9JhqPwxLvMtj axMk9U3eb2VfmeuEWbzS+Cs2guzAGgGJixtyA0uuUxqp3pddLfYwZwpWhh6frsfU5XUA 2+lXyqii2NF9e/85gncsd1MDuCC7vwRp9Aw1cwUgQk6KHxxUGhv4NiFBu5vlQ7NV+jpb mi202xSwIqnFclYkdBwWkK0svQHNKBhiDhGSH3t4qIDhlGYE5O/hGWpcnQfbB+Odvi3U huVWkfpeVppVT8mQMo6RYEVgThCx3FkOpSxKrKYr7JsaS+tiSlBu7gZiSqpPm4gXRHA2 f0Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=C1w486I9F+xsgSCip5HP2aR4mqxScD4feOYx2dJJj8M=; b=YSedCZBCqjKuMcP39HICSnAySlh8V8SZuz6+vJo15uYfrdVM8nv56RySDo8nDj8/1U VW4cgbNLT5zDX7iZqPb+/YP9kmQzb6tQdbVBrli9JdE4laa9cIZMJoMar9N99pD8+I1S TJu9Q2dpUGdmFJLJx5IwG4hGdNmLqBhfj1ijUJNy9hZ5OynFq4yW+rKdSZoCYb6pEauP OuSNwrqfvYBB1BHz8X2mm9wXIdvg01HielaD1rkP+QI48JPAS0S4G+QjdWXE/18wYVww XAAeenMgE0GoJftArea5sAD7yw3iOxPhaJLLxDbVt8wbm4QBSM75h53Y/OX0D7k4Pr24 OySQ== X-Gm-Message-State: AJcUukd/XOrFWwuGOVrnC3sP6Lt1e9K6x5gSpPmuqvF4QDP+CzlVQMk1 SJuF/xUk3wgLjBWVIo8E6V5CoJbtkpc= X-Google-Smtp-Source: ALg8bN4NdYt9IClbWKmlBh5shOy2CYi/m2CAzY9kGNkH49ep25LJfcoXwj98u+lphyorKnNYF5TnlA== X-Received: by 2002:a5d:4f10:: with SMTP id c16mr3800483wru.177.1547014419739; Tue, 08 Jan 2019 22:13:39 -0800 (PST) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id o16sm69053727wrn.11.2019.01.08.22.13.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Jan 2019 22:13:39 -0800 (PST) From: Pete Heist Message-Id: <86EB08E7-BD75-4F6C-A5CC-6996BF5B2AC3@heistp.net> Content-Type: multipart/alternative; boundary="Apple-Mail=_3F040CD2-DCCF-410A-BF23-F18F703D429A" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Date: Wed, 9 Jan 2019 07:13:37 +0100 In-Reply-To: Cc: Cake List To: Dave Taht References: <87wonjxvss.fsf@toke.dk> <0077CC34-490F-4D76-82D3-BE37B27F2E1C@heistp.net> <49A6DCF8-BE98-47F4-9C66-6B4288390A58@heistp.net> <87tvinxos7.fsf@toke.dk> <87r2drxnal.fsf@toke.dk> <45D43135-318B-48AD-B09B-69BBB034CE12@heistp.net> <87o98vxm57.fsf@toke.dk> <797FCC60-0048-4EF6-80BC-19707E9173FB@heistp.net> <87lg3zxdyr.fsf@toke.dk> <87imz2yiet.fsf@toke.dk> <252DC221-7024-4834-9757-96335372A5A7@heistp.net> <87ftu6yc2y.fsf@toke.dk> <11DD478A-E61D-4D62-92B3-30B9A9A9572E@heistp.net> <874laly07v.fsf@toke.dk> <75345B18-E171-44C8-B059-9F6A6A663CAC@heistp.net> <87sgy4wvqq.fsf@toke.dk> <4FB89A19-314E-4B45-BEF3-83DB04A90C08@heistp.net> <39B9038B-42DF-41B6-9D78-6B71DCB06B2A@heistp.net> <749130B2-8A84-4F88-AF23-385B438C04DA@heistp.net> X-Mailer: Apple Mail (2.3445.9.1) Subject: Re: [Cake] cake infinite loop(?) with hfsc on one-armed router X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jan 2019 06:13:41 -0000 --Apple-Mail=_3F040CD2-DCCF-410A-BF23-F18F703D429A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jan 8, 2019, at 11:33 PM, Dave Taht wrote: >=20 > On Tue, Jan 8, 2019 at 2:01 PM Pete Heist > wrote: >>=20 >> I should have done that: = https://www.heistp.net/downloads/htb_split_gso_patched/ = >>=20 >> Note that I changed the names in the plots to match the convention of = my first email, but it should be clear which is which and I left all = plots in. The text output is there too as I sometimes like to open = several up in different browser tabs and switch between tabs to compare = values. >>=20 >> It looks like about 100 usec to me. Throughput also looks = consistently about 0.3 Mbit higher (~1.3%) in the split results. >=20 > My guess is with ecn on would have the highest latency and the same > throughput. ? I see a slight increase in icmp/udp rtt (1.04ms-1.02ms =3D 20us) and a = slight increase in throughput (180.12Mbit - 179.50Mbit =3D 620kbit). https://www.heistp.net/downloads/htb_cakep2_ecnon/ https://www.heistp.net/downloads/htb_cakep2_ecnon2/ These results are not five sigma. :) > Since we started this effort in an era when seconds of added latency > was common, a mere 100us improvement seems insignificant, except that > that's a 10% improvement over the present-day baseline, and *that's > worth it*. ;) This is also a function of the number of flows, kernel Who knows how many timeouts are avoided in the future, just because of = this 100us. :) > I have noticed that BQL's values can get quite large with cake doing > the shaping, btw, much larger than they do with htb. Wonder why that is. So far it=E2=80=99s harder to use cake=E2=80=99s shaper in the current = setup I=E2=80=99m working on, because it appears =E2=80=9Cbetter" to = have eth0 and eth0.3300 under one link sharing hierarchy for eth0: tc qdisc add dev eth0 root handle 1: htb default 1 tc class add dev eth0 parent 1: classid 1:1 htb rate $RATE tc class add dev eth0 parent 1: classid 1:2 htb rate $RATE tc qdisc add dev eth0 parent 1:1 cake besteffort tc qdisc add dev eth0 parent 1:2 cake besteffort tc filter add dev eth0 parent 1:0 prio 1 protocol all \ basic match not "meta(vlan mask 0xfff gt 0x0)" flowid 1:1 tc filter add dev eth0 parent 1:0 prio 2 protocol all \ basic match "meta(vlan mask 0xfff eq `printf \"0x%x\" = $VLAN_TAG`)" flowid 1:2 but I can=E2=80=99t do that with cake=E2=80=99s shaper. It=E2=80=99s = better in the sense that when you=E2=80=99re out of CPU, latency = doesn=E2=80=99t increase suddenly. I _can_ use cake=E2=80=99s shaper by = adding cake to the VLAN interface and filtering out vlan traffic from = the main interface: tc qdisc add dev eth0 root handle 1: prio bands 2 priomap 1 1 1 1 1 1 1 = 1 1 1 1 1 1 1 1 1 tc qdisc add dev eth0 parent 1:1 handle 10: cake besteffort bandwidth = $RATE tc filter add dev eth0 parent 1:0 prio 1 protocol all \ basic match not "meta(vlan mask 0xfff gt 0x0)" flowid 10:1 tc qdisc add dev eth0.3300 root cake besteffort bandwidth $RATE but when you=E2=80=99re out of CPU, you get starvation and inter-flow = latency increases. So so far, hfsc or htb is working better in this = case. It might be something to think about for an =E2=80=9CISP = Cake=E2=80=9D=E2=80=A6 --Apple-Mail=_3F040CD2-DCCF-410A-BF23-F18F703D429A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On = Jan 8, 2019, at 11:33 PM, Dave Taht <dave.taht@gmail.com>= wrote:

On Tue, Jan 8, 2019 at 2:01 PM = Pete Heist <pete@heistp.net> wrote:

I should have done = that: https://www.heistp.net/downloads/htb_split_gso_patched/

Note that I changed the names in the plots to = match the convention of my first email, but it should be clear which is = which and I left all plots in. The text output is there too as I = sometimes like to open several up in different browser tabs and switch = between tabs to compare values.

It looks = like about 100 usec to me. Throughput also looks consistently about 0.3 = Mbit higher (~1.3%) in the split results.

My guess is with ecn on would = have the highest latency and the same
throughput. ?

I see a slight increase in icmp/udp rtt = (1.04ms-1.02ms =3D 20us) and a slight increase in throughput (180.12Mbit = - 179.50Mbit =3D 620kbit).

https://www.heistp.net/downloads/htb_cakep2_ecnon2/

These results are not five sigma. = :)

Since we started this effort in an era when seconds of added = latency
was common, a = mere 100us improvement seems insignificant, except that
that's a 10% improvement over = the present-day baseline, and *that's
worth it*. ;) This is also a function of the number of flows, = kernel

Who = knows how many timeouts are avoided in the future, just because of this = 100us. :)

I have = noticed that BQL's values can get quite large with cake doing
the shaping, btw, much larger = than they do with htb.

Wonder why that is.

So far it=E2=80=99s harder to use = cake=E2=80=99s shaper in the current setup I=E2=80=99m working on, = because it appears =E2=80=9Cbetter" to have eth0 and eth0.3300 under one = link sharing hierarchy for eth0:

tc qdisc add dev eth0 root handle 1: = htb default 1
tc class add dev eth0 parent 1: = classid 1:1 htb rate $RATE
tc class add = dev eth0 parent 1: classid 1:2 htb rate $RATE
tc = qdisc add dev eth0 parent 1:1 cake besteffort
tc = qdisc add dev eth0 parent 1:2 cake besteffort
tc = filter add dev eth0 parent 1:0 prio 1 protocol all \
= basic match not "meta(vlan mask 0xfff gt 0x0)" flowid 1:1
tc filter add dev eth0 parent 1:0 prio 2 protocol = all \
basic match "meta(vlan mask 0xfff = eq `printf \"0x%x\" $VLAN_TAG`)" flowid 1:2

but I can=E2=80=99t do that with = cake=E2=80=99s shaper. It=E2=80=99s better in the sense that when = you=E2=80=99re out of CPU, latency doesn=E2=80=99t increase suddenly. I = _can_ use cake=E2=80=99s shaper by adding cake to the VLAN interface and = filtering out vlan traffic from the main interface:

tc qdisc add = dev eth0 root handle 1: prio bands 2 priomap 1 1 1 1 1 1 1 1 1 = 1 1 1 1 1 1 1
tc qdisc add dev eth0 parent 1:1 = handle 10: cake besteffort bandwidth $RATE
tc filter add = dev eth0 parent 1:0 prio 1 protocol all \
basic = match not "meta(vlan mask 0xfff gt 0x0)" flowid 10:1
tc = qdisc add dev eth0.3300 root cake besteffort bandwidth $RATE

but when you=E2=80=99re = out of CPU, you get starvation and inter-flow latency increases. So so = far, hfsc or htb is working better in this case. It might be something = to think about for an =E2=80=9CISP Cake=E2=80=9D=E2=80=A6

= --Apple-Mail=_3F040CD2-DCCF-410A-BF23-F18F703D429A--