From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 607743B29E for ; Fri, 28 Dec 2018 17:42:13 -0500 (EST) Received: by mail-wm1-x335.google.com with SMTP id n190so19887511wmd.0 for ; Fri, 28 Dec 2018 14:42:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=LCLsVBvsKZvmDV9ZJzxYwdaxnPuAph1s6LBrdBBPbR8=; b=Z9Z1ppLvdA9Hdrh7WakLTPdy8F6vTBsHuKCP+A+yCrHeEhx4r7Xufj7/qa1x6RFTZq 83nm0An90oFCIdQh3G77tvCFj9HIT6F56RoFOaXpeX1MzsDTit9axPQSk+Z0bA4/r+y0 7p5PhoZLSlY+jQBMVrGDVVyUO4C2yx/QRp8OmY5HGspQIs1IyF5JRVsLdKOIcByxZHiX Lc+Iz4Evvpar4z2xlPQLZsxc0WBU/3ebMQVysZp302TRi+3egnfyf7I/5kOEX7t9DeVX ZamsgsdxiLuIdNFiUmhuzX0XO2bbsqbFB8c173XCwrTuhOOWYANQiP0mKiCK9vEeDpxJ 4PAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=LCLsVBvsKZvmDV9ZJzxYwdaxnPuAph1s6LBrdBBPbR8=; b=c5ys+AkAgYAXl2dDDUeYhfx7nTr5mBFMH9gGoJ971qKhoaNoDR2w1Bz1p3Qfmd75JM 5sepiws5HkUlitzy3K0fpox28FXcWNN6trs1erikyfuqzAn24ljWmQvr4khUR68rPulU 4NBytPUFmqJUNnB4Kv3rz7n9e83GtIsrCpRKntXsFarMDV4XbzWK4XcaaFu9FP9dvrsu r4XJZYTMoMQoyZEOW7atjaY2YVed9HVT6NJGHBtlUviiNSJ6lFIWqjx2Tk1zvnP0hn7n L6nW/ETCIENguqQDuhm0jiHyRM9f5MCrkehyrtDs4UfWxwFV/cQKsCW8RlRg6A5hPu+l vM1w== X-Gm-Message-State: AJcUukeJnoRon+6fwUHiYYZOplLMPaj3dClqZTqa4RAlRRG4rKT1USo3 M+hOUI8Z2RDl38JpeWCr8QkNog== X-Google-Smtp-Source: AFSGD/UTqm8/7aYsTJrGzXWTvu5Ntaxb5exDxGRXoEbrVgz5QTJYggzhOaqbkQhY1ATmRzgfWGEX5Q== X-Received: by 2002:a1c:1593:: with SMTP id 141mr24763177wmv.85.1546036932494; Fri, 28 Dec 2018 14:42:12 -0800 (PST) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id k135sm52048107wmd.42.2018.12.28.14.42.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 28 Dec 2018 14:42:11 -0800 (PST) From: Pete Heist Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_E812F87B-66F7-4764-B051-E0161DC75CA7" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Date: Fri, 28 Dec 2018 23:42:10 +0100 In-Reply-To: Cc: Cake List To: Jonathan Morton References: <5482A3CA-9C36-4DDE-A858-24D8467F70C7@heistp.net> X-Mailer: Apple Mail (2.3445.9.1) Subject: Re: [Cake] cake infinite loop(?) with hfsc on one-armed router X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 22:42:13 -0000 --Apple-Mail=_E812F87B-66F7-4764-B051-E0161DC75CA7 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Dec 28, 2018, at 11:07 PM, Jonathan Morton = wrote: >=20 >> On 28 Dec, 2018, at 11:22 pm, Pete Heist wrote: >>=20 >> This smells of a bug Toke fixed on Sep 12, 2018 in = 42e87f12ea5c390bf5eeb658c942bc810046160a, but then reverted in the next = commit because it was fixed upstream. However, if I re-apply that = commit, it still doesn=E2=80=99t fix it. >>=20 >> Perhaps there are more cases where skb_reset_mac_len(skb) needs to be = called somewhere for VLAN support? >=20 > I did wonder if there was something about the old kernel messing = things up. >=20 > Do you still get acceptable throughput if you disable GRO/GSO at the = interfaces? Turning off GRO does avoid the problem, but unlimited throughput through = the one-armed router gets slashed: GRO on: 920mbit up / 935mbit down GRO off: 510mbit / 534mbit down Incidentally, for those using the v1 PCEngines APU, turning off = rx-vlan-offload provides much better throughput for traffic routed from = VLANs (presumably a quirk of the Realtek r8169 driver): ethtool -K eth0 rxvlan on: 494mbit ethtool -K eth0 rxvlan off: 936mbit It would be great if there=E2=80=99s a way to work around this in cake = on older kernels, but I=E2=80=99m not holding my breath. It=E2=80=99s = also very possible that the current fix in newer kernels takes care of = this case, even if the original workaround in cake does not. This was = the kernel change: https://patchwork.ozlabs.org/patch/968734/ = So far, I=E2=80=99ll just use prio for this at lower throughputs and htb = or drr if the CPU is taxed. This works well enough on 3.16, all things = considered=E2=80=A6 --Apple-Mail=_E812F87B-66F7-4764-B051-E0161DC75CA7 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On = Dec 28, 2018, at 11:07 PM, Jonathan Morton <chromatix99@gmail.com> wrote:

On 28 Dec, 2018, at = 11:22 pm, Pete Heist <pete@heistp.net> wrote:

This smells of a bug Toke fixed on Sep 12, 2018 in = 42e87f12ea5c390bf5eeb658c942bc810046160a, but then reverted in the next = commit because it was fixed upstream. However, if I re-apply that = commit, it still doesn=E2=80=99t fix it.

Perhaps there are more cases where skb_reset_mac_len(skb) = needs to be called somewhere for VLAN support?

I did wonder if there was = something about the old kernel messing things up.

Do you still get acceptable throughput if you disable GRO/GSO = at the interfaces?

Turning off GRO does avoid the problem, but = unlimited throughput through the one-armed router gets = slashed:

GRO = on: 920mbit up / 935mbit down
GRO off: 510mbit / = 534mbit down

Incidentally, for those using the v1 PCEngines APU, turning = off rx-vlan-offload provides much better throughput for traffic = routed from VLANs (presumably a quirk of the Realtek r8169 = driver):

ethtool= -K eth0 rxvlan on: 494mbit
ethtool -K eth0 rxvlan = off: 936mbit

It = would be great if there=E2=80=99s a way to work around this in cake on = older kernels, but I=E2=80=99m not holding my breath. It=E2=80=99s also = very possible that the current fix in newer kernels takes care of this = case, even if the original workaround in cake does not. This was the = kernel change:

https://patchwork.ozlabs.org/patch/968734/

So far, I=E2=80=99ll = just use prio for this at lower throughputs and htb or drr if the CPU is = taxed. This works well enough on 3.16, all things = considered=E2=80=A6

= --Apple-Mail=_E812F87B-66F7-4764-B051-E0161DC75CA7--