From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-x22d.google.com (mail-wr0-x22d.google.com [IPv6:2a00:1450:400c:c0c::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 62F053B2A3 for ; Sun, 12 Feb 2017 09:11:54 -0500 (EST) Received: by mail-wr0-x22d.google.com with SMTP id k90so135203260wrc.3 for ; Sun, 12 Feb 2017 06:11:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=q0BxQ8eVTgPX6vNgXPhtUlHgKWoPhm0S0fSQBT1VoNo=; b=KrXUfqWp4YzJu7eTqNU9xfhrh58l5LMNIlV8aCYHMC/sGResE+/xhyrWx0azE1ZHLl y+d7lFl3YQr1UKbXQaZ/Wd+cqDW++EBBh/cSGtwmUevYkGG8TlHwPvQJ7tFvaDdt9toj KKkxgs9QY8Xjvr0p4yE5QSDbGwLX6cAUI4B5lj+ohADJG8DhFgOeBWPVgDIO0uMYjy4D oTnTct6g33X7aluLK7aE8IqnJT4HlzI8gRXiBzeuS5Ba/WA6rAbatJO6K/a6gLzt3xpY CdVo9ZrTpfb0f+xPyG0oOVkLJGNVnbWS3U8CDJQCRv7YDhqOEsO2SNqr6nPEM00TTV+0 Tj3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=q0BxQ8eVTgPX6vNgXPhtUlHgKWoPhm0S0fSQBT1VoNo=; b=NCxiLNpDaAorDpsm7DuNRUhYqFEWbxDH08Omg7atDTczVzBXk6yGdx5Vtl87txwD2K m66b3dwiep+xNIRBbnkPRuYQROWo8XI4isvcEFObtgz5OyeM2ECK/LVyPJPeFc0M0k6B H1MrR8UlTuiKV1YNSir21af0aHWLGcRdY5f3aw63lgoxZ7qvgSzoIL4P3tiRrXMzNEZz xv9RiE6CKEpVBXrSXdi8K1XSX+VUzEWBC0uAK24F2CMp2y8xgYRRSEI3N/AM6A6GunWZ KDdAtBSGnqHlrxJDvIgaS43z0FUzsOkNjLJzIBAzNag98sTeQ65OdUeXjqJbSJ2Egi1M D8jw== X-Gm-Message-State: AMke39netPwRqLpcjRC43BtnjZGeEXh1C19fBTG8yw01y9zV++06gTcGEmkJqkVoc0OdqA== X-Received: by 10.223.138.241 with SMTP id z46mr18087323wrz.30.1486908713494; Sun, 12 Feb 2017 06:11:53 -0800 (PST) Received: from [10.72.0.34] (h-1169.lbcfree.net. [185.99.119.68]) by smtp.gmail.com with ESMTPSA id y126sm1506547wmg.19.2017.02.12.06.11.52 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 12 Feb 2017 06:11:52 -0800 (PST) Content-Type: multipart/alternative; boundary="Apple-Mail=_D4A98F2D-3C55-48B2-960F-5E2979C9D184" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) From: Pete Heist In-Reply-To: Date: Sun, 12 Feb 2017 15:12:01 +0100 Cc: Jonathan Morton , Cake List Message-Id: <8305DE32-10AB-42C3-8D08-5FF3391914BF@gmail.com> References: <459B9F17-317F-465E-8D2F-361CF47E5F32@gmail.com> <3D9E1A43-0182-4A1F-8262-6F587A79254E@gmail.com> <830143EE-20F2-42A5-A4FC-ECE7DF50C632@gmail.com> <652AA7A2-60C5-460F-AE60-CF4CB1D1D781@gmail.com> <5BE2A225-4B9C-4F0F-ACC5-C23CCC873DF5@gmail.com> <4B18C549-4CEF-4275-B9B3-CB8A046EB4EC@gmail.com> <856BB65A-569E-4633-B104-5E3BD15B649F@gmail.com> <70520D3D-D381-44DC-A789-BB1E24FBE3F4@gmx.de> <531AF998-D1B2-43A6-A55B-F0471C0164E1@gmail.com> <967D2491-5DB5-45CC-B4C5-E8FF48743504@gmail.com> To: Dave Taht X-Mailer: Apple Mail (2.3124) Subject: Re: [Cake] Cake latency update X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Feb 2017 14:11:54 -0000 --Apple-Mail=_D4A98F2D-3C55-48B2-960F-5E2979C9D184 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Feb 12, 2017, at 2:08 PM, Dave Taht wrote: >=20 > Disable offloads on the sky hardware and see what happens? >=20 > ethtool -K gro off tso off gso off your_device I=E2=80=99d already had them disabled for testing in = /etc/network/interfaces: post-up ethtool -K eth0 tso off gso off gro off sg off On a whim I tried _enabling_ offloads again but it happens in both = cases. > How old is the OS on that hardware - offloads have always been = tricksy. Pretty new: Ubuntu 16.10 (GNU/Linux 4.8.0-37-generic x86_64) > as to why you might be seeing it more with cake, with this stuff on, > you are not necessarily checking every packet for checksums, and flows > are "finer" - more mixed up packets. >=20 > capturing these events with tcpdump at various points on the path = might help. >=20 > Still, these are the kinds of baseline deployment issues that block > progress elsewhere. The whole first stage of the rocket has to succeed > in order to test the second. Doesn't matter how good your second stage > is, if you RUD the first. It must be a challenge for you guys sometimes! Unless I can find an = obvious solution soon it=E2=80=99s probably going to mean a hardware = change for me. But there are only a few options I see with what=E2=80=99s = available to me now: 1) Using my Apple USB Ethernet adapter for testing instead of just = management. Not excited about that- no BQL? USB latency? fq_codel on = this adapter over Ethernet reduces Flent RRUL average latency to a = pretty solid 1ms, looks sufficient? (Perhaps no coincidence that USB 2.0 = start-of-frame is sent every 1 ms.) 2) Using a 1.25 GHz Mac Mini PPC G4 I have laying around. I successfully = ran fq_codel for ADSL on that box in the past, but at 5 / 0.5 Mbps. = Accurate Flent results running Cake at 80 Mbps? Timer issues? Also I = think no BQL support with the Sun GEM chipset: = https://github.com/torvalds/linux/blob/master/drivers/net/ethernet/sun/sun= gem.c = . 3) Using two of these for my routers instead: = https://pcengines.ch/alix2d2.htm , = which I=E2=80=99ll want to test later anyway. They=E2=80=99re not new. = 500 MHz AMD Geode LX800. Pre-Obama (June 2008). Not even sure yet if = I=E2=80=99ll rate limit properly at 80-90 Mbit with these. Any opinion on a =E2=80=98best=E2=80=99 alternative among these? I=E2=80=99= m leaning towards #1 for ease. Otherwise I=E2=80=99ll make my way, and = may have to dig up some better hardware. Pete --Apple-Mail=_D4A98F2D-3C55-48B2-960F-5E2979C9D184 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On = Feb 12, 2017, at 2:08 PM, Dave Taht <dave.taht@gmail.com>= wrote:

Disable offloads on the sky hardware and see what happens?

ethtool -K gro off tso off gso off = your_device

I=E2=80=99d already had them disabled for testing = in /etc/network/interfaces:

post-up = ethtool -K eth0 tso off gso off gro off sg off

On a whim I tried _enabling_ offloads again but it = happens in both cases.

How old is the OS on that = hardware - offloads have always been tricksy.

Pretty = new: Ubuntu 16.10 (GNU/Linux 4.8.0-37-generic x86_64)

as to why you might be seeing it more with cake, with this = stuff on,
you are not necessarily checking every packet = for checksums, and flows
are "finer" - more mixed up = packets.

capturing these events with = tcpdump at various points on the path might help.

Still, these are the kinds of baseline deployment issues that = block
progress elsewhere. The whole first stage of the = rocket has to succeed
in order to test the second. Doesn't = matter how good your second stage
is, if you RUD the = first.

It must be a challenge for you guys sometimes! = Unless I can find an obvious solution soon it=E2=80=99s probably going = to mean a hardware change for me. But there are only a few options I see = with what=E2=80=99s available to me now:

1) Using my Apple USB Ethernet adapter for testing = instead of just management. Not excited about that- no BQL? USB latency? = fq_codel on this adapter over Ethernet reduces Flent RRUL average = latency to a pretty solid 1ms, looks sufficient? (Perhaps no coincidence = that USB 2.0 start-of-frame is sent every 1 ms.)

2) Using a 1.25 GHz Mac Mini PPC G4 I have laying = around. I successfully ran fq_codel for ADSL on that box in the past, = but at 5 / 0.5 Mbps. Accurate Flent results running Cake at 80 Mbps? = Timer issues? Also I think no BQL support with the Sun GEM = chipset: https://github.com/torvalds/linux/blob/master/drivers/net/ether= net/sun/sungem.c.

3) Using two = of these for my routers instead: https://pcengines.ch/alix2d2.htm, which I=E2=80=99ll want = to test later anyway. They=E2=80=99re not new. 500 MHz AMD Geode LX800. = Pre-Obama (June 2008). Not even sure yet if I=E2=80=99ll rate limit = properly at 80-90 Mbit with these.

Any= opinion on a =E2=80=98best=E2=80=99 alternative among these? I=E2=80=99m = leaning towards #1 for ease. Otherwise I=E2=80=99ll make my way, and may = have to dig up some better hardware.

Pete

= --Apple-Mail=_D4A98F2D-3C55-48B2-960F-5E2979C9D184--