From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com [IPv6:2a00:1450:400c:c09::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 425273B2A3 for ; Sun, 12 Feb 2017 11:37:46 -0500 (EST) Received: by mail-wm0-x22f.google.com with SMTP id 196so1511541wmm.1 for ; Sun, 12 Feb 2017 08:37:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=Wwnp+AG3RxMsHOThJMfPt3U9f9NBJ69UVUCnc5A+6Cw=; b=p6ts47jbzg1DZ7Qy2NsdgwyRoWTsoltquqyHlnrQdNZiONapjFyyE1R0NCQ+BJM9Eo 4CP1GAsQ3zH4p5qnFG/0zUGm5DmzQF1yhP6UfavNa2ziro5F310dyb2DGpGmcpUJdKkN 12NWBTxucZ6bjknomW/tq9fjwsmEBOjB3duhFYx5Xdlj4o6g3b/jXLZsMQIudUH7Vxal pO9saZ0eZsQCCZVaa4IUJVd58UygWIoaEb+SWyRvH88dwG7ECeIxu+C59V6u8QVJtZsX PkaVqWx/iC2eWL98zEoBc4i8M3Tu+XTHjmiSpoJd7QkJYzdQkXZE7vVN18gtXcgmqKB9 qVrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=Wwnp+AG3RxMsHOThJMfPt3U9f9NBJ69UVUCnc5A+6Cw=; b=fJXh2rpneO/DpKFvvfpOM/3h0oLYsGQ4yyw2+O97yMuLjFrr2YdDkqodqlS+dEGOKB 6Yx9DHsS0prewidIMToHQqjS9pDrKUHVEKTGVlKzoaiRSGwPzZBRwQdohd1wK/DqAmjY bmDOZmctjZei/sfFW0FupGz4UjP+MdL3xIeEiibA7VGAq9g/TSwo2FE/It//P5D7zIck sH3eKsVCGm8EQVmq7ulZlHeyiSbq86LinCBUFjHHdIui8tOZ/Mv+r2HxVaEnV+YdAa6n zKQHW4WNqs3DwtTF15GViJ8EeGz0VLq9kfgzTH5oizZwDuPgItzQ3ediX+Xfvtzep9eZ nA1A== X-Gm-Message-State: AMke39n04R9+d6TA0Pw0hKEh4ymgF9fc0Obh4bcwIcuMYJ3ZqkulucKZ22SM2jxqgviqug== X-Received: by 10.28.207.70 with SMTP id f67mr14786180wmg.72.1486917464441; Sun, 12 Feb 2017 08:37:44 -0800 (PST) Received: from [10.72.0.34] (h-1169.lbcfree.net. [185.99.119.68]) by smtp.gmail.com with ESMTPSA id o143sm1960421wmd.3.2017.02.12.08.37.43 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 12 Feb 2017 08:37:43 -0800 (PST) Content-Type: multipart/alternative; boundary="Apple-Mail=_6AA64F30-01D8-4EFA-9A81-9DB3390A4989" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) From: Pete Heist In-Reply-To: <8305DE32-10AB-42C3-8D08-5FF3391914BF@gmail.com> Date: Sun, 12 Feb 2017 17:37:53 +0100 Cc: Jonathan Morton , Cake List Message-Id: References: <459B9F17-317F-465E-8D2F-361CF47E5F32@gmail.com> <3D9E1A43-0182-4A1F-8262-6F587A79254E@gmail.com> <830143EE-20F2-42A5-A4FC-ECE7DF50C632@gmail.com> <652AA7A2-60C5-460F-AE60-CF4CB1D1D781@gmail.com> <5BE2A225-4B9C-4F0F-ACC5-C23CCC873DF5@gmail.com> <4B18C549-4CEF-4275-B9B3-CB8A046EB4EC@gmail.com> <856BB65A-569E-4633-B104-5E3BD15B649F@gmail.com> <70520D3D-D381-44DC-A789-BB1E24FBE3F4@gmx.de> <531AF998-D1B2-43A6-A55B-F0471C0164E1@gmail.com> <967D2491-5DB5-45CC-B4C5-E8FF48743504@gmail.com> <8305DE32-10AB-42C3-8D08-5FF3391914BF@gmail.com> To: Dave Taht X-Mailer: Apple Mail (2.3124) Subject: Re: [Cake] Cake latency update X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Feb 2017 16:37:46 -0000 --Apple-Mail=_6AA64F30-01D8-4EFA-9A81-9DB3390A4989 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Or, I can do: ethtool -K eth0 tx off rx off and disable the checksums entirely. That stops the messages, but = unfortunately it doesn=E2=80=99t appear to be the end of the throughput = shifts. But this experience has made me want to look at all of this on other = hardware, so that=E2=80=99s next. > On Feb 12, 2017, at 3:12 PM, Pete Heist wrote: >=20 >> On Feb 12, 2017, at 2:08 PM, Dave Taht > wrote: >>=20 >> Disable offloads on the sky hardware and see what happens? >>=20 >> ethtool -K gro off tso off gso off your_device >=20 > I=E2=80=99d already had them disabled for testing in = /etc/network/interfaces: >=20 > post-up ethtool -K eth0 tso off gso off gro off sg off >=20 > On a whim I tried _enabling_ offloads again but it happens in both = cases. >=20 >> How old is the OS on that hardware - offloads have always been = tricksy. >=20 > Pretty new: Ubuntu 16.10 (GNU/Linux 4.8.0-37-generic x86_64) >=20 >> as to why you might be seeing it more with cake, with this stuff on, >> you are not necessarily checking every packet for checksums, and = flows >> are "finer" - more mixed up packets. >>=20 >> capturing these events with tcpdump at various points on the path = might help. >>=20 >> Still, these are the kinds of baseline deployment issues that block >> progress elsewhere. The whole first stage of the rocket has to = succeed >> in order to test the second. Doesn't matter how good your second = stage >> is, if you RUD the first. >=20 > It must be a challenge for you guys sometimes! Unless I can find an = obvious solution soon it=E2=80=99s probably going to mean a hardware = change for me. But there are only a few options I see with what=E2=80=99s = available to me now: >=20 > 1) Using my Apple USB Ethernet adapter for testing instead of just = management. Not excited about that- no BQL? USB latency? fq_codel on = this adapter over Ethernet reduces Flent RRUL average latency to a = pretty solid 1ms, looks sufficient? (Perhaps no coincidence that USB 2.0 = start-of-frame is sent every 1 ms.) >=20 > 2) Using a 1.25 GHz Mac Mini PPC G4 I have laying around. I = successfully ran fq_codel for ADSL on that box in the past, but at 5 / = 0.5 Mbps. Accurate Flent results running Cake at 80 Mbps? Timer issues? = Also I think no BQL support with the Sun GEM chipset: = https://github.com/torvalds/linux/blob/master/drivers/net/ethernet/sun/sun= gem.c = . >=20 > 3) Using two of these for my routers instead: = https://pcengines.ch/alix2d2.htm , = which I=E2=80=99ll want to test later anyway. They=E2=80=99re not new. = 500 MHz AMD Geode LX800. Pre-Obama (June 2008). Not even sure yet if = I=E2=80=99ll rate limit properly at 80-90 Mbit with these. >=20 > Any opinion on a =E2=80=98best=E2=80=99 alternative among these? I=E2=80= =99m leaning towards #1 for ease. Otherwise I=E2=80=99ll make my way, = and may have to dig up some better hardware. >=20 > Pete >=20 --Apple-Mail=_6AA64F30-01D8-4EFA-9A81-9DB3390A4989 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Or, I can do: ethtool -K eth0 tx off rx = off

and = disable the checksums entirely. That stops the messages, but = unfortunately it doesn=E2=80=99t appear to be the end of the throughput = shifts.

But = this experience has made me want to look at all of this on other = hardware, so that=E2=80=99s next.

On Feb 12, 2017, at 3:12 PM, = Pete Heist <peteheist@gmail.com> wrote:

On Feb = 12, 2017, at 2:08 PM, Dave Taht <dave.taht@gmail.com> wrote:

Disable offloads on the sky hardware and see what happens?

ethtool -K gro off tso off gso off = your_device

I=E2=80=99d already had them disabled = for testing in /etc/network/interfaces:

post-up ethtool -K eth0 tso off gso off = gro off sg off

On a whim I tried _enabling_ offloads again but it happens in = both cases.

How old is the OS on that hardware - offloads = have always been tricksy.

Pretty new: Ubuntu = 16.10 (GNU/Linux 4.8.0-37-generic x86_64)

as to why you = might be seeing it more with cake, with this stuff on,
you = are not necessarily checking every packet for checksums, and flows
are "finer" - more mixed up packets.

capturing these events with tcpdump at various points on the = path might help.

Still, these are the kinds = of baseline deployment issues that block
progress = elsewhere. The whole first stage of the rocket has to succeed
in order to test the second. Doesn't matter how good your = second stage
is, if you RUD the first.

It must be a challenge for you guys = sometimes! Unless I can find an obvious solution soon it=E2=80=99s = probably going to mean a hardware change for me. But there are only a = few options I see with what=E2=80=99s available to me now:

1) Using my Apple USB = Ethernet adapter for testing instead of just management. Not excited = about that- no BQL? USB latency? fq_codel on this adapter over Ethernet = reduces Flent RRUL average latency to a pretty solid 1ms, looks = sufficient? (Perhaps no coincidence that USB 2.0 start-of-frame is sent = every 1 ms.)

2) = Using a 1.25 GHz Mac Mini PPC G4 I have laying around. I successfully = ran fq_codel for ADSL on that box in the past, but at 5 / 0.5 Mbps. = Accurate Flent results running Cake at 80 Mbps? Timer issues? Also I = think no BQL support with the Sun GEM chipset: https://github.com/torvalds/linux/blob/master/drivers/net/ether= net/sun/sungem.c.

3) Using two of these for my routers instead: https://pcengines.ch/alix2d2.htm, which I=E2=80=99ll want = to test later anyway. They=E2=80=99re not new. 500 MHz AMD Geode LX800. = Pre-Obama (June 2008). Not even sure yet if I=E2=80=99ll rate limit = properly at 80-90 Mbit with these.

Any opinion on a =E2=80=98best=E2=80=99 = alternative among these? I=E2=80=99m leaning towards #1 for ease. = Otherwise I=E2=80=99ll make my way, and may have to dig up some better = hardware.

Pete


= --Apple-Mail=_6AA64F30-01D8-4EFA-9A81-9DB3390A4989--