From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ot0-x22b.google.com (mail-ot0-x22b.google.com [IPv6:2607:f8b0:4003:c0f::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id BFEE23B29E for ; Mon, 13 Feb 2017 12:51:55 -0500 (EST) Received: by mail-ot0-x22b.google.com with SMTP id f9so73990172otd.1 for ; Mon, 13 Feb 2017 09:51:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=eMk09cBVGQ8cxEI8luHuzT2IB3rhhl+STiKEPrqPIYY=; b=RixJM8Gp8O1LG+e6r1YJTmBJEwfi45f4BuTrTGbMybhnW4pBAxhTqf0WNC5xWxHQsG XldM1SLfYq1hhdB6fI4skADr/IsniPhAG0jw1aZt5PqdipGArh3JVaSXJhojqu0TYn6Q DGG2ghYU7Q8kTYxAt8qwOOfxPAXaLCZ1thx+5SJt1rUDRpswnSfab2NXInqkZNXqWAoj YuBDvC0caCW5zS8Jq5QSydTjgYiZX7UGl4qacO3gQ7ad5jVIgLgApAgyt3GS6hexzF4H CN7WJulcl6ThKP5LKhmzVUmoXnsRekoODRORhFN5HMcYiHgkimdL8J2nYaqAahrLwKsl mYeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=eMk09cBVGQ8cxEI8luHuzT2IB3rhhl+STiKEPrqPIYY=; b=MEXqHRBMAHhgqj1nPKc74l4kDexsFQSbZVDgCLRS2FA1gHcLBlZ32BVFxGS1tFryBF O5PcTCR5RqLB2hS6beJ9PkXHp5gzsgGnqMs3u+p4i9wlBSWrJG0s/DbCfxAbxQBIcbWN NBUmXxkF0SAApuw7NNeFga5b3LJaOnwFaBO+u6eRpoZpX2KTnNWeeHmRbcMF/kK7oQCp 1Q/v64KhH/p7Bqa1giFUd9gU9YCZXqwrJXgKpJly/XhDCtmg3+IN990lrxrhuKFXFtBT kgbT1dL5NAN9JrqOPKQU8NI2aHKcwSDo8UuaFmz5faqyEjYFwJl7y5SKlmKrMCkRJDSa p6Cw== X-Gm-Message-State: AMke39lLS2iEyO1IG6JUGG9mNFC2K+j/+5N2gU9ytDcpzU9C0teQee7Wci1tipSowEb2EsDiGznh9Ek0jsMS1A== X-Received: by 10.157.53.42 with SMTP id o39mr14493596otc.157.1487008314928; Mon, 13 Feb 2017 09:51:54 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.0.39 with HTTP; Mon, 13 Feb 2017 09:51:54 -0800 (PST) In-Reply-To: References: From: Hans-Kristian Bakke Date: Mon, 13 Feb 2017 18:51:54 +0100 Message-ID: To: bloat Content-Type: multipart/alternative; boundary=001a11c00e4e63b5a205486d1af2 Subject: Re: [Bloat] sch_fq, pacing and inconsistent TSO continued... X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Feb 2017 17:51:55 -0000 --001a11c00e4e63b5a205486d1af2 Content-Type: text/plain; charset=UTF-8 Yep.. downgrading systemd from 232.15 to 231.10 fixed it. There you have it. Systemd 232 not only got the ability to set offloads, but it also disables tx-tcp-segmentation by default for both physical NICs and bonds. These packages from Debian Snapshots did the trick: libsystemd0_231-10_amd64.deb libudev1_231-10_amd64.deb systemd_231-10_amd64.deb udev_231-10_amd64.deb This is ...fun.... No wonder file my server performance got more unreliable the last couple of months. On 13 February 2017 at 18:39, Hans-Kristian Bakke wrote: > After some further testing i see that this is not igb specific! Also bond > interfaces and e1000e have tso disabled like this: > > tx-tcp-segmentation: off > tx-tcp-mangleid-segmentation: off [requested on] > > bonded vlan interfaces however have tso enabled, but again vlan interfaces > seems to have everything enabled. > > As downgrading the kernel to a known good did not do the trick there must > be something else systemwide in Debian Stretch. Perhaps I should really try > to get all the dependencies together to downgrade systemd. > > Regards, > Hans-Kristian > > On 13 February 2017 at 17:54, Hans-Kristian Bakke > wrote: > >> In a previous thread on this mailing list (Excessive throttling with fq) >> it was concluded that the reason for the bad performance was that tso was >> somewhat inconcistent between the bond and the physical interfaces. I never >> really knew why, but I knew I had been experimenting with traffic shaping >> and offloads so I naturally thought that was the culprit. >> >> However, I have now rebooted some of my systems and in both of my Debian >> Stretch (testing) systems (I tried both kernel 4.8 and 4.9, one with >> fq_codel and one with sch_fq) I end up with this combination of >> segmentation offload settings: >> >> ... >> tcp-segmentation-offload: on >> tx-tcp-segmentation: off <---- ??? >> tx-tcp-ecn-segmentation: off [fixed] >> tx-tcp-mangleid-segmentation: off >> tx-tcp6-segmentation: on >> udp-fragmentation-offload: off [fixed] >> generic-segmentation-offload: on >> ... >> >> If I do the following... >> ethtool -K eth0 tx-tcp-segmentation on >> >> ...I end up with what I expect and the performance returns for sch_fq >> with pacing. >> ... >> tcp-segmentation-offload: on >> tx-tcp-segmentation: on >> tx-tcp-ecn-segmentation: off [fixed] >> tx-tcp-mangleid-segmentation: off >> tx-tcp6-segmentation: on >> udp-fragmentation-offload: off [fixed] >> generic-segmentation-offload: on >> ... >> >> I have some other Debian Jessie systems also using the igb-driver and tso >> is always enabled here by default (but not the same NIC). (Kernel 3.16) >> >> I also have a Proxmox VE system on kernel 4.4 with exactly the same quad >> NIC that i suspect do not use the kernel source igb-drivers as it has a lot >> more params that also has tso enabled correctly. >> >> # Proxmox VE system (full TSO by default) >> #ethtool -i eth1 >> driver: igb >> version: 5.3.5.3 >> firmware-version: 3.19, 0x00013cbf >> bus-info: 0000:03:00.1 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: yes >> supports-register-dump: yes >> supports-priv-flags: no >> >> # Debian Stretch systems (half-enabled TSO by default) >> # ethtool -i eth2 >> driver: igb >> version: 5.4.0-k >> firmware-version: 0.0.0 >> expansion-rom-version: >> bus-info: 0000:00:14.2 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: yes >> supports-register-dump: yes >> supports-priv-flags: no >> >> The NIC that the "working" and the "non-working" system have is HP >> NC365T, which is based on intel 82580. >> The other "non-working" Debian Stretch system is a Intel Rangeley Atom >> c2758 supermicros system with a quad intel I354 network controller embedded. >> >> I tried booting the stretch system with kernel 3.16 from Debian Jessie to >> get the same drivers like my working systems but no change. Then I noticed >> that systemd got the ability to change exactly these offloads in the last >> systemd version 232 which arrived in testing in the end of 2016 so I am >> thinking that this might have something to do with this, but I have not >> found anything (mostly because I don't even really know where to look) >> >> I also tried to not having one of the interfaces in any kind of bond or >> vlan subinterface in case those were changing some stuff but no changes in >> behaviour. >> >> So to my questions: >> - Can anyone think of a reason why tx-tcp-segmentation offload is >> disabled by default (and not tcp segmentation offload in general) >> - Do you have any tips to troubleshoot this? >> - As this default combination of out-of-the-box settings is bad for fq >> with pacing performance, and it seems to somewhat be the default now for >> two different igb-systems I thought I should give you a heads up. >> >> Regards, >> Hans-Kristian >> > > --001a11c00e4e63b5a205486d1af2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Yep.. downgrading systemd from 232.15 to 231.10 fixed it.
=

<= /div>
= There you have it. Systemd 232 not only got the ability to set offloads, bu= t it also disables tx-tcp-segmentation by default for both physical NICs an= d bonds.

These packages from Debian Snapshots did the trick:
libsystemd0_231-10_amd64.deb
libudev1_231-10_amd64.deb
systemd= _231-10_amd64.deb
udev_231-10_amd64.deb

This is ...fun.... No wonder file my server perform= ance got more unreliable the last couple of months.


On 13 Fe= bruary 2017 at 18:39, Hans-Kristian Bakke <hkbakke@gmail.com> wrote:
After some further t= esting i see that this is not igb specific! Also bond interfaces and e1000e= have tso disabled like this:

tx-tcp-segmentation: off
tx-tcp-mangleid-= segmentation: off [requested on]
<= font face=3D"verdana, sans-serif">
bonded vlan interfaces however have= tso enabled, but again vlan interfaces seems to have everything enabled.
=
As downgrading the kernel to a known good did not do the trick there = must be something else systemwide in Debian Stretch. Perhaps I should reall= y try to get all the dependencies together to downgrade systemd.

Reg= ards,
Hans-Kristian

On 13 Februar= y 2017 at 17:54, Hans-Kristian Bakke <hkbakke@gmail.com> wro= te:
In a previous thread on t= his mailing list (Excessive throttling with fq) it was concluded that the r= eason for the bad performance was that tso was somewhat inconcistent betwee= n the bond and the physical interfaces. I never really knew why, but I knew= I had been experimenting with traffic shaping and offloads so I naturally = thought that was the culprit.

However, I have now rebooted some of my= systems and in both of my Debian Stretch (testing) systems (I tried both k= ernel 4.8 and 4.9, one with fq_codel and one with sch_fq) I end up with thi= s combination of segmentation offload settings:

...
tcp-segmentation-offload: on
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-= segmentation: off <---- ???
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-ecn-segme= ntation: off [fixed]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-mangleid-segmentati= on: off
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp6-segmentation: on
udp-fragmentat= ion-offload: off [fixed]
generic-segmentation-offload: on
...

<= div class=3D"gmail_default" style=3D"font-family:verdana,sans-serif">If I d= o the following...
ethtool= -K eth0 tx-tcp-segmentation on=C2=A0

...I end up with what I expect and the performance returns for sch_fq with= pacing.
...
=
tcp-segmentation-offload: on
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-= tcp-segmentation: on
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-ecn-segme= ntation: off [fixed]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-mangleid-= segmentation: off
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp6-segmentatio= n: on
udp-fragmentation-offload: off [fixed]
generic-se= gmentation-offload: on
...

I have some o= ther Debian Jessie systems also using the igb-driver and tso is always enab= led here by default (but not the same NIC). (Kernel 3.16)
I also have a Proxmox VE system on kernel 4.4 with exactly the = same quad NIC that i suspect do not use the kernel source igb-drivers as it= has a lot more params that also has tso enabled correctly.
=
# Proxmox VE system (full TSO by default)
#et= htool -i eth1
driver: igb
version: 5.3.5.3
fi= rmware-version: 3.19, 0x00013cbf
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-e= eprom-access: yes
supports-register-dump: yes
supports-= priv-flags: no

# Debian Stretch system= s (half-enabled TSO by default)
# ethtool -i eth2
= driver: igb
version: 5.4.0-k
firmware-version: 0.0.0
expansion-rom-version:
bus-info: 0000:00:14.2
s= upports-statistics: yes
supports-test: yes
supports-eep= rom-access: yes
supports-register-dump: yes
supports-pr= iv-flags: no

The NIC that the "working&= quot; and the "non-working" system have is=C2=A0HP NC365T, which = is based on intel 82580.=C2=A0
The other "non-working&qu= ot; Debian Stretch system is a Intel Rangeley Atom c2758 supermicros system= with a quad intel I354 network controller embedded.

I tried booting the stretch system with kernel 3.16 from Debian Jessie t= o get the same drivers like my working systems but no change. Then I notice= d that systemd got the ability to change exactly these offloads in the last= systemd version 232 which arrived in testing in the end of 2016 so I am th= inking that this might have something to do with this, but I have not found= anything (mostly because I don't even really know where to look)

I also tried to not having one of the interfaces in= any kind of bond or vlan subinterface in case those were changing some stu= ff but no changes in behaviour.

So to my questions= :
- Can anyone think of a reason why tx-tcp-segmentation offload = is disabled by default (and not tcp segmentation offload in general)
<= div>- Do you have any tips to troubleshoot this?
- As this defaul= t combination of out-of-the-box settings is bad for fq with pacing performa= nce, and it seems to somewhat be the default now for two different igb-syst= ems I thought I should give you a heads up.

Regard= s,
Hans-Kristian


--001a11c00e4e63b5a205486d1af2--