From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ot0-x232.google.com (mail-ot0-x232.google.com [IPv6:2607:f8b0:4003:c0f::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id B0BF33B29E for ; Mon, 13 Feb 2017 11:54:35 -0500 (EST) Received: by mail-ot0-x232.google.com with SMTP id f9so72370725otd.1 for ; Mon, 13 Feb 2017 08:54:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=PKd6IJcWChzu1fo94x93BHlDwrT9WyK5/bXsQyum/Ik=; b=IjbDxileRew/400/lMBc9+Qk8wneGyerrO5e4D7IZkqWKnNc2yUByULtAMjCwgzEH4 mPurne2p8Q8DW0Tmzk77PgHGQFW0zfdgti0y2Dxo7iIpiFIrkXFLMuP2HrftAQ4ijlYX k2NSydVNKI+MMkkB1aSx+4lbl/N5faIHi0Of7XDOiLMFwiO+vv3BCY/x6dUlv6je2oIh DOKqi5KJpkYndjrhtENPOsQvqo26BW3BC/BIT7+VJ0CygWHdsjcmz+a5pOVJDGIuj2Vs 6t6qnA07nGXqcKasWsGimK1Pz2pLGLJa5Jf5QQO6QNqqUm3ym7hJ7uBnwcxp/jiIuWNx Y9KQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=PKd6IJcWChzu1fo94x93BHlDwrT9WyK5/bXsQyum/Ik=; b=JuGtHnWiaIx4HKEY2DOWK1iolPPstrWtSLpG4W0dAvtfeZ/AP5fI6IolkrTd7oyVUa rScWPPrAXb7Lv8DWp3Gp6/oFtGzLip2fPyVsQkUaklBpNIBv67yn+YmkQsbmbjR9kSq3 RxQjZfWHz7jgxkMx4Ce5gqKBbpAK1nBKMOcbdDi749BSb8WoWUSNWof7DAeqnwtFyo+S b/mR9wqxLmB2fEEtC0SQyXfp/yFGez5C02ds4lsPJ1Z1WFmN9xx74KJv0AIqu7Um90pM 5mfpfl0C5rUic0n4NWbkpCIiw4DuP5FGFfIUbSmLLP8GbXOA7RB4jL0YYM6BBe7CzVd7 qihA== X-Gm-Message-State: AMke39lJDqFEvfGKBkrJNn+rfvdMOpPe/a+Cq/OJxLMsscgxj+gug2lcbNXoOsBQ/8PxsUW7/YesA/2hUpLPdQ== X-Received: by 10.157.18.246 with SMTP id g109mr14422866otg.10.1487004874858; Mon, 13 Feb 2017 08:54:34 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.0.39 with HTTP; Mon, 13 Feb 2017 08:54:34 -0800 (PST) From: Hans-Kristian Bakke Date: Mon, 13 Feb 2017 17:54:34 +0100 Message-ID: To: bloat Content-Type: multipart/alternative; boundary=94eb2c05539c58688605486c4da2 Subject: [Bloat] sch_fq, pacing and inconsistent TSO continued... X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Feb 2017 16:54:35 -0000 --94eb2c05539c58688605486c4da2 Content-Type: text/plain; charset=UTF-8 In a previous thread on this mailing list (Excessive throttling with fq) it was concluded that the reason for the bad performance was that tso was somewhat inconcistent between the bond and the physical interfaces. I never really knew why, but I knew I had been experimenting with traffic shaping and offloads so I naturally thought that was the culprit. However, I have now rebooted some of my systems and in both of my Debian Stretch (testing) systems (I tried both kernel 4.8 and 4.9, one with fq_codel and one with sch_fq) I end up with this combination of segmentation offload settings: ... tcp-segmentation-offload: on tx-tcp-segmentation: off <---- ??? tx-tcp-ecn-segmentation: off [fixed] tx-tcp-mangleid-segmentation: off tx-tcp6-segmentation: on udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on ... If I do the following... ethtool -K eth0 tx-tcp-segmentation on ...I end up with what I expect and the performance returns for sch_fq with pacing. ... tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: off [fixed] tx-tcp-mangleid-segmentation: off tx-tcp6-segmentation: on udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on ... I have some other Debian Jessie systems also using the igb-driver and tso is always enabled here by default (but not the same NIC). (Kernel 3.16) I also have a Proxmox VE system on kernel 4.4 with exactly the same quad NIC that i suspect do not use the kernel source igb-drivers as it has a lot more params that also has tso enabled correctly. # Proxmox VE system (full TSO by default) #ethtool -i eth1 driver: igb version: 5.3.5.3 firmware-version: 3.19, 0x00013cbf bus-info: 0000:03:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no # Debian Stretch systems (half-enabled TSO by default) # ethtool -i eth2 driver: igb version: 5.4.0-k firmware-version: 0.0.0 expansion-rom-version: bus-info: 0000:00:14.2 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no The NIC that the "working" and the "non-working" system have is HP NC365T, which is based on intel 82580. The other "non-working" Debian Stretch system is a Intel Rangeley Atom c2758 supermicros system with a quad intel I354 network controller embedded. I tried booting the stretch system with kernel 3.16 from Debian Jessie to get the same drivers like my working systems but no change. Then I noticed that systemd got the ability to change exactly these offloads in the last systemd version 232 which arrived in testing in the end of 2016 so I am thinking that this might have something to do with this, but I have not found anything (mostly because I don't even really know where to look) I also tried to not having one of the interfaces in any kind of bond or vlan subinterface in case those were changing some stuff but no changes in behaviour. So to my questions: - Can anyone think of a reason why tx-tcp-segmentation offload is disabled by default (and not tcp segmentation offload in general) - Do you have any tips to troubleshoot this? - As this default combination of out-of-the-box settings is bad for fq with pacing performance, and it seems to somewhat be the default now for two different igb-systems I thought I should give you a heads up. Regards, Hans-Kristian --94eb2c05539c58688605486c4da2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
In a previous thread on this mailing list (Excessive throttling= with fq) it was concluded that the reason for the bad performance was that= tso was somewhat inconcistent between the bond and the physical interfaces= . I never really knew why, but I knew I had been experimenting with traffic= shaping and offloads so I naturally thought that was the culprit.

How= ever, I have now rebooted some of my systems and in both of my Debian Stret= ch (testing) systems (I tried both kernel 4.8 and 4.9, one with fq_codel an= d one with sch_fq) I end up with this combination of segmentation offload s= ettings:

...
tcp-segmentation-offload: on=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-segmentation: off <---- ???
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 tx-tcp-ecn-segmentation: off [fixed]
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 tx-tcp-mangleid-segmentation: off
=C2=A0 =C2=A0 =C2=A0 =C2=A0= tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-se= gmentation-offload: on
...

If I do the following...
ethtool -K eth0 tx-tcp-segmentation on=C2=A0=

<= div>...I end up with what I expect and t= he performance returns for sch_fq with pacing.
...
tcp-segmentation-offload: on
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-segmentation: on
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-ecn-segmentation: off [fixed]
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-mangleid-segmentation: off
=C2=A0= =C2=A0 =C2=A0 =C2=A0 tx-tcp6-segmentation: on
udp-fragmentation-= offload: off [fixed]
generic-segmentation-offload: on
.= ..

I have some other Debian Jessie systems also us= ing the igb-driver and tso is always enabled here by default (but not the s= ame NIC). (Kernel 3.16)

I also have a Proxmox = VE system on kernel 4.4 with exactly the same quad NIC that i suspect do no= t use the kernel source igb-drivers as it has a lot more params that also h= as tso enabled correctly.

# Proxmox VE system= (full TSO by default)
#ethtool -i eth1
driver: ig= b
version: 5.3.5.3
firmware-version: 3.19, 0x00013cbf
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
suppor= ts-register-dump: yes
supports-priv-flags: no

# Debian Stretch systems (half-enabled TSO by default)
# ethtool -i eth2
driver: igb
version: 5.4= .0-k
firmware-version: 0.0.0
expansion-rom-version:
bus-info: 0000:00:14.2
supports-statistics: yes
= supports-test: yes
supports-eeprom-access: yes
supports= -register-dump: yes
supports-priv-flags: no

<= /div>
The NIC that the "working" and the "non-working&qu= ot; system have is=C2=A0HP NC365T, which is based on intel 82580.=C2=A0
=
The other "non-working" Debian Stretch system is a Int= el Rangeley Atom c2758 supermicros system with a quad intel I354 network co= ntroller embedded.

I tried booting the stretch sys= tem with kernel 3.16 from Debian Jessie to get the same drivers like my wor= king systems but no change. Then I noticed that systemd got the ability to = change exactly these offloads in the last systemd version 232 which arrived= in testing in the end of 2016 so I am thinking that this might have someth= ing to do with this, but I have not found anything (mostly because I don= 9;t even really know where to look)

I also tri= ed to not having one of the interfaces in any kind of bond or vlan subinter= face in case those were changing some stuff but no changes in behaviour.

So to my questions:
- Can anyone think of = a reason why tx-tcp-segmentation offload is disabled by default (and not tc= p segmentation offload in general)
- Do you have any tips to trou= bleshoot this?
- As this default combination of out-of-the-box se= ttings is bad for fq with pacing performance, and it seems to somewhat be t= he default now for two different igb-systems I thought I should give you a = heads up.

Regards,
Hans-Kristian
--94eb2c05539c58688605486c4da2--