From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-x231.google.com (mail-oi0-x231.google.com [IPv6:2607:f8b0:4003:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 5EB833B29E for ; Mon, 13 Feb 2017 12:39:17 -0500 (EST) Received: by mail-oi0-x231.google.com with SMTP id j15so54767119oih.2 for ; Mon, 13 Feb 2017 09:39:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=5OFirR8uTjrTzgLDMisLxkpA+nMYRI5l/Rtf9V2Ra5U=; b=fHFsBRdNIac3KiSZxOkv70pn5/7HoBkmZYrV7ZKKsmoX9/BTWMwR8q+h1aujPXO2W+ txVyLtZ0LQVCNcoNmcOhuV9FJwL4Suugobasp3VcYlDSsB+n+xZz+UDdwtBvjwjIQjpD bnC47b3XPYy+qZafjIJ+UDrfhx1T42rCxrW8YrLkxEW370JBKIqhILGd/XIj6s+F2h69 Sr7bfdBb9m53XM4xwRM9MOhVRrVfdC3SIP1tyGoE7PxCKLcH5J9y0c5ZVcPfPs+vKVqN sxOc6j9/Kq1o+mJei21The9wYdZLfrNvb+NYSohMxuI3f8sCNgEOnX5xq02cBzqjH0d7 XjIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=5OFirR8uTjrTzgLDMisLxkpA+nMYRI5l/Rtf9V2Ra5U=; b=XprK3D7lHsTbzBBKDSbM534izqbkQGZOzA4NW/SYIxUWTnH3hCGBbCe82E2LucRXfE x7mAwwPvBpKFC3hjRvggtqpSp6W7rcj4mxzZ1okXKVjchTcougty5NhezK5/xvkQa58i JowjVXxkD7v/hR1h9rexezDHhu10iWnUkoGKs3SYLPANxlWFAPpPsplSCdA/wqL1CS6p 5vEK31Fi6/hkHRwFvdCxqv8jH1WTiOGHH3eThWwa3E4tyqL6eRsvmkL4FkLbgxlNyvn2 K+a1nXdiwzS47i9ImcfzmFE0yU2mXGdvB4VA9icKs3NiI4/2QyM27lStTNT5BpfBwnfx ET3A== X-Gm-Message-State: AMke39l8DNSrkA7BcachkLcLl85KgI5iIbzzyzD9uw6KRrogKX92QHxEM+w+WQo99+obdgNLCtNxp2Ycbodccg== X-Received: by 10.202.84.143 with SMTP id i137mr13153656oib.202.1487007556446; Mon, 13 Feb 2017 09:39:16 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.0.39 with HTTP; Mon, 13 Feb 2017 09:39:16 -0800 (PST) In-Reply-To: References: From: Hans-Kristian Bakke Date: Mon, 13 Feb 2017 18:39:16 +0100 Message-ID: To: bloat Content-Type: multipart/alternative; boundary=001a113debb82e337a05486ced27 Subject: Re: [Bloat] sch_fq, pacing and inconsistent TSO continued... X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Feb 2017 17:39:17 -0000 --001a113debb82e337a05486ced27 Content-Type: text/plain; charset=UTF-8 After some further testing i see that this is not igb specific! Also bond interfaces and e1000e have tso disabled like this: tx-tcp-segmentation: off tx-tcp-mangleid-segmentation: off [requested on] bonded vlan interfaces however have tso enabled, but again vlan interfaces seems to have everything enabled. As downgrading the kernel to a known good did not do the trick there must be something else systemwide in Debian Stretch. Perhaps I should really try to get all the dependencies together to downgrade systemd. Regards, Hans-Kristian On 13 February 2017 at 17:54, Hans-Kristian Bakke wrote: > In a previous thread on this mailing list (Excessive throttling with fq) > it was concluded that the reason for the bad performance was that tso was > somewhat inconcistent between the bond and the physical interfaces. I never > really knew why, but I knew I had been experimenting with traffic shaping > and offloads so I naturally thought that was the culprit. > > However, I have now rebooted some of my systems and in both of my Debian > Stretch (testing) systems (I tried both kernel 4.8 and 4.9, one with > fq_codel and one with sch_fq) I end up with this combination of > segmentation offload settings: > > ... > tcp-segmentation-offload: on > tx-tcp-segmentation: off <---- ??? > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp-mangleid-segmentation: off > tx-tcp6-segmentation: on > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: on > ... > > If I do the following... > ethtool -K eth0 tx-tcp-segmentation on > > ...I end up with what I expect and the performance returns for sch_fq with > pacing. > ... > tcp-segmentation-offload: on > tx-tcp-segmentation: on > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp-mangleid-segmentation: off > tx-tcp6-segmentation: on > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: on > ... > > I have some other Debian Jessie systems also using the igb-driver and tso > is always enabled here by default (but not the same NIC). (Kernel 3.16) > > I also have a Proxmox VE system on kernel 4.4 with exactly the same quad > NIC that i suspect do not use the kernel source igb-drivers as it has a lot > more params that also has tso enabled correctly. > > # Proxmox VE system (full TSO by default) > #ethtool -i eth1 > driver: igb > version: 5.3.5.3 > firmware-version: 3.19, 0x00013cbf > bus-info: 0000:03:00.1 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: no > > # Debian Stretch systems (half-enabled TSO by default) > # ethtool -i eth2 > driver: igb > version: 5.4.0-k > firmware-version: 0.0.0 > expansion-rom-version: > bus-info: 0000:00:14.2 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: no > > The NIC that the "working" and the "non-working" system have is HP NC365T, > which is based on intel 82580. > The other "non-working" Debian Stretch system is a Intel Rangeley Atom > c2758 supermicros system with a quad intel I354 network controller embedded. > > I tried booting the stretch system with kernel 3.16 from Debian Jessie to > get the same drivers like my working systems but no change. Then I noticed > that systemd got the ability to change exactly these offloads in the last > systemd version 232 which arrived in testing in the end of 2016 so I am > thinking that this might have something to do with this, but I have not > found anything (mostly because I don't even really know where to look) > > I also tried to not having one of the interfaces in any kind of bond or > vlan subinterface in case those were changing some stuff but no changes in > behaviour. > > So to my questions: > - Can anyone think of a reason why tx-tcp-segmentation offload is disabled > by default (and not tcp segmentation offload in general) > - Do you have any tips to troubleshoot this? > - As this default combination of out-of-the-box settings is bad for fq > with pacing performance, and it seems to somewhat be the default now for > two different igb-systems I thought I should give you a heads up. > > Regards, > Hans-Kristian > --001a113debb82e337a05486ced27 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
After some further testing i see that this is not igb specific!= Also bond interfaces and e1000e have tso disabled like this:

tx-tcp-segment= ation: off
tx-tcp-mangleid-segmentation: off [requested on]

bon= ded vlan interfaces however have tso enabled, but again vlan interfaces see= ms to have everything enabled.

As downgrading the kernel to a known = good did not do the trick there must be something else systemwide in Debian= Stretch. Perhaps I should really try to get all the dependencies together = to downgrade systemd.

Regards,
Hans-Kristian

On 13 February 2017= at 17:54, Hans-Kristian Bakke <hkbakke@gmail.com> wrote:
In a previous thread on this ma= iling list (Excessive throttling with fq) it was concluded that the reason = for the bad performance was that tso was somewhat inconcistent between the = bond and the physical interfaces. I never really knew why, but I knew I had= been experimenting with traffic shaping and offloads so I naturally though= t that was the culprit.

However, I have now rebooted some of my system= s and in both of my Debian Stretch (testing) systems (I tried both kernel 4= .8 and 4.9, one with fq_codel and one with sch_fq) I end up with this combi= nation of segmentation offload settings:
=

<= /div>
= ...
tcp-segmentation-offload: on
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-segment= ation: off <---- ???
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-ecn-segmentation= : off [fixed]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-mangleid-segmentation: off=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp6-segmentation: on
udp-fragmentation-off= load: off [fixed]
generic-segmentation-offload: on
...

If I do the fol= lowing...
ethtool -K eth0 = tx-tcp-segmentation on=C2=A0

...I end= up with what I expect and the performance returns for sch_fq with pacing.<= /font>
...
tcp-= segmentation-offload: on
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-segme= ntation: on
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-ecn-segmentation: = off [fixed]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp-mangleid-segmentat= ion: off
=C2=A0 =C2=A0 =C2=A0 =C2=A0 tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentatio= n-offload: on
...

I have some other Debi= an Jessie systems also using the igb-driver and tso is always enabled here = by default (but not the same NIC). (Kernel 3.16)

I also have a Proxmox VE system on kernel 4.4 with exactly the same quad= NIC that i suspect do not use the kernel source igb-drivers as it has a lo= t more params that also has tso enabled correctly.

# Proxmox VE system (full TSO by default)
#ethtool -i = eth1
driver: igb
version: 5.3.5.3
firmware-ve= rsion: 3.19, 0x00013cbf
bus-info: 0000:03:00.1
supports= -statistics: yes
supports-test: yes
supports-eeprom-acc= ess: yes
supports-register-dump: yes
supports-priv-flag= s: no

# Debian Stretch systems (half-e= nabled TSO by default)
# ethtool -i eth2
driver: i= gb
version: 5.4.0-k
firmware-version: 0.0.0
e= xpansion-rom-version:
bus-info: 0000:00:14.2
supports-s= tatistics: yes
supports-test: yes
supports-eeprom-acces= s: yes
supports-register-dump: yes
supports-priv-flags:= no

The NIC that the "working" and= the "non-working" system have is=C2=A0HP NC365T, which is based = on intel 82580.=C2=A0
The other "non-working" Debia= n Stretch system is a Intel Rangeley Atom c2758 supermicros system with a q= uad intel I354 network controller embedded.

I trie= d booting the stretch system with kernel 3.16 from Debian Jessie to get the= same drivers like my working systems but no change. Then I noticed that sy= stemd got the ability to change exactly these offloads in the last systemd = version 232 which arrived in testing in the end of 2016 so I am thinking th= at this might have something to do with this, but I have not found anything= (mostly because I don't even really know where to look)
=
I also tried to not having one of the interfaces in any kind= of bond or vlan subinterface in case those were changing some stuff but no= changes in behaviour.

So to my questions:
- Can anyone think of a reason why tx-tcp-segmentation offload is disabl= ed by default (and not tcp segmentation offload in general)
- Do = you have any tips to troubleshoot this?
- As this default combina= tion of out-of-the-box settings is bad for fq with pacing performance, and = it seems to somewhat be the default now for two different igb-systems I tho= ught I should give you a heads up.

Regards,
<= div>Hans-Kristian

--001a113debb82e337a05486ced27--