[Bloat] sch_fq, pacing and inconsistent TSO continued...

[Hans-Kristian Bakke]

Yep.. downgrading systemd from 232.15 to 231.10 fixed it.

There you have it. Systemd 232 not only got the ability to set offloads,
but it also disables tx-tcp-segmentation by default for both physical NICs
and bonds.

These packages from Debian Snapshots did the trick:
libsystemd0_231-10_amd64.deb
libudev1_231-10_amd64.deb
systemd_231-10_amd64.deb
udev_231-10_amd64.deb

This is ...fun.... No wonder file my server performance got more unreliable
the last couple of months.


On 13 February 2017 at 18:39, Hans-Kristian Bakke <hkbakke at gmail.com> wrote:

> After some further testing i see that this is not igb specific! Also bond
> interfaces and e1000e have tso disabled like this:
>
> tx-tcp-segmentation: off
> tx-tcp-mangleid-segmentation: off [requested on]
>
> bonded vlan interfaces however have tso enabled, but again vlan interfaces
> seems to have everything enabled.
>
> As downgrading the kernel to a known good did not do the trick there must
> be something else systemwide in Debian Stretch. Perhaps I should really try
> to get all the dependencies together to downgrade systemd.
>
> Regards,
> Hans-Kristian
>
> On 13 February 2017 at 17:54, Hans-Kristian Bakke <hkbakke at gmail.com>
> wrote:
>
>> In a previous thread on this mailing list (Excessive throttling with fq)
>> it was concluded that the reason for the bad performance was that tso was
>> somewhat inconcistent between the bond and the physical interfaces. I never
>> really knew why, but I knew I had been experimenting with traffic shaping
>> and offloads so I naturally thought that was the culprit.
>>
>> However, I have now rebooted some of my systems and in both of my Debian
>> Stretch (testing) systems (I tried both kernel 4.8 and 4.9, one with
>> fq_codel and one with sch_fq) I end up with this combination of
>> segmentation offload settings:
>>
>> ...
>> tcp-segmentation-offload: on
>>         tx-tcp-segmentation: off <---- ???
>>         tx-tcp-ecn-segmentation: off [fixed]
>>         tx-tcp-mangleid-segmentation: off
>>         tx-tcp6-segmentation: on
>> udp-fragmentation-offload: off [fixed]
>> generic-segmentation-offload: on
>> ...
>>
>> If I do the following...
>> ethtool -K eth0 tx-tcp-segmentation on
>>
>> ...I end up with what I expect and the performance returns for sch_fq
>> with pacing.
>> ...
>> tcp-segmentation-offload: on
>>         tx-tcp-segmentation: on
>>         tx-tcp-ecn-segmentation: off [fixed]
>>         tx-tcp-mangleid-segmentation: off
>>         tx-tcp6-segmentation: on
>> udp-fragmentation-offload: off [fixed]
>> generic-segmentation-offload: on
>> ...
>>
>> I have some other Debian Jessie systems also using the igb-driver and tso
>> is always enabled here by default (but not the same NIC). (Kernel 3.16)
>>
>> I also have a Proxmox VE system on kernel 4.4 with exactly the same quad
>> NIC that i suspect do not use the kernel source igb-drivers as it has a lot
>> more params that also has tso enabled correctly.
>>
>> # Proxmox VE system (full TSO by default)
>> #ethtool -i eth1
>> driver: igb
>> version: 5.3.5.3
>> firmware-version: 3.19, 0x00013cbf
>> bus-info: 0000:03:00.1
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: yes
>> supports-register-dump: yes
>> supports-priv-flags: no
>>
>> # Debian Stretch systems (half-enabled TSO by default)
>> # ethtool -i eth2
>> driver: igb
>> version: 5.4.0-k
>> firmware-version: 0.0.0
>> expansion-rom-version:
>> bus-info: 0000:00:14.2
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: yes
>> supports-register-dump: yes
>> supports-priv-flags: no
>>
>> The NIC that the "working" and the "non-working" system have is HP
>> NC365T, which is based on intel 82580.
>> The other "non-working" Debian Stretch system is a Intel Rangeley Atom
>> c2758 supermicros system with a quad intel I354 network controller embedded.
>>
>> I tried booting the stretch system with kernel 3.16 from Debian Jessie to
>> get the same drivers like my working systems but no change. Then I noticed
>> that systemd got the ability to change exactly these offloads in the last
>> systemd version 232 which arrived in testing in the end of 2016 so I am
>> thinking that this might have something to do with this, but I have not
>> found anything (mostly because I don't even really know where to look)
>>
>> I also tried to not having one of the interfaces in any kind of bond or
>> vlan subinterface in case those were changing some stuff but no changes in
>> behaviour.
>>
>> So to my questions:
>> - Can anyone think of a reason why tx-tcp-segmentation offload is
>> disabled by default (and not tcp segmentation offload in general)
>> - Do you have any tips to troubleshoot this?
>> - As this default combination of out-of-the-box settings is bad for fq
>> with pacing performance, and it seems to somewhat be the default now for
>> two different igb-systems I thought I should give you a heads up.
>>
>> Regards,
>> Hans-Kristian
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/bloat/attachments/20170213/2f6675da/attachment-0002.html>