[Bloat] sch_fq, pacing and inconsistent TSO continued...

[Hans-Kristian Bakke <hkbakke@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3475 bytes --]

In a previous thread on this mailing list (Excessive throttling with fq) it
was concluded that the reason for the bad performance was that tso was
somewhat inconcistent between the bond and the physical interfaces. I never
really knew why, but I knew I had been experimenting with traffic shaping
and offloads so I naturally thought that was the culprit.

However, I have now rebooted some of my systems and in both of my Debian
Stretch (testing) systems (I tried both kernel 4.8 and 4.9, one with
fq_codel and one with sch_fq) I end up with this combination of
segmentation offload settings:

...
tcp-segmentation-offload: on
        tx-tcp-segmentation: off <---- ???
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
...

If I do the following...
ethtool -K eth0 tx-tcp-segmentation on

...I end up with what I expect and the performance returns for sch_fq with
pacing.
...
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
...

I have some other Debian Jessie systems also using the igb-driver and tso
is always enabled here by default (but not the same NIC). (Kernel 3.16)

I also have a Proxmox VE system on kernel 4.4 with exactly the same quad
NIC that i suspect do not use the kernel source igb-drivers as it has a lot
more params that also has tso enabled correctly.

# Proxmox VE system (full TSO by default)
#ethtool -i eth1
driver: igb
version: 5.3.5.3
firmware-version: 3.19, 0x00013cbf
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

# Debian Stretch systems (half-enabled TSO by default)
# ethtool -i eth2
driver: igb
version: 5.4.0-k
firmware-version: 0.0.0
expansion-rom-version:
bus-info: 0000:00:14.2
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

The NIC that the "working" and the "non-working" system have is HP NC365T,
which is based on intel 82580.
The other "non-working" Debian Stretch system is a Intel Rangeley Atom
c2758 supermicros system with a quad intel I354 network controller embedded.

I tried booting the stretch system with kernel 3.16 from Debian Jessie to
get the same drivers like my working systems but no change. Then I noticed
that systemd got the ability to change exactly these offloads in the last
systemd version 232 which arrived in testing in the end of 2016 so I am
thinking that this might have something to do with this, but I have not
found anything (mostly because I don't even really know where to look)

I also tried to not having one of the interfaces in any kind of bond or
vlan subinterface in case those were changing some stuff but no changes in
behaviour.

So to my questions:
- Can anyone think of a reason why tx-tcp-segmentation offload is disabled
by default (and not tcp segmentation offload in general)
- Do you have any tips to troubleshoot this?
- As this default combination of out-of-the-box settings is bad for fq with
pacing performance, and it seems to somewhat be the default now for two
different igb-systems I thought I should give you a heads up.

Regards,
Hans-Kristian

[-- Attachment #2: Type: text/html, Size: 5506 bytes --]