[Bloat] sch_fq, pacing and inconsistent TSO continued...

[Bloat] sch_fq, pacing and inconsistent TSO continued...

[Hans-Kristian Bakke]

[-- Attachment #1: Type: text/plain, Size: 3475 bytes --]

In a previous thread on this mailing list (Excessive throttling with fq) it
was concluded that the reason for the bad performance was that tso was
somewhat inconcistent between the bond and the physical interfaces. I never
really knew why, but I knew I had been experimenting with traffic shaping
and offloads so I naturally thought that was the culprit.

However, I have now rebooted some of my systems and in both of my Debian
Stretch (testing) systems (I tried both kernel 4.8 and 4.9, one with
fq_codel and one with sch_fq) I end up with this combination of
segmentation offload settings:

...
tcp-segmentation-offload: on
        tx-tcp-segmentation: off <---- ???
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
...

If I do the following...
ethtool -K eth0 tx-tcp-segmentation on

...I end up with what I expect and the performance returns for sch_fq with
pacing.
...
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
...

I have some other Debian Jessie systems also using the igb-driver and tso
is always enabled here by default (but not the same NIC). (Kernel 3.16)

I also have a Proxmox VE system on kernel 4.4 with exactly the same quad
NIC that i suspect do not use the kernel source igb-drivers as it has a lot
more params that also has tso enabled correctly.

# Proxmox VE system (full TSO by default)
#ethtool -i eth1
driver: igb
version: 5.3.5.3
firmware-version: 3.19, 0x00013cbf
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

# Debian Stretch systems (half-enabled TSO by default)
# ethtool -i eth2
driver: igb
version: 5.4.0-k
firmware-version: 0.0.0
expansion-rom-version:
bus-info: 0000:00:14.2
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

The NIC that the "working" and the "non-working" system have is HP NC365T,
which is based on intel 82580.
The other "non-working" Debian Stretch system is a Intel Rangeley Atom
c2758 supermicros system with a quad intel I354 network controller embedded.

I tried booting the stretch system with kernel 3.16 from Debian Jessie to
get the same drivers like my working systems but no change. Then I noticed
that systemd got the ability to change exactly these offloads in the last
systemd version 232 which arrived in testing in the end of 2016 so I am
thinking that this might have something to do with this, but I have not
found anything (mostly because I don't even really know where to look)

I also tried to not having one of the interfaces in any kind of bond or
vlan subinterface in case those were changing some stuff but no changes in
behaviour.

So to my questions:
- Can anyone think of a reason why tx-tcp-segmentation offload is disabled
by default (and not tcp segmentation offload in general)
- Do you have any tips to troubleshoot this?
- As this default combination of out-of-the-box settings is bad for fq with
pacing performance, and it seems to somewhat be the default now for two
different igb-systems I thought I should give you a heads up.

Regards,
Hans-Kristian

[-- Attachment #2: Type: text/html, Size: 5506 bytes --]

Re: [Bloat] sch_fq, pacing and inconsistent TSO continued...

[Hans-Kristian Bakke]

[-- Attachment #1: Type: text/plain, Size: 4275 bytes --]

After some further testing i see that this is not igb specific! Also bond
interfaces and e1000e have tso disabled like this:

tx-tcp-segmentation: off
tx-tcp-mangleid-segmentation: off [requested on]

bonded vlan interfaces however have tso enabled, but again vlan interfaces
seems to have everything enabled.

As downgrading the kernel to a known good did not do the trick there must
be something else systemwide in Debian Stretch. Perhaps I should really try
to get all the dependencies together to downgrade systemd.

Regards,
Hans-Kristian

On 13 February 2017 at 17:54, Hans-Kristian Bakke <hkbakke@gmail.com> wrote:

> In a previous thread on this mailing list (Excessive throttling with fq)
> it was concluded that the reason for the bad performance was that tso was
> somewhat inconcistent between the bond and the physical interfaces. I never
> really knew why, but I knew I had been experimenting with traffic shaping
> and offloads so I naturally thought that was the culprit.
>
> However, I have now rebooted some of my systems and in both of my Debian
> Stretch (testing) systems (I tried both kernel 4.8 and 4.9, one with
> fq_codel and one with sch_fq) I end up with this combination of
> segmentation offload settings:
>
> ...
> tcp-segmentation-offload: on
>         tx-tcp-segmentation: off <---- ???
>         tx-tcp-ecn-segmentation: off [fixed]
>         tx-tcp-mangleid-segmentation: off
>         tx-tcp6-segmentation: on
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
> ...
>
> If I do the following...
> ethtool -K eth0 tx-tcp-segmentation on
>
> ...I end up with what I expect and the performance returns for sch_fq with
> pacing.
> ...
> tcp-segmentation-offload: on
>         tx-tcp-segmentation: on
>         tx-tcp-ecn-segmentation: off [fixed]
>         tx-tcp-mangleid-segmentation: off
>         tx-tcp6-segmentation: on
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
> ...
>
> I have some other Debian Jessie systems also using the igb-driver and tso
> is always enabled here by default (but not the same NIC). (Kernel 3.16)
>
> I also have a Proxmox VE system on kernel 4.4 with exactly the same quad
> NIC that i suspect do not use the kernel source igb-drivers as it has a lot
> more params that also has tso enabled correctly.
>
> # Proxmox VE system (full TSO by default)
> #ethtool -i eth1
> driver: igb
> version: 5.3.5.3
> firmware-version: 3.19, 0x00013cbf
> bus-info: 0000:03:00.1
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
> # Debian Stretch systems (half-enabled TSO by default)
> # ethtool -i eth2
> driver: igb
> version: 5.4.0-k
> firmware-version: 0.0.0
> expansion-rom-version:
> bus-info: 0000:00:14.2
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
> The NIC that the "working" and the "non-working" system have is HP NC365T,
> which is based on intel 82580.
> The other "non-working" Debian Stretch system is a Intel Rangeley Atom
> c2758 supermicros system with a quad intel I354 network controller embedded.
>
> I tried booting the stretch system with kernel 3.16 from Debian Jessie to
> get the same drivers like my working systems but no change. Then I noticed
> that systemd got the ability to change exactly these offloads in the last
> systemd version 232 which arrived in testing in the end of 2016 so I am
> thinking that this might have something to do with this, but I have not
> found anything (mostly because I don't even really know where to look)
>
> I also tried to not having one of the interfaces in any kind of bond or
> vlan subinterface in case those were changing some stuff but no changes in
> behaviour.
>
> So to my questions:
> - Can anyone think of a reason why tx-tcp-segmentation offload is disabled
> by default (and not tcp segmentation offload in general)
> - Do you have any tips to troubleshoot this?
> - As this default combination of out-of-the-box settings is bad for fq
> with pacing performance, and it seems to somewhat be the default now for
> two different igb-systems I thought I should give you a heads up.
>
> Regards,
> Hans-Kristian
>

[-- Attachment #2: Type: text/html, Size: 7232 bytes --]

Re: [Bloat] sch_fq, pacing and inconsistent TSO continued...

[Hans-Kristian Bakke]

[-- Attachment #1: Type: text/plain, Size: 4963 bytes --]

Yep.. downgrading systemd from 232.15 to 231.10 fixed it.

There you have it. Systemd 232 not only got the ability to set offloads,
but it also disables tx-tcp-segmentation by default for both physical NICs
and bonds.

These packages from Debian Snapshots did the trick:
libsystemd0_231-10_amd64.deb
libudev1_231-10_amd64.deb
systemd_231-10_amd64.deb
udev_231-10_amd64.deb

This is ...fun.... No wonder file my server performance got more unreliable
the last couple of months.


On 13 February 2017 at 18:39, Hans-Kristian Bakke <hkbakke@gmail.com> wrote:

> After some further testing i see that this is not igb specific! Also bond
> interfaces and e1000e have tso disabled like this:
>
> tx-tcp-segmentation: off
> tx-tcp-mangleid-segmentation: off [requested on]
>
> bonded vlan interfaces however have tso enabled, but again vlan interfaces
> seems to have everything enabled.
>
> As downgrading the kernel to a known good did not do the trick there must
> be something else systemwide in Debian Stretch. Perhaps I should really try
> to get all the dependencies together to downgrade systemd.
>
> Regards,
> Hans-Kristian
>
> On 13 February 2017 at 17:54, Hans-Kristian Bakke <hkbakke@gmail.com>
> wrote:
>
>> In a previous thread on this mailing list (Excessive throttling with fq)
>> it was concluded that the reason for the bad performance was that tso was
>> somewhat inconcistent between the bond and the physical interfaces. I never
>> really knew why, but I knew I had been experimenting with traffic shaping
>> and offloads so I naturally thought that was the culprit.
>>
>> However, I have now rebooted some of my systems and in both of my Debian
>> Stretch (testing) systems (I tried both kernel 4.8 and 4.9, one with
>> fq_codel and one with sch_fq) I end up with this combination of
>> segmentation offload settings:
>>
>> ...
>> tcp-segmentation-offload: on
>>         tx-tcp-segmentation: off <---- ???
>>         tx-tcp-ecn-segmentation: off [fixed]
>>         tx-tcp-mangleid-segmentation: off
>>         tx-tcp6-segmentation: on
>> udp-fragmentation-offload: off [fixed]
>> generic-segmentation-offload: on
>> ...
>>
>> If I do the following...
>> ethtool -K eth0 tx-tcp-segmentation on
>>
>> ...I end up with what I expect and the performance returns for sch_fq
>> with pacing.
>> ...
>> tcp-segmentation-offload: on
>>         tx-tcp-segmentation: on
>>         tx-tcp-ecn-segmentation: off [fixed]
>>         tx-tcp-mangleid-segmentation: off
>>         tx-tcp6-segmentation: on
>> udp-fragmentation-offload: off [fixed]
>> generic-segmentation-offload: on
>> ...
>>
>> I have some other Debian Jessie systems also using the igb-driver and tso
>> is always enabled here by default (but not the same NIC). (Kernel 3.16)
>>
>> I also have a Proxmox VE system on kernel 4.4 with exactly the same quad
>> NIC that i suspect do not use the kernel source igb-drivers as it has a lot
>> more params that also has tso enabled correctly.
>>
>> # Proxmox VE system (full TSO by default)
>> #ethtool -i eth1
>> driver: igb
>> version: 5.3.5.3
>> firmware-version: 3.19, 0x00013cbf
>> bus-info: 0000:03:00.1
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: yes
>> supports-register-dump: yes
>> supports-priv-flags: no
>>
>> # Debian Stretch systems (half-enabled TSO by default)
>> # ethtool -i eth2
>> driver: igb
>> version: 5.4.0-k
>> firmware-version: 0.0.0
>> expansion-rom-version:
>> bus-info: 0000:00:14.2
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: yes
>> supports-register-dump: yes
>> supports-priv-flags: no
>>
>> The NIC that the "working" and the "non-working" system have is HP
>> NC365T, which is based on intel 82580.
>> The other "non-working" Debian Stretch system is a Intel Rangeley Atom
>> c2758 supermicros system with a quad intel I354 network controller embedded.
>>
>> I tried booting the stretch system with kernel 3.16 from Debian Jessie to
>> get the same drivers like my working systems but no change. Then I noticed
>> that systemd got the ability to change exactly these offloads in the last
>> systemd version 232 which arrived in testing in the end of 2016 so I am
>> thinking that this might have something to do with this, but I have not
>> found anything (mostly because I don't even really know where to look)
>>
>> I also tried to not having one of the interfaces in any kind of bond or
>> vlan subinterface in case those were changing some stuff but no changes in
>> behaviour.
>>
>> So to my questions:
>> - Can anyone think of a reason why tx-tcp-segmentation offload is
>> disabled by default (and not tcp segmentation offload in general)
>> - Do you have any tips to troubleshoot this?
>> - As this default combination of out-of-the-box settings is bad for fq
>> with pacing performance, and it seems to somewhat be the default now for
>> two different igb-systems I thought I should give you a heads up.
>>
>> Regards,
>> Hans-Kristian
>>
>
>

[-- Attachment #2: Type: text/html, Size: 8731 bytes --]