* [Bloat] Excessive throttling with fq
@ 2017-01-26 18:36 Hans-Kristian Bakke
2017-01-26 18:38 ` Hans-Kristian Bakke
0 siblings, 1 reply; 15+ messages in thread
From: Hans-Kristian Bakke @ 2017-01-26 18:36 UTC (permalink / raw)
To: bloat
[-- Attachment #1: Type: text/plain, Size: 1447 bytes --]
Another day, another fq issue (or user error).
I try to do the seeminlig simple task of downloading a single large file
over local gigabit LAN from a physical server running kernel 4.8 and
sch_fq on intel server NICs.
For some reason it wouldn't go past around 25 MB/s. After having replaced
SSL with no SSL, replaced apache with nginx and verified that there is
plenty of bandwith available between my client and the server I tried to
change qdisc from fq to pfifo_fast. It instantly shot up to around the
expected 85-90 MB/s. The same happened with fq_codel in place of fq.
I then checked the statistics for fq and the throttled counter is
increasing massively every second (eth0 and eth1 is LACPed using Linux
bonding so both is seen here):
qdisc fq 8007: root refcnt 2 limit 10000p flow_limit 100p buckets 1024
orphan_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms
Sent 787131797 bytes 520082 pkt (dropped 15, overlimits 0 requeues 0)
backlog 98410b 65p requeues 0
15 flows (14 inactive, 1 throttled)
0 gc, 2 highprio, 259920 throttled, 15 flows_plimit
qdisc fq 8008: root refcnt 2 limit 10000p flow_limit 100p buckets 1024
orphan_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms
Sent 2533167 bytes 6731 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
24 flows (24 inactive, 0 throttled)
0 gc, 2 highprio, 397 throttled
Do you have any suggestions?
Regards,
Hans-Kristian
[-- Attachment #2: Type: text/html, Size: 2517 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bloat] Excessive throttling with fq
2017-01-26 18:36 [Bloat] Excessive throttling with fq Hans-Kristian Bakke
@ 2017-01-26 18:38 ` Hans-Kristian Bakke
2017-01-26 18:55 ` Hans-Kristian Bakke
0 siblings, 1 reply; 15+ messages in thread
From: Hans-Kristian Bakke @ 2017-01-26 18:38 UTC (permalink / raw)
To: bloat
[-- Attachment #1: Type: text/plain, Size: 1656 bytes --]
I can add that this is without BBR, just plain old kernel 4.8 cubic.
On 26 January 2017 at 19:36, Hans-Kristian Bakke <hkbakke@gmail.com> wrote:
> Another day, another fq issue (or user error).
>
> I try to do the seeminlig simple task of downloading a single large file
> over local gigabit LAN from a physical server running kernel 4.8 and
> sch_fq on intel server NICs.
>
> For some reason it wouldn't go past around 25 MB/s. After having replaced
> SSL with no SSL, replaced apache with nginx and verified that there is
> plenty of bandwith available between my client and the server I tried to
> change qdisc from fq to pfifo_fast. It instantly shot up to around the
> expected 85-90 MB/s. The same happened with fq_codel in place of fq.
>
> I then checked the statistics for fq and the throttled counter is
> increasing massively every second (eth0 and eth1 is LACPed using Linux
> bonding so both is seen here):
>
> qdisc fq 8007: root refcnt 2 limit 10000p flow_limit 100p buckets 1024
> orphan_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms
> Sent 787131797 bytes 520082 pkt (dropped 15, overlimits 0 requeues 0)
> backlog 98410b 65p requeues 0
> 15 flows (14 inactive, 1 throttled)
> 0 gc, 2 highprio, 259920 throttled, 15 flows_plimit
> qdisc fq 8008: root refcnt 2 limit 10000p flow_limit 100p buckets 1024
> orphan_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms
> Sent 2533167 bytes 6731 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> 24 flows (24 inactive, 0 throttled)
> 0 gc, 2 highprio, 397 throttled
>
> Do you have any suggestions?
>
> Regards,
> Hans-Kristian
>
[-- Attachment #2: Type: text/html, Size: 3028 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bloat] Excessive throttling with fq
2017-01-26 18:38 ` Hans-Kristian Bakke
@ 2017-01-26 18:55 ` Hans-Kristian Bakke
2017-01-26 19:18 ` Eric Dumazet
0 siblings, 1 reply; 15+ messages in thread
From: Hans-Kristian Bakke @ 2017-01-26 18:55 UTC (permalink / raw)
To: bloat
[-- Attachment #1: Type: text/plain, Size: 2346 bytes --]
After some more testing I see that if I disable fq pacing the performance
is restored to the expected levels:
# for i in eth0 eth1; do tc qdisc replace dev $i root fq nopacing; done
Is this expected behaviour? There is some background traffic, but only in
the sub 100 mbit/s on the switches and gateway between the server and
client.
The chain:
Windows 10 client -> 1000 mbit/s -> switch -> 2xgigabit LACP -> switch -> 4
x gigabit LACP -> gw (fq_codel on all nics) -> 4 x gigabit LACP (the same
as in) -> switch -> 2 x lacp -> server (with misbehaving fq pacing)
On 26 January 2017 at 19:38, Hans-Kristian Bakke <hkbakke@gmail.com> wrote:
> I can add that this is without BBR, just plain old kernel 4.8 cubic.
>
> On 26 January 2017 at 19:36, Hans-Kristian Bakke <hkbakke@gmail.com>
> wrote:
>
>> Another day, another fq issue (or user error).
>>
>> I try to do the seeminlig simple task of downloading a single large file
>> over local gigabit LAN from a physical server running kernel 4.8 and
>> sch_fq on intel server NICs.
>>
>> For some reason it wouldn't go past around 25 MB/s. After having replaced
>> SSL with no SSL, replaced apache with nginx and verified that there is
>> plenty of bandwith available between my client and the server I tried to
>> change qdisc from fq to pfifo_fast. It instantly shot up to around the
>> expected 85-90 MB/s. The same happened with fq_codel in place of fq.
>>
>> I then checked the statistics for fq and the throttled counter is
>> increasing massively every second (eth0 and eth1 is LACPed using Linux
>> bonding so both is seen here):
>>
>> qdisc fq 8007: root refcnt 2 limit 10000p flow_limit 100p buckets 1024
>> orphan_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms
>> Sent 787131797 bytes 520082 pkt (dropped 15, overlimits 0 requeues 0)
>> backlog 98410b 65p requeues 0
>> 15 flows (14 inactive, 1 throttled)
>> 0 gc, 2 highprio, 259920 throttled, 15 flows_plimit
>> qdisc fq 8008: root refcnt 2 limit 10000p flow_limit 100p buckets 1024
>> orphan_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms
>> Sent 2533167 bytes 6731 pkt (dropped 0, overlimits 0 requeues 0)
>> backlog 0b 0p requeues 0
>> 24 flows (24 inactive, 0 throttled)
>> 0 gc, 2 highprio, 397 throttled
>>
>> Do you have any suggestions?
>>
>> Regards,
>> Hans-Kristian
>>
>
>
[-- Attachment #2: Type: text/html, Size: 4625 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bloat] Excessive throttling with fq
2017-01-26 18:55 ` Hans-Kristian Bakke
@ 2017-01-26 19:18 ` Eric Dumazet
2017-01-26 19:58 ` David Lang
0 siblings, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2017-01-26 19:18 UTC (permalink / raw)
To: Hans-Kristian Bakke; +Cc: bloat
Nothing jumps on my head.
We use FQ on links varying from 1Gbit to 100Gbit, and we have no such
issues.
You could probably check on the server the TCP various infos given by ss
command
ss -temoi dst <remoteip>
pacing rate is shown. You might have some issues, but it is hard to say.
On Thu, 2017-01-26 at 19:55 +0100, Hans-Kristian Bakke wrote:
> After some more testing I see that if I disable fq pacing the
> performance is restored to the expected levels:
> # for i in eth0 eth1; do tc qdisc replace dev $i root fq nopacing;
> done
>
>
> Is this expected behaviour? There is some background traffic, but only
> in the sub 100 mbit/s on the switches and gateway between the server
> and client.
>
>
> The chain:
> Windows 10 client -> 1000 mbit/s -> switch -> 2xgigabit LACP -> switch
> -> 4 x gigabit LACP -> gw (fq_codel on all nics) -> 4 x gigabit LACP
> (the same as in) -> switch -> 2 x lacp -> server (with misbehaving fq
> pacing)
>
>
>
> On 26 January 2017 at 19:38, Hans-Kristian Bakke <hkbakke@gmail.com>
> wrote:
> I can add that this is without BBR, just plain old kernel 4.8
> cubic.
>
> On 26 January 2017 at 19:36, Hans-Kristian Bakke
> <hkbakke@gmail.com> wrote:
> Another day, another fq issue (or user error).
>
>
> I try to do the seeminlig simple task of downloading a
> single large file over local gigabit LAN from a
> physical server running kernel 4.8 and sch_fq on intel
> server NICs.
>
>
> For some reason it wouldn't go past around 25 MB/s.
> After having replaced SSL with no SSL, replaced apache
> with nginx and verified that there is plenty of
> bandwith available between my client and the server I
> tried to change qdisc from fq to pfifo_fast. It
> instantly shot up to around the expected 85-90 MB/s.
> The same happened with fq_codel in place of fq.
>
>
> I then checked the statistics for fq and the throttled
> counter is increasing massively every second (eth0 and
> eth1 is LACPed using Linux bonding so both is seen
> here):
>
>
> qdisc fq 8007: root refcnt 2 limit 10000p flow_limit
> 100p buckets 1024 orphan_mask 1023 quantum 3028
> initial_quantum 15140 refill_delay 40.0ms
> Sent 787131797 bytes 520082 pkt (dropped 15,
> overlimits 0 requeues 0)
> backlog 98410b 65p requeues 0
> 15 flows (14 inactive, 1 throttled)
> 0 gc, 2 highprio, 259920 throttled, 15 flows_plimit
> qdisc fq 8008: root refcnt 2 limit 10000p flow_limit
> 100p buckets 1024 orphan_mask 1023 quantum 3028
> initial_quantum 15140 refill_delay 40.0ms
> Sent 2533167 bytes 6731 pkt (dropped 0, overlimits 0
> requeues 0)
> backlog 0b 0p requeues 0
> 24 flows (24 inactive, 0 throttled)
> 0 gc, 2 highprio, 397 throttled
>
>
> Do you have any suggestions?
>
>
> Regards,
> Hans-Kristian
>
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bloat] Excessive throttling with fq
2017-01-26 19:18 ` Eric Dumazet
@ 2017-01-26 19:58 ` David Lang
2017-01-26 20:19 ` Hans-Kristian Bakke
0 siblings, 1 reply; 15+ messages in thread
From: David Lang @ 2017-01-26 19:58 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Hans-Kristian Bakke, bloat
Is there any CPU bottleneck?
pacing causing this sort of problem makes me thing that the CPU either can't
keep up or that something (Hz setting type of thing) is delaying when the CPU
can get used.
It's not clear from the posts if the problem is with sending data or receiving
data.
David Lang
On Thu, 26 Jan 2017, Eric Dumazet wrote:
> Nothing jumps on my head.
>
> We use FQ on links varying from 1Gbit to 100Gbit, and we have no such
> issues.
>
> You could probably check on the server the TCP various infos given by ss
> command
>
>
> ss -temoi dst <remoteip>
>
>
> pacing rate is shown. You might have some issues, but it is hard to say.
>
>
> On Thu, 2017-01-26 at 19:55 +0100, Hans-Kristian Bakke wrote:
>> After some more testing I see that if I disable fq pacing the
>> performance is restored to the expected levels:
>> # for i in eth0 eth1; do tc qdisc replace dev $i root fq nopacing;
>> done
>>
>>
>> Is this expected behaviour? There is some background traffic, but only
>> in the sub 100 mbit/s on the switches and gateway between the server
>> and client.
>>
>>
>> The chain:
>> Windows 10 client -> 1000 mbit/s -> switch -> 2xgigabit LACP -> switch
>> -> 4 x gigabit LACP -> gw (fq_codel on all nics) -> 4 x gigabit LACP
>> (the same as in) -> switch -> 2 x lacp -> server (with misbehaving fq
>> pacing)
>>
>>
>>
>> On 26 January 2017 at 19:38, Hans-Kristian Bakke <hkbakke@gmail.com>
>> wrote:
>> I can add that this is without BBR, just plain old kernel 4.8
>> cubic.
>>
>> On 26 January 2017 at 19:36, Hans-Kristian Bakke
>> <hkbakke@gmail.com> wrote:
>> Another day, another fq issue (or user error).
>>
>>
>> I try to do the seeminlig simple task of downloading a
>> single large file over local gigabit LAN from a
>> physical server running kernel 4.8 and sch_fq on intel
>> server NICs.
>>
>>
>> For some reason it wouldn't go past around 25 MB/s.
>> After having replaced SSL with no SSL, replaced apache
>> with nginx and verified that there is plenty of
>> bandwith available between my client and the server I
>> tried to change qdisc from fq to pfifo_fast. It
>> instantly shot up to around the expected 85-90 MB/s.
>> The same happened with fq_codel in place of fq.
>>
>>
>> I then checked the statistics for fq and the throttled
>> counter is increasing massively every second (eth0 and
>> eth1 is LACPed using Linux bonding so both is seen
>> here):
>>
>>
>> qdisc fq 8007: root refcnt 2 limit 10000p flow_limit
>> 100p buckets 1024 orphan_mask 1023 quantum 3028
>> initial_quantum 15140 refill_delay 40.0ms
>> Sent 787131797 bytes 520082 pkt (dropped 15,
>> overlimits 0 requeues 0)
>> backlog 98410b 65p requeues 0
>> 15 flows (14 inactive, 1 throttled)
>> 0 gc, 2 highprio, 259920 throttled, 15 flows_plimit
>> qdisc fq 8008: root refcnt 2 limit 10000p flow_limit
>> 100p buckets 1024 orphan_mask 1023 quantum 3028
>> initial_quantum 15140 refill_delay 40.0ms
>> Sent 2533167 bytes 6731 pkt (dropped 0, overlimits 0
>> requeues 0)
>> backlog 0b 0p requeues 0
>> 24 flows (24 inactive, 0 throttled)
>> 0 gc, 2 highprio, 397 throttled
>>
>>
>> Do you have any suggestions?
>>
>>
>> Regards,
>> Hans-Kristian
>>
>>
>>
>>
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bloat] Excessive throttling with fq
2017-01-26 19:58 ` David Lang
@ 2017-01-26 20:19 ` Hans-Kristian Bakke
2017-01-26 20:41 ` Eric Dumazet
2017-01-26 20:54 ` Eric Dumazet
0 siblings, 2 replies; 15+ messages in thread
From: Hans-Kristian Bakke @ 2017-01-26 20:19 UTC (permalink / raw)
To: David Lang; +Cc: Eric Dumazet, bloat
[-- Attachment #1: Type: text/plain, Size: 6281 bytes --]
There are two packet captures from fq with and without pacing here:
https://owncloud.proikt.com/index.php/s/KuXIl8h8bSFH1fM
The server (with fq pacing/nopacing) is 10.0.5.10 and is running a Apache2
webserver at port tcp port 443. The tcp client is nginx reverse proxy at
10.0.5.13 on the same subnet which again is proxying the connection from
the Windows 10 client.
- I did try to connect directly to the server with the client (via a linux
gateway router) avoiding the nginx proxy and just using plain no-ssl http.
That did not change anything.
- I also tried stopping the eth0 interface to force the traffic to the eth1
interface in the LACP which changed nothing.
- I also pulled each of the cable on the switch to force the traffic to
switch between interfaces in the LACP link between the client switch and
the server switch.
The CPU is a 5-6 year old Intel Xeon X3430 CPU @ 4x2.40GHz on a SuperMicro
platform. It is not very loaded and the results are always in the same
ballpark with fq pacing on.
top - 21:12:38 up 12 days, 11:08, 4 users, load average: 0.56, 0.68, 0.77
Tasks: 1344 total, 1 running, 1343 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
%Cpu1 : 0.0 us, 0.3 sy, 0.0 ni, 97.4 id, 2.0 wa, 0.0 hi, 0.3 si,
0.0 st
%Cpu2 : 0.0 us, 2.0 sy, 0.0 ni, 96.4 id, 1.3 wa, 0.0 hi, 0.3 si,
0.0 st
%Cpu3 : 0.7 us, 2.3 sy, 0.0 ni, 94.1 id, 3.0 wa, 0.0 hi, 0.0 si,
0.0 st
KiB Mem : 16427572 total, 173712 free, 9739976 used, 6513884 buff/cache
KiB Swap: 6369276 total, 6126736 free, 242540 used. 6224836 avail Mem
This seems OK to me. It does have 24 drives in 3 ZFS pools at 144TB raw
storage in total with several SAS HBAs that is pretty much always poking
the system in some way or the other.
There are around 32K interrupts when running @23 MB/s (as seen in chrome
downloads) with pacing on and about 25K interrupts when running @105 MB/s
with fq nopacing. Is that normal?
Hans-Kristian
On 26 January 2017 at 20:58, David Lang <david@lang.hm> wrote:
> Is there any CPU bottleneck?
>
> pacing causing this sort of problem makes me thing that the CPU either
> can't keep up or that something (Hz setting type of thing) is delaying when
> the CPU can get used.
>
> It's not clear from the posts if the problem is with sending data or
> receiving data.
>
> David Lang
>
>
> On Thu, 26 Jan 2017, Eric Dumazet wrote:
>
> Nothing jumps on my head.
>>
>> We use FQ on links varying from 1Gbit to 100Gbit, and we have no such
>> issues.
>>
>> You could probably check on the server the TCP various infos given by ss
>> command
>>
>>
>> ss -temoi dst <remoteip>
>>
>>
>> pacing rate is shown. You might have some issues, but it is hard to say.
>>
>>
>> On Thu, 2017-01-26 at 19:55 +0100, Hans-Kristian Bakke wrote:
>>
>>> After some more testing I see that if I disable fq pacing the
>>> performance is restored to the expected levels: # for i in eth0 eth1; do
>>> tc qdisc replace dev $i root fq nopacing;
>>> done
>>>
>>>
>>> Is this expected behaviour? There is some background traffic, but only
>>> in the sub 100 mbit/s on the switches and gateway between the server
>>> and client.
>>>
>>>
>>> The chain:
>>> Windows 10 client -> 1000 mbit/s -> switch -> 2xgigabit LACP -> switch
>>> -> 4 x gigabit LACP -> gw (fq_codel on all nics) -> 4 x gigabit LACP
>>> (the same as in) -> switch -> 2 x lacp -> server (with misbehaving fq
>>> pacing)
>>>
>>>
>>>
>>> On 26 January 2017 at 19:38, Hans-Kristian Bakke <hkbakke@gmail.com>
>>> wrote:
>>> I can add that this is without BBR, just plain old kernel 4.8
>>> cubic.
>>>
>>> On 26 January 2017 at 19:36, Hans-Kristian Bakke
>>> <hkbakke@gmail.com> wrote:
>>> Another day, another fq issue (or user error).
>>>
>>>
>>> I try to do the seeminlig simple task of downloading a
>>> single large file over local gigabit LAN from a
>>> physical server running kernel 4.8 and sch_fq on intel
>>> server NICs.
>>>
>>>
>>> For some reason it wouldn't go past around 25 MB/s.
>>> After having replaced SSL with no SSL, replaced apache
>>> with nginx and verified that there is plenty of
>>> bandwith available between my client and the server I
>>> tried to change qdisc from fq to pfifo_fast. It
>>> instantly shot up to around the expected 85-90 MB/s.
>>> The same happened with fq_codel in place of fq.
>>>
>>>
>>> I then checked the statistics for fq and the throttled
>>> counter is increasing massively every second (eth0 and
>>> eth1 is LACPed using Linux bonding so both is seen
>>> here):
>>>
>>>
>>> qdisc fq 8007: root refcnt 2 limit 10000p flow_limit
>>> 100p buckets 1024 orphan_mask 1023 quantum 3028
>>> initial_quantum 15140 refill_delay 40.0ms
>>> Sent 787131797 bytes 520082 pkt (dropped 15,
>>> overlimits 0 requeues 0)
>>> backlog 98410b 65p requeues 0
>>> 15 flows (14 inactive, 1 throttled)
>>> 0 gc, 2 highprio, 259920 throttled, 15 flows_plimit
>>> qdisc fq 8008: root refcnt 2 limit 10000p flow_limit
>>> 100p buckets 1024 orphan_mask 1023 quantum 3028
>>> initial_quantum 15140 refill_delay 40.0ms
>>> Sent 2533167 bytes 6731 pkt (dropped 0, overlimits 0
>>> requeues 0)
>>> backlog 0b 0p requeues 0
>>> 24 flows (24 inactive, 0 throttled)
>>> 0 gc, 2 highprio, 397 throttled
>>>
>>>
>>> Do you have any suggestions?
>>>
>>>
>>> Regards,
>>> Hans-Kristian
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>>>
>>
>>
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>>
>
[-- Attachment #2: Type: text/html, Size: 10399 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bloat] Excessive throttling with fq
2017-01-26 20:19 ` Hans-Kristian Bakke
@ 2017-01-26 20:41 ` Eric Dumazet
2017-01-26 20:46 ` Hans-Kristian Bakke
2017-01-26 20:54 ` Eric Dumazet
1 sibling, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2017-01-26 20:41 UTC (permalink / raw)
To: Hans-Kristian Bakke; +Cc: David Lang, bloat
Can you post :
ethtool -i eth0
ethtool -k eth0
grep HZ /boot/config.... (what is the HZ value of your kernel)
I suspect a possible problem with TSO autodefer when/if HZ < 1000
Thanks.
On Thu, 2017-01-26 at 21:19 +0100, Hans-Kristian Bakke wrote:
> There are two packet captures from fq with and without pacing here:
>
>
> https://owncloud.proikt.com/index.php/s/KuXIl8h8bSFH1fM
>
>
>
> The server (with fq pacing/nopacing) is 10.0.5.10 and is running a
> Apache2 webserver at port tcp port 443. The tcp client is nginx
> reverse proxy at 10.0.5.13 on the same subnet which again is proxying
> the connection from the Windows 10 client.
> - I did try to connect directly to the server with the client (via a
> linux gateway router) avoiding the nginx proxy and just using plain
> no-ssl http. That did not change anything.
> - I also tried stopping the eth0 interface to force the traffic to the
> eth1 interface in the LACP which changed nothing.
> - I also pulled each of the cable on the switch to force the traffic
> to switch between interfaces in the LACP link between the client
> switch and the server switch.
>
>
> The CPU is a 5-6 year old Intel Xeon X3430 CPU @ 4x2.40GHz on a
> SuperMicro platform. It is not very loaded and the results are always
> in the same ballpark with fq pacing on.
>
>
>
> top - 21:12:38 up 12 days, 11:08, 4 users, load average: 0.56, 0.68,
> 0.77
> Tasks: 1344 total, 1 running, 1343 sleeping, 0 stopped, 0 zombie
> %Cpu0 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0
> si, 0.0 st
> %Cpu1 : 0.0 us, 0.3 sy, 0.0 ni, 97.4 id, 2.0 wa, 0.0 hi, 0.3
> si, 0.0 st
> %Cpu2 : 0.0 us, 2.0 sy, 0.0 ni, 96.4 id, 1.3 wa, 0.0 hi, 0.3
> si, 0.0 st
> %Cpu3 : 0.7 us, 2.3 sy, 0.0 ni, 94.1 id, 3.0 wa, 0.0 hi, 0.0
> si, 0.0 st
> KiB Mem : 16427572 total, 173712 free, 9739976 used, 6513884
> buff/cache
> KiB Swap: 6369276 total, 6126736 free, 242540 used. 6224836 avail
> Mem
>
>
> This seems OK to me. It does have 24 drives in 3 ZFS pools at 144TB
> raw storage in total with several SAS HBAs that is pretty much always
> poking the system in some way or the other.
>
>
> There are around 32K interrupts when running @23 MB/s (as seen in
> chrome downloads) with pacing on and about 25K interrupts when running
> @105 MB/s with fq nopacing. Is that normal?
>
>
> Hans-Kristian
>
>
>
> On 26 January 2017 at 20:58, David Lang <david@lang.hm> wrote:
> Is there any CPU bottleneck?
>
> pacing causing this sort of problem makes me thing that the
> CPU either can't keep up or that something (Hz setting type of
> thing) is delaying when the CPU can get used.
>
> It's not clear from the posts if the problem is with sending
> data or receiving data.
>
> David Lang
>
>
> On Thu, 26 Jan 2017, Eric Dumazet wrote:
>
> Nothing jumps on my head.
>
> We use FQ on links varying from 1Gbit to 100Gbit, and
> we have no such
> issues.
>
> You could probably check on the server the TCP various
> infos given by ss
> command
>
>
> ss -temoi dst <remoteip>
>
>
> pacing rate is shown. You might have some issues, but
> it is hard to say.
>
>
> On Thu, 2017-01-26 at 19:55 +0100, Hans-Kristian Bakke
> wrote:
> After some more testing I see that if I
> disable fq pacing the
> performance is restored to the expected
> levels: # for i in eth0 eth1; do tc qdisc
> replace dev $i root fq nopacing;
> done
>
>
> Is this expected behaviour? There is some
> background traffic, but only
> in the sub 100 mbit/s on the switches and
> gateway between the server
> and client.
>
>
> The chain:
> Windows 10 client -> 1000 mbit/s -> switch ->
> 2xgigabit LACP -> switch
> -> 4 x gigabit LACP -> gw (fq_codel on all
> nics) -> 4 x gigabit LACP
> (the same as in) -> switch -> 2 x lacp ->
> server (with misbehaving fq
> pacing)
>
>
>
> On 26 January 2017 at 19:38, Hans-Kristian
> Bakke <hkbakke@gmail.com>
> wrote:
> I can add that this is without BBR,
> just plain old kernel 4.8
> cubic.
>
> On 26 January 2017 at 19:36,
> Hans-Kristian Bakke
> <hkbakke@gmail.com> wrote:
> Another day, another fq issue
> (or user error).
>
>
> I try to do the seeminlig
> simple task of downloading a
> single large file over local
> gigabit LAN from a
> physical server running kernel
> 4.8 and sch_fq on intel
> server NICs.
>
>
> For some reason it wouldn't go
> past around 25 MB/s.
> After having replaced SSL with
> no SSL, replaced apache
> with nginx and verified that
> there is plenty of
> bandwith available between my
> client and the server I
> tried to change qdisc from fq
> to pfifo_fast. It
> instantly shot up to around
> the expected 85-90 MB/s.
> The same happened with
> fq_codel in place of fq.
>
>
> I then checked the statistics
> for fq and the throttled
> counter is increasing
> massively every second (eth0 and
> eth1 is LACPed using Linux
> bonding so both is seen
> here):
>
>
> qdisc fq 8007: root refcnt 2
> limit 10000p flow_limit
> 100p buckets 1024 orphan_mask
> 1023 quantum 3028
> initial_quantum 15140
> refill_delay 40.0ms
> Sent 787131797 bytes 520082
> pkt (dropped 15,
> overlimits 0 requeues 0)
> backlog 98410b 65p requeues 0
> 15 flows (14 inactive, 1
> throttled)
> 0 gc, 2 highprio, 259920
> throttled, 15 flows_plimit
> qdisc fq 8008: root refcnt 2
> limit 10000p flow_limit
> 100p buckets 1024 orphan_mask
> 1023 quantum 3028
> initial_quantum 15140
> refill_delay 40.0ms
> Sent 2533167 bytes 6731 pkt
> (dropped 0, overlimits 0
> requeues 0)
> backlog 0b 0p requeues 0
> 24 flows (24 inactive, 0
> throttled)
> 0 gc, 2 highprio, 397
> throttled
>
>
> Do you have any suggestions?
>
>
> Regards,
> Hans-Kristian
>
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bloat] Excessive throttling with fq
2017-01-26 20:41 ` Eric Dumazet
@ 2017-01-26 20:46 ` Hans-Kristian Bakke
2017-01-26 21:00 ` Eric Dumazet
0 siblings, 1 reply; 15+ messages in thread
From: Hans-Kristian Bakke @ 2017-01-26 20:46 UTC (permalink / raw)
To: Eric Dumazet; +Cc: David Lang, bloat
[-- Attachment #1: Type: text/plain, Size: 11588 bytes --]
# ethtool -i eth0
driver: e1000e
version: 3.2.6-k
firmware-version: 1.9-0
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
# ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
hw-tc-offload: off [fixed]
# grep HZ /boot/config-4.8.0-2-amd64
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_FULL is not set
# CONFIG_NO_HZ is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_MACHZ_WDT=m
On 26 January 2017 at 21:41, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>
> Can you post :
>
> ethtool -i eth0
> ethtool -k eth0
>
> grep HZ /boot/config.... (what is the HZ value of your kernel)
>
> I suspect a possible problem with TSO autodefer when/if HZ < 1000
>
> Thanks.
>
> On Thu, 2017-01-26 at 21:19 +0100, Hans-Kristian Bakke wrote:
> > There are two packet captures from fq with and without pacing here:
> >
> >
> > https://owncloud.proikt.com/index.php/s/KuXIl8h8bSFH1fM
> >
> >
> >
> > The server (with fq pacing/nopacing) is 10.0.5.10 and is running a
> > Apache2 webserver at port tcp port 443. The tcp client is nginx
> > reverse proxy at 10.0.5.13 on the same subnet which again is proxying
> > the connection from the Windows 10 client.
> > - I did try to connect directly to the server with the client (via a
> > linux gateway router) avoiding the nginx proxy and just using plain
> > no-ssl http. That did not change anything.
> > - I also tried stopping the eth0 interface to force the traffic to the
> > eth1 interface in the LACP which changed nothing.
> > - I also pulled each of the cable on the switch to force the traffic
> > to switch between interfaces in the LACP link between the client
> > switch and the server switch.
> >
> >
> > The CPU is a 5-6 year old Intel Xeon X3430 CPU @ 4x2.40GHz on a
> > SuperMicro platform. It is not very loaded and the results are always
> > in the same ballpark with fq pacing on.
> >
> >
> >
> > top - 21:12:38 up 12 days, 11:08, 4 users, load average: 0.56, 0.68,
> > 0.77
> > Tasks: 1344 total, 1 running, 1343 sleeping, 0 stopped, 0 zombie
> > %Cpu0 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0
> > si, 0.0 st
> > %Cpu1 : 0.0 us, 0.3 sy, 0.0 ni, 97.4 id, 2.0 wa, 0.0 hi, 0.3
> > si, 0.0 st
> > %Cpu2 : 0.0 us, 2.0 sy, 0.0 ni, 96.4 id, 1.3 wa, 0.0 hi, 0.3
> > si, 0.0 st
> > %Cpu3 : 0.7 us, 2.3 sy, 0.0 ni, 94.1 id, 3.0 wa, 0.0 hi, 0.0
> > si, 0.0 st
> > KiB Mem : 16427572 total, 173712 free, 9739976 used, 6513884
> > buff/cache
> > KiB Swap: 6369276 total, 6126736 free, 242540 used. 6224836 avail
> > Mem
> >
> >
> > This seems OK to me. It does have 24 drives in 3 ZFS pools at 144TB
> > raw storage in total with several SAS HBAs that is pretty much always
> > poking the system in some way or the other.
> >
> >
> > There are around 32K interrupts when running @23 MB/s (as seen in
> > chrome downloads) with pacing on and about 25K interrupts when running
> > @105 MB/s with fq nopacing. Is that normal?
> >
> >
> > Hans-Kristian
> >
> >
> >
> > On 26 January 2017 at 20:58, David Lang <david@lang.hm> wrote:
> > Is there any CPU bottleneck?
> >
> > pacing causing this sort of problem makes me thing that the
> > CPU either can't keep up or that something (Hz setting type of
> > thing) is delaying when the CPU can get used.
> >
> > It's not clear from the posts if the problem is with sending
> > data or receiving data.
> >
> > David Lang
> >
> >
> > On Thu, 26 Jan 2017, Eric Dumazet wrote:
> >
> > Nothing jumps on my head.
> >
> > We use FQ on links varying from 1Gbit to 100Gbit, and
> > we have no such
> > issues.
> >
> > You could probably check on the server the TCP various
> > infos given by ss
> > command
> >
> >
> > ss -temoi dst <remoteip>
> >
> >
> > pacing rate is shown. You might have some issues, but
> > it is hard to say.
> >
> >
> > On Thu, 2017-01-26 at 19:55 +0100, Hans-Kristian Bakke
> > wrote:
> > After some more testing I see that if I
> > disable fq pacing the
> > performance is restored to the expected
> > levels: # for i in eth0 eth1; do tc qdisc
> > replace dev $i root fq nopacing;
> > done
> >
> >
> > Is this expected behaviour? There is some
> > background traffic, but only
> > in the sub 100 mbit/s on the switches and
> > gateway between the server
> > and client.
> >
> >
> > The chain:
> > Windows 10 client -> 1000 mbit/s -> switch ->
> > 2xgigabit LACP -> switch
> > -> 4 x gigabit LACP -> gw (fq_codel on all
> > nics) -> 4 x gigabit LACP
> > (the same as in) -> switch -> 2 x lacp ->
> > server (with misbehaving fq
> > pacing)
> >
> >
> >
> > On 26 January 2017 at 19:38, Hans-Kristian
> > Bakke <hkbakke@gmail.com>
> > wrote:
> > I can add that this is without BBR,
> > just plain old kernel 4.8
> > cubic.
> >
> > On 26 January 2017 at 19:36,
> > Hans-Kristian Bakke
> > <hkbakke@gmail.com> wrote:
> > Another day, another fq issue
> > (or user error).
> >
> >
> > I try to do the seeminlig
> > simple task of downloading a
> > single large file over local
> > gigabit LAN from a
> > physical server running kernel
> > 4.8 and sch_fq on intel
> > server NICs.
> >
> >
> > For some reason it wouldn't go
> > past around 25 MB/s.
> > After having replaced SSL with
> > no SSL, replaced apache
> > with nginx and verified that
> > there is plenty of
> > bandwith available between my
> > client and the server I
> > tried to change qdisc from fq
> > to pfifo_fast. It
> > instantly shot up to around
> > the expected 85-90 MB/s.
> > The same happened with
> > fq_codel in place of fq.
> >
> >
> > I then checked the statistics
> > for fq and the throttled
> > counter is increasing
> > massively every second (eth0 and
> > eth1 is LACPed using Linux
> > bonding so both is seen
> > here):
> >
> >
> > qdisc fq 8007: root refcnt 2
> > limit 10000p flow_limit
> > 100p buckets 1024 orphan_mask
> > 1023 quantum 3028
> > initial_quantum 15140
> > refill_delay 40.0ms
> > Sent 787131797 bytes 520082
> > pkt (dropped 15,
> > overlimits 0 requeues 0)
> > backlog 98410b 65p requeues 0
> > 15 flows (14 inactive, 1
> > throttled)
> > 0 gc, 2 highprio, 259920
> > throttled, 15 flows_plimit
> > qdisc fq 8008: root refcnt 2
> > limit 10000p flow_limit
> > 100p buckets 1024 orphan_mask
> > 1023 quantum 3028
> > initial_quantum 15140
> > refill_delay 40.0ms
> > Sent 2533167 bytes 6731 pkt
> > (dropped 0, overlimits 0
> > requeues 0)
> > backlog 0b 0p requeues 0
> > 24 flows (24 inactive, 0
> > throttled)
> > 0 gc, 2 highprio, 397
> > throttled
> >
> >
> > Do you have any suggestions?
> >
> >
> > Regards,
> > Hans-Kristian
> >
> >
> >
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >
> >
>
>
>
[-- Attachment #2: Type: text/html, Size: 20139 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bloat] Excessive throttling with fq
2017-01-26 20:19 ` Hans-Kristian Bakke
2017-01-26 20:41 ` Eric Dumazet
@ 2017-01-26 20:54 ` Eric Dumazet
2017-01-26 20:57 ` Hans-Kristian Bakke
1 sibling, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2017-01-26 20:54 UTC (permalink / raw)
To: Hans-Kristian Bakke; +Cc: David Lang, bloat
It looks like the receiver is seriously limiting receive window in the
"pacing" case.
Only after 30 Mbytes are transfered, it finally increases it, from
359168 to 1328192
DRS is not working as expected. Again maybe related to HZ value.
On Thu, 2017-01-26 at 21:19 +0100, Hans-Kristian Bakke wrote:
> There are two packet captures from fq with and without pacing here:
>
>
> https://owncloud.proikt.com/index.php/s/KuXIl8h8bSFH1fM
>
>
>
> The server (with fq pacing/nopacing) is 10.0.5.10 and is running a
> Apache2 webserver at port tcp port 443. The tcp client is nginx
> reverse proxy at 10.0.5.13 on the same subnet which again is proxying
> the connection from the Windows 10 client.
> - I did try to connect directly to the server with the client (via a
> linux gateway router) avoiding the nginx proxy and just using plain
> no-ssl http. That did not change anything.
> - I also tried stopping the eth0 interface to force the traffic to the
> eth1 interface in the LACP which changed nothing.
> - I also pulled each of the cable on the switch to force the traffic
> to switch between interfaces in the LACP link between the client
> switch and the server switch.
>
>
> The CPU is a 5-6 year old Intel Xeon X3430 CPU @ 4x2.40GHz on a
> SuperMicro platform. It is not very loaded and the results are always
> in the same ballpark with fq pacing on.
>
>
>
> top - 21:12:38 up 12 days, 11:08, 4 users, load average: 0.56, 0.68,
> 0.77
> Tasks: 1344 total, 1 running, 1343 sleeping, 0 stopped, 0 zombie
> %Cpu0 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0
> si, 0.0 st
> %Cpu1 : 0.0 us, 0.3 sy, 0.0 ni, 97.4 id, 2.0 wa, 0.0 hi, 0.3
> si, 0.0 st
> %Cpu2 : 0.0 us, 2.0 sy, 0.0 ni, 96.4 id, 1.3 wa, 0.0 hi, 0.3
> si, 0.0 st
> %Cpu3 : 0.7 us, 2.3 sy, 0.0 ni, 94.1 id, 3.0 wa, 0.0 hi, 0.0
> si, 0.0 st
> KiB Mem : 16427572 total, 173712 free, 9739976 used, 6513884
> buff/cache
> KiB Swap: 6369276 total, 6126736 free, 242540 used. 6224836 avail
> Mem
>
>
> This seems OK to me. It does have 24 drives in 3 ZFS pools at 144TB
> raw storage in total with several SAS HBAs that is pretty much always
> poking the system in some way or the other.
>
>
> There are around 32K interrupts when running @23 MB/s (as seen in
> chrome downloads) with pacing on and about 25K interrupts when running
> @105 MB/s with fq nopacing. Is that normal?
>
>
> Hans-Kristian
>
>
>
> On 26 January 2017 at 20:58, David Lang <david@lang.hm> wrote:
> Is there any CPU bottleneck?
>
> pacing causing this sort of problem makes me thing that the
> CPU either can't keep up or that something (Hz setting type of
> thing) is delaying when the CPU can get used.
>
> It's not clear from the posts if the problem is with sending
> data or receiving data.
>
> David Lang
>
>
> On Thu, 26 Jan 2017, Eric Dumazet wrote:
>
> Nothing jumps on my head.
>
> We use FQ on links varying from 1Gbit to 100Gbit, and
> we have no such
> issues.
>
> You could probably check on the server the TCP various
> infos given by ss
> command
>
>
> ss -temoi dst <remoteip>
>
>
> pacing rate is shown. You might have some issues, but
> it is hard to say.
>
>
> On Thu, 2017-01-26 at 19:55 +0100, Hans-Kristian Bakke
> wrote:
> After some more testing I see that if I
> disable fq pacing the
> performance is restored to the expected
> levels: # for i in eth0 eth1; do tc qdisc
> replace dev $i root fq nopacing;
> done
>
>
> Is this expected behaviour? There is some
> background traffic, but only
> in the sub 100 mbit/s on the switches and
> gateway between the server
> and client.
>
>
> The chain:
> Windows 10 client -> 1000 mbit/s -> switch ->
> 2xgigabit LACP -> switch
> -> 4 x gigabit LACP -> gw (fq_codel on all
> nics) -> 4 x gigabit LACP
> (the same as in) -> switch -> 2 x lacp ->
> server (with misbehaving fq
> pacing)
>
>
>
> On 26 January 2017 at 19:38, Hans-Kristian
> Bakke <hkbakke@gmail.com>
> wrote:
> I can add that this is without BBR,
> just plain old kernel 4.8
> cubic.
>
> On 26 January 2017 at 19:36,
> Hans-Kristian Bakke
> <hkbakke@gmail.com> wrote:
> Another day, another fq issue
> (or user error).
>
>
> I try to do the seeminlig
> simple task of downloading a
> single large file over local
> gigabit LAN from a
> physical server running kernel
> 4.8 and sch_fq on intel
> server NICs.
>
>
> For some reason it wouldn't go
> past around 25 MB/s.
> After having replaced SSL with
> no SSL, replaced apache
> with nginx and verified that
> there is plenty of
> bandwith available between my
> client and the server I
> tried to change qdisc from fq
> to pfifo_fast. It
> instantly shot up to around
> the expected 85-90 MB/s.
> The same happened with
> fq_codel in place of fq.
>
>
> I then checked the statistics
> for fq and the throttled
> counter is increasing
> massively every second (eth0 and
> eth1 is LACPed using Linux
> bonding so both is seen
> here):
>
>
> qdisc fq 8007: root refcnt 2
> limit 10000p flow_limit
> 100p buckets 1024 orphan_mask
> 1023 quantum 3028
> initial_quantum 15140
> refill_delay 40.0ms
> Sent 787131797 bytes 520082
> pkt (dropped 15,
> overlimits 0 requeues 0)
> backlog 98410b 65p requeues 0
> 15 flows (14 inactive, 1
> throttled)
> 0 gc, 2 highprio, 259920
> throttled, 15 flows_plimit
> qdisc fq 8008: root refcnt 2
> limit 10000p flow_limit
> 100p buckets 1024 orphan_mask
> 1023 quantum 3028
> initial_quantum 15140
> refill_delay 40.0ms
> Sent 2533167 bytes 6731 pkt
> (dropped 0, overlimits 0
> requeues 0)
> backlog 0b 0p requeues 0
> 24 flows (24 inactive, 0
> throttled)
> 0 gc, 2 highprio, 397
> throttled
>
>
> Do you have any suggestions?
>
>
> Regards,
> Hans-Kristian
>
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bloat] Excessive throttling with fq
2017-01-26 20:54 ` Eric Dumazet
@ 2017-01-26 20:57 ` Hans-Kristian Bakke
2020-02-19 6:58 ` Alexey Ivanov
0 siblings, 1 reply; 15+ messages in thread
From: Hans-Kristian Bakke @ 2017-01-26 20:57 UTC (permalink / raw)
To: Eric Dumazet; +Cc: David Lang, bloat
[-- Attachment #1: Type: text/plain, Size: 10025 bytes --]
The receiver (as in the nginx proxy in the dumps) is actually running fq
qdisc with BBR on kernel 4.9. Could that explain what you are seeing?
Changing it to cubic does not change the resulting throughput though, and
it also was not involved at all in the Windows 10 -> linux router -> Apache
server tests which also gives the same 23-ish MB/s with pacing.
On 26 January 2017 at 21:54, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> It looks like the receiver is seriously limiting receive window in the
> "pacing" case.
>
> Only after 30 Mbytes are transfered, it finally increases it, from
> 359168 to 1328192
>
> DRS is not working as expected. Again maybe related to HZ value.
>
>
> On Thu, 2017-01-26 at 21:19 +0100, Hans-Kristian Bakke wrote:
> > There are two packet captures from fq with and without pacing here:
> >
> >
> > https://owncloud.proikt.com/index.php/s/KuXIl8h8bSFH1fM
> >
> >
> >
> > The server (with fq pacing/nopacing) is 10.0.5.10 and is running a
> > Apache2 webserver at port tcp port 443. The tcp client is nginx
> > reverse proxy at 10.0.5.13 on the same subnet which again is proxying
> > the connection from the Windows 10 client.
> > - I did try to connect directly to the server with the client (via a
> > linux gateway router) avoiding the nginx proxy and just using plain
> > no-ssl http. That did not change anything.
> > - I also tried stopping the eth0 interface to force the traffic to the
> > eth1 interface in the LACP which changed nothing.
> > - I also pulled each of the cable on the switch to force the traffic
> > to switch between interfaces in the LACP link between the client
> > switch and the server switch.
> >
> >
> > The CPU is a 5-6 year old Intel Xeon X3430 CPU @ 4x2.40GHz on a
> > SuperMicro platform. It is not very loaded and the results are always
> > in the same ballpark with fq pacing on.
> >
> >
> >
> > top - 21:12:38 up 12 days, 11:08, 4 users, load average: 0.56, 0.68,
> > 0.77
> > Tasks: 1344 total, 1 running, 1343 sleeping, 0 stopped, 0 zombie
> > %Cpu0 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0
> > si, 0.0 st
> > %Cpu1 : 0.0 us, 0.3 sy, 0.0 ni, 97.4 id, 2.0 wa, 0.0 hi, 0.3
> > si, 0.0 st
> > %Cpu2 : 0.0 us, 2.0 sy, 0.0 ni, 96.4 id, 1.3 wa, 0.0 hi, 0.3
> > si, 0.0 st
> > %Cpu3 : 0.7 us, 2.3 sy, 0.0 ni, 94.1 id, 3.0 wa, 0.0 hi, 0.0
> > si, 0.0 st
> > KiB Mem : 16427572 total, 173712 free, 9739976 used, 6513884
> > buff/cache
> > KiB Swap: 6369276 total, 6126736 free, 242540 used. 6224836 avail
> > Mem
> >
> >
> > This seems OK to me. It does have 24 drives in 3 ZFS pools at 144TB
> > raw storage in total with several SAS HBAs that is pretty much always
> > poking the system in some way or the other.
> >
> >
> > There are around 32K interrupts when running @23 MB/s (as seen in
> > chrome downloads) with pacing on and about 25K interrupts when running
> > @105 MB/s with fq nopacing. Is that normal?
> >
> >
> > Hans-Kristian
> >
> >
> >
> > On 26 January 2017 at 20:58, David Lang <david@lang.hm> wrote:
> > Is there any CPU bottleneck?
> >
> > pacing causing this sort of problem makes me thing that the
> > CPU either can't keep up or that something (Hz setting type of
> > thing) is delaying when the CPU can get used.
> >
> > It's not clear from the posts if the problem is with sending
> > data or receiving data.
> >
> > David Lang
> >
> >
> > On Thu, 26 Jan 2017, Eric Dumazet wrote:
> >
> > Nothing jumps on my head.
> >
> > We use FQ on links varying from 1Gbit to 100Gbit, and
> > we have no such
> > issues.
> >
> > You could probably check on the server the TCP various
> > infos given by ss
> > command
> >
> >
> > ss -temoi dst <remoteip>
> >
> >
> > pacing rate is shown. You might have some issues, but
> > it is hard to say.
> >
> >
> > On Thu, 2017-01-26 at 19:55 +0100, Hans-Kristian Bakke
> > wrote:
> > After some more testing I see that if I
> > disable fq pacing the
> > performance is restored to the expected
> > levels: # for i in eth0 eth1; do tc qdisc
> > replace dev $i root fq nopacing;
> > done
> >
> >
> > Is this expected behaviour? There is some
> > background traffic, but only
> > in the sub 100 mbit/s on the switches and
> > gateway between the server
> > and client.
> >
> >
> > The chain:
> > Windows 10 client -> 1000 mbit/s -> switch ->
> > 2xgigabit LACP -> switch
> > -> 4 x gigabit LACP -> gw (fq_codel on all
> > nics) -> 4 x gigabit LACP
> > (the same as in) -> switch -> 2 x lacp ->
> > server (with misbehaving fq
> > pacing)
> >
> >
> >
> > On 26 January 2017 at 19:38, Hans-Kristian
> > Bakke <hkbakke@gmail.com>
> > wrote:
> > I can add that this is without BBR,
> > just plain old kernel 4.8
> > cubic.
> >
> > On 26 January 2017 at 19:36,
> > Hans-Kristian Bakke
> > <hkbakke@gmail.com> wrote:
> > Another day, another fq issue
> > (or user error).
> >
> >
> > I try to do the seeminlig
> > simple task of downloading a
> > single large file over local
> > gigabit LAN from a
> > physical server running kernel
> > 4.8 and sch_fq on intel
> > server NICs.
> >
> >
> > For some reason it wouldn't go
> > past around 25 MB/s.
> > After having replaced SSL with
> > no SSL, replaced apache
> > with nginx and verified that
> > there is plenty of
> > bandwith available between my
> > client and the server I
> > tried to change qdisc from fq
> > to pfifo_fast. It
> > instantly shot up to around
> > the expected 85-90 MB/s.
> > The same happened with
> > fq_codel in place of fq.
> >
> >
> > I then checked the statistics
> > for fq and the throttled
> > counter is increasing
> > massively every second (eth0 and
> > eth1 is LACPed using Linux
> > bonding so both is seen
> > here):
> >
> >
> > qdisc fq 8007: root refcnt 2
> > limit 10000p flow_limit
> > 100p buckets 1024 orphan_mask
> > 1023 quantum 3028
> > initial_quantum 15140
> > refill_delay 40.0ms
> > Sent 787131797 bytes 520082
> > pkt (dropped 15,
> > overlimits 0 requeues 0)
> > backlog 98410b 65p requeues 0
> > 15 flows (14 inactive, 1
> > throttled)
> > 0 gc, 2 highprio, 259920
> > throttled, 15 flows_plimit
> > qdisc fq 8008: root refcnt 2
> > limit 10000p flow_limit
> > 100p buckets 1024 orphan_mask
> > 1023 quantum 3028
> > initial_quantum 15140
> > refill_delay 40.0ms
> > Sent 2533167 bytes 6731 pkt
> > (dropped 0, overlimits 0
> > requeues 0)
> > backlog 0b 0p requeues 0
> > 24 flows (24 inactive, 0
> > throttled)
> > 0 gc, 2 highprio, 397
> > throttled
> >
> >
> > Do you have any suggestions?
> >
> >
> > Regards,
> > Hans-Kristian
> >
> >
> >
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >
> >
>
>
>
[-- Attachment #2: Type: text/html, Size: 14390 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bloat] Excessive throttling with fq
2017-01-26 20:46 ` Hans-Kristian Bakke
@ 2017-01-26 21:00 ` Eric Dumazet
[not found] ` <CAD_cGvFXR+Qb9_gnp=k4UttJZnrRRm4i19of7D4v8MK9EjeZ6Q@mail.gmail.com>
0 siblings, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2017-01-26 21:00 UTC (permalink / raw)
To: Hans-Kristian Bakke; +Cc: David Lang, bloat
For some reason, even though this NIC advertises TSO support,
tcpdump clearly shows TSO is not used at all.
Oh wait, maybe TSO is not enabled on the bonding device ?
On Thu, 2017-01-26 at 21:46 +0100, Hans-Kristian Bakke wrote:
> # ethtool -i eth0
> driver: e1000e
> version: 3.2.6-k
> firmware-version: 1.9-0
> expansion-rom-version:
> bus-info: 0000:04:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
>
> # ethtool -k eth0
> Features for eth0:
> rx-checksumming: on
> tx-checksumming: on
> tx-checksum-ipv4: off [fixed]
> tx-checksum-ip-generic: on
> tx-checksum-ipv6: off [fixed]
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: off [fixed]
> tx-tcp-mangleid-segmentation: on
> tx-tcp6-segmentation: on
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: off [fixed]
> receive-hashing: on
> highdma: on [fixed]
> rx-vlan-filter: on [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-gre-csum-segmentation: off [fixed]
> tx-ipxip4-segmentation: off [fixed]
> tx-ipxip6-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> tx-udp_tnl-csum-segmentation: off [fixed]
> tx-gso-partial: off [fixed]
> tx-sctp-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off
> rx-all: off
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> busy-poll: off [fixed]
> hw-tc-offload: off [fixed]
>
>
> # grep HZ /boot/config-4.8.0-2-amd64
> CONFIG_NO_HZ_COMMON=y
> # CONFIG_HZ_PERIODIC is not set
> CONFIG_NO_HZ_IDLE=y
> # CONFIG_NO_HZ_FULL is not set
> # CONFIG_NO_HZ is not set
> # CONFIG_HZ_100 is not set
> CONFIG_HZ_250=y
> # CONFIG_HZ_300 is not set
> # CONFIG_HZ_1000 is not set
> CONFIG_HZ=250
> CONFIG_MACHZ_WDT=m
>
>
>
> On 26 January 2017 at 21:41, Eric Dumazet <eric.dumazet@gmail.com>
> wrote:
>
> Can you post :
>
> ethtool -i eth0
> ethtool -k eth0
>
> grep HZ /boot/config.... (what is the HZ value of your kernel)
>
> I suspect a possible problem with TSO autodefer when/if HZ <
> 1000
>
> Thanks.
>
> On Thu, 2017-01-26 at 21:19 +0100, Hans-Kristian Bakke wrote:
> > There are two packet captures from fq with and without
> pacing here:
> >
> >
> > https://owncloud.proikt.com/index.php/s/KuXIl8h8bSFH1fM
> >
> >
> >
> > The server (with fq pacing/nopacing) is 10.0.5.10 and is
> running a
> > Apache2 webserver at port tcp port 443. The tcp client is
> nginx
> > reverse proxy at 10.0.5.13 on the same subnet which again is
> proxying
> > the connection from the Windows 10 client.
> > - I did try to connect directly to the server with the
> client (via a
> > linux gateway router) avoiding the nginx proxy and just
> using plain
> > no-ssl http. That did not change anything.
> > - I also tried stopping the eth0 interface to force the
> traffic to the
> > eth1 interface in the LACP which changed nothing.
> > - I also pulled each of the cable on the switch to force the
> traffic
> > to switch between interfaces in the LACP link between the
> client
> > switch and the server switch.
> >
> >
> > The CPU is a 5-6 year old Intel Xeon X3430 CPU @ 4x2.40GHz
> on a
> > SuperMicro platform. It is not very loaded and the results
> are always
> > in the same ballpark with fq pacing on.
> >
> >
> >
> > top - 21:12:38 up 12 days, 11:08, 4 users, load average:
> 0.56, 0.68,
> > 0.77
> > Tasks: 1344 total, 1 running, 1343 sleeping, 0
> stopped, 0 zombie
> > %Cpu0 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0
> hi, 0.0
> > si, 0.0 st
> > %Cpu1 : 0.0 us, 0.3 sy, 0.0 ni, 97.4 id, 2.0 wa, 0.0
> hi, 0.3
> > si, 0.0 st
> > %Cpu2 : 0.0 us, 2.0 sy, 0.0 ni, 96.4 id, 1.3 wa, 0.0
> hi, 0.3
> > si, 0.0 st
> > %Cpu3 : 0.7 us, 2.3 sy, 0.0 ni, 94.1 id, 3.0 wa, 0.0
> hi, 0.0
> > si, 0.0 st
> > KiB Mem : 16427572 total, 173712 free, 9739976 used,
> 6513884
> > buff/cache
> > KiB Swap: 6369276 total, 6126736 free, 242540 used.
> 6224836 avail
> > Mem
> >
> >
> > This seems OK to me. It does have 24 drives in 3 ZFS pools
> at 144TB
> > raw storage in total with several SAS HBAs that is pretty
> much always
> > poking the system in some way or the other.
> >
> >
> > There are around 32K interrupts when running @23 MB/s (as
> seen in
> > chrome downloads) with pacing on and about 25K interrupts
> when running
> > @105 MB/s with fq nopacing. Is that normal?
> >
> >
> > Hans-Kristian
> >
> >
> >
> > On 26 January 2017 at 20:58, David Lang <david@lang.hm>
> wrote:
> > Is there any CPU bottleneck?
> >
> > pacing causing this sort of problem makes me thing
> that the
> > CPU either can't keep up or that something (Hz
> setting type of
> > thing) is delaying when the CPU can get used.
> >
> > It's not clear from the posts if the problem is with
> sending
> > data or receiving data.
> >
> > David Lang
> >
> >
> > On Thu, 26 Jan 2017, Eric Dumazet wrote:
> >
> > Nothing jumps on my head.
> >
> > We use FQ on links varying from 1Gbit to
> 100Gbit, and
> > we have no such
> > issues.
> >
> > You could probably check on the server the
> TCP various
> > infos given by ss
> > command
> >
> >
> > ss -temoi dst <remoteip>
> >
> >
> > pacing rate is shown. You might have some
> issues, but
> > it is hard to say.
> >
> >
> > On Thu, 2017-01-26 at 19:55 +0100,
> Hans-Kristian Bakke
> > wrote:
> > After some more testing I see that
> if I
> > disable fq pacing the
> > performance is restored to the
> expected
> > levels: # for i in eth0 eth1; do tc
> qdisc
> > replace dev $i root fq nopacing;
> > done
> >
> >
> > Is this expected behaviour? There is
> some
> > background traffic, but only
> > in the sub 100 mbit/s on the
> switches and
> > gateway between the server
> > and client.
> >
> >
> > The chain:
> > Windows 10 client -> 1000 mbit/s ->
> switch ->
> > 2xgigabit LACP -> switch
> > -> 4 x gigabit LACP -> gw (fq_codel
> on all
> > nics) -> 4 x gigabit LACP
> > (the same as in) -> switch -> 2 x
> lacp ->
> > server (with misbehaving fq
> > pacing)
> >
> >
> >
> > On 26 January 2017 at 19:38,
> Hans-Kristian
> > Bakke <hkbakke@gmail.com>
> > wrote:
> > I can add that this is
> without BBR,
> > just plain old kernel 4.8
> > cubic.
> >
> > On 26 January 2017 at 19:36,
> > Hans-Kristian Bakke
> > <hkbakke@gmail.com> wrote:
> > Another day, another
> fq issue
> > (or user error).
> >
> >
> > I try to do the
> seeminlig
> > simple task of downloading a
> > single large file
> over local
> > gigabit LAN from a
> > physical server
> running kernel
> > 4.8 and sch_fq on intel
> > server NICs.
> >
> >
> > For some reason it
> wouldn't go
> > past around 25 MB/s.
> > After having
> replaced SSL with
> > no SSL, replaced apache
> > with nginx and
> verified that
> > there is plenty of
> > bandwith available
> between my
> > client and the server I
> > tried to change
> qdisc from fq
> > to pfifo_fast. It
> > instantly shot up to
> around
> > the expected 85-90 MB/s.
> > The same happened
> with
> > fq_codel in place of fq.
> >
> >
> > I then checked the
> statistics
> > for fq and the throttled
> > counter is
> increasing
> > massively every second (eth0 and
> > eth1 is LACPed using
> Linux
> > bonding so both is seen
> > here):
> >
> >
> > qdisc fq 8007: root
> refcnt 2
> > limit 10000p flow_limit
> > 100p buckets 1024
> orphan_mask
> > 1023 quantum 3028
> > initial_quantum
> 15140
> > refill_delay 40.0ms
> > Sent 787131797
> bytes 520082
> > pkt (dropped 15,
> > overlimits 0
> requeues 0)
> > backlog 98410b 65p
> requeues 0
> > 15 flows (14
> inactive, 1
> > throttled)
> > 0 gc, 2 highprio,
> 259920
> > throttled, 15 flows_plimit
> > qdisc fq 8008: root
> refcnt 2
> > limit 10000p flow_limit
> > 100p buckets 1024
> orphan_mask
> > 1023 quantum 3028
> > initial_quantum
> 15140
> > refill_delay 40.0ms
> > Sent 2533167 bytes
> 6731 pkt
> > (dropped 0, overlimits 0
> > requeues 0)
> > backlog 0b 0p
> requeues 0
> > 24 flows (24
> inactive, 0
> > throttled)
> > 0 gc, 2 highprio,
> 397
> > throttled
> >
> >
> > Do you have any
> suggestions?
> >
> >
> > Regards,
> > Hans-Kristian
> >
> >
> >
> >
> >
> _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> >
> https://lists.bufferbloat.net/listinfo/bloat
> >
> >
> >
> _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >
> >
>
>
>
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bloat] Excessive throttling with fq
[not found] ` <CAD_cGvFXR+Qb9_gnp=k4UttJZnrRRm4i19of7D4v8MK9EjeZ6Q@mail.gmail.com>
@ 2017-01-26 21:07 ` Eric Dumazet
[not found] ` <CAD_cGvGuCU+R=ddTGTnLF3C8avmJ=UZyAYAkD0FQzd-v6fknPw@mail.gmail.com>
0 siblings, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2017-01-26 21:07 UTC (permalink / raw)
To: Hans-Kristian Bakke; +Cc: David Lang, bloat
On Thu, 2017-01-26 at 22:02 +0100, Hans-Kristian Bakke wrote:
> It seems like it is not:
>
It really should ;)
This is normally the default. Do you know why it is off ?
ethtool -K bond0 tso on
>
> Features for bond0:
> rx-checksumming: off [fixed]
> tx-checksumming: on
> tx-checksum-ipv4: off [fixed]
> tx-checksum-ip-generic: on
> tx-checksum-ipv6: off [fixed]
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [requested on]
> tcp-segmentation-offload: on
> tx-tcp-segmentation: off
> tx-tcp-ecn-segmentation: on
> tx-tcp-mangleid-segmentation: off [requested on]
> tx-tcp6-segmentation: on
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: on
> rx-vlan-filter: on
> vlan-challenged: off [fixed]
> tx-lockless: on [fixed]
> netns-local: on [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: on
> tx-gre-csum-segmentation: on
> tx-ipxip4-segmentation: on
> tx-ipxip6-segmentation: on
> tx-udp_tnl-segmentation: on
> tx-udp_tnl-csum-segmentation: on
> tx-gso-partial: off [fixed]
> tx-sctp-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> busy-poll: off [fixed]
> hw-tc-offload: off [fixed]
>
>
>
> On 26 January 2017 at 22:00, Eric Dumazet <eric.dumazet@gmail.com>
> wrote:
> For some reason, even though this NIC advertises TSO support,
> tcpdump clearly shows TSO is not used at all.
>
> Oh wait, maybe TSO is not enabled on the bonding device ?
>
> On Thu, 2017-01-26 at 21:46 +0100, Hans-Kristian Bakke wrote:
> > # ethtool -i eth0
> > driver: e1000e
> > version: 3.2.6-k
> > firmware-version: 1.9-0
> > expansion-rom-version:
> > bus-info: 0000:04:00.0
> > supports-statistics: yes
> > supports-test: yes
> > supports-eeprom-access: yes
> > supports-register-dump: yes
> > supports-priv-flags: no
> >
> >
> > # ethtool -k eth0
> > Features for eth0:
> > rx-checksumming: on
> > tx-checksumming: on
> > tx-checksum-ipv4: off [fixed]
> > tx-checksum-ip-generic: on
> > tx-checksum-ipv6: off [fixed]
> > tx-checksum-fcoe-crc: off [fixed]
> > tx-checksum-sctp: off [fixed]
> > scatter-gather: on
> > tx-scatter-gather: on
> > tx-scatter-gather-fraglist: off [fixed]
> > tcp-segmentation-offload: on
> > tx-tcp-segmentation: on
> > tx-tcp-ecn-segmentation: off [fixed]
> > tx-tcp-mangleid-segmentation: on
> > tx-tcp6-segmentation: on
> > udp-fragmentation-offload: off [fixed]
> > generic-segmentation-offload: on
> > generic-receive-offload: on
> > large-receive-offload: off [fixed]
> > rx-vlan-offload: on
> > tx-vlan-offload: on
> > ntuple-filters: off [fixed]
> > receive-hashing: on
> > highdma: on [fixed]
> > rx-vlan-filter: on [fixed]
> > vlan-challenged: off [fixed]
> > tx-lockless: off [fixed]
> > netns-local: off [fixed]
> > tx-gso-robust: off [fixed]
> > tx-fcoe-segmentation: off [fixed]
> > tx-gre-segmentation: off [fixed]
> > tx-gre-csum-segmentation: off [fixed]
> > tx-ipxip4-segmentation: off [fixed]
> > tx-ipxip6-segmentation: off [fixed]
> > tx-udp_tnl-segmentation: off [fixed]
> > tx-udp_tnl-csum-segmentation: off [fixed]
> > tx-gso-partial: off [fixed]
> > tx-sctp-segmentation: off [fixed]
> > fcoe-mtu: off [fixed]
> > tx-nocache-copy: off
> > loopback: off [fixed]
> > rx-fcs: off
> > rx-all: off
> > tx-vlan-stag-hw-insert: off [fixed]
> > rx-vlan-stag-hw-parse: off [fixed]
> > rx-vlan-stag-filter: off [fixed]
> > l2-fwd-offload: off [fixed]
> > busy-poll: off [fixed]
> > hw-tc-offload: off [fixed]
> >
> >
> > # grep HZ /boot/config-4.8.0-2-amd64
> > CONFIG_NO_HZ_COMMON=y
> > # CONFIG_HZ_PERIODIC is not set
> > CONFIG_NO_HZ_IDLE=y
> > # CONFIG_NO_HZ_FULL is not set
> > # CONFIG_NO_HZ is not set
> > # CONFIG_HZ_100 is not set
> > CONFIG_HZ_250=y
> > # CONFIG_HZ_300 is not set
> > # CONFIG_HZ_1000 is not set
> > CONFIG_HZ=250
> > CONFIG_MACHZ_WDT=m
> >
> >
> >
> > On 26 January 2017 at 21:41, Eric Dumazet
> <eric.dumazet@gmail.com>
> > wrote:
> >
> > Can you post :
> >
> > ethtool -i eth0
> > ethtool -k eth0
> >
> > grep HZ /boot/config.... (what is the HZ value of
> your kernel)
> >
> > I suspect a possible problem with TSO autodefer
> when/if HZ <
> > 1000
> >
> > Thanks.
> >
> > On Thu, 2017-01-26 at 21:19 +0100, Hans-Kristian
> Bakke wrote:
> > > There are two packet captures from fq with and
> without
> > pacing here:
> > >
> > >
> > >
> https://owncloud.proikt.com/index.php/s/KuXIl8h8bSFH1fM
> > >
> > >
> > >
> > > The server (with fq pacing/nopacing) is 10.0.5.10
> and is
> > running a
> > > Apache2 webserver at port tcp port 443. The tcp
> client is
> > nginx
> > > reverse proxy at 10.0.5.13 on the same subnet
> which again is
> > proxying
> > > the connection from the Windows 10 client.
> > > - I did try to connect directly to the server with
> the
> > client (via a
> > > linux gateway router) avoiding the nginx proxy and
> just
> > using plain
> > > no-ssl http. That did not change anything.
> > > - I also tried stopping the eth0 interface to
> force the
> > traffic to the
> > > eth1 interface in the LACP which changed nothing.
> > > - I also pulled each of the cable on the switch to
> force the
> > traffic
> > > to switch between interfaces in the LACP link
> between the
> > client
> > > switch and the server switch.
> > >
> > >
> > > The CPU is a 5-6 year old Intel Xeon X3430 CPU @
> 4x2.40GHz
> > on a
> > > SuperMicro platform. It is not very loaded and the
> results
> > are always
> > > in the same ballpark with fq pacing on.
> > >
> > >
> > >
> > > top - 21:12:38 up 12 days, 11:08, 4 users, load
> average:
> > 0.56, 0.68,
> > > 0.77
> > > Tasks: 1344 total, 1 running, 1343 sleeping, 0
> > stopped, 0 zombie
> > > %Cpu0 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0
> wa, 0.0
> > hi, 0.0
> > > si, 0.0 st
> > > %Cpu1 : 0.0 us, 0.3 sy, 0.0 ni, 97.4 id, 2.0
> wa, 0.0
> > hi, 0.3
> > > si, 0.0 st
> > > %Cpu2 : 0.0 us, 2.0 sy, 0.0 ni, 96.4 id, 1.3
> wa, 0.0
> > hi, 0.3
> > > si, 0.0 st
> > > %Cpu3 : 0.7 us, 2.3 sy, 0.0 ni, 94.1 id, 3.0
> wa, 0.0
> > hi, 0.0
> > > si, 0.0 st
> > > KiB Mem : 16427572 total, 173712 free, 9739976
> used,
> > 6513884
> > > buff/cache
> > > KiB Swap: 6369276 total, 6126736 free, 242540
> used.
> > 6224836 avail
> > > Mem
> > >
> > >
> > > This seems OK to me. It does have 24 drives in 3
> ZFS pools
> > at 144TB
> > > raw storage in total with several SAS HBAs that is
> pretty
> > much always
> > > poking the system in some way or the other.
> > >
> > >
> > > There are around 32K interrupts when running @23
> MB/s (as
> > seen in
> > > chrome downloads) with pacing on and about 25K
> interrupts
> > when running
> > > @105 MB/s with fq nopacing. Is that normal?
> > >
> > >
> > > Hans-Kristian
> > >
> > >
> > >
> > > On 26 January 2017 at 20:58, David Lang
> <david@lang.hm>
> > wrote:
> > > Is there any CPU bottleneck?
> > >
> > > pacing causing this sort of problem makes
> me thing
> > that the
> > > CPU either can't keep up or that something
> (Hz
> > setting type of
> > > thing) is delaying when the CPU can get
> used.
> > >
> > > It's not clear from the posts if the
> problem is with
> > sending
> > > data or receiving data.
> > >
> > > David Lang
> > >
> > >
> > > On Thu, 26 Jan 2017, Eric Dumazet wrote:
> > >
> > > Nothing jumps on my head.
> > >
> > > We use FQ on links varying from
> 1Gbit to
> > 100Gbit, and
> > > we have no such
> > > issues.
> > >
> > > You could probably check on the
> server the
> > TCP various
> > > infos given by ss
> > > command
> > >
> > >
> > > ss -temoi dst <remoteip>
> > >
> > >
> > > pacing rate is shown. You might
> have some
> > issues, but
> > > it is hard to say.
> > >
> > >
> > > On Thu, 2017-01-26 at 19:55 +0100,
> > Hans-Kristian Bakke
> > > wrote:
> > > After some more testing I
> see that
> > if I
> > > disable fq pacing the
> > > performance is restored to
> the
> > expected
> > > levels: # for i in eth0
> eth1; do tc
> > qdisc
> > > replace dev $i root fq
> nopacing;
> > > done
> > >
> > >
> > > Is this expected
> behaviour? There is
> > some
> > > background traffic, but
> only
> > > in the sub 100 mbit/s on
> the
> > switches and
> > > gateway between the server
> > > and client.
> > >
> > >
> > > The chain:
> > > Windows 10 client -> 1000
> mbit/s ->
> > switch ->
> > > 2xgigabit LACP -> switch
> > > -> 4 x gigabit LACP -> gw
> (fq_codel
> > on all
> > > nics) -> 4 x gigabit LACP
> > > (the same as in) -> switch
> -> 2 x
> > lacp ->
> > > server (with misbehaving
> fq
> > > pacing)
> > >
> > >
> > >
> > > On 26 January 2017 at
> 19:38,
> > Hans-Kristian
> > > Bakke <hkbakke@gmail.com>
> > > wrote:
> > > I can add that
> this is
> > without BBR,
> > > just plain old kernel 4.8
> > > cubic.
> > >
> > > On 26 January 2017
> at 19:36,
> > > Hans-Kristian Bakke
> > >
> <hkbakke@gmail.com> wrote:
> > > Another
> day, another
> > fq issue
> > > (or user error).
> > >
> > >
> > > I try to
> do the
> > seeminlig
> > > simple task of downloading
> a
> > > single
> large file
> > over local
> > > gigabit LAN from a
> > > physical
> server
> > running kernel
> > > 4.8 and sch_fq on intel
> > > server
> NICs.
> > >
> > >
> > > For some
> reason it
> > wouldn't go
> > > past around 25 MB/s.
> > > After
> having
> > replaced SSL with
> > > no SSL, replaced apache
> > > with nginx
> and
> > verified that
> > > there is plenty of
> > > bandwith
> available
> > between my
> > > client and the server I
> > > tried to
> change
> > qdisc from fq
> > > to pfifo_fast. It
> > > instantly
> shot up to
> > around
> > > the expected 85-90 MB/s.
> > > The same
> happened
> > with
> > > fq_codel in place of fq.
> > >
> > >
> > > I then
> checked the
> > statistics
> > > for fq and the throttled
> > > counter is
> > increasing
> > > massively every second
> (eth0 and
> > > eth1 is
> LACPed using
> > Linux
> > > bonding so both is seen
> > > here):
> > >
> > >
> > > qdisc fq
> 8007: root
> > refcnt 2
> > > limit 10000p flow_limit
> > > 100p
> buckets 1024
> > orphan_mask
> > > 1023 quantum 3028
> > >
> initial_quantum
> > 15140
> > > refill_delay 40.0ms
> > > Sent
> 787131797
> > bytes 520082
> > > pkt (dropped 15,
> > > overlimits
> 0
> > requeues 0)
> > > backlog
> 98410b 65p
> > requeues 0
> > > 15 flows
> (14
> > inactive, 1
> > > throttled)
> > > 0 gc, 2
> highprio,
> > 259920
> > > throttled, 15 flows_plimit
> > > qdisc fq
> 8008: root
> > refcnt 2
> > > limit 10000p flow_limit
> > > 100p
> buckets 1024
> > orphan_mask
> > > 1023 quantum 3028
> > >
> initial_quantum
> > 15140
> > > refill_delay 40.0ms
> > > Sent
> 2533167 bytes
> > 6731 pkt
> > > (dropped 0, overlimits 0
> > > requeues
> 0)
> > > backlog
> 0b 0p
> > requeues 0
> > > 24 flows
> (24
> > inactive, 0
> > > throttled)
> > > 0 gc, 2
> highprio,
> > 397
> > > throttled
> > >
> > >
> > > Do you
> have any
> > suggestions?
> > >
> > >
> > > Regards,
> > >
> Hans-Kristian
> > >
> > >
> > >
> > >
> > >
> > _______________________________________________
> > > Bloat mailing list
> > >
> Bloat@lists.bufferbloat.net
> > >
> > https://lists.bufferbloat.net/listinfo/bloat
> > >
> > >
> > >
> > _______________________________________________
> > > Bloat mailing list
> > > Bloat@lists.bufferbloat.net
> > >
> https://lists.bufferbloat.net/listinfo/bloat
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bloat] Excessive throttling with fq
[not found] ` <CAD_cGvGuCU+R=ddTGTnLF3C8avmJ=UZyAYAkD0FQzd-v6fknPw@mail.gmail.com>
@ 2017-01-26 21:33 ` Eric Dumazet
0 siblings, 0 replies; 15+ messages in thread
From: Eric Dumazet @ 2017-01-26 21:33 UTC (permalink / raw)
To: Hans-Kristian Bakke; +Cc: David Lang, bloat
On Thu, 2017-01-26 at 22:20 +0100, Hans-Kristian Bakke wrote:
> Wow, that was it (after seeing your previous mail I disabled and
> reenabled tso and gso on all eth0, eth1 AND bond0 to reset all to the
> same state and it cleared up all the issues.
>
>
> In other words my issue was that my physical nics eth0 and eth1 had
> gso/tso enabled but my bond0 interface gso/tso disabled which
> everything else but fq with pacing did not seem to care about.
>
>
> The reason why is probably from my traffic shaper script in previous
> experiments the last couple of days.
> I actually think my gateway may have the same latent issue for fq with
> pacing as my HTB + fq_codel WAN traffic shaper script is automatically
> disabling tso and gso on the shaped interface, which in my case is
> bond0.12 (bonding AND VLANs) while the underlying physical interfaces
> still have tso and gso enabled as the script does not know that the
> interface happens to be bound to one or more layers of interfaces
> below that.
>
>
>
>
> This is the difference in the ethtool -k output between the
> non-working fq pacing settings and the working version.
>
>
> diff ethtool_k_bond0.txt ethtool_k_bond0-2.txt
> 13c13
> < tx-tcp-segmentation: off
> ---
> > tx-tcp-segmentation: on
> 15c15
> < tx-tcp-mangleid-segmentation: off [requested on]
> ---
> > tx-tcp-mangleid-segmentation: on
>
>
> Thank you for pointing me in the right direction! I don't know is this
> is a "wont-fix" issue because of non-logical user configuration or if
> it should be looked into to be handled better in the future.
>
>
non TSO devices are supported, but we would generally install FQ on the
bonding device, and let TSO enabled on the bonding.
This is because setting timers is expensive, and our design choices for
pacing tried hard to avoid setting timers for every 2 packets sent (as
in 1-MSS packets) ;)
( https://lwn.net/Articles/564978/ )
Of course this does not really matter for slow links (Like 10Mbit or
100Mbit NIC)
>
>
> On 26 January 2017 at 22:07, Eric Dumazet <eric.dumazet@gmail.com>
> wrote:
> On Thu, 2017-01-26 at 22:02 +0100, Hans-Kristian Bakke wrote:
> > It seems like it is not:
> >
>
> It really should ;)
>
> This is normally the default. Do you know why it is off ?
>
> ethtool -K bond0 tso on
>
>
> >
> > Features for bond0:
> > rx-checksumming: off [fixed]
> > tx-checksumming: on
> > tx-checksum-ipv4: off [fixed]
> > tx-checksum-ip-generic: on
> > tx-checksum-ipv6: off [fixed]
> > tx-checksum-fcoe-crc: off [fixed]
> > tx-checksum-sctp: off [fixed]
> > scatter-gather: on
> > tx-scatter-gather: on
> > tx-scatter-gather-fraglist: off [requested on]
> > tcp-segmentation-offload: on
> > tx-tcp-segmentation: off
> > tx-tcp-ecn-segmentation: on
> > tx-tcp-mangleid-segmentation: off [requested on]
> > tx-tcp6-segmentation: on
> > udp-fragmentation-offload: off [fixed]
> > generic-segmentation-offload: on
> > generic-receive-offload: on
> > large-receive-offload: off
> > rx-vlan-offload: on
> > tx-vlan-offload: on
> > ntuple-filters: off [fixed]
> > receive-hashing: off [fixed]
> > highdma: on
> > rx-vlan-filter: on
> > vlan-challenged: off [fixed]
> > tx-lockless: on [fixed]
> > netns-local: on [fixed]
> > tx-gso-robust: off [fixed]
> > tx-fcoe-segmentation: off [fixed]
> > tx-gre-segmentation: on
> > tx-gre-csum-segmentation: on
> > tx-ipxip4-segmentation: on
> > tx-ipxip6-segmentation: on
> > tx-udp_tnl-segmentation: on
> > tx-udp_tnl-csum-segmentation: on
> > tx-gso-partial: off [fixed]
> > tx-sctp-segmentation: off [fixed]
> > fcoe-mtu: off [fixed]
> > tx-nocache-copy: off
> > loopback: off [fixed]
> > rx-fcs: off [fixed]
> > rx-all: off [fixed]
> > tx-vlan-stag-hw-insert: off [fixed]
> > rx-vlan-stag-hw-parse: off [fixed]
> > rx-vlan-stag-filter: off [fixed]
> > l2-fwd-offload: off [fixed]
> > busy-poll: off [fixed]
> > hw-tc-offload: off [fixed]
> >
> >
> >
> > On 26 January 2017 at 22:00, Eric Dumazet
> <eric.dumazet@gmail.com>
> > wrote:
> > For some reason, even though this NIC advertises TSO
> support,
> > tcpdump clearly shows TSO is not used at all.
> >
> > Oh wait, maybe TSO is not enabled on the bonding
> device ?
> >
> > On Thu, 2017-01-26 at 21:46 +0100, Hans-Kristian
> Bakke wrote:
> > > # ethtool -i eth0
> > > driver: e1000e
> > > version: 3.2.6-k
> > > firmware-version: 1.9-0
> > > expansion-rom-version:
> > > bus-info: 0000:04:00.0
> > > supports-statistics: yes
> > > supports-test: yes
> > > supports-eeprom-access: yes
> > > supports-register-dump: yes
> > > supports-priv-flags: no
> > >
> > >
> > > # ethtool -k eth0
> > > Features for eth0:
> > > rx-checksumming: on
> > > tx-checksumming: on
> > > tx-checksum-ipv4: off [fixed]
> > > tx-checksum-ip-generic: on
> > > tx-checksum-ipv6: off [fixed]
> > > tx-checksum-fcoe-crc: off [fixed]
> > > tx-checksum-sctp: off [fixed]
> > > scatter-gather: on
> > > tx-scatter-gather: on
> > > tx-scatter-gather-fraglist: off [fixed]
> > > tcp-segmentation-offload: on
> > > tx-tcp-segmentation: on
> > > tx-tcp-ecn-segmentation: off [fixed]
> > > tx-tcp-mangleid-segmentation: on
> > > tx-tcp6-segmentation: on
> > > udp-fragmentation-offload: off [fixed]
> > > generic-segmentation-offload: on
> > > generic-receive-offload: on
> > > large-receive-offload: off [fixed]
> > > rx-vlan-offload: on
> > > tx-vlan-offload: on
> > > ntuple-filters: off [fixed]
> > > receive-hashing: on
> > > highdma: on [fixed]
> > > rx-vlan-filter: on [fixed]
> > > vlan-challenged: off [fixed]
> > > tx-lockless: off [fixed]
> > > netns-local: off [fixed]
> > > tx-gso-robust: off [fixed]
> > > tx-fcoe-segmentation: off [fixed]
> > > tx-gre-segmentation: off [fixed]
> > > tx-gre-csum-segmentation: off [fixed]
> > > tx-ipxip4-segmentation: off [fixed]
> > > tx-ipxip6-segmentation: off [fixed]
> > > tx-udp_tnl-segmentation: off [fixed]
> > > tx-udp_tnl-csum-segmentation: off [fixed]
> > > tx-gso-partial: off [fixed]
> > > tx-sctp-segmentation: off [fixed]
> > > fcoe-mtu: off [fixed]
> > > tx-nocache-copy: off
> > > loopback: off [fixed]
> > > rx-fcs: off
> > > rx-all: off
> > > tx-vlan-stag-hw-insert: off [fixed]
> > > rx-vlan-stag-hw-parse: off [fixed]
> > > rx-vlan-stag-filter: off [fixed]
> > > l2-fwd-offload: off [fixed]
> > > busy-poll: off [fixed]
> > > hw-tc-offload: off [fixed]
> > >
> > >
> > > # grep HZ /boot/config-4.8.0-2-amd64
> > > CONFIG_NO_HZ_COMMON=y
> > > # CONFIG_HZ_PERIODIC is not set
> > > CONFIG_NO_HZ_IDLE=y
> > > # CONFIG_NO_HZ_FULL is not set
> > > # CONFIG_NO_HZ is not set
> > > # CONFIG_HZ_100 is not set
> > > CONFIG_HZ_250=y
> > > # CONFIG_HZ_300 is not set
> > > # CONFIG_HZ_1000 is not set
> > > CONFIG_HZ=250
> > > CONFIG_MACHZ_WDT=m
> > >
> > >
> > >
> > > On 26 January 2017 at 21:41, Eric Dumazet
> > <eric.dumazet@gmail.com>
> > > wrote:
> > >
> > > Can you post :
> > >
> > > ethtool -i eth0
> > > ethtool -k eth0
> > >
> > > grep HZ /boot/config.... (what is the HZ
> value of
> > your kernel)
> > >
> > > I suspect a possible problem with TSO
> autodefer
> > when/if HZ <
> > > 1000
> > >
> > > Thanks.
> > >
> > > On Thu, 2017-01-26 at 21:19 +0100,
> Hans-Kristian
> > Bakke wrote:
> > > > There are two packet captures from fq
> with and
> > without
> > > pacing here:
> > > >
> > > >
> > > >
> >
> https://owncloud.proikt.com/index.php/s/KuXIl8h8bSFH1fM
> > > >
> > > >
> > > >
> > > > The server (with fq pacing/nopacing) is
> 10.0.5.10
> > and is
> > > running a
> > > > Apache2 webserver at port tcp port 443.
> The tcp
> > client is
> > > nginx
> > > > reverse proxy at 10.0.5.13 on the same
> subnet
> > which again is
> > > proxying
> > > > the connection from the Windows 10
> client.
> > > > - I did try to connect directly to the
> server with
> > the
> > > client (via a
> > > > linux gateway router) avoiding the nginx
> proxy and
> > just
> > > using plain
> > > > no-ssl http. That did not change
> anything.
> > > > - I also tried stopping the eth0
> interface to
> > force the
> > > traffic to the
> > > > eth1 interface in the LACP which changed
> nothing.
> > > > - I also pulled each of the cable on the
> switch to
> > force the
> > > traffic
> > > > to switch between interfaces in the LACP
> link
> > between the
> > > client
> > > > switch and the server switch.
> > > >
> > > >
> > > > The CPU is a 5-6 year old Intel Xeon
> X3430 CPU @
> > 4x2.40GHz
> > > on a
> > > > SuperMicro platform. It is not very
> loaded and the
> > results
> > > are always
> > > > in the same ballpark with fq pacing on.
> > > >
> > > >
> > > >
> > > > top - 21:12:38 up 12 days, 11:08, 4
> users, load
> > average:
> > > 0.56, 0.68,
> > > > 0.77
> > > > Tasks: 1344 total, 1 running, 1343
> sleeping, 0
> > > stopped, 0 zombie
> > > > %Cpu0 : 0.0 us, 1.0 sy, 0.0 ni, 99.0
> id, 0.0
> > wa, 0.0
> > > hi, 0.0
> > > > si, 0.0 st
> > > > %Cpu1 : 0.0 us, 0.3 sy, 0.0 ni, 97.4
> id, 2.0
> > wa, 0.0
> > > hi, 0.3
> > > > si, 0.0 st
> > > > %Cpu2 : 0.0 us, 2.0 sy, 0.0 ni, 96.4
> id, 1.3
> > wa, 0.0
> > > hi, 0.3
> > > > si, 0.0 st
> > > > %Cpu3 : 0.7 us, 2.3 sy, 0.0 ni, 94.1
> id, 3.0
> > wa, 0.0
> > > hi, 0.0
> > > > si, 0.0 st
> > > > KiB Mem : 16427572 total, 173712 free,
> 9739976
> > used,
> > > 6513884
> > > > buff/cache
> > > > KiB Swap: 6369276 total, 6126736
> free, 242540
> > used.
> > > 6224836 avail
> > > > Mem
> > > >
> > > >
> > > > This seems OK to me. It does have 24
> drives in 3
> > ZFS pools
> > > at 144TB
> > > > raw storage in total with several SAS
> HBAs that is
> > pretty
> > > much always
> > > > poking the system in some way or the
> other.
> > > >
> > > >
> > > > There are around 32K interrupts when
> running @23
> > MB/s (as
> > > seen in
> > > > chrome downloads) with pacing on and
> about 25K
> > interrupts
> > > when running
> > > > @105 MB/s with fq nopacing. Is that
> normal?
> > > >
> > > >
> > > > Hans-Kristian
> > > >
> > > >
> > > >
> > > > On 26 January 2017 at 20:58, David Lang
> > <david@lang.hm>
> > > wrote:
> > > > Is there any CPU bottleneck?
> > > >
> > > > pacing causing this sort of
> problem makes
> > me thing
> > > that the
> > > > CPU either can't keep up or that
> something
> > (Hz
> > > setting type of
> > > > thing) is delaying when the CPU
> can get
> > used.
> > > >
> > > > It's not clear from the posts if
> the
> > problem is with
> > > sending
> > > > data or receiving data.
> > > >
> > > > David Lang
> > > >
> > > >
> > > > On Thu, 26 Jan 2017, Eric
> Dumazet wrote:
> > > >
> > > > Nothing jumps on my
> head.
> > > >
> > > > We use FQ on links
> varying from
> > 1Gbit to
> > > 100Gbit, and
> > > > we have no such
> > > > issues.
> > > >
> > > > You could probably check
> on the
> > server the
> > > TCP various
> > > > infos given by ss
> > > > command
> > > >
> > > >
> > > > ss -temoi dst <remoteip>
> > > >
> > > >
> > > > pacing rate is shown.
> You might
> > have some
> > > issues, but
> > > > it is hard to say.
> > > >
> > > >
> > > > On Thu, 2017-01-26 at
> 19:55 +0100,
> > > Hans-Kristian Bakke
> > > > wrote:
> > > > After some more
> testing I
> > see that
> > > if I
> > > > disable fq
> pacing the
> > > > performance is
> restored to
> > the
> > > expected
> > > > levels: # for i
> in eth0
> > eth1; do tc
> > > qdisc
> > > > replace dev $i
> root fq
> > nopacing;
> > > > done
> > > >
> > > >
> > > > Is this expected
> > behaviour? There is
> > > some
> > > > background
> traffic, but
> > only
> > > > in the sub 100
> mbit/s on
> > the
> > > switches and
> > > > gateway between
> the server
> > > > and client.
> > > >
> > > >
> > > > The chain:
> > > > Windows 10
> client -> 1000
> > mbit/s ->
> > > switch ->
> > > > 2xgigabit LACP
> -> switch
> > > > -> 4 x gigabit
> LACP -> gw
> > (fq_codel
> > > on all
> > > > nics) -> 4 x
> gigabit LACP
> > > > (the same as in)
> -> switch
> > -> 2 x
> > > lacp ->
> > > > server (with
> misbehaving
> > fq
> > > > pacing)
> > > >
> > > >
> > > >
> > > > On 26 January
> 2017 at
> > 19:38,
> > > Hans-Kristian
> > > > Bakke
> <hkbakke@gmail.com>
> > > > wrote:
> > > > I can
> add that
> > this is
> > > without BBR,
> > > > just plain old
> kernel 4.8
> > > > cubic.
> > > >
> > > > On 26
> January 2017
> > at 19:36,
> > > > Hans-Kristian
> Bakke
> > > >
> > <hkbakke@gmail.com> wrote:
> > > >
> Another
> > day, another
> > > fq issue
> > > > (or user error).
> > > >
> > > >
> > > >
> I try to
> > do the
> > > seeminlig
> > > > simple task of
> downloading
> > a
> > > >
> single
> > large file
> > > over local
> > > > gigabit LAN
> from a
> > > >
> physical
> > server
> > > running kernel
> > > > 4.8 and sch_fq
> on intel
> > > >
> server
> > NICs.
> > > >
> > > >
> > > >
> For some
> > reason it
> > > wouldn't go
> > > > past around 25
> MB/s.
> > > >
> After
> > having
> > > replaced SSL with
> > > > no SSL, replaced
> apache
> > > >
> with nginx
> > and
> > > verified that
> > > > there is plenty
> of
> > > >
> bandwith
> > available
> > > between my
> > > > client and the
> server I
> > > >
> tried to
> > change
> > > qdisc from fq
> > > > to pfifo_fast.
> It
> > > >
> instantly
> > shot up to
> > > around
> > > > the expected
> 85-90 MB/s.
> > > >
> The same
> > happened
> > > with
> > > > fq_codel in
> place of fq.
> > > >
> > > >
> > > >
> I then
> > checked the
> > > statistics
> > > > for fq and the
> throttled
> > > >
> counter is
> > > increasing
> > > > massively every
> second
> > (eth0 and
> > > >
> eth1 is
> > LACPed using
> > > Linux
> > > > bonding so both
> is seen
> > > >
> here):
> > > >
> > > >
> > > >
> qdisc fq
> > 8007: root
> > > refcnt 2
> > > > limit 10000p
> flow_limit
> > > >
> 100p
> > buckets 1024
> > > orphan_mask
> > > > 1023 quantum
> 3028
> > > >
> > initial_quantum
> > > 15140
> > > > refill_delay
> 40.0ms
> > > >
> Sent
> > 787131797
> > > bytes 520082
> > > > pkt (dropped 15,
> > > >
> overlimits
> > 0
> > > requeues 0)
> > > >
> backlog
> > 98410b 65p
> > > requeues 0
> > > >
> 15 flows
> > (14
> > > inactive, 1
> > > > throttled)
> > > >
> 0 gc, 2
> > highprio,
> > > 259920
> > > > throttled, 15
> flows_plimit
> > > >
> qdisc fq
> > 8008: root
> > > refcnt 2
> > > > limit 10000p
> flow_limit
> > > >
> 100p
> > buckets 1024
> > > orphan_mask
> > > > 1023 quantum
> 3028
> > > >
> > initial_quantum
> > > 15140
> > > > refill_delay
> 40.0ms
> > > >
> Sent
> > 2533167 bytes
> > > 6731 pkt
> > > > (dropped 0,
> overlimits 0
> > > >
> requeues
> > 0)
> > > >
> backlog
> > 0b 0p
> > > requeues 0
> > > >
> 24 flows
> > (24
> > > inactive, 0
> > > > throttled)
> > > >
> 0 gc, 2
> > highprio,
> > > 397
> > > > throttled
> > > >
> > > >
> > > >
> Do you
> > have any
> > > suggestions?
> > > >
> > > >
> > > >
> Regards,
> > > >
> > Hans-Kristian
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> _______________________________________________
> > > > Bloat mailing
> list
> > > >
> > Bloat@lists.bufferbloat.net
> > > >
> > >
> https://lists.bufferbloat.net/listinfo/bloat
> > > >
> > > >
> > > >
> > >
> _______________________________________________
> > > > Bloat mailing list
> > > >
> Bloat@lists.bufferbloat.net
> > > >
> > https://lists.bufferbloat.net/listinfo/bloat
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Bloat] Excessive throttling with fq
2017-01-26 20:57 ` Hans-Kristian Bakke
@ 2020-02-19 6:58 ` Alexey Ivanov
2020-02-19 13:52 ` Neal Cardwell
0 siblings, 1 reply; 15+ messages in thread
From: Alexey Ivanov @ 2020-02-19 6:58 UTC (permalink / raw)
To: bloat
[-- Attachment #1.1: Type: text/plain, Size: 1055 bytes --]
FWIW, we recently noticed a similar issue with our CUBIC senders after
we've enabled FQ on them (4.15 kernel.)
Disabling train detection in hystart did fix the problem:
# echo 2 > /sys/module/tcp_cubic/parameters/hystart_detect
[1]
https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_cubic.c#L76-L77
> The receiver (as in the nginx proxy in the dumps) is actually running fq
> qdisc with BBR on kernel 4.9. Could that explain what you are seeing?
>
> Changing it to cubic does not change the resulting throughput though, and
> it also was not involved at all in the Windows 10 -> linux router -> Apache
> server tests which also gives the same 23-ish MB/s with pacing.
>
>
> On 26 January 2017 at 21:54, Eric Dumazet <eric.dumazet at gmail.com> wrote:
>
>> It looks like the receiver is seriously limiting receive window in the
>> "pacing" case.
>>
>> Only after 30 Mbytes are transfered, it finally increases it, from
>> 359168 to 1328192
>>
>> DRS is not working as expected. Again maybe related to HZ value.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Bloat] Excessive throttling with fq
2020-02-19 6:58 ` Alexey Ivanov
@ 2020-02-19 13:52 ` Neal Cardwell
0 siblings, 0 replies; 15+ messages in thread
From: Neal Cardwell @ 2020-02-19 13:52 UTC (permalink / raw)
To: Alexey Ivanov; +Cc: bloat
On Wed, Feb 19, 2020 at 1:58 AM Alexey Ivanov <savetherbtz@gmail.com> wrote:
>
> FWIW, we recently noticed a similar issue with our CUBIC senders after
> we've enabled FQ on them (4.15 kernel.)
>
> Disabling train detection in hystart did fix the problem:
>
> # echo 2 > /sys/module/tcp_cubic/parameters/hystart_detect
>
> [1]
> https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_cubic.c#L76-L77
Yes, Hystart was not designed to work with pacing, so when fq is used
Hystart ACK train detection needs to be disabled.
Eric Dumazet made some recent fixes (Linux v5.6) to make Hystart
dynamically adapt to the presence of pacing to avoid this issue:
473900a504e5 tcp_cubic: optimize hystart_update()
35821fc2b41c tcp_cubic: remove one conditional from hystart_update()
cff04e2da308 tcp_cubic: switch bictcp_clock() to usec resolution
42f3a8aaae66 tcp_cubic: tweak Hystart detection for short RTT flows
ede656e84658 tcp_cubic: make Hystart aware of pacing
f278b99ca6b2 tcp_cubic: refactor code to perform a divide only when needed
neal
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2020-02-19 13:52 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-01-26 18:36 [Bloat] Excessive throttling with fq Hans-Kristian Bakke
2017-01-26 18:38 ` Hans-Kristian Bakke
2017-01-26 18:55 ` Hans-Kristian Bakke
2017-01-26 19:18 ` Eric Dumazet
2017-01-26 19:58 ` David Lang
2017-01-26 20:19 ` Hans-Kristian Bakke
2017-01-26 20:41 ` Eric Dumazet
2017-01-26 20:46 ` Hans-Kristian Bakke
2017-01-26 21:00 ` Eric Dumazet
[not found] ` <CAD_cGvFXR+Qb9_gnp=k4UttJZnrRRm4i19of7D4v8MK9EjeZ6Q@mail.gmail.com>
2017-01-26 21:07 ` Eric Dumazet
[not found] ` <CAD_cGvGuCU+R=ddTGTnLF3C8avmJ=UZyAYAkD0FQzd-v6fknPw@mail.gmail.com>
2017-01-26 21:33 ` Eric Dumazet
2017-01-26 20:54 ` Eric Dumazet
2017-01-26 20:57 ` Hans-Kristian Bakke
2020-02-19 6:58 ` Alexey Ivanov
2020-02-19 13:52 ` Neal Cardwell
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox