<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif"><div class="gmail_default" style="font-family:arial,sans-serif"><font face="verdana, sans-serif">There are two packet captures from fq with and without pacing here:</font></div><div class="gmail_default" style="font-family:arial,sans-serif"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default" style="font-family:arial,sans-serif"><font face="verdana, sans-serif"><a href="https://owncloud.proikt.com/index.php/s/KuXIl8h8bSFH1fM">https://owncloud.proikt.com/index.php/s/KuXIl8h8bSFH1fM</a></font><br></div><div class="gmail_default" style="font-family:arial,sans-serif"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default" style="font-family:arial,sans-serif"><font face="verdana, sans-serif">The server (with fq pacing/nopacing) is 10.0.5.10 and is running a Apache2 webserver at port tcp port 443. The tcp client is nginx reverse proxy at 10.0.5.13 on the same subnet which again is proxying the connection from the Windows 10 client. </font></div><div class="gmail_default" style="font-family:arial,sans-serif"><font face="verdana, sans-serif">- I did try to connect directly to the server with the client (via a linux gateway router) avoiding the nginx proxy and just using plain no-ssl http. That did not change anything. </font></div><div class="gmail_default" style="font-family:arial,sans-serif"><font face="verdana, sans-serif">- I also tried stopping the eth0 interface to force the traffic to the eth1 interface in the LACP which changed nothing.</font></div><div class="gmail_default" style="font-family:arial,sans-serif"><font face="verdana, sans-serif">- I also pulled each of the cable on the switch to force the traffic to switch between interfaces in the LACP link between the client switch and the server switch.</font></div><div class="gmail_default" style="font-family:arial,sans-serif"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default" style="font-family:arial,sans-serif"><span style="font-family:verdana,sans-serif">The CPU is a 5-6 year old Intel Xeon X3430 CPU @ 4x2.40GHz on a SuperMicro platform. It is not very loaded and the results are always in the same ballpark with fq pacing on. </span><br></div><div class="gmail_default" style="font-family:arial,sans-serif"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default" style="font-family:arial,sans-serif"><font face="verdana, sans-serif"><div class="gmail_default" style="font-family:arial,sans-serif">top - 21:12:38 up 12 days, 11:08, 4 users, load average: 0.56, 0.68, 0.77</div><div class="gmail_default" style="font-family:arial,sans-serif">Tasks: 1344 total, 1 running, 1343 sleeping, 0 stopped, 0 zombie</div><div class="gmail_default" style="font-family:arial,sans-serif">%Cpu0 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st</div><div class="gmail_default" style="font-family:arial,sans-serif">%Cpu1 : 0.0 us, 0.3 sy, 0.0 ni, 97.4 id, 2.0 wa, 0.0 hi, 0.3 si, 0.0 st</div><div class="gmail_default" style="font-family:arial,sans-serif">%Cpu2 : 0.0 us, 2.0 sy, 0.0 ni, 96.4 id, 1.3 wa, 0.0 hi, 0.3 si, 0.0 st</div><div class="gmail_default" style="font-family:arial,sans-serif">%Cpu3 : 0.7 us, 2.3 sy, 0.0 ni, 94.1 id, 3.0 wa, 0.0 hi, 0.0 si, 0.0 st</div><div class="gmail_default" style="font-family:arial,sans-serif"><div class="gmail_default">KiB Mem : 16427572 total, 173712 free, 9739976 used, 6513884 buff/cache</div><div class="gmail_default">KiB Swap: 6369276 total, 6126736 free, 242540 used. 6224836 avail Mem</div><div class="gmail_default"><br></div><div class="gmail_default">This seems OK to me. It does have 24 drives in 3 ZFS pools at 144TB raw storage in total with several SAS HBAs that is pretty much always poking the system in some way or the other.</div></div><div class="gmail_default" style="font-family:arial,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,sans-serif">There are around 32K interrupts when running @23 MB/s (as seen in chrome downloads) with pacing on and about 25K interrupts when running @105 MB/s with fq nopacing. Is that normal?</div><div class="gmail_default" style="font-family:arial,sans-serif"><br></div><div>Hans-Kristian</div></font></div><div><font face="verdana, sans-serif"><br></font></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 26 January 2017 at 20:58, David Lang <span dir="ltr"><<a href="mailto:david@lang.hm" target="_blank">david@lang.hm</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Is there any CPU bottleneck?<br>
<br>
pacing causing this sort of problem makes me thing that the CPU either can't keep up or that something (Hz setting type of thing) is delaying when the CPU can get used.<br>
<br>
It's not clear from the posts if the problem is with sending data or receiving data.<span class="gmail-HOEnZb"><font color="#888888"><br>
<br>
David Lang</font></span><div class="gmail-HOEnZb"><div class="gmail-h5"><br>
<br>
On Thu, 26 Jan 2017, Eric Dumazet wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Nothing jumps on my head.<br>
<br>
We use FQ on links varying from 1Gbit to 100Gbit, and we have no such<br>
issues.<br>
<br>
You could probably check on the server the TCP various infos given by ss<br>
command<br>
<br>
<br>
ss -temoi dst <remoteip><br>
<br>
<br>
pacing rate is shown. You might have some issues, but it is hard to say.<br>
<br>
<br>
On Thu, 2017-01-26 at 19:55 +0100, Hans-Kristian Bakke wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
After some more testing I see that if I disable fq pacing the<br>
performance is restored to the expected levels: # for i in eth0 eth1; do tc qdisc replace dev $i root fq nopacing;<br>
done<br>
<br>
<br>
Is this expected behaviour? There is some background traffic, but only<br>
in the sub 100 mbit/s on the switches and gateway between the server<br>
and client.<br>
<br>
<br>
The chain:<br>
Windows 10 client -> 1000 mbit/s -> switch -> 2xgigabit LACP -> switch<br>
-> 4 x gigabit LACP -> gw (fq_codel on all nics) -> 4 x gigabit LACP<br>
(the same as in) -> switch -> 2 x lacp -> server (with misbehaving fq<br>
pacing)<br>
<br>
<br>
<br>
On 26 January 2017 at 19:38, Hans-Kristian Bakke <<a href="mailto:hkbakke@gmail.com" target="_blank">hkbakke@gmail.com</a>><br>
wrote:<br>
I can add that this is without BBR, just plain old kernel 4.8<br>
cubic.<br>
<br>
On 26 January 2017 at 19:36, Hans-Kristian Bakke<br>
<<a href="mailto:hkbakke@gmail.com" target="_blank">hkbakke@gmail.com</a>> wrote:<br>
Another day, another fq issue (or user error).<br>
<br>
<br>
I try to do the seeminlig simple task of downloading a<br>
single large file over local gigabit LAN from a<br>
physical server running kernel 4.8 and sch_fq on intel<br>
server NICs.<br>
<br>
<br>
For some reason it wouldn't go past around 25 MB/s.<br>
After having replaced SSL with no SSL, replaced apache<br>
with nginx and verified that there is plenty of<br>
bandwith available between my client and the server I<br>
tried to change qdisc from fq to pfifo_fast. It<br>
instantly shot up to around the expected 85-90 MB/s.<br>
The same happened with fq_codel in place of fq.<br>
<br>
<br>
I then checked the statistics for fq and the throttled<br>
counter is increasing massively every second (eth0 and<br>
eth1 is LACPed using Linux bonding so both is seen<br>
here):<br>
<br>
<br>
qdisc fq 8007: root refcnt 2 limit 10000p flow_limit<br>
100p buckets 1024 orphan_mask 1023 quantum 3028<br>
initial_quantum 15140 refill_delay 40.0ms<br>
Sent 787131797 bytes 520082 pkt (dropped 15,<br>
overlimits 0 requeues 0)<br>
backlog 98410b 65p requeues 0<br>
15 flows (14 inactive, 1 throttled)<br>
0 gc, 2 highprio, 259920 throttled, 15 flows_plimit<br>
qdisc fq 8008: root refcnt 2 limit 10000p flow_limit<br>
100p buckets 1024 orphan_mask 1023 quantum 3028<br>
initial_quantum 15140 refill_delay 40.0ms<br>
Sent 2533167 bytes 6731 pkt (dropped 0, overlimits 0<br>
requeues 0)<br>
backlog 0b 0p requeues 0<br>
24 flows (24 inactive, 0 throttled)<br>
0 gc, 2 highprio, 397 throttled<br>
<br>
<br>
Do you have any suggestions?<br>
<br>
<br>
Regards,<br>
Hans-Kristian<br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Bloat mailing list<br>
<a href="mailto:Bloat@lists.bufferbloat.net" target="_blank">Bloat@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/bloat" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/<wbr>listinfo/bloat</a><br>
</blockquote>
<br>
<br>
______________________________<wbr>_________________<br>
Bloat mailing list<br>
<a href="mailto:Bloat@lists.bufferbloat.net" target="_blank">Bloat@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/bloat" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/<wbr>listinfo/bloat</a><br>
</blockquote>
</div></div></blockquote></div><br></div></div>