From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-x229.google.com (mail-oi0-x229.google.com [IPv6:2607:f8b0:4003:c06::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 220143B2A4 for ; Thu, 26 Jan 2017 13:55:19 -0500 (EST) Received: by mail-oi0-x229.google.com with SMTP id w204so143968032oiw.0 for ; Thu, 26 Jan 2017 10:55:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=gD+Bd83/CDEBWkYFApfl3470sOylnAqqEy39ezBY+i8=; b=Sggr8HLAJoo7lO4ZIz8t2qywUDiT9XOwbrv8ItY6nxvwVHhNw6rTKZH7KmgtuAAt9N eLWzzpRupgLbPcMfU/MgxPV6DMYdy4ZVIa6A7FVDNQlb9Xxr9yodJlZfQuQpd5rzad+H wvaFUV2YIs4meS+d90Yv0M1CFpVQ5AttILrFCb/qzs2VyKIp8MS53xEN3R668rkeYeHo uxdcH3DBawkzZ8FyzqE3WQM4nD9vRIIc7K4mrz2cWlGSnqcbbaUi+tsT5lpJj2qBkX0/ l3Zr143C4Fg6zxdofDtHsh4bktHViD8INFy5LL+yhXDUsqrS7d/RYqf3dh7kDTwjEWo3 uGBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=gD+Bd83/CDEBWkYFApfl3470sOylnAqqEy39ezBY+i8=; b=hGl8RPewbWYsbHeHLRFTwVRRtbWaocU4zTXqtRX1mFr3IJ+BOfJe/h4ANwEYHwUvBV thI/EEl9hYxvKtTYbv8dggxXAiIEyRBDI6kG9t8BuHth5r2RVn2cWbQYdPRFoz2jn0R2 11bvq9SlixNwYrPv3VHOH4ATG1yUdC7zP8OW19Ai+j+LOQ3k34jFXU27k9Vp1jLAQpRZ HrNfZF3LgkKT85ziVLo3YlU6z5MbrrdvESYtcCidvzWwMo4VdFV9mbC8YZprXGFchLWJ cfUqTcAt+X+hEcA8IqNWHIYQy2lMHZL9cuygN5LRgJVlZLdWIiPLRmE7NTTnQ+XbIHvK 8DoQ== X-Gm-Message-State: AIkVDXILR7f+9y1KPPB7y5jXlunwZJ0TzeBhI9OGGQ8OfgO5kyNeW8IejkDcmaVxp5+1CWp+sRpHK1BWJYWIoA== X-Received: by 10.202.207.211 with SMTP id f202mr2843434oig.162.1485456918364; Thu, 26 Jan 2017 10:55:18 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.1.21 with HTTP; Thu, 26 Jan 2017 10:55:18 -0800 (PST) In-Reply-To: References: From: Hans-Kristian Bakke Date: Thu, 26 Jan 2017 19:55:18 +0100 Message-ID: To: bloat Content-Type: multipart/alternative; boundary=001a113dea42f2c91b054703e374 Subject: Re: [Bloat] Excessive throttling with fq X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Jan 2017 18:55:19 -0000 --001a113dea42f2c91b054703e374 Content-Type: text/plain; charset=UTF-8 After some more testing I see that if I disable fq pacing the performance is restored to the expected levels: # for i in eth0 eth1; do tc qdisc replace dev $i root fq nopacing; done Is this expected behaviour? There is some background traffic, but only in the sub 100 mbit/s on the switches and gateway between the server and client. The chain: Windows 10 client -> 1000 mbit/s -> switch -> 2xgigabit LACP -> switch -> 4 x gigabit LACP -> gw (fq_codel on all nics) -> 4 x gigabit LACP (the same as in) -> switch -> 2 x lacp -> server (with misbehaving fq pacing) On 26 January 2017 at 19:38, Hans-Kristian Bakke wrote: > I can add that this is without BBR, just plain old kernel 4.8 cubic. > > On 26 January 2017 at 19:36, Hans-Kristian Bakke > wrote: > >> Another day, another fq issue (or user error). >> >> I try to do the seeminlig simple task of downloading a single large file >> over local gigabit LAN from a physical server running kernel 4.8 and >> sch_fq on intel server NICs. >> >> For some reason it wouldn't go past around 25 MB/s. After having replaced >> SSL with no SSL, replaced apache with nginx and verified that there is >> plenty of bandwith available between my client and the server I tried to >> change qdisc from fq to pfifo_fast. It instantly shot up to around the >> expected 85-90 MB/s. The same happened with fq_codel in place of fq. >> >> I then checked the statistics for fq and the throttled counter is >> increasing massively every second (eth0 and eth1 is LACPed using Linux >> bonding so both is seen here): >> >> qdisc fq 8007: root refcnt 2 limit 10000p flow_limit 100p buckets 1024 >> orphan_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms >> Sent 787131797 bytes 520082 pkt (dropped 15, overlimits 0 requeues 0) >> backlog 98410b 65p requeues 0 >> 15 flows (14 inactive, 1 throttled) >> 0 gc, 2 highprio, 259920 throttled, 15 flows_plimit >> qdisc fq 8008: root refcnt 2 limit 10000p flow_limit 100p buckets 1024 >> orphan_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms >> Sent 2533167 bytes 6731 pkt (dropped 0, overlimits 0 requeues 0) >> backlog 0b 0p requeues 0 >> 24 flows (24 inactive, 0 throttled) >> 0 gc, 2 highprio, 397 throttled >> >> Do you have any suggestions? >> >> Regards, >> Hans-Kristian >> > > --001a113dea42f2c91b054703e374 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
After some more testing I see that if I disable fq pacing the p= erformance is restored to the expected levels:=C2=A0
# for i in eth0 eth1; = do tc qdisc replace dev $i root fq nopacing; done

Is this expected beh= aviour? There is some background traffic, but only in the sub 100 mbit/s on= the switches and gateway between the server and client.

The chain:
Wi= ndows 10 client -> 1000 mbit/s -> switch -> 2xgigabit LACP -> s= witch -> 4 x gigabit LACP -> gw (fq_codel on all nics) -> 4 x giga= bit LACP (the same as in) -> switch -> 2 x lacp -> server (with mi= sbehaving fq pacing)


On 26 January 2017 at 19:38, Hans-Kristian Bakke <hkba= kke@gmail.com> wrote:
I can add that this is without BBR, just plain old kernel 4.8 cubic.=

On 26 January 2017 at 19:36, Hans-Kristi= an Bakke <hkbakke@gmail.com> wrote:
Another day, another fq issue (or user error).
=

<= /div>
= I try to do the seeminlig simple task of downloading a single large file ov= er local gigabit =C2=A0LAN from a physical server running kernel 4.8 and sc= h_fq on intel server NICs.

For some reason it wouldn't go past aro= und 25 MB/s. After having replaced SSL with no SSL, replaced apache with ng= inx and verified that there is plenty of bandwith available between my clie= nt and the server I tried to change qdisc from fq to pfifo_fast. It instant= ly shot up to around the expected 85-90 MB/s. The same happened with fq_cod= el in place of fq.

I then checked the statistics for fq and the thrott= led counter is increasing massively every second (eth0 and eth1 is LACPed u= sing Linux bonding so both is seen here):

qdisc fq 8007: root refcnt 2 limit 10000p flow_limit 100p buckets 1024 orp= han_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms
<= div class=3D"gmail_default">=C2=A0Sent 787131797 bytes 520082 pkt (dropped = 15, overlimits 0 requeues 0)
=C2=A0backlo= g 98410b 65p requeues 0
=C2=A0 15 flows (= 14 inactive, 1 throttled)
=C2=A0 0 gc, 2 = highprio, 259920 throttled, 15 flows_plimit
qdisc fq 8008: root refcnt 2 limit 10000p flow_limit 100p buckets 1024 o= rphan_mask 1023 quantum 3028 initial_quantum 15140 refill_delay 40.0ms
=C2=A0Sent 2533167 bytes 6731 pkt (dropped 0,= overlimits 0 requeues 0)
=C2=A0backlog 0= b 0p requeues 0
=C2=A0 24 flows (24 inact= ive, 0 throttled)
=C2=A0 0 gc, 2 highprio= , 397 throttled

Do you have any suggestions?
=

Regards,
Hans-Kristian


--001a113dea42f2c91b054703e374--