From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ot0-x22a.google.com (mail-ot0-x22a.google.com [IPv6:2607:f8b0:4003:c0f::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id EAF313B2A3 for ; Fri, 27 Jan 2017 14:56:09 -0500 (EST) Received: by mail-ot0-x22a.google.com with SMTP id f9so205189236otd.1 for ; Fri, 27 Jan 2017 11:56:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=Eyn/VxS0XBxYOJyuhta0aYtI+0CUSYnz3YujdiedTtY=; b=IUHztIzwDg/QkaSlvP4EUX98hXAaifCIRg/t7+PijsXwm3veSUmWtnP1uxs4Pjn55c MeVOBoZu/0xy1jr/eCNvMxJbr/y+gCnKLYUQqU8yKFr0cxpZ1MZlglOVahuK2+cNNW8A 0H/bq6KbGMT+DHZRmWqq+1mLfUa/xQ1BzszAoZXsnQjhZDmYYeUhTcjwRQcQbox8hvu8 ZDHnwkUZQeTBdvD6JfjaqdIQ/rsSx5E7dK4d7MtRieiqOuDt9MD8UVKl2U0P/d41QUYZ Bee/NA01Lp73cxj3GMTi4UrHFfKKhBnE6NMVU6xXMQOan0G3vEWCmjQRWdKa2vRZkg67 F6og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=Eyn/VxS0XBxYOJyuhta0aYtI+0CUSYnz3YujdiedTtY=; b=Cwb4YEbK4AShMrLPUonW668HBaMlAbjjHK9A7pebKXK2Xcl8eBSICHnFLDeC3Me6nR comrBEmcp0HtPCRUUGvUqtiol7ODLn64HyIlnsHFIu8flssn4Up3njvHVed8k1fg4Z4o SVRwRvXqSdGGSMyoHP/jOrN+TWnEtcPrnUqe5CgvKSfxMtgfU0ezsiNSj7JLrfULRlk5 On6TygRZq7SjjgZUATCSu0LOl1RY6F5vn85hsq4uJ5LQ7TdDKR2rkYZJHr/wtI9Thtaw fCIRv8sE7I9dITL0V6jNvN6D6aIFhrLyRMvrm+zZD+T6vcnytsuuj4A8dZk3Ura837Xn qlDA== X-Gm-Message-State: AIkVDXIwW1yMyr3MDERBoooujttcG5rTYa4cfToKKjRRkZzT8GPFENfwcAgqUWsWhkanYo2zVHYdcSSQsYmd8A== X-Received: by 10.157.43.198 with SMTP id u64mr5608390ota.80.1485546968255; Fri, 27 Jan 2017 11:56:08 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.1.21 with HTTP; Fri, 27 Jan 2017 11:56:06 -0800 (PST) In-Reply-To: References: <0496946b-827a-8527-643d-0b186f52e192@taht.net> From: Hans-Kristian Bakke Date: Fri, 27 Jan 2017 20:56:06 +0100 Message-ID: To: bloat Content-Type: multipart/alternative; boundary=001a113ac1c259b474054718dbde Subject: [Bloat] Fwd: Recommendations for fq_codel and tso/gso in 2017 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jan 2017 19:56:10 -0000 --001a113ac1c259b474054718dbde Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thank you for answering! On 27 January 2017 at 08:55, Dave T=C3=A4ht wrote: > > > On 1/26/17 11:21 PM, Hans-Kristian Bakke wrote: > > Hi > > > > After having had some issues with inconcistent tso/gso configuration > > causing performance issues for sch_fq with pacing in one of my systems, > > I wonder if is it still recommended to disable gso/tso for interfaces > > used with fq_codel qdiscs and shaping using HTB etc. > > At lower bandwidths gro can do terrible things. Say you have a 1Mbit > uplink, and IW10. (At least one device (mvneta) will synthesise 64k of > gro packets) > > a single IW10 burst from one flow injects 130ms of latency. > > > > > If there is a trade off, at which bandwith does it generally make more > > sense to enable tso/gso than to have it disabled when doing HTB shaped > > fq_codel qdiscs? > > I stopped caring about tuning params at > 40Mbit. < 10 gbit, or rather, > trying get below 200usec of jitter|latency. (Others care) > > And: My expectation was generally that people would ignore our > recommendations on disabling offloads! > > Yes, we should revise the sample sqm code and recommendations for a post > gigabit era to not bother with changing network offloads. Were you > modifying the old debloat script? > =E2=80=8BI just picked it up from just about any bufferbloat script or intr= oduction =E2=80=8BI have seen in the last 4 years. In addition it seemed to bring the bandwith accuracy of the shaped stream a little bit closer to the bandwith I actually configured in HTB in my own testing, which, if I remember correctly, was then done on a symmetrical link that was shaped to around 25 mbit/s, so I just took it for granted. However, the fq pacing issue I had when I had a bond interface with tso and gso disabled on top of physical nics with tso and gso enabled, made me think that disabling tso and gso perhaps is not really expected behaviour for new implentations in the linux network stack. Perhaps it works nicely for my shaping needs, but also gives me other not so obvious issues in other ways. > TBF & sch_Cake do peeling of gro/tso/gso back into packets, and then > interleave their scheduling, so GRO is both helpful (transiting the > stack faster) and harmless, at all bandwidths. > > HTB doesn't peel. We just ripped out hsfc for sqm-scripts (too buggy), > alsp. Leaving: tbf + fq_codel, htb+fq_codel, and cake models there. > > ... > > Cake is coming along nicely. I'd love a test in your 2Gbit bonding > scenario, particularly in a per host fairness test, at line or shaped > rates. We recently got cake working well with nat. > > =E2=80=8BIs this something I can do for you? This is a system in production= . Non-critical enough to play with some qdiscs and generate some bandwith usage, but still in production=E2=80=8B. It is not really possible for me t= o remove all other traffic and factors that may interfere with the results (or is a real life scenario perhaps the point?). But running a few scripts is no problem if that is what is required! > http://blog.cerowrt.org/flent/steam/down_working.svg (ignore the latency > figure, the 6 flows were to spots all over the world) > > > Regards, > > Hans-Kristian > > > > > > _______________________________________________ > > Bloat mailing list > > Bloat@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat > > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > --001a113ac1c259b474054718dbde Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thank you for answ= ering!

On 27 January 201= 7 at 08:55, Dave T=C3=A4ht <dave= @taht.net> wrote= :


On 1/26/17 11:21 PM, Hans-Kristian Bakke wrote:
> Hi
>
> After having had some issues with inconcistent tso/gso configuration > causing performance issues for sch_fq with pacing in one of my systems= ,
> I wonder if is it still recommended to disable gso/tso for interfaces<= br> > used with fq_codel qdiscs and shaping using HTB etc.

At lower bandwidths gro can do terrible things. Say you have a 1Mbit=
uplink, and IW10. (At least one device (mvneta) will synthesise 64k of
gro packets)

a single IW10 burst from one flow injects 130ms of latency.

>
> If there is a trade off, at which bandwith does it generally make more=
> sense to enable tso/gso than to have it disabled when doing HTB shaped=
> fq_codel qdiscs?

I stopped caring about tuning params at > 40Mbit. < 10 gbit, o= r rather,
trying get below 200usec of jitter|latency. (Others care)

And: My expectation was generally that people would ignore our
recommendations on disabling offloads!

Yes, we should revise the sample sqm code and recommendations for a post gigabit era to not bother with changing network offloads. Were you
modifying the old debloat script?

=E2=80=8BI just picked it u= p from just about any bufferbloat script or introduction =E2=80=8BI have se= en in the last 4 years.=C2=A0
In addition it seemed to bring the bandwith accuracy of the shaped st= ream a little bit closer to the bandwith I actually configured in HTB in my= own testing, which, if I remember correctly, was then done on a symmetrica= l link that was shaped to around 25 mbit/s, so I just took it for granted.= =C2=A0

However, the fq pacing issue I had w= hen I had a bond interface with tso and gso disabled on top of physical nic= s with tso and gso enabled, made me think that disabling tso and gso perhap= s is not really expected behaviour for new implentations in the linux netwo= rk stack. Perhaps it works nicely for my shaping needs, but also gives me o= ther not so obvious issues in other ways.


TBF & sch_Cake do peeling of gro/tso/gso back into packets, and then interleave their scheduling, so GRO is both helpful (transiting the
stack faster) and harmless, at all bandwidths.

HTB doesn't peel. We just ripped out hsfc for sqm-scripts (too buggy),<= br> alsp. Leaving: tbf + fq_codel, htb+fq_codel, and cake models there.

...

Cake is coming along nicely. I'd love a test in your 2Gbit bonding
scenario, particularly in a per host fairness test, at line or shaped
rates. We recently got cake working well with nat.


=E2=80=8BIs this something I can do for you? This is a syste= m in production. Non-critical enough to play with some qdiscs and generate = some bandwith usage, but still in production=E2=80=8B. It is not really pos= sible for me to remove all other traffic and factors that may interfere wit= h the results (or is a real life scenario perhaps the point?). But running = a few scripts is no problem if that is what is required!

=C2=A0
http://blog.cerowrt.org/flent/steam/down_wo= rking.svg (ignore the latency
figure, the 6 flows were to spots all over the world)

> Regards,
> Hans-Kristian
>
>
> _______________________________________________
> Bloat mailing list
> Bloat= @lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
_______________________________________________
Bloat mailing list
Bloat@list= s.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat


--001a113ac1c259b474054718dbde--