From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 87DC63B29E for ; Thu, 16 Nov 2017 13:40:27 -0500 (EST) Received: by mail-wm0-x22a.google.com with SMTP id z3so2097162wme.5 for ; Thu, 16 Nov 2017 10:40:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=PYI5fIa+kRyT6ofn9kq7TOXfXWtrnkbxb+h+WuNKU4k=; b=fVpiZhOyrRsYS+avtOBXMykzPY+snWR/i2YASfHl801fK/StjMJ2scINiDSokpnwls sO4zJblwNZh7383o4pU81wlfW/iYHq8V5YU7tviACMG+IHFx7I6q/5bXAERonM3ok0Wm PPfnSA2i51l3CFl6gbZwnC0A0xscyrP8EUSwgChkccjHbSo5jfpx8cIF3h9Sg+fenNoG hbeLFraFQdD8hWPUqvOydflYbDbi/MmttOku5Qp16OdglTd9+kuLSYhMsbD1xAJTwSkG 7Uj2/y1W27D7GGnaH8geb0kggiAk43InQADbGoE9/tEmlY3XXzQtZdAD55s2FFN+NeRB u+PQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=PYI5fIa+kRyT6ofn9kq7TOXfXWtrnkbxb+h+WuNKU4k=; b=azcCWVlpqqJOM/KLfZAznwhgQddHmnmm+bcUUIVxdqQGGx7p+LYbXDr3g5PsTzBrVm sq4tY2we+Cmb7sqsvM5R+tC82NAMSJPW2vaQt3MEkyCw3ViW2mwvZuUoFk2xicx93PIa HtSogC3KScm8SbkEfTFbsIGk0j4KEQ8yxFVuF0EWph2+mjHkkBTbU7FNJD0Oq2iXRDd8 4HJ+HV+M+oCrb1Bsn4/0q404srTy1Ewo3RffIxoUrsgB/NKX/TD6JVoVwzFNd0kfD7Cu V5vKtChXjgxUABbK5+ZJjYm3nQgXiEn2EdHpawwg+0yD1M+EX4akUMWcoIidLTkvUyMj zoiQ== X-Gm-Message-State: AJaThX7SafBG1w6cDMBml4ptD4bvGHN46eN7QWpOAEzqf3jCBDSy1gGf 5SuxqglvsB6RmJiveLPO2OY= X-Google-Smtp-Source: AGs4zMZZrAZHL8BQHlSwotS58fxFIG5ZRRZV05fzUr7+nGFt+5CU4DzMZ+o75sb8+8jxa3CgbymFRg== X-Received: by 10.28.139.144 with SMTP id n138mr2327353wmd.78.1510857626021; Thu, 16 Nov 2017 10:40:26 -0800 (PST) Received: from [10.72.0.130] (h-1169.lbcfree.net. [185.99.119.68]) by smtp.gmail.com with ESMTPSA id j21sm1184036wre.86.2017.11.16.10.40.25 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 16 Nov 2017 10:40:25 -0800 (PST) Content-Type: multipart/alternative; boundary="Apple-Mail=_92DEEB40-3884-4FA4-AC67-9FFED33CAFBD" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) From: Pete Heist In-Reply-To: <87shde2nj7.fsf@nemesis.taht.net> Date: Thu, 16 Nov 2017 19:40:24 +0100 Cc: cake@lists.bufferbloat.net Message-Id: <1BED94C0-BF21-44BE-A0B0-22224C24C160@gmail.com> References: <87shde2nj7.fsf@nemesis.taht.net> To: Dave Taht X-Mailer: Apple Mail (2.3124) Subject: Re: [Cake] Cake upstream Planning X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Nov 2017 18:40:27 -0000 --Apple-Mail=_92DEEB40-3884-4FA4-AC67-9FFED33CAFBD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Nov 16, 2017, at 5:31 PM, Dave Taht wrote: >=20 > Pete Heist > writes: >=20 >>>> On Nov 15, 2017, at 9:04 PM, Dave Taht wrote: >>>>=20 >>>>=20 >>>> Dave Taht writes: >>>>=20 >>> https://github.com/ffainelli/bqlmon was a tool for looking at bql = more >>> directly. >>>=20 >>> I had forked it for some reason or another: >>>=20 >>> https://github.com/dtaht/bqlmon >>=20 >> Nice, that does work for me. It=E2=80=99s interesting that there are = four queues for the >> igb driver, 00 - 03, and when I try an rrul_be_nflows test, not all = four queues >> are necessarily used. Once I get >=3D 8 flows in each direction they = usually are >> though. I suppose this is the driver deciding when to start using = another queue >> or not. >=20 > Usually it is selected via a hash. In more than a few cases, however, > the designer of the hardware intended it as a strict priority queue. = In > other cases, it's based on the CPU. >=20 > In all cases such a limited number of queues tends to cause oddities. >=20 > I think it was the mvneta (?) that had the strict priority queue idea = baked > into it, which we ended up disabling entirely and going with just one > hardware queue. I noticed when I went to buy the APU2 that the two lower-end models = (apu2c2 and apu2c0) have I211 NICs instead of a I210. The I211 is a = =E2=80=9Cvalue part=E2=80=9D that among other things has 2 tx and rx = queues per port instead of 4. I wasn=E2=80=99t sure of the real effect = of this when I purchased them, but for an extra few bucks the I210 = seemed worth it. Table 1-6 on page 13: = https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/i2= 10-ethernet-controller-datasheet.pdf >> Cake does seem to visibly reduce the size of the queues. >=20 > I generally observe that TSO/GRO/etc tends to make BQL's queues 3-5 > times larger than they are without those offloads - no way to fix it, > short of doing what cake does to peel those apart. A real nicety of Cake that the world should benefit from. >> For whatever >> terminal/ncurses weirdness reason though, the bar graphs may be = sometimes >> blowing off the top of my 45 row screen, but it doesn=E2=80=99t = entirely ruin the >> experience. >=20 > Maybe that was why I forked it? Looks like you forked it to fix a multi-queue problem. I forked your = fork to add a scaling parameter to fix the bar height. -s 4096 works = well for me. --Apple-Mail=_92DEEB40-3884-4FA4-AC67-9FFED33CAFBD Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On Nov 16, 2017, at 5:31 PM, Dave Taht <dave@taht.net> = wrote:

Pete Heist <peteheist@gmail.com> writes:

  On Nov 15, 2017, at 9:04 PM, Dave = Taht <dave@taht.net>= wrote:


  Dave = Taht <dave@taht.net>= writes:

https://github.com/ffainelli/bqlmon was a tool for = looking at bql more
directly.

I= had forked it for some reason or another:

https://github.com/dtaht/bqlmon

Nice, that does work for me. = It=E2=80=99s interesting that there are four queues for the
igb driver, 00 - 03, and when I try an rrul_be_nflows test, = not all four queues
are necessarily used. Once I get >=3D= 8 flows in each direction they usually are
though. I = suppose this is the driver deciding when to start using another queue
or not.

Usually it is selected via a hash. In = more than a few cases, however,
the designer of the hardware intended it = as a strict priority queue. In
other cases, it's based on the = CPU.

In all cases such a = limited number of queues tends to cause oddities.

I think it was the mvneta (?) that had = the strict priority queue idea baked
into it, which we ended up disabling = entirely and going with just one
hardware queue.

I noticed when I went to buy the APU2 that the two = lower-end models (apu2c2 and apu2c0) have I211 NICs instead of a I210. = The I211 is a =E2=80=9Cvalue part=E2=80=9D that among other things has 2 = tx and rx queues per port instead of 4. I wasn=E2=80=99t sure of the = real effect of this when I purchased them, but for an extra few bucks = the I210 seemed worth it. Table 1-6 on page 13:


Cake does seem to visibly reduce the size of the queues.

I generally observe that TSO/GRO/etc tends to = make BQL's queues 3-5
times larger than they are without those = offloads - no way to fix it,
short of doing what cake does to peel = those apart.

A = real nicety of Cake that the world should benefit from.

For whatever
terminal/ncurses weirdness reason = though, the bar graphs may be sometimes
blowing off the = top of my 45 row screen, but it doesn=E2=80=99t entirely ruin the
experience.

Maybe that was why = I forked it?

Looks like you forked it to fix a multi-queue problem. I = forked your fork to add a scaling parameter to fix the bar height. -s = 4096 works well for me.

= --Apple-Mail=_92DEEB40-3884-4FA4-AC67-9FFED33CAFBD--