From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-x234.google.com (mail-wr0-x234.google.com [IPv6:2a00:1450:400c:c0c::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 602383B2A4 for ; Fri, 24 Nov 2017 08:49:56 -0500 (EST) Received: by mail-wr0-x234.google.com with SMTP id z75so19082675wrc.5 for ; Fri, 24 Nov 2017 05:49:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=ETYdF+WgMJT4KK79u08mAfUR/25XLYDm+wR81jSFbxE=; b=kWBnQKj4v8nbwQMWHqUw6T6LSdlHDjYVUhV0TrG0rYDq8kwxnTF5xaoqeGStC6sWoN TFMNa3xNx4UU/0I6X5TKL34o69FbleY07h+fv/RcUzdILp0iu/CHwopO4JzMUoEUpMpA DK4/PWrjQVghgEk6JWoZcAJIKia+T4WETMW1l5VSNFbMwwrrDs+OdxfxwJVUC/va1AQV fIN1WyR3v6nP2sRaE5CpmTCoQDt+54ezk9CakRQcHrtIi5vWmRLSCMlpmIL2G2rwEVts 3NOXb/xN1nPx1+j2+C3XGKt3QtXanEV1qnOIEbl2D7gpV2Y7KKuoAJPdWJ7NFEuccQvZ Y4tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=ETYdF+WgMJT4KK79u08mAfUR/25XLYDm+wR81jSFbxE=; b=sX0qYY7VKUEVjkFnL32hPLPlsK3xL4pl2yWW51oErI0iI1g1uzR1WJfmqbJ2gp6qaH XNCe9/5SL2RAbMCfNXNUs1IfHdE6WiqDNQkJivmb3w0x/mcipLXwk/gmraQr4uhDlXfn dLD4REH5Q6aYuAOqP6BKfd0JecNzsF1xBGpzOouCvdEtM+5Kiq2p2G9O1ph86tsn9sOs cGI2bHZadXFbKgGmwyaQsjqGOHlPBNgO5KDEfXv7VzPdl14aam1y3D2XBkO9QahRMEcg qbRoCubQttzgp1NS3+A1yb9Zzei8AGD86kQCewKGWyMlARUXbpa4WAqm1j5WQLv73rDx qbtg== X-Gm-Message-State: AJaThX7V5RhuqIadTtlbejNjs1O4LjNITxMxOAWxiTnhiuYrAyT9DQvb wBJtI3U3XdzS4rLvtJImCY74XWHU X-Google-Smtp-Source: AGs4zMYEe1N27tr48YSBGnX7RRrAn3tNnsBdSrU2WcHiBciFC9e20U4gYmnIvzpJb+iVefzdcIVodg== X-Received: by 10.223.153.100 with SMTP id x91mr23846366wrb.189.1511531395238; Fri, 24 Nov 2017 05:49:55 -0800 (PST) Received: from [10.72.0.130] (h-1169.lbcfree.net. [185.99.119.68]) by smtp.gmail.com with ESMTPSA id e124sm4166408wmg.34.2017.11.24.05.49.54 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 24 Nov 2017 05:49:54 -0800 (PST) Content-Type: multipart/alternative; boundary="Apple-Mail=_51A42CF9-B1AF-4980-9380-502C47A01791" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) From: Pete Heist In-Reply-To: <87609zapmt.fsf@toke.dk> Date: Fri, 24 Nov 2017 14:49:53 +0100 Cc: =?utf-8?Q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , Cake List Message-Id: <0D339F2F-F2CB-4800-84B9-E7321AE4D15E@gmail.com> References: <71FB183D-F848-4513-A6F6-D03FD0F10769@gmail.com> <52C2B216-220C-4C17-882C-9994867E86BB@gmail.com> <87tvxlvsex.fsf@nemesis.taht.net> <87609zapmt.fsf@toke.dk> To: Dave Taht X-Mailer: Apple Mail (2.3124) Subject: Re: [Cake] lan keyword affects host fairness X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Nov 2017 13:49:56 -0000 --Apple-Mail=_51A42CF9-B1AF-4980-9380-502C47A01791 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Nov 24, 2017, at 12:21 PM, Toke H=C3=B8iland-J=C3=B8rgensen = wrote: >=20 > Dave Taht writes: >=20 >> Pete Heist writes: >>=20 >>> On Nov 23, 2017, at 10:44 AM, Jonathan Morton = wrote: >>>=20 >>> This is most likely an interaction of the AQM with Linux' = scheduling >>> latency. >>>=20 >>> At the 'lan' setting, the time comstants are similar in magnitude = to the >>> delays induced by Linux itself, so congestion might be signalled >>> prematurely. The flows will then become sparse and total = throughput reduced, >>> leaving little or no back-pressure for the fairness logic to work = against. >>=20 >> Agreed.=20 >>=20 >> man page add: >>=20 >> At the 'lan' setting(1ms), the time constants are similar in = magnitude >> to the jitter in the Linux kernel itself, so congestion might be >> signalled prematurely. The flows will then become sparse and total >> throughput reduced, leaving little or no back-pressure for the = fairness >> logic to work against. Use the "metro" setting for local lans unless = you >> have a custom kernel. >=20 > Erm, doesn't this make the 'lan' keyword pretty much useless? So why = not > just remove it? Or redefine it to something that actually works? 3ms? Removing the bandwidth keywords altogether and going back to = fq_codel=E2=80=99s specification of target and interval would be my = personal preference (unless we can figure out how to make the keywords = work well with one another in all cases). If we want to keep them, I=E2=80=99ll look for more holes where things = might not behave as people expect so we can further clear up the docs. Also, note that even at =E2=80=98metro=E2=80=99, fairness is better, but = still not fully fair on my hardware: = https://docs.google.com/spreadsheets/d/1SMXWw2fLfmBRU622urfdvA_Ujsuf_KQ4P3= uyOH1skOM/edit#gid=3D2072687073 = With 950mbit rate limiting, it=E2=80=99s pretty good at 1.16:1. But with = bql it=E2=80=99s still ~2:1. At what rtt will fairness actually work? = That may depend on hardware, OS version, number of flows, whether bql is = being used or not, or other factors. At rtt 100ms fairness worked well = for this test with both bql and rate limiting. Also note that I=E2=80=99m glossing over the fact that on my hardware = (Ethernet device with four queues), sometimes there was a bandwidth = asymmetry and corresponding fluctuation in fairness when using bql that = changed with each flent run, so I had to retry the test once or twice to = get a =E2=80=9Cgood=E2=80=9D run. I would try to quantify that = separately before I get into it further. But that=E2=80=99s something = that may also throw folks for a loop. --Apple-Mail=_51A42CF9-B1AF-4980-9380-502C47A01791 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On Nov 24, 2017, at 12:21 PM, Toke H=C3=B8iland-J=C3=B8rgensen = <toke@toke.dk> = wrote:

Dave Taht <dave@taht.net> writes:

Pete Heist <peteheist@gmail.com>= writes:

   On Nov 23, 2017, at 10:44 AM, Jonathan = Morton <chromatix99@gmail.com> wrote:

   This is most likely an interaction of the = AQM with Linux' scheduling
   latency.

   At the 'lan' setting, the = time comstants are similar in magnitude to the
=    delays induced by Linux itself, so congestion might be = signalled
   prematurely. The flows will = then become sparse and total throughput reduced,
=    leaving little or no back-pressure for the fairness = logic to work against.

Agreed. =

man page add:

At the 'lan' setting(1ms), the time constants are similar in = magnitude
to the jitter in the Linux kernel itself, so = congestion might be
signalled prematurely. The flows will = then become sparse and total
throughput reduced, leaving = little or no back-pressure for the fairness
logic to work = against. Use the "metro" setting for local lans unless you
have a custom kernel.

Erm, doesn't this make the 'lan' keyword pretty much useless? = So why not
just remove it? Or redefine it to something = that actually works? 3ms?
Removing the bandwidth keywords altogether = and going back to fq_codel=E2=80=99s specification of target and = interval would be my personal preference (unless we can figure out how = to make the keywords work well with one another in all cases).

If we want to keep them, = I=E2=80=99ll look for more holes where things might not behave as people = expect so we can further clear up the docs.

Also, note that even at =E2=80=98metro=E2= =80=99, fairness is better, but still not fully fair on my = hardware:

https://docs.google.com/spreadsheets/d/1SMXWw2fLfmBRU622urfdvA_= Ujsuf_KQ4P3uyOH1skOM/edit#gid=3D2072687073

With 950mbit rate limiting, it=E2=80=99s = pretty good at 1.16:1. But with bql it=E2=80=99s still ~2:1. At what rtt = will fairness actually work? That may depend on hardware, OS version, = number of flows, whether bql is being used or not, or other factors. At = rtt 100ms fairness worked well for this test with both bql and rate = limiting.

Also = note that I=E2=80=99m glossing over the fact that on my hardware = (Ethernet device with four queues), sometimes there was a bandwidth = asymmetry and corresponding fluctuation in fairness when using bql that = changed with each flent run, so I had to retry the test once or twice to = get a =E2=80=9Cgood=E2=80=9D run. I would try to quantify that = separately before I get into it further. But that=E2=80=99s something = that may also throw folks for a loop.

= --Apple-Mail=_51A42CF9-B1AF-4980-9380-502C47A01791--