From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id E952F3BA8E for ; Tue, 11 Sep 2018 14:09:25 -0400 (EDT) Received: by mail-wm0-x236.google.com with SMTP id y139-v6so1962120wmc.2 for ; Tue, 11 Sep 2018 11:09:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=Eji1U9owBS43/PH4ijb/LZuQqxNQD7kBQRbGYYBeePw=; b=OQ6pCPLtnIwVDWs9wmaQHRzUQxd4exUJRd35x9tzD+eJJ0qPUuuzOLYuMvyZYByYFi agIvisyVcVuyOeH9iig6hpyVfu28b4xw4Vm6upVEvZ8/yhgVCPzZx/UV6yaC31zSpSBu iwUTu5QNRyvRa7c0hW5/srcpQySYb3Eg7i2SCXg2WjBxEGhGWA/qtebpGecl944x6dD+ HuR/L5mkyx9+J8+nSGh+cGBkaBqxe+Tgo2uS/6omPYq0TCiyRpUwOw0MR8J6TdQks9dq HsSuw4dlOR5RYzE2jQ40ZtEtuj04DQjYOUcLWqkwQHy6l4lBfsJOsyXeqbCJHVOyFntL EtQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=Eji1U9owBS43/PH4ijb/LZuQqxNQD7kBQRbGYYBeePw=; b=b2aJrUCwXHXHKnb32WwF/+Qph6gpiZe76xNuoP7jejEiG9CcQqB/dG3t31jT2j9KBs +uhO/Lx88z8GuCcJNIyVbdt8aZUBKxAHrQujWSKyqzfHn+YADRxr5U7M9bRZeQaL97gS MR8E7MQ0Ib1l1jgvWFuWNCo7ogdWe47QD6hGo5arZrtRMBxf7EBc9jsjoAU48p5qMs9D SG0lzCibvvppDWu1b0ePTi0pBJzNWkK6ITsx5t3VBVkLqwzzhYa3f6cFI8PCRdBr3XxJ u0uzgjG9kRSqx+L/ZIWSFDvqlB/KI2t4gbkxRphlsBTr1xl/6EjFWYom/Cvi2KdtQB48 8+Tw== X-Gm-Message-State: APzg51BBCCWce4aD1g0/sK/Fw1Upbp8kD/EjixRQY+YZexYksOWhuZLc 7oXMAWqCPy+lix9MIiCAR7G85w== X-Google-Smtp-Source: ANB0VdZJOBr222ld3XdwXpaWP81PgAqkJG4R/Em9LPpD6eZfRBS4hKJfxhZmLG22T8BG9loGucFdtw== X-Received: by 2002:a1c:91cd:: with SMTP id t196-v6mr1973353wmd.100.1536689364964; Tue, 11 Sep 2018 11:09:24 -0700 (PDT) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id 66-v6sm2632737wmw.34.2018.09.11.11.09.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 11:09:24 -0700 (PDT) From: Pete Heist Message-Id: <8E0ECFFC-37A0-4DC3-91C9-27793B1C18E5@heistp.net> Content-Type: multipart/alternative; boundary="Apple-Mail=_87991015-9B94-4293-9BD5-EC904DC119CB" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Date: Tue, 11 Sep 2018 20:09:22 +0200 In-Reply-To: <79F47FB1-6B00-4753-930B-950FB8CD3850@gmx.de> Cc: Cake List To: Sebastian Moeller References: <87zhwxzh8o.fsf@toke.dk> <139B295B-7371-43DE-B472-DE629C9B8432@heistp.net> <87efe65wol.fsf@toke.dk> <6C556301-015B-4903-AE5A-F22D3517FFCC@heistp.net> <79F47FB1-6B00-4753-930B-950FB8CD3850@gmx.de> X-Mailer: Apple Mail (2.3445.9.1) Subject: Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Sep 2018 18:09:26 -0000 --Apple-Mail=_87991015-9B94-4293-9BD5-EC904DC119CB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Sep 11, 2018, at 9:54 AM, Sebastian Moeller = wrote: >>=20 >> So this has turned info an interesting exercise that produced a = result counter to what the common wisdom has been (that fq_codel is = =E2=80=9Cfaster=E2=80=9D than cake >=20 > I believe the argument is more about htb+fq_codel versus cake = instead of fq_codel versus cake, as it seems to be the shaper = functionality that incurs the highest cost. I sometimes loosely use fq_codel to mean htb+fq_codel. >> It occurs to me that what I really want to know is the maximum set = bitrate for the shaper where it still appears to be behaving properly = and consistently, meaning, the actual measured TCP throughput is held = steady, and at the expected percentage less than the set bitrate. I = first find this out by setting a =E2=80=9Ccomfortable=E2=80=9D rate of = 100Mbit and checking TCP throughput with iperf3, which is typically = around 5% less than the set bitrate. >=20 > So the expected values somewhat depend on the exact = configuration, but over all the expected TCP/IPv4 goodput is calculated = as follows (I assume you are well aware of that, but I believe this = worth repeating to calibrate the expectancy): Yes, it=E2=80=99s pretty much right on the money. >> Then I increase the shaper=E2=80=99s bitrate 5Mbit at a time and = re-run the test until I find the last bitrate at which iperf3 reports a = stable (within a few percent) and correct rate over 10 seconds for = several runs in a row. See the attached iperf3 results for sample runs = around the threshold rates. >=20 > Except for the 10 seconds this sounds reasonable, I would aim = for at least 30, even tough this will be more important once you also = monitor the latency under load concurrently to the bandwidth-probing = flows=E2=80=A6 I agree, so far I was just trying not to spend an all-nighter on it last = night. :) I was actually running 3-5 trials of ten seconds, also because = sometimes with fq_codel once it=E2=80=99s slightly above its limit, = results would vary from run to run. >> cake nat dual-srchost / cake nat dual-dsthost ingress: 135 / 145 >=20 > On your box is there actual NAT masquerading happening? Yeah, good point, I left nat there because I had one port configured for = routing and the other for the bridge and was sometimes swapping between = the two. I realize now I actually sent the numbers for routing, not = bridging. Bridging without =E2=80=98nat=E2=80=99 looks a bit higher (155 = Mbit for cake instead of 135 Mbit). I would re-do all these tests for = completeness but I=E2=80=99m out of time now. > The last time we discussed the bust issue, I could not manage to = see any difference with or without a specified burst, but I strongly = believe I simply did not properly test. Btw, this is unidirectional = shaping or with bidirectional saturation? Unidirectional. I definitely see a difference, but I wonder what = criteria we (and I) used for =E2=80=9Cout of CPU=E2=80=99 in the past. > > I am quite curious about these files, but I seem incapable of = downloading/opening them... Ok, no more sending attachments to the list I see. If this link = doesn=E2=80=99t work I might be replacing a disk at the time=E2=80=A6 :) https://www.heistp.net/downloads/erx-sqm.tgz --Apple-Mail=_87991015-9B94-4293-9BD5-EC904DC119CB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On = Sep 11, 2018, at 9:54 AM, Sebastian Moeller <moeller0@gmx.de> = wrote:

So this has turned = info an interesting exercise that produced a result counter to what the = common wisdom has been (that fq_codel is =E2=80=9Cfaster=E2=80=9D than = cake

= I believe the = argument is more about htb+fq_codel versus cake instead of fq_codel = versus cake, as it seems to be the shaper functionality that incurs the = highest cost.

I = sometimes loosely use fq_codel to mean htb+fq_codel.

It = occurs to me that what I really want to know is the maximum set bitrate = for the shaper where it still appears to be behaving properly and = consistently, meaning, the actual measured TCP throughput is held = steady, and at the expected percentage less than the set bitrate. I = first find this out by setting a =E2=80=9Ccomfortable=E2=80=9D rate of = 100Mbit and checking TCP throughput with iperf3, which is typically = around 5% less than the set bitrate.

So the = expected values somewhat depend on the exact configuration, but over all = the expected TCP/IPv4 goodput is calculated as follows (I assume you are = well aware of that, but I believe this worth repeating to calibrate the = expectancy):

Yes, = it=E2=80=99s pretty much right on the money.

Then = I increase the shaper=E2=80=99s bitrate 5Mbit at a time and re-run the = test until I find the last bitrate at which iperf3 reports a stable = (within a few percent) and correct rate over 10 seconds for several runs = in a row. See the attached iperf3 results for sample runs around the = threshold rates.

= Except for = the 10 seconds this sounds reasonable, I would aim for at least 30, even = tough this will be more important once you also monitor the latency = under load concurrently to the bandwidth-probing = flows=E2=80=A6

I = agree, so far I was just trying not to spend an all-nighter on it last = night. :) I was actually running 3-5 trials of ten seconds, also because = sometimes with fq_codel once it=E2=80=99s slightly above its limit, = results would vary from run to run.

cake nat dual-srchost / cake nat = dual-dsthost ingress: 135 / 145

On your box = is there actual NAT masquerading happening?

Yeah, good = point, I left nat there because I had one port configured for routing = and the other for the bridge and was sometimes swapping between the two. = I realize now I actually sent the numbers for routing, not bridging. = Bridging without =E2=80=98nat=E2=80=99 looks a bit higher (155 Mbit for = cake instead of 135 Mbit). I would re-do all these tests for = completeness but I=E2=80=99m out of time now.

The last time = we discussed the bust issue, I could not manage to see any difference = with or without a specified burst, but I strongly believe I simply did = not properly test. Btw, this is unidirectional shaping or with = bidirectional saturation?

Unidirectional. I definitely see a difference, but = I wonder what criteria we (and I) used for =E2=80=9Cout of CPU=E2=80=99 = in the past.

<fq_code= l_125.txt><fq_code= l_130.txt><cake_13= 5.txt><cake_14= 0.txt>
 = I am quite = curious about these files, but I seem incapable of downloading/opening = them...

Ok, no more sending attachments to the list I see. If = this link doesn=E2=80=99t work I might be replacing a disk at the = time=E2=80=A6 :)
https://www.heistp.net/downloads/erx-sqm.tgz

= --Apple-Mail=_87991015-9B94-4293-9BD5-EC904DC119CB--