From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 58F5D3BA8E for ; Tue, 11 Sep 2018 14:28:55 -0400 (EDT) Received: from hms-beagle2.lan ([79.235.228.21]) by mail.gmx.com (mrgmx102 [212.227.17.168]) with ESMTPSA (Nemesis) id 0MKt5A-1fznPB2kUK-0000Gx; Tue, 11 Sep 2018 20:28:53 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) From: Sebastian Moeller In-Reply-To: <8E0ECFFC-37A0-4DC3-91C9-27793B1C18E5@heistp.net> Date: Tue, 11 Sep 2018 20:28:52 +0200 Cc: Cake List Content-Transfer-Encoding: quoted-printable Message-Id: <5C7BA623-F83E-4B93-9997-1099DEC2392C@gmx.de> References: <87zhwxzh8o.fsf@toke.dk> <139B295B-7371-43DE-B472-DE629C9B8432@heistp.net> <87efe65wol.fsf@toke.dk> <6C556301-015B-4903-AE5A-F22D3517FFCC@heistp.net> <79F47FB1-6B00-4753-930B-950FB8CD3850@gmx.de> <8E0ECFFC-37A0-4DC3-91C9-27793B1C18E5@heistp.net> To: Pete Heist X-Mailer: Apple Mail (2.3445.9.1) X-Provags-ID: V03:K1:wGQW6c0cjpSnBuf3wELwgSIdSEhirdCOXjWPFPPTK6WpBEuDJ1/ APwr/zPQqVe5qMEHajhEy6jQLnq12QhJZpSbtK52LL+Iq8Dx4Z3jCFWO2YsqQPAvsSCZ/MN NrRZJbODqpfzct4CqLiqmAbqtJqbtTrvZHWudqP2i2NXpDEV/QT2RiTGcRHdjL/rOfE+cdO W7ENnqMcRLUXYiuUPVTlw== X-UI-Out-Filterresults: notjunk:1;V01:K0:g8EWLoU8gm0=:3Eydyn8gijfGPO1d8LSMMQ 8zxicoz5mUCr05bAtPtW7ZyR+WV9OWRhhOJ/CV9fsN8Uyk/2V8Ow7YfoAprxVKnVc0Q2/ykuX iV+ssvjCAA7cqqegg3APr/3Bqp5UyQIsAdj7ehPNe7M2DxqjJziyrMENVCjjGs3UDEbcpkp1n 0FRl1x11gJ7Jreh3lbYxfUEIUuE+BYazHH4DA+SklIhWg7EPz6+J6UDMY2+aNPdchxYdaOCQJ s1XU+UKHBv9LQa6ZuANngm1tke89x1N++7RJBVrd+u5wVj7c97LvbtRK1AvBGLaW9Y3d1NYwl MmpRgUFeR/IRAs6ow1b6+oxA9YGIz2o38YINUJ6+w7V5g+P5weAC6FUuNtXl908j7zwx2MAYZ /Fbew4wCoJXUD8yJQKAKMHcuEuvCXJpPunaj4Nzdzmw4rf99ADxFrtxNFhdFMdVulTHFyD9uA 4GlZA/UyDj+AvoeSTkLlKRUl3zuaTnE58bukYViWIisdXiRcsVp1w12CkwGCk0+2cC7qFDUZ+ VbZV1K4uVUFaZpBpHzHmuTN17c8y5EtDq2WlV09tOmQFWIW9xlZmGiM96NgNvcy2zfuBvl/R2 AeZs/HBEkFEMwyuzVOPHhOVSMX/Ga+eoARJtn7sIv60WcysfuEm2rHOXFXZSj16jqPDsEAdVu zWsj5Cy4F/25r3gPuXfeeStNX2vTtQGltGp0EquqjL3o4BMuiH3X5v2AElojhadONkT3RihKP SgclWioE3xypULcITP8x0Qon9JbwUaG/LtBOKqiT5cy/nKLx9NoGyDDc3HYQ/GsPsxqpNK/ew VMXIYIo Subject: Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Sep 2018 18:28:55 -0000 Hi Pete, > On Sep 11, 2018, at 20:09, Pete Heist wrote: >=20 >=20 >> On Sep 11, 2018, at 9:54 AM, Sebastian Moeller = wrote: >>>=20 >>> So this has turned info an interesting exercise that produced a = result counter to what the common wisdom has been (that fq_codel is = =E2=80=9Cfaster=E2=80=9D than cake >>=20 >> I believe the argument is more about htb+fq_codel versus cake = instead of fq_codel versus cake, as it seems to be the shaper = functionality that incurs the highest cost. >=20 > I sometimes loosely use fq_codel to mean htb+fq_codel. Sorry for being a smart aleck, unfortunately it is my second = nature. >=20 >>> It occurs to me that what I really want to know is the maximum set = bitrate for the shaper where it still appears to be behaving properly = and consistently, meaning, the actual measured TCP throughput is held = steady, and at the expected percentage less than the set bitrate. I = first find this out by setting a =E2=80=9Ccomfortable=E2=80=9D rate of = 100Mbit and checking TCP throughput with iperf3, which is typically = around 5% less than the set bitrate. >>=20 >> So the expected values somewhat depend on the exact = configuration, but over all the expected TCP/IPv4 goodput is calculated = as follows (I assume you are well aware of that, but I believe this = worth repeating to calibrate the expectancy): >=20 > Yes, it=E2=80=99s pretty much right on the money. >=20 >>> Then I increase the shaper=E2=80=99s bitrate 5Mbit at a time and = re-run the test until I find the last bitrate at which iperf3 reports a = stable (within a few percent) and correct rate over 10 seconds for = several runs in a row. See the attached iperf3 results for sample runs = around the threshold rates. >>=20 >> Except for the 10 seconds this sounds reasonable, I would aim = for at least 30, even tough this will be more important once you also = monitor the latency under load concurrently to the bandwidth-probing = flows=E2=80=A6 >=20 > I agree, so far I was just trying not to spend an all-nighter on it = last night. :) I was actually running 3-5 trials of ten seconds, also = because sometimes with fq_codel once it=E2=80=99s slightly above its = limit, results would vary from run to run. Ah, why is it so often reality that crosses the most diligent = plans ;) ? >=20 >>> cake nat dual-srchost / cake nat dual-dsthost ingress: 135 / 145 >>=20 >> On your box is there actual NAT masquerading happening? >=20 > Yeah, good point, I left nat there because I had one port configured = for routing and the other for the bridge and was sometimes swapping = between the two. I realize now I actually sent the numbers for routing, = not bridging. Bridging without =E2=80=98nat=E2=80=99 looks a bit higher = (155 Mbit for cake instead of 135 Mbit). I would re-do all these tests = for completeness but I=E2=80=99m out of time now. Ouch, a ten percent bandwidth cost for the nat feature certainly = answeers the question whether nat should be the default... >=20 >> The last time we discussed the bust issue, I could not manage to = see any difference with or without a specified burst, but I strongly = believe I simply did not properly test. Btw, this is unidirectional = shaping or with bidirectional saturation? >=20 > Unidirectional. I definitely see a difference, but I wonder what = criteria we (and I) used for =E2=80=9Cout of CPU=E2=80=99 in the past. So totally unscientifically me yardstick was as long as = throughput increases more or less linearly with configured shaper = bandwidth things are fine, and then at the candidate bandwidths I ran = "top -d 1" and monitored both idle% ad sirq% with idle falling below 5% = being a strong indicator of bottlenecking on cpu cycles. Dlakelan over = at github (https://github.com/dlakelan/routerperf) is working on a small = side project that aims for tighter multi-core aware logging of cpu usage = on a router, but that has not left the early prototype stage. >=20 >> >> I am quite curious about these files, but I seem incapable of = downloading/opening them... >=20 > Ok, no more sending attachments to the list I see. If this link = doesn=E2=80=99t work I might be replacing a disk at the time=E2=80=A6 :) > https://www.heistp.net/downloads/erx-sqm.tgz Great, that worked; but now I realize that I need to get acquinted with = iperf output ;) Best Regards Sebastian >=20