From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 9EB5B3CB35 for ; Tue, 11 Sep 2018 03:54:54 -0400 (EDT) Received: from [172.16.11.53] ([134.76.241.253]) by mail.gmx.com (mrgmx003 [212.227.17.190]) with ESMTPSA (Nemesis) id 0MegbQ-1gExRE1BXs-00OGb2; Tue, 11 Sep 2018 09:54:52 +0200 From: Sebastian Moeller Message-Id: <79F47FB1-6B00-4753-930B-950FB8CD3850@gmx.de> Content-Type: multipart/mixed; boundary="Apple-Mail=_335DD3EC-9750-4EE3-BA40-951190D233CD" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Date: Tue, 11 Sep 2018 09:54:51 +0200 In-Reply-To: Cc: =?utf-8?Q?Dave_T=C3=A4ht?= , Cake List To: Pete Heist References: <87zhwxzh8o.fsf@toke.dk> <139B295B-7371-43DE-B472-DE629C9B8432@heistp.net> <87efe65wol.fsf@toke.dk> <6C556301-015B-4903-AE5A-F22D3517FFCC@heistp.net> X-Mailer: Apple Mail (2.3445.9.1) X-Provags-ID: V03:K1:A23xKm8vWtmTR66+G/9QcvemUwk052TK/dRAQtHg/Sm2GCaeYxR 6FXU2RGarrrLT66Um/f4sZ9wBU4PXLm/NZyBERdcxCBXJfwdKIZcf8zeE4kZMjshKi6Udkq YCFN+0IxM0p/A1bKN51QycsezbUZG+1vgiDwBJDCBpEL7Ftk3FxLsx+rCmqF11U0sjtiDE6 yMlwXLF0kvQgvLecSGiTA== X-UI-Out-Filterresults: notjunk:1;V01:K0:Al5TXVnoaOY=:7hFrzCgs1RKj6acOGQ/0EG 9VMwiqYuwjKpzpPVkXc5ueuPkQHsoa2e2BWQJR2fk6oP9QNeqljm8YM1pX6edDy46catRh7DR GiQiX5qQ/JoWxgFHvERB/nf05SoKIXQ9n7Hv44FPQWpTOKilbK347Vya7ArLQWTW22XxK61Ji Ognl5NHTRh+4crbmFWqxEdhKrqSoegOqbasrVqkpCTJkpS8wjmrcgB4Y6pyRjziQvI4LslnHc nsXv6ZyouFwlcyVZWT8HEY7vxEifOBgdrgpGmxdjKSImIk8klfOwM9C5C7qJzmj9v657yDFFb o/PlMvD2VHWOXOjVsMGBHMh3BtEMRXLWWrWYAJKxga2tdtLeP13qkC1rGL0Smur3xIKCO45rJ Uwrj5kYd9NfCOOK2y/RjX6DjYRTYtuYCE8nuFIMI+sxihZc+TF9oS5KMA+mEVV0MKPaF5mEsu fXdLDQPjYOnYDYpIYwc6SJwYo8xOkzWpOXXErV8gUmoVsDh+zEEfa7El/cTyZhJRm59Kt+Dyo FzRRVTUJJB/3u/WjiMAAwsfCwJPNAxrDxZbDSFrnUNr/OE7nCniMnopUWgOsNHj5qXQtFmeaJ sLNrFJabFXiKqyajT53Or6lDXecsw8FvgxjLI9r6SYQJkbjIIXvTH876lNoaj/CWgANKClwkr A+doxKM3Q08ZsYost1qIEAPqd8sfnCDjLvTFR5H1der5dcLKd8v2Uaf/g7QqVciUOnbVae32f ol1RdyWMg3hQsG0T7HALk3cP2uYfiBhuzwvfEugqsEk7dJ8+XCAduQ0sNB8= Subject: Re: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Sep 2018 07:54:54 -0000 --Apple-Mail=_335DD3EC-9750-4EE3-BA40-951190D233CD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Pete, > On Sep 11, 2018, at 00:40, Pete Heist wrote: >=20 > Subject changed from =E2=80=9CCake on elements of a bridge=E2=80=9D... >=20 > On Sep 10, 2018, at 9:55 PM, Dave Taht wrote: >>=20 >> On Mon, Sep 10, 2018 at 12:29 PM Pete Heist wrote: >>>=20 >>> For anyone who followed this, yes, the regular soft bridge (i.e. set = interfaces bridge br0) works fine on the ER-X, as I suspect it would on = most any Linux. A few notes about it: >>>=20 >>> - Your qdisc must be added to the physical interface (e.g. eth4), = not the bridge interface >>> - Unlike the hardware bridge which has its own MAC, the soft bridge = seems to take the MAC of the lowest (or first listed?) interface port >>> - On ER-X, bridge-nf-call-iptables=3D1 is the default so nothing = needs to be changed there for firewalling >>> - When firewalling the bridged WAN interface, =E2=80=98in=E2=80=99 = corresponds to bridged traffic and =E2=80=98local=E2=80=99 to routed = traffic, which is different from the semantics for ordinary routed = traffic >>> - I can do stateful firewalling for bridged hosts with =E2=80=9Caccept= established and related=E2=80=9D, but have to explicitly allow DHCP = (UDP source/dest port 67-68) in the WAN interface=E2=80=99s =E2=80=98in=E2= =80=99 rules for DHCP traffic to pass through the bridge >>>=20 >>> Performance: >>>=20 >>> Using Cake with this setup, the fun ends at around 110 Mbit with = ksoftirqd thrashing. Unsurprisingly, there=E2=80=99s probably some = overhead here with the soft bridge. For my purposes though (50 Mbit), = it=E2=80=99s enough, barely=E2=80=A6 >>=20 >> Can I encourage you to give regular ole htb+fq_codel sqm a shot with = a >> bigger burst and cburst size for htb? Fiddling with the htb quantum >> isn't helping much, >> but try this, from: https://github.com/tohojo/sqm-scripts/issues/71 >>=20 >> (I am thinking burst and cburst should be about 1.1ms of buffering in = size) >=20 > So this has turned info an interesting exercise that produced a result = counter to what the common wisdom has been (that fq_codel is = =E2=80=9Cfaster=E2=80=9D than cake I believe the argument is more about htb+fq_codel versus cake = instead of fq_codel versus cake, as it seems to be the shaper = functionality that incurs the highest cost. > ). Because of that, I=E2=80=99m open to criticism of my methodology = and different criteria for a successful bitrate for the shaper. >=20 > First, note that these tests still through a bridge as above, but are = for a more typical setup with separate qdisc instances on egress and = ingress, as opposed to my =E2=80=9C110 Mbit=E2=80=9D result from above, = which was for egress and ingress through a common IFB. >=20 > It occurs to me that what I really want to know is the maximum set = bitrate for the shaper where it still appears to be behaving properly = and consistently, meaning, the actual measured TCP throughput is held = steady, and at the expected percentage less than the set bitrate. I = first find this out by setting a =E2=80=9Ccomfortable=E2=80=9D rate of = 100Mbit and checking TCP throughput with iperf3, which is typically = around 5% less than the set bitrate. So the expected values somewhat depend on the exact = configuration, but over all the expected TCP/IPv4 goodput is calculated = as follows (I assume you are well aware of that, but I believe this = worth repeating to calibrate the expectancy): Expected overhead percentage: 100 - 100 * ((MTU - IP-Overhead - = TCP-Overhead) / (MTU + Framing-Overhead)) assuming MTU 1500, IPv4, no TCP options, and ethernet framing (of which = the kernel only accounts for 14 bytes) we get 100 - 100 * ((1500 - 20 - 20) / (1500 + 14)) =3D 3.57 % so the observed = difference between set gross-rate and measured net-tcp payload rate = matches the theory reasonably well. with tcp timestamps which you might/should have enabled you get: 100 - 100 * ((1500 - 20 - 20 - 12) / (1500 + 14)) =3D 4.39 % and with proper accounting for an ethernet carrier: 100 - 100 * ((1500 - 20 - 20 - 12) / (1500 + 38)) =3D 5.85175552666 % all of these are close enough to 5% to make the 5% rule a reasonable = threshold to compare against, at least to me. > Then I increase the shaper=E2=80=99s bitrate 5Mbit at a time and = re-run the test until I find the last bitrate at which iperf3 reports a = stable (within a few percent) and correct rate over 10 seconds for = several runs in a row. See the attached iperf3 results for sample runs = around the threshold rates. Except for the 10 seconds this sounds reasonable, I would aim = for at least 30, even tough this will be more important once you also = monitor the latency under load concurrently to the bandwidth-probing = flows... >=20 > qdisc: egress Mbit / ingress Mbit >=20 > cake nat dual-srchost / cake nat dual-dsthost ingress: 135 / 145 On your box is there actual NAT masquerading happening? > htb+fq_codel: 125 / 125 > htb+fq_codel with burst/cburst=3D96000: 155 / 155 The last time we discussed the bust issue, I could not manage to = see any difference with or without a specified burst, but I strongly = believe I simply did not properly test. Btw, this is unidirectional = shaping or with bidirectional saturation?=20 >=20 > So with this testing criteria, I=E2=80=99m actually seeing cake = =E2=80=9Cwin=E2=80=9D (with the exception of setting htb's burst/cburst = to 96000, which shows a clear improvement, probably at the expense of = something). I could be wrong but the cost of burst/cburst is basically = potentially more delay and jitter with the delay bound by the time = required to empty the burst bucket, so at 96000 or 96kb and a set rate = of 155Mbps I expect an additional delay of 1000 * 96/(155*1000) =3D 0.62 = milliseconds, which fits as I believe that the 96k were targeted for 1ms = @ 100Mbps. Also it will make the output of the shaper be less smooth and = more choppy as sending is not paced ideally any more. My (too simplistic) mental model is that burst allows the shaper = to work better in sirq-constrained conditions, as the issue basically = seems to be that the shaper does not run often enough but without any = overcommit permission will continue putting less data to the NIC than = can be sent (at the desired shaper rate) in the (longer than = expected/desired) interval between shaper executions. Te bust basically = allow the shaper to dump in a batch of packets, hopefully getting = allowing the interface to keep sending on the average desired sending = rate. > I also see that the ingress rate for cake can be held steady to a bit = higher of a bitrate than egress. I am using the =E2=80=98ingress=E2=80=99 = keyword on ingress. I have to be careful here because from run to run = there can be slight variations in behavior, but having repeated it = several times at each bitrate around the threshold, I=E2=80=99m fairly = certain about the results. >=20 > In the ER-X manual (https://dl.ubnt.com/guides/edgemax/EdgeOS_UG.pdf), = they give a guideline of 100-250Mbps on the =E2=80=9Cexpected Smart = Queue shaping performance=E2=80=9D (which means fq_codel) for the ER-X. = In reality, 100Mbps is comfortable, and 250Mbps seems impossible. You = might be able to get that rate by setting fq_codel to 300+Mbit (and you = can=E2=80=99t, through a bridge anyway), but is the queue really = controlled? I think I=E2=80=99m applying at least a little more = consistent criteria for =E2=80=9Csuccess" here at a given bitrate than = we have before. >=20 > I suppose I should repeat this test with different hardware to be = surer of the claim, but I=E2=80=99m not sure when I=E2=80=99ll have the = time. I will say that Cake=E2=80=99s shaper overall produces more = satisfyingly consistent rates, and given its NAT support and host = fairness, that=E2=80=99s why I=E2=80=99m likely to continue to use it = when I can. >=20 --Apple-Mail=_335DD3EC-9750-4EE3-BA40-951190D233CD Content-Disposition: attachment; filename=fq_codel_125.txt Content-Type: text/plain; x-unix-mode=0644; name="fq_codel_125.txt" Content-Transfer-Encoding: 7bit --Apple-Mail=_335DD3EC-9750-4EE3-BA40-951190D233CD Content-Disposition: attachment; filename=fq_codel_130.txt Content-Type: text/plain; x-unix-mode=0644; name="fq_codel_130.txt" Content-Transfer-Encoding: 7bit --Apple-Mail=_335DD3EC-9750-4EE3-BA40-951190D233CD Content-Disposition: attachment; filename=cake_135.txt Content-Type: text/plain; x-unix-mode=0644; name="cake_135.txt" Content-Transfer-Encoding: 7bit --Apple-Mail=_335DD3EC-9750-4EE3-BA40-951190D233CD Content-Disposition: attachment; filename=cake_140.txt Content-Type: text/plain; x-unix-mode=0644; name="cake_140.txt" Content-Transfer-Encoding: 7bit --Apple-Mail=_335DD3EC-9750-4EE3-BA40-951190D233CD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii >=20 I am quite curious about these files, but I seem incapable of = downloading/opening them... Best Regards Sebastian >=20 > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake --Apple-Mail=_335DD3EC-9750-4EE3-BA40-951190D233CD--