[Cake] Cake vs fq_codel and c/burst on an ER-X bridge

Pete Heist pete at heistp.net
Mon Sep 10 18:40:42 EDT 2018

Subject changed from “Cake on elements of a bridge”...

On Sep 10, 2018, at 9:55 PM, Dave Taht <dave.taht at gmail.com> wrote:
> On Mon, Sep 10, 2018 at 12:29 PM Pete Heist <pete at heistp.net> wrote:
>> For anyone who followed this, yes, the regular soft bridge (i.e. set interfaces bridge br0) works fine on the ER-X, as I suspect it would on most any Linux. A few notes about it:
>> - Your qdisc must be added to the physical interface (e.g. eth4), not the bridge interface
>> - Unlike the hardware bridge which has its own MAC, the soft bridge seems to take the MAC of the lowest (or first listed?) interface port
>> - On ER-X, bridge-nf-call-iptables=1 is the default so nothing needs to be changed there for firewalling
>> - When firewalling the bridged WAN interface, ‘in’ corresponds to bridged traffic and ‘local’ to routed traffic, which is different from the semantics for ordinary routed traffic
>> - I can do stateful firewalling for bridged hosts with “accept established and related”, but have to explicitly allow DHCP (UDP source/dest port 67-68) in the WAN interface’s ‘in’ rules for DHCP traffic to pass through the bridge
>> Performance:
>> Using Cake with this setup, the fun ends at around 110 Mbit with ksoftirqd thrashing. Unsurprisingly, there’s probably some overhead here with the soft bridge. For my purposes though (50 Mbit), it’s enough, barely…
> Can I encourage you to give regular ole htb+fq_codel sqm a shot with a
> bigger burst and cburst size for htb? Fiddling with the htb quantum
> isn't helping much,
> but try this, from: https://github.com/tohojo/sqm-scripts/issues/71
> (I am thinking burst and cburst should be about 1.1ms of buffering in size)

So this has turned info an interesting exercise that produced a result counter to what the common wisdom has been (that fq_codel is “faster” than cake). Because of that, I’m open to criticism of my methodology and different criteria for a successful bitrate for the shaper.

First, note that these tests still through a bridge as above, but are for a more typical setup with separate qdisc instances on egress and ingress, as opposed to my “110 Mbit” result from above, which was for egress and ingress through a common IFB.

It occurs to me that what I really want to know is the maximum set bitrate for the shaper where it still appears to be behaving properly and consistently, meaning, the actual measured TCP throughput is held steady, and at the expected percentage less than the set bitrate. I first find this out by setting a “comfortable” rate of 100Mbit and checking TCP throughput with iperf3, which is typically around 5% less than the set bitrate. Then I increase the shaper’s bitrate 5Mbit at a time and re-run the test until I find the last bitrate at which iperf3 reports a stable (within a few percent) and correct rate over 10 seconds for several runs in a row. See the attached iperf3 results for sample runs around the threshold rates.

qdisc: egress Mbit / ingress Mbit

cake nat dual-srchost / cake nat dual-dsthost ingress: 135 / 145
htb+fq_codel: 125 / 125
htb+fq_codel with burst/cburst=96000: 155 / 155

So with this testing criteria, I’m actually seeing cake “win” (with the exception of setting htb's burst/cburst to 96000, which shows a clear improvement, probably at the expense of something). I also see that the ingress rate for cake can be held steady to a bit higher of a bitrate than egress. I am using the ‘ingress’ keyword on ingress. I have to be careful here because from run to run there can be slight variations in behavior, but having repeated it several times at each bitrate around the threshold, I’m fairly certain about the results.

In the ER-X manual (https://dl.ubnt.com/guides/edgemax/EdgeOS_UG.pdf), they give a guideline of 100-250Mbps on the “expected Smart Queue shaping performance” (which means fq_codel) for the ER-X. In reality, 100Mbps is comfortable, and 250Mbps seems impossible. You might be able to get that rate by setting fq_codel to 300+Mbit (and you can’t, through a bridge anyway), but is the queue really controlled? I think I’m applying at least a little more consistent criteria for “success" here at a given bitrate than we have before.

I suppose I should repeat this test with different hardware to be surer of the claim, but I’m not sure when I’ll have the time. I will say that Cake’s shaper overall produces more satisfyingly consistent rates, and given its NAT support and host fairness, that’s why I’m likely to continue to use it when I can.

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fq_codel_125.txt
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20180911/ac8f7e44/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fq_codel_130.txt
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20180911/ac8f7e44/attachment-0001.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: cake_135.txt
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20180911/ac8f7e44/attachment-0002.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: cake_140.txt
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20180911/ac8f7e44/attachment-0003.txt>
-------------- next part --------------

More information about the Cake mailing list