[Cake] Cake vs fq_codel and c/burst on an ER-X bridge

Tue Sep 11 03:54:51 EDT 2018

Hi Pete,

> On Sep 11, 2018, at 00:40, Pete Heist <pete at heistp.net> wrote:
> 
> Subject changed from “Cake on elements of a bridge”...
> 
> On Sep 10, 2018, at 9:55 PM, Dave Taht <dave.taht at gmail.com> wrote:
>> 
>> On Mon, Sep 10, 2018 at 12:29 PM Pete Heist <pete at heistp.net> wrote:
>>> 
>>> For anyone who followed this, yes, the regular soft bridge (i.e. set interfaces bridge br0) works fine on the ER-X, as I suspect it would on most any Linux. A few notes about it:
>>> 
>>> - Your qdisc must be added to the physical interface (e.g. eth4), not the bridge interface
>>> - Unlike the hardware bridge which has its own MAC, the soft bridge seems to take the MAC of the lowest (or first listed?) interface port
>>> - On ER-X, bridge-nf-call-iptables=1 is the default so nothing needs to be changed there for firewalling
>>> - When firewalling the bridged WAN interface, ‘in’ corresponds to bridged traffic and ‘local’ to routed traffic, which is different from the semantics for ordinary routed traffic
>>> - I can do stateful firewalling for bridged hosts with “accept established and related”, but have to explicitly allow DHCP (UDP source/dest port 67-68) in the WAN interface’s ‘in’ rules for DHCP traffic to pass through the bridge
>>> 
>>> Performance:
>>> 
>>> Using Cake with this setup, the fun ends at around 110 Mbit with ksoftirqd thrashing. Unsurprisingly, there’s probably some overhead here with the soft bridge. For my purposes though (50 Mbit), it’s enough, barely…
>> 
>> Can I encourage you to give regular ole htb+fq_codel sqm a shot with a
>> bigger burst and cburst size for htb? Fiddling with the htb quantum
>> isn't helping much,
>> but try this, from: https://github.com/tohojo/sqm-scripts/issues/71
>> 
>> (I am thinking burst and cburst should be about 1.1ms of buffering in size)
> 
> So this has turned info an interesting exercise that produced a result counter to what the common wisdom has been (that fq_codel is “faster” than cake

	I believe the argument is more about htb+fq_codel versus cake instead of fq_codel versus cake, as it seems to be the shaper functionality that incurs the highest cost.

> ). Because of that, I’m open to criticism of my methodology and different criteria for a successful bitrate for the shaper.
> 
> First, note that these tests still through a bridge as above, but are for a more typical setup with separate qdisc instances on egress and ingress, as opposed to my “110 Mbit” result from above, which was for egress and ingress through a common IFB.
> 
> It occurs to me that what I really want to know is the maximum set bitrate for the shaper where it still appears to be behaving properly and consistently, meaning, the actual measured TCP throughput is held steady, and at the expected percentage less than the set bitrate. I first find this out by setting a “comfortable” rate of 100Mbit and checking TCP throughput with iperf3, which is typically around 5% less than the set bitrate.

	So the expected values somewhat depend on the exact configuration, but over all the expected TCP/IPv4 goodput is calculated as follows (I assume you are well aware of that, but I believe this worth repeating to calibrate the expectancy):
Expected overhead percentage: 100 - 100 * ((MTU - IP-Overhead - TCP-Overhead) / (MTU + Framing-Overhead))
assuming MTU 1500, IPv4, no TCP options, and ethernet framing (of which the kernel only accounts for 14 bytes) we get
100 - 100 * ((1500 - 20 - 20) / (1500 + 14)) = 3.57 % so the observed difference between set gross-rate and measured net-tcp payload rate matches the theory reasonably well.
with tcp timestamps which you might/should have enabled you get:
100 - 100 * ((1500 - 20 - 20 - 12) / (1500 + 14)) = 4.39 %
and with proper accounting for an ethernet carrier:
100 - 100 * ((1500 - 20 - 20 - 12) / (1500 + 38)) = 5.85175552666 %
all of these are close enough to 5% to make the 5% rule a reasonable threshold to compare against, at least to me.

> Then I increase the shaper’s bitrate 5Mbit at a time and re-run the test until I find the last bitrate at which iperf3 reports a stable (within a few percent) and correct rate over 10 seconds for several runs in a row. See the attached iperf3 results for sample runs around the threshold rates.

	Except for the 10 seconds this sounds reasonable, I would aim for at least 30, even tough this will be more important once you also monitor the latency under load concurrently to the bandwidth-probing flows...

> 
> qdisc: egress Mbit / ingress Mbit
> 
> cake nat dual-srchost / cake nat dual-dsthost ingress: 135 / 145

	On your box is there actual NAT masquerading happening?

> htb+fq_codel: 125 / 125
> htb+fq_codel with burst/cburst=96000: 155 / 155

	The last time we discussed the bust issue, I could not manage to see any difference with or without a specified burst, but I strongly believe I simply did not properly test. Btw, this is unidirectional shaping or with bidirectional saturation? 

> 
> So with this testing criteria, I’m actually seeing cake “win” (with the exception of setting htb's burst/cburst to 96000, which shows a clear improvement, probably at the expense of something).

	I could be wrong but the cost of burst/cburst is basically potentially more delay and jitter with the delay bound by the time required to empty the burst bucket, so at 96000 or 96kb and a set rate of 155Mbps I expect an additional delay of 1000 * 96/(155*1000) = 0.62 milliseconds, which fits as I believe that the 96k were targeted for 1ms @ 100Mbps. Also it will make the output of the shaper be less smooth and more choppy as sending is not paced ideally any more.
	My (too simplistic) mental model is that burst allows the shaper to work better in sirq-constrained conditions, as the issue basically seems to be that the shaper does not run often enough but without any overcommit permission will continue putting less data to the NIC than can be sent (at the desired shaper rate) in the (longer than expected/desired) interval between shaper executions. Te bust basically allow the shaper to dump in a batch of packets, hopefully getting allowing the interface to keep sending on the average desired sending rate.

> I also see that the ingress rate for cake can be held steady to a bit higher of a bitrate than egress. I am using the ‘ingress’ keyword on ingress. I have to be careful here because from run to run there can be slight variations in behavior, but having repeated it several times at each bitrate around the threshold, I’m fairly certain about the results.
> 
> In the ER-X manual (https://dl.ubnt.com/guides/edgemax/EdgeOS_UG.pdf), they give a guideline of 100-250Mbps on the “expected Smart Queue shaping performance” (which means fq_codel) for the ER-X. In reality, 100Mbps is comfortable, and 250Mbps seems impossible. You might be able to get that rate by setting fq_codel to 300+Mbit (and you can’t, through a bridge anyway), but is the queue really controlled? I think I’m applying at least a little more consistent criteria for “success" here at a given bitrate than we have before.
> 
> I suppose I should repeat this test with different hardware to be surer of the claim, but I’m not sure when I’ll have the time. I will say that Cake’s shaper overall produces more satisfyingly consistent rates, and given its NAT support and host fairness, that’s why I’m likely to continue to use it when I can.
> 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fq_codel_125.txt
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20180911/933a2056/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fq_codel_130.txt
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20180911/933a2056/attachment-0001.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: cake_135.txt
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20180911/933a2056/attachment-0002.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: cake_140.txt
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20180911/933a2056/attachment-0003.txt>
-------------- next part --------------
> 

	I am quite curious about these files, but I seem incapable of downloading/opening them...

Best Regards
	Sebastian

> 
> _______________________________________________
> Cake mailing list
> Cake at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake