[Cake] Cake vs fq_codel and c/burst on an ER-X bridge

[Pete Heist <pete@heistp.net>]

[-- Attachment #1: Type: text/plain, Size: 4650 bytes --]

Subject changed from “Cake on elements of a bridge”...

On Sep 10, 2018, at 9:55 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Mon, Sep 10, 2018 at 12:29 PM Pete Heist <pete@heistp.net> wrote:
>> 
>> For anyone who followed this, yes, the regular soft bridge (i.e. set interfaces bridge br0) works fine on the ER-X, as I suspect it would on most any Linux. A few notes about it:
>> 
>> - Your qdisc must be added to the physical interface (e.g. eth4), not the bridge interface
>> - Unlike the hardware bridge which has its own MAC, the soft bridge seems to take the MAC of the lowest (or first listed?) interface port
>> - On ER-X, bridge-nf-call-iptables=1 is the default so nothing needs to be changed there for firewalling
>> - When firewalling the bridged WAN interface, ‘in’ corresponds to bridged traffic and ‘local’ to routed traffic, which is different from the semantics for ordinary routed traffic
>> - I can do stateful firewalling for bridged hosts with “accept established and related”, but have to explicitly allow DHCP (UDP source/dest port 67-68) in the WAN interface’s ‘in’ rules for DHCP traffic to pass through the bridge
>> 
>> Performance:
>> 
>> Using Cake with this setup, the fun ends at around 110 Mbit with ksoftirqd thrashing. Unsurprisingly, there’s probably some overhead here with the soft bridge. For my purposes though (50 Mbit), it’s enough, barely…
> 
> Can I encourage you to give regular ole htb+fq_codel sqm a shot with a
> bigger burst and cburst size for htb? Fiddling with the htb quantum
> isn't helping much,
> but try this, from: https://github.com/tohojo/sqm-scripts/issues/71
> 
> (I am thinking burst and cburst should be about 1.1ms of buffering in size)

So this has turned info an interesting exercise that produced a result counter to what the common wisdom has been (that fq_codel is “faster” than cake). Because of that, I’m open to criticism of my methodology and different criteria for a successful bitrate for the shaper.

First, note that these tests still through a bridge as above, but are for a more typical setup with separate qdisc instances on egress and ingress, as opposed to my “110 Mbit” result from above, which was for egress and ingress through a common IFB.

It occurs to me that what I really want to know is the maximum set bitrate for the shaper where it still appears to be behaving properly and consistently, meaning, the actual measured TCP throughput is held steady, and at the expected percentage less than the set bitrate. I first find this out by setting a “comfortable” rate of 100Mbit and checking TCP throughput with iperf3, which is typically around 5% less than the set bitrate. Then I increase the shaper’s bitrate 5Mbit at a time and re-run the test until I find the last bitrate at which iperf3 reports a stable (within a few percent) and correct rate over 10 seconds for several runs in a row. See the attached iperf3 results for sample runs around the threshold rates.

qdisc: egress Mbit / ingress Mbit

cake nat dual-srchost / cake nat dual-dsthost ingress: 135 / 145
htb+fq_codel: 125 / 125
htb+fq_codel with burst/cburst=96000: 155 / 155

So with this testing criteria, I’m actually seeing cake “win” (with the exception of setting htb's burst/cburst to 96000, which shows a clear improvement, probably at the expense of something). I also see that the ingress rate for cake can be held steady to a bit higher of a bitrate than egress. I am using the ‘ingress’ keyword on ingress. I have to be careful here because from run to run there can be slight variations in behavior, but having repeated it several times at each bitrate around the threshold, I’m fairly certain about the results.

In the ER-X manual (https://dl.ubnt.com/guides/edgemax/EdgeOS_UG.pdf), they give a guideline of 100-250Mbps on the “expected Smart Queue shaping performance” (which means fq_codel) for the ER-X. In reality, 100Mbps is comfortable, and 250Mbps seems impossible. You might be able to get that rate by setting fq_codel to 300+Mbit (and you can’t, through a bridge anyway), but is the queue really controlled? I think I’m applying at least a little more consistent criteria for “success" here at a given bitrate than we have before.

I suppose I should repeat this test with different hardware to be surer of the claim, but I’m not sure when I’ll have the time. I will say that Cake’s shaper overall produces more satisfyingly consistent rates, and given its NAT support and host fairness, that’s why I’m likely to continue to use it when I can.


[-- Attachment #2: fq_codel_125.txt --]
[-- Type: text/plain, Size: 1173 bytes --]

hostname:~/.ssh:% iperf3 -c a.b.c.230
Connecting to host a.b.c.230, port 5201
[  5] local 192.168.1.20 port 51310 connected to a.b.c.230 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  14.5 MBytes   122 Mbits/sec                  
[  5]   1.00-2.00   sec  14.2 MBytes   119 Mbits/sec                  
[  5]   2.00-3.00   sec  14.3 MBytes   120 Mbits/sec                  
[  5]   3.00-4.00   sec  14.3 MBytes   120 Mbits/sec                  
[  5]   4.00-5.00   sec  14.3 MBytes   120 Mbits/sec                  
[  5]   5.00-6.00   sec  14.2 MBytes   119 Mbits/sec                  
[  5]   6.00-7.00   sec  14.3 MBytes   120 Mbits/sec                  
[  5]   7.00-8.00   sec  14.3 MBytes   120 Mbits/sec                  
[  5]   8.00-9.00   sec  14.2 MBytes   119 Mbits/sec                  
[  5]   9.00-10.00  sec  14.3 MBytes   120 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   143 MBytes   120 Mbits/sec                  sender
[  5]   0.00-10.00  sec   143 MBytes   120 Mbits/sec                  receiver

[-- Attachment #3: fq_codel_130.txt --]
[-- Type: text/plain, Size: 1178 bytes --]

hostname:~/Downloads:% iperf3 -c a.b.c.230
Connecting to host a.b.c.230, port 5201
[  5] local 192.168.1.20 port 51324 connected to a.b.c.230 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  13.3 MBytes   111 Mbits/sec                  
[  5]   1.00-2.00   sec  14.0 MBytes   117 Mbits/sec                  
[  5]   2.00-3.00   sec  13.9 MBytes   117 Mbits/sec                  
[  5]   3.00-4.00   sec  13.9 MBytes   116 Mbits/sec                  
[  5]   4.00-5.00   sec  13.6 MBytes   114 Mbits/sec                  
[  5]   5.00-6.00   sec  13.9 MBytes   116 Mbits/sec                  
[  5]   6.00-7.00   sec  14.0 MBytes   117 Mbits/sec                  
[  5]   7.00-8.00   sec  13.9 MBytes   116 Mbits/sec                  
[  5]   8.00-9.00   sec  13.8 MBytes   116 Mbits/sec                  
[  5]   9.00-10.00  sec  13.8 MBytes   116 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   138 MBytes   116 Mbits/sec                  sender
[  5]   0.00-10.00  sec   138 MBytes   116 Mbits/sec                  receiver

[-- Attachment #4: cake_135.txt --]
[-- Type: text/plain, Size: 1183 bytes --]

hostname:~/Downloads:% iperf3 -c a.b.c.230     
Connecting to host a.b.c.230, port 5201
[  5] local 192.168.1.20 port 51328 connected to a.b.c.230 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  15.5 MBytes   130 Mbits/sec                  
[  5]   1.00-2.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   2.00-3.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   3.00-4.00   sec  15.2 MBytes   128 Mbits/sec                  
[  5]   4.00-5.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   5.00-6.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   6.00-7.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   7.00-8.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   8.00-9.00   sec  15.3 MBytes   128 Mbits/sec                  
[  5]   9.00-10.00  sec  14.8 MBytes   124 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   153 MBytes   129 Mbits/sec                  sender
[  5]   0.00-10.00  sec   153 MBytes   128 Mbits/sec                  receiver

[-- Attachment #5: cake_140.txt --]
[-- Type: text/plain, Size: 1178 bytes --]

hostname:~/Downloads:% iperf3 -c a.b.c.230
Connecting to host a.b.c.230, port 5201
[  5] local 192.168.1.20 port 51332 connected to a.b.c.230 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  16.3 MBytes   136 Mbits/sec                  
[  5]   1.00-2.00   sec  16.3 MBytes   137 Mbits/sec                  
[  5]   2.00-3.00   sec  15.6 MBytes   131 Mbits/sec                  
[  5]   3.00-4.00   sec  15.6 MBytes   131 Mbits/sec                  
[  5]   4.00-5.00   sec  15.3 MBytes   128 Mbits/sec                  
[  5]   5.00-6.00   sec  16.5 MBytes   139 Mbits/sec                  
[  5]   6.00-7.00   sec  15.5 MBytes   130 Mbits/sec                  
[  5]   7.00-8.00   sec  15.3 MBytes   129 Mbits/sec                  
[  5]   8.00-9.00   sec  16.6 MBytes   139 Mbits/sec                  
[  5]   9.00-10.00  sec  15.3 MBytes   128 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   158 MBytes   133 Mbits/sec                  sender
[  5]   0.00-10.00  sec   158 MBytes   133 Mbits/sec                  receiver

[-- Attachment #6: Type: text/plain, Size: 2 bytes --]