[Cake] cake and hfsc rate limiters outperforming htb on one-armed router

[Pete Heist]

It’s a bit more complicated than this. It looks like the htb rate limiter is different in that as rates increase the actual rate starts to deviate from the specified rate early on, but it rather gracefully handles the “out of CPU” situation, where it still maintains control of the queue, just gradually fails to meet the rate specified by greater and greater percentages.

Instead of a single flow test with iperf3, here are rates that each limiter can reach on egress of both apu1a interfaces during an rrul_be test:

# - max limit on APU for one-armed routing, rrul_be test 4+4 flows (firewall on):
#   - cake: 210mbit
#   - htb+fq_codel: 93%@100mbit, 90%@200mbit, 84%@300mbit, 72%@400mbit, 59%@500mbit
#   - hfsc+fq_codel: 310mbit
#   - hfsc+cake: 300mbit

The numbers for cake and hfsc are right before loss of queue, and with htb the queue isn’t lost even at 500mbit, for example, just the actual rate is only 59% of what was specified.

I really need to graph the specified rate vs the actual rate, inter-flow and intra-flow latency, stepped 25mbit at a time. I think it would be interesting, so this is on my todo list if there’s time after the ISP config gets done.

> On Dec 28, 2018, at 1:17 AM, Pete Heist <pete at heistp.net> wrote:
> 
> For whatever reason, I’m seeing the rate limiters in cake and hfsc vastly outperform htb in the one-armed router configuration I described in my previous thread. To simplify things, I apply the qdiscs with a single class only at egress of eth0 on apu1a:
> 
> apu2a   <— default VLAN —>   apu1a   <— VLAN 3300 —>   apu2b
> 
> I use iperf3 from apu2a to apu2b and find the rate at which things break down. Whereas cake and hfsc can both reach around 850mbit, htb is breaking down at around 200mbit, which seems rather strange. This could be a function of the older kernel I have to use, the hardware, or maybe htb just isn’t suited well to this task for some reason. I wish I knew, as I’d rather be using htb for this task than hfsc (especially given the lockup issue with cake)...
> 
> ——
> 
> #!/bin/bash 
> 
> # point where iperf3 throughput drops below ~93% of theoretical:
> # htb: 200mbit
> # hfsc: 850mbit
> # cake: 850mbit
> 
> IFACE=eth0
> RATE=850mbit
> 
> start_htb() {
>        stop
>        tc qdisc add dev $IFACE root handle 1: htb default 1
>        tc class add dev $IFACE parent 1: classid 1:1 htb rate $RATE ceil $RATE
>        tc qdisc add dev $IFACE parent 1:1 handle 10: fq_codel
> }
> 
> start_hfsc() {
>        stop
>        tc qdisc add dev $IFACE root handle 1: hfsc default 1
>        tc class add dev $IFACE parent 1: classid 1:1 hfsc sc rate $RATE ul rate $RATE
>        tc qdisc add dev $IFACE parent 1:1 handle 10: fq_codel
> }
> 
> start_cake() {
>        stop
>        tc qdisc add dev $IFACE root cake bandwidth $RATE
> }
> 
> stop() {
>        tc qdisc del dev $IFACE root &>/dev/null
>        tc qdisc del dev $IFACE ingress &>/dev/null
> }
> 
> "$@“
> ——
> 
> root at apu1a:~/rate_limiters# uname -a
> Linux apu1a 3.16.7-ckt9-voyage #1 SMP Thu Apr 23 11:10:44 HKT 2015 i686 GNU/Linux
> 
> root at apu1a:~/rate_limiters# cat /proc/cpuinfo 
> processor	: 0
> vendor_id	: AuthenticAMD
> cpu family	: 20
> model		: 2
> model name	: AMD G-T40E Processor
> stepping	: 0
> microcode	: 0x5000101
> cpu MHz		: 800.000
> cache size	: 512 KB
> physical id	: 0
> siblings	: 2
> core id		: 0
> cpu cores	: 2
> apicid		: 0
> initial apicid	: 0
> fdiv_bug	: no
> f00f_bug	: no
> coma_bug	: no
> fpu		: yes
> fpu_exception	: yes
> cpuid level	: 6
> wp		: yes
> flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc nonstop_tsc extd_apicid aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt arat hw_pstate npt lbrv svm_lock nrip_save pausefilter vmmcall
> bogomips	: 1999.83
> clflush size	: 64
> cache_alignment	: 64
> address sizes	: 36 bits physical, 48 bits virtual
> power management: ts ttp tm stc 100mhzsteps hwpstate
> 
> processor	: 1
> vendor_id	: AuthenticAMD
> cpu family	: 20
> model		: 2
> model name	: AMD G-T40E Processor
> stepping	: 0
> microcode	: 0x5000101
> cpu MHz		: 800.000
> cache size	: 512 KB
> physical id	: 0
> siblings	: 2
> core id		: 1
> cpu cores	: 2
> apicid		: 1
> initial apicid	: 1
> fdiv_bug	: no
> f00f_bug	: no
> coma_bug	: no
> fpu		: yes
> fpu_exception	: yes
> cpuid level	: 6
> wp		: yes
> flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc nonstop_tsc extd_apicid aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt arat hw_pstate npt lbrv svm_lock nrip_save pausefilter vmmcall
> bogomips	: 1999.83
> clflush size	: 64
> cache_alignment	: 64
> address sizes	: 36 bits physical, 48 bits virtual
> power management: ts ttp tm stc 100mhzsteps hwpstate
> 
> root at apu1a:~/rate_limiters# ethtool -i eth0
> driver: r8169
> version: 2.3LK-NAPI
> firmware-version: rtl_nic/rtl8168e-2.fw
> bus-info: 0000:01:00.0
> supports-statistics: yes
> supports-test: no
> supports-eeprom-access: no
> supports-register-dump: yes
> supports-priv-flags: no
>