From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 25ED93B2A4 for ; Sun, 30 Dec 2018 16:51:28 -0500 (EST) Received: from hms-beagle2.lan ([77.181.129.114]) by mail.gmx.com (mrgmx101 [212.227.17.168]) with ESMTPSA (Nemesis) id 0M4ScS-1hXzpi0WtA-00ygJU; Sun, 30 Dec 2018 22:51:26 +0100 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) From: Sebastian Moeller In-Reply-To: <555BACDF-7A1E-4C6B-BFB1-0C5ACB77715E@heistp.net> Date: Sun, 30 Dec 2018 22:51:24 +0100 Cc: Cake List Content-Transfer-Encoding: quoted-printable Message-Id: <79875813-63E8-48D4-9A1F-B7C18F1325D0@gmx.de> References: <555BACDF-7A1E-4C6B-BFB1-0C5ACB77715E@heistp.net> To: Pete Heist X-Mailer: Apple Mail (2.3445.9.1) X-Provags-ID: V03:K1:QYjxGN2JwXQlm+Vi128OdHJBZvMK2gbo5LPJiTOndIIbYahX7nk Y9ZweJQWo0fQDtC6x/HILDx/lxcfTG814pkuMLlLEOzLF0EL6ken23um4MZ5W5incNecohq X9JazuUX4iQHx8vTXqwNfoCRUFQweNXKsaZPwowjWx3fGVVmqMfDIHOx+fFSgqNalx9+9WU ktF0o5ZEadmSVd4X3TXdQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:p1xp1v9PFV8=:RbxWD3X9wF2wMKwn7c85Zv ynG+IixRkphWkYqdRvrVIyedFJ9wsr1n7qf3N562BxHPZg6lEJqh+P3QFpfw0Ga4jonhUGX5A OAXPrzUr5RMYOBdE30bZdFSTmJtM4p9Duyl2JXBUQK/4VteovyFkkf2cfQ9rE8P5wy6GxHD1k N9B78rHIuqxszTMdMqrkAMMuoxBkseAvJbagu+V4Hxo7zYpY0nAjXUwvBXIrPNYhvS7CM7L92 +t+XTqkZW9txB/b49wgJn4Xpo46BFxsXHnQQfZCLLiE3+b/ARVwONdmnRjN54KMP2rLaIERFg ysUR4gWGz1edn8tFSCQOSRYEEUP0rwOHolIjUVW5CZ2/7nMg55F+dzpLN2LUt7ccKD3UPLLRI FTbpKdpT+W9gBmlYNhKAlMMIreMKCPhsV7UiKIg9XR743cRGT+LfDLqq58Z+jRdx2USHfNnUA COh1R4bed1g4V6XQ8OBcc+NzFkJll08p5i0Hx0BH3bdRuPDSUFkx9EkWiroa6fHPuwpeeZFst jf+YecYjfjK82p1fA1MrZzaFf3n0kuXiG/fkwHzZwrWLVNsn5j2BiKDcgmBpvg64PMDgc8Ev4 vraXkae0xnPSeYMEehYqzj5KnbVMbtzWVoDFnSCMvyTp0542+9LMP0sa6DWLsle1MHAIVJ7CQ UebgeRYlzvWrzyrQTsPeFQ5IuhzFMAWu9p1JkyAaJc95wm4UpXy5Q7siDFH2N+12tXhVbsp2a JXVsp0fmSA4GGJzLlmwwwmlxgpRC/wBJygRAzjfeDD9dcLiTY6z4/imoF6t1r4r9Avyx18Kkf 7lwUoGMZjDOlyaHrB0Rzf5zRWnzR1PURzKOlymm6lsRdcROlsQHxZU3UALGSY7NmLJDQrl7HD d6HrXS8imhK+P3KRjVhcbbz8DchXnRqDAiu2CuXHPED5Wu3wkIiDm7dIN373jj Subject: Re: [Cake] cake and hfsc rate limiters outperforming htb on one-armed router X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2018 21:51:28 -0000 Hi Pete, you might want to have a look at htb's burst and cburst parameters, as = these should allow to trade in latency under load for bandwidth = utilization. Best Regards Sebastian > On Dec 30, 2018, at 21:42, Pete Heist wrote: >=20 > It=E2=80=99s a bit more complicated than this. It looks like the htb = rate limiter is different in that as rates increase the actual rate = starts to deviate from the specified rate early on, but it rather = gracefully handles the =E2=80=9Cout of CPU=E2=80=9D situation, where it = still maintains control of the queue, just gradually fails to meet the = rate specified by greater and greater percentages. >=20 > Instead of a single flow test with iperf3, here are rates that each = limiter can reach on egress of both apu1a interfaces during an rrul_be = test: >=20 > # - max limit on APU for one-armed routing, rrul_be test 4+4 flows = (firewall on): > # - cake: 210mbit > # - htb+fq_codel: 93%@100mbit, 90%@200mbit, 84%@300mbit, = 72%@400mbit, 59%@500mbit > # - hfsc+fq_codel: 310mbit > # - hfsc+cake: 300mbit >=20 > The numbers for cake and hfsc are right before loss of queue, and with = htb the queue isn=E2=80=99t lost even at 500mbit, for example, just the = actual rate is only 59% of what was specified. >=20 > I really need to graph the specified rate vs the actual rate, = inter-flow and intra-flow latency, stepped 25mbit at a time. I think it = would be interesting, so this is on my todo list if there=E2=80=99s time = after the ISP config gets done. >=20 >> On Dec 28, 2018, at 1:17 AM, Pete Heist wrote: >>=20 >> For whatever reason, I=E2=80=99m seeing the rate limiters in cake and = hfsc vastly outperform htb in the one-armed router configuration I = described in my previous thread. To simplify things, I apply the qdiscs = with a single class only at egress of eth0 on apu1a: >>=20 >> apu2a <=E2=80=94 default VLAN =E2=80=94> apu1a <=E2=80=94 VLAN = 3300 =E2=80=94> apu2b >>=20 >> I use iperf3 from apu2a to apu2b and find the rate at which things = break down. Whereas cake and hfsc can both reach around 850mbit, htb is = breaking down at around 200mbit, which seems rather strange. This could = be a function of the older kernel I have to use, the hardware, or maybe = htb just isn=E2=80=99t suited well to this task for some reason. I wish = I knew, as I=E2=80=99d rather be using htb for this task than hfsc = (especially given the lockup issue with cake)... >>=20 >> =E2=80=94=E2=80=94 >>=20 >> #!/bin/bash=20 >>=20 >> # point where iperf3 throughput drops below ~93% of theoretical: >> # htb: 200mbit >> # hfsc: 850mbit >> # cake: 850mbit >>=20 >> IFACE=3Deth0 >> RATE=3D850mbit >>=20 >> start_htb() { >> stop >> tc qdisc add dev $IFACE root handle 1: htb default 1 >> tc class add dev $IFACE parent 1: classid 1:1 htb rate $RATE = ceil $RATE >> tc qdisc add dev $IFACE parent 1:1 handle 10: fq_codel >> } >>=20 >> start_hfsc() { >> stop >> tc qdisc add dev $IFACE root handle 1: hfsc default 1 >> tc class add dev $IFACE parent 1: classid 1:1 hfsc sc rate = $RATE ul rate $RATE >> tc qdisc add dev $IFACE parent 1:1 handle 10: fq_codel >> } >>=20 >> start_cake() { >> stop >> tc qdisc add dev $IFACE root cake bandwidth $RATE >> } >>=20 >> stop() { >> tc qdisc del dev $IFACE root &>/dev/null >> tc qdisc del dev $IFACE ingress &>/dev/null >> } >>=20 >> "$@=E2=80=9C >> =E2=80=94=E2=80=94 >>=20 >> root@apu1a:~/rate_limiters# uname -a >> Linux apu1a 3.16.7-ckt9-voyage #1 SMP Thu Apr 23 11:10:44 HKT 2015 = i686 GNU/Linux >>=20 >> root@apu1a:~/rate_limiters# cat /proc/cpuinfo=20 >> processor : 0 >> vendor_id : AuthenticAMD >> cpu family : 20 >> model : 2 >> model name : AMD G-T40E Processor >> stepping : 0 >> microcode : 0x5000101 >> cpu MHz : 800.000 >> cache size : 512 KB >> physical id : 0 >> siblings : 2 >> core id : 0 >> cpu cores : 2 >> apicid : 0 >> initial apicid : 0 >> fdiv_bug : no >> f00f_bug : no >> coma_bug : no >> fpu : yes >> fpu_exception : yes >> cpuid level : 6 >> wp : yes >> flags : fpu vme de pse tsc msr pae mce cx8 apic sep = mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx = mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc nonstop_tsc extd_apicid = aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic = cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt arat = hw_pstate npt lbrv svm_lock nrip_save pausefilter vmmcall >> bogomips : 1999.83 >> clflush size : 64 >> cache_alignment : 64 >> address sizes : 36 bits physical, 48 bits virtual >> power management: ts ttp tm stc 100mhzsteps hwpstate >>=20 >> processor : 1 >> vendor_id : AuthenticAMD >> cpu family : 20 >> model : 2 >> model name : AMD G-T40E Processor >> stepping : 0 >> microcode : 0x5000101 >> cpu MHz : 800.000 >> cache size : 512 KB >> physical id : 0 >> siblings : 2 >> core id : 1 >> cpu cores : 2 >> apicid : 1 >> initial apicid : 1 >> fdiv_bug : no >> f00f_bug : no >> coma_bug : no >> fpu : yes >> fpu_exception : yes >> cpuid level : 6 >> wp : yes >> flags : fpu vme de pse tsc msr pae mce cx8 apic sep = mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx = mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc nonstop_tsc extd_apicid = aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic = cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt arat = hw_pstate npt lbrv svm_lock nrip_save pausefilter vmmcall >> bogomips : 1999.83 >> clflush size : 64 >> cache_alignment : 64 >> address sizes : 36 bits physical, 48 bits virtual >> power management: ts ttp tm stc 100mhzsteps hwpstate >>=20 >> root@apu1a:~/rate_limiters# ethtool -i eth0 >> driver: r8169 >> version: 2.3LK-NAPI >> firmware-version: rtl_nic/rtl8168e-2.fw >> bus-info: 0000:01:00.0 >> supports-statistics: yes >> supports-test: no >> supports-eeprom-access: no >> supports-register-dump: yes >> supports-priv-flags: no >>=20 >=20 > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake