From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 2615D3B2A4 for ; Sun, 30 Dec 2018 17:36:30 -0500 (EST) Received: by mail-wm1-x329.google.com with SMTP id y139so22423678wmc.5 for ; Sun, 30 Dec 2018 14:36:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=bhfSCQRwu9yMDYj/u16/43/3gy257IxU7n8mOu7W5H0=; b=iJhy6OgoL1pIfTzFIoghyIzsYD9f27GNy7fwm2xJZzV8vMgMQMaKoc7pW452cj8NxZ RxfIk2987E52tPLwF7lcJy2se4SBvgBV7lDH1Id0o3t0sfSM9Y7q/FFsw+RVY2UvDYYt LW3XawIdjCjA4yBpGrCm0/VWuIml8hRgHmy+FyRHl0Mgu6hvp2Nypw0WRmoHmAN3uG7w j9WLylMN8lwKcp/6O+U6g2ASBMAOAaCg9WoTQrlgk6/YHvAuRmOLDX3JYi3gC6vqfs55 tUyVb0BCTk19Z3dRFmqX/lpD3i7Pf/rBeiTWj0WUNcSxS40iviQgHQLsjtYzHOnxZTcK qS/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=bhfSCQRwu9yMDYj/u16/43/3gy257IxU7n8mOu7W5H0=; b=pnviBs7JdKvngDYAtlgT4JvvAGroaY4vRHf6kawpfMha68aPaobFDa4lhyBpB981Yu ARSYg+lpkG8UUiGwlUy80496m9+PZdGAL/gYdd6OlCiX2Q8mYiGO/BiboDvOX7obWmjD tWn2wqpTBDxHNnIXfWPoqt2mJfiWmPqE3QSG3kMz7mDTmEyRiu2NBGU8/F1a2qhOXKGU tV4+F3w+DhTFCui/w/UL5X5tGYHRlYIWmJOau5Sp/8i6KxUtr65UVTt5uX835PlALuCb IcGb6Wcf94rntVzJDL0l6ol++SEIIvNMbbPbYn61NQNrs3K4fxjfT26MzNBokagK1YSM h+QA== X-Gm-Message-State: AA+aEWbl+3pPlN9zjrBZ/0iF0Wb1JlZUlcJFpg9Ri+46w+g9odmTReuJ yI0nvnjKaMKaJUP7YXao5ducZQ== X-Google-Smtp-Source: AFSGD/VClS3aSJz6tI5jMEY6DfEZnhjzKdqZbiOc1q1Xe2P9p3bemw33I3qUcE3d/KXGfshaxVX80Q== X-Received: by 2002:a1c:d74a:: with SMTP id o71mr29642108wmg.73.1546209389135; Sun, 30 Dec 2018 14:36:29 -0800 (PST) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id q9sm27514891wrv.26.2018.12.30.14.36.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Dec 2018 14:36:28 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) From: Pete Heist In-Reply-To: <79875813-63E8-48D4-9A1F-B7C18F1325D0@gmx.de> Date: Sun, 30 Dec 2018 23:36:27 +0100 Cc: Cake List Content-Transfer-Encoding: quoted-printable Message-Id: <52F7AEE6-D156-4569-94C4-9E9E1590C84F@heistp.net> References: <555BACDF-7A1E-4C6B-BFB1-0C5ACB77715E@heistp.net> <79875813-63E8-48D4-9A1F-B7C18F1325D0@gmx.de> To: Sebastian Moeller X-Mailer: Apple Mail (2.3445.9.1) Subject: Re: [Cake] cake and hfsc rate limiters outperforming htb on one-armed router X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2018 22:36:30 -0000 The experiments I did with those didn=E2=80=99t yield great results, = with changing a value by one MTU sometimes causing sudden throughput or = inter-flow latency increases, with the tradeoffs not being clear. I=E2=80=99= m afraid admins could easily cause problems fiddling with these. = Fortunately most customer facing routers have aggregate bitrates that an = APU1 can handle even with default htb, or cake. I also appreciate that = such settings don=E2=80=99t exist in cake=E2=80=A6 :) > On Dec 30, 2018, at 10:51 PM, Sebastian Moeller = wrote: >=20 > Hi Pete, >=20 > you might want to have a look at htb's burst and cburst parameters, as = these should allow to trade in latency under load for bandwidth = utilization. >=20 >=20 > Best Regards > Sebastian >=20 >> On Dec 30, 2018, at 21:42, Pete Heist wrote: >>=20 >> It=E2=80=99s a bit more complicated than this. It looks like the htb = rate limiter is different in that as rates increase the actual rate = starts to deviate from the specified rate early on, but it rather = gracefully handles the =E2=80=9Cout of CPU=E2=80=9D situation, where it = still maintains control of the queue, just gradually fails to meet the = rate specified by greater and greater percentages. >>=20 >> Instead of a single flow test with iperf3, here are rates that each = limiter can reach on egress of both apu1a interfaces during an rrul_be = test: >>=20 >> # - max limit on APU for one-armed routing, rrul_be test 4+4 flows = (firewall on): >> # - cake: 210mbit >> # - htb+fq_codel: 93%@100mbit, 90%@200mbit, 84%@300mbit, = 72%@400mbit, 59%@500mbit >> # - hfsc+fq_codel: 310mbit >> # - hfsc+cake: 300mbit >>=20 >> The numbers for cake and hfsc are right before loss of queue, and = with htb the queue isn=E2=80=99t lost even at 500mbit, for example, just = the actual rate is only 59% of what was specified. >>=20 >> I really need to graph the specified rate vs the actual rate, = inter-flow and intra-flow latency, stepped 25mbit at a time. I think it = would be interesting, so this is on my todo list if there=E2=80=99s time = after the ISP config gets done. >>=20 >>> On Dec 28, 2018, at 1:17 AM, Pete Heist wrote: >>>=20 >>> For whatever reason, I=E2=80=99m seeing the rate limiters in cake = and hfsc vastly outperform htb in the one-armed router configuration I = described in my previous thread. To simplify things, I apply the qdiscs = with a single class only at egress of eth0 on apu1a: >>>=20 >>> apu2a <=E2=80=94 default VLAN =E2=80=94> apu1a <=E2=80=94 VLAN = 3300 =E2=80=94> apu2b >>>=20 >>> I use iperf3 from apu2a to apu2b and find the rate at which things = break down. Whereas cake and hfsc can both reach around 850mbit, htb is = breaking down at around 200mbit, which seems rather strange. This could = be a function of the older kernel I have to use, the hardware, or maybe = htb just isn=E2=80=99t suited well to this task for some reason. I wish = I knew, as I=E2=80=99d rather be using htb for this task than hfsc = (especially given the lockup issue with cake)... >>>=20 >>> =E2=80=94=E2=80=94 >>>=20 >>> #!/bin/bash=20 >>>=20 >>> # point where iperf3 throughput drops below ~93% of theoretical: >>> # htb: 200mbit >>> # hfsc: 850mbit >>> # cake: 850mbit >>>=20 >>> IFACE=3Deth0 >>> RATE=3D850mbit >>>=20 >>> start_htb() { >>> stop >>> tc qdisc add dev $IFACE root handle 1: htb default 1 >>> tc class add dev $IFACE parent 1: classid 1:1 htb rate $RATE = ceil $RATE >>> tc qdisc add dev $IFACE parent 1:1 handle 10: fq_codel >>> } >>>=20 >>> start_hfsc() { >>> stop >>> tc qdisc add dev $IFACE root handle 1: hfsc default 1 >>> tc class add dev $IFACE parent 1: classid 1:1 hfsc sc rate = $RATE ul rate $RATE >>> tc qdisc add dev $IFACE parent 1:1 handle 10: fq_codel >>> } >>>=20 >>> start_cake() { >>> stop >>> tc qdisc add dev $IFACE root cake bandwidth $RATE >>> } >>>=20 >>> stop() { >>> tc qdisc del dev $IFACE root &>/dev/null >>> tc qdisc del dev $IFACE ingress &>/dev/null >>> } >>>=20 >>> "$@=E2=80=9C >>> =E2=80=94=E2=80=94 >>>=20 >>> root@apu1a:~/rate_limiters# uname -a >>> Linux apu1a 3.16.7-ckt9-voyage #1 SMP Thu Apr 23 11:10:44 HKT 2015 = i686 GNU/Linux >>>=20 >>> root@apu1a:~/rate_limiters# cat /proc/cpuinfo=20 >>> processor : 0 >>> vendor_id : AuthenticAMD >>> cpu family : 20 >>> model : 2 >>> model name : AMD G-T40E Processor >>> stepping : 0 >>> microcode : 0x5000101 >>> cpu MHz : 800.000 >>> cache size : 512 KB >>> physical id : 0 >>> siblings : 2 >>> core id : 0 >>> cpu cores : 2 >>> apicid : 0 >>> initial apicid : 0 >>> fdiv_bug : no >>> f00f_bug : no >>> coma_bug : no >>> fpu : yes >>> fpu_exception : yes >>> cpuid level : 6 >>> wp : yes >>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep = mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx = mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc nonstop_tsc extd_apicid = aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic = cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt arat = hw_pstate npt lbrv svm_lock nrip_save pausefilter vmmcall >>> bogomips : 1999.83 >>> clflush size : 64 >>> cache_alignment : 64 >>> address sizes : 36 bits physical, 48 bits virtual >>> power management: ts ttp tm stc 100mhzsteps hwpstate >>>=20 >>> processor : 1 >>> vendor_id : AuthenticAMD >>> cpu family : 20 >>> model : 2 >>> model name : AMD G-T40E Processor >>> stepping : 0 >>> microcode : 0x5000101 >>> cpu MHz : 800.000 >>> cache size : 512 KB >>> physical id : 0 >>> siblings : 2 >>> core id : 1 >>> cpu cores : 2 >>> apicid : 1 >>> initial apicid : 1 >>> fdiv_bug : no >>> f00f_bug : no >>> coma_bug : no >>> fpu : yes >>> fpu_exception : yes >>> cpuid level : 6 >>> wp : yes >>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep = mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx = mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc nonstop_tsc extd_apicid = aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic = cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt arat = hw_pstate npt lbrv svm_lock nrip_save pausefilter vmmcall >>> bogomips : 1999.83 >>> clflush size : 64 >>> cache_alignment : 64 >>> address sizes : 36 bits physical, 48 bits virtual >>> power management: ts ttp tm stc 100mhzsteps hwpstate >>>=20 >>> root@apu1a:~/rate_limiters# ethtool -i eth0 >>> driver: r8169 >>> version: 2.3LK-NAPI >>> firmware-version: rtl_nic/rtl8168e-2.fw >>> bus-info: 0000:01:00.0 >>> supports-statistics: yes >>> supports-test: no >>> supports-eeprom-access: no >>> supports-register-dump: yes >>> supports-priv-flags: no >>>=20 >>=20 >> _______________________________________________ >> Cake mailing list >> Cake@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/cake >=20