From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id D084D3B2A4 for ; Sun, 30 Dec 2018 15:42:39 -0500 (EST) Received: by mail-wr1-x431.google.com with SMTP id j2so25300260wrw.1 for ; Sun, 30 Dec 2018 12:42:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=from:content-transfer-encoding:mime-version:subject:date:references :to:in-reply-to:message-id; bh=2fhkdCcmxgNzNpZCQRgBUvj06ZBaLTGMyqNjsZGzPFQ=; b=Gq+ikJAShX9SkgMhpwLF0XeFvk7BNHj0handmw+L0/mw5z6omwqz5q/+2+/QdwRcN6 nCNyn+1oZ01oo6/AmmIKC0GXlZqnflFK75arm6YG8KezAB4hlr/KoRdJV/lr+Nuzf8+U tIBl8ojilUjQErLe4vuqczNK+0vjkGnhOzIEVqf3JYIJcN2vFOkc0VSAoNGZZLDV481U U8oClIwti75G/zhiE2smnYicRl5zXkIMsYMnz5EabymCOs6rXrvc6h1jUmzaY/tsya3v 3bd1ogxlO3IMoiVnblfs4in8H8KArhYlTBn3p3efETzDooXwD1e7AR8VZDGmyDSWqL7d /9mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:date:references:to:in-reply-to:message-id; bh=2fhkdCcmxgNzNpZCQRgBUvj06ZBaLTGMyqNjsZGzPFQ=; b=sJWh3+nm+QKK0VIlWRAtcz3yDKg3p1LuW8qrgr37IUJR6GDxHK1IRimRt0wz1CcTkf 7TYjgs4FZYhNppRhwPOTfgLKLrZ1nioZie8fu10arLApx29HfMfi7tM+3M86ZoLCJnqn ifz9LN+TXGfdDG6uQ3SFXWaVYonpedmfEhK4339UQKJb8y3D6oz2MRx+q62N5rKRTlxp tgemQjYgsv0JEmH+N2X0A5fR69N0iwWRUj+pjozokh6lvUhGEZR9r/i5yuSS5rVsv4Ic hVkklzNm8za0d0kXDuZOIuYfYyRX+DKrNnK6umF9LcPoPrIahEr3DIhWUELs3RD26HXW a7Cg== X-Gm-Message-State: AJcUukfUjHi370P3qakVNKKbCsSOdCNLURG6TQ1D5WvYzIRrBwz4eC9y STNYVYZK5ksr4lF6HUVbXOk6S5avLhg= X-Google-Smtp-Source: ALg8bN40a8ZvDoVecXbttFyq87ai+afwSZPNfBNgLn8EYEP7+pltRbP67YPc18/gOoXqUnh8QV+EkA== X-Received: by 2002:adf:9d4c:: with SMTP id o12mr29471899wre.94.1546202558555; Sun, 30 Dec 2018 12:42:38 -0800 (PST) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id y34sm106570544wrd.68.2018.12.30.12.42.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Dec 2018 12:42:37 -0800 (PST) From: Pete Heist Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Date: Sun, 30 Dec 2018 21:42:36 +0100 References: To: Cake List In-Reply-To: Message-Id: <555BACDF-7A1E-4C6B-BFB1-0C5ACB77715E@heistp.net> X-Mailer: Apple Mail (2.3445.9.1) Subject: Re: [Cake] cake and hfsc rate limiters outperforming htb on one-armed router X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2018 20:42:40 -0000 It=E2=80=99s a bit more complicated than this. It looks like the htb = rate limiter is different in that as rates increase the actual rate = starts to deviate from the specified rate early on, but it rather = gracefully handles the =E2=80=9Cout of CPU=E2=80=9D situation, where it = still maintains control of the queue, just gradually fails to meet the = rate specified by greater and greater percentages. Instead of a single flow test with iperf3, here are rates that each = limiter can reach on egress of both apu1a interfaces during an rrul_be = test: # - max limit on APU for one-armed routing, rrul_be test 4+4 flows = (firewall on): # - cake: 210mbit # - htb+fq_codel: 93%@100mbit, 90%@200mbit, 84%@300mbit, 72%@400mbit, = 59%@500mbit # - hfsc+fq_codel: 310mbit # - hfsc+cake: 300mbit The numbers for cake and hfsc are right before loss of queue, and with = htb the queue isn=E2=80=99t lost even at 500mbit, for example, just the = actual rate is only 59% of what was specified. I really need to graph the specified rate vs the actual rate, inter-flow = and intra-flow latency, stepped 25mbit at a time. I think it would be = interesting, so this is on my todo list if there=E2=80=99s time after = the ISP config gets done. > On Dec 28, 2018, at 1:17 AM, Pete Heist wrote: >=20 > For whatever reason, I=E2=80=99m seeing the rate limiters in cake and = hfsc vastly outperform htb in the one-armed router configuration I = described in my previous thread. To simplify things, I apply the qdiscs = with a single class only at egress of eth0 on apu1a: >=20 > apu2a <=E2=80=94 default VLAN =E2=80=94> apu1a <=E2=80=94 VLAN = 3300 =E2=80=94> apu2b >=20 > I use iperf3 from apu2a to apu2b and find the rate at which things = break down. Whereas cake and hfsc can both reach around 850mbit, htb is = breaking down at around 200mbit, which seems rather strange. This could = be a function of the older kernel I have to use, the hardware, or maybe = htb just isn=E2=80=99t suited well to this task for some reason. I wish = I knew, as I=E2=80=99d rather be using htb for this task than hfsc = (especially given the lockup issue with cake)... >=20 > =E2=80=94=E2=80=94 >=20 > #!/bin/bash=20 >=20 > # point where iperf3 throughput drops below ~93% of theoretical: > # htb: 200mbit > # hfsc: 850mbit > # cake: 850mbit >=20 > IFACE=3Deth0 > RATE=3D850mbit >=20 > start_htb() { > stop > tc qdisc add dev $IFACE root handle 1: htb default 1 > tc class add dev $IFACE parent 1: classid 1:1 htb rate $RATE = ceil $RATE > tc qdisc add dev $IFACE parent 1:1 handle 10: fq_codel > } >=20 > start_hfsc() { > stop > tc qdisc add dev $IFACE root handle 1: hfsc default 1 > tc class add dev $IFACE parent 1: classid 1:1 hfsc sc rate = $RATE ul rate $RATE > tc qdisc add dev $IFACE parent 1:1 handle 10: fq_codel > } >=20 > start_cake() { > stop > tc qdisc add dev $IFACE root cake bandwidth $RATE > } >=20 > stop() { > tc qdisc del dev $IFACE root &>/dev/null > tc qdisc del dev $IFACE ingress &>/dev/null > } >=20 > "$@=E2=80=9C > =E2=80=94=E2=80=94 >=20 > root@apu1a:~/rate_limiters# uname -a > Linux apu1a 3.16.7-ckt9-voyage #1 SMP Thu Apr 23 11:10:44 HKT 2015 = i686 GNU/Linux >=20 > root@apu1a:~/rate_limiters# cat /proc/cpuinfo=20 > processor : 0 > vendor_id : AuthenticAMD > cpu family : 20 > model : 2 > model name : AMD G-T40E Processor > stepping : 0 > microcode : 0x5000101 > cpu MHz : 800.000 > cache size : 512 KB > physical id : 0 > siblings : 2 > core id : 0 > cpu cores : 2 > apicid : 0 > initial apicid : 0 > fdiv_bug : no > f00f_bug : no > coma_bug : no > fpu : yes > fpu_exception : yes > cpuid level : 6 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge = mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext = fxsr_opt pdpe1gb rdtscp lm constant_tsc nonstop_tsc extd_apicid = aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic = cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt arat = hw_pstate npt lbrv svm_lock nrip_save pausefilter vmmcall > bogomips : 1999.83 > clflush size : 64 > cache_alignment : 64 > address sizes : 36 bits physical, 48 bits virtual > power management: ts ttp tm stc 100mhzsteps hwpstate >=20 > processor : 1 > vendor_id : AuthenticAMD > cpu family : 20 > model : 2 > model name : AMD G-T40E Processor > stepping : 0 > microcode : 0x5000101 > cpu MHz : 800.000 > cache size : 512 KB > physical id : 0 > siblings : 2 > core id : 1 > cpu cores : 2 > apicid : 1 > initial apicid : 1 > fdiv_bug : no > f00f_bug : no > coma_bug : no > fpu : yes > fpu_exception : yes > cpuid level : 6 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge = mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext = fxsr_opt pdpe1gb rdtscp lm constant_tsc nonstop_tsc extd_apicid = aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic = cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt arat = hw_pstate npt lbrv svm_lock nrip_save pausefilter vmmcall > bogomips : 1999.83 > clflush size : 64 > cache_alignment : 64 > address sizes : 36 bits physical, 48 bits virtual > power management: ts ttp tm stc 100mhzsteps hwpstate >=20 > root@apu1a:~/rate_limiters# ethtool -i eth0 > driver: r8169 > version: 2.3LK-NAPI > firmware-version: rtl_nic/rtl8168e-2.fw > bus-info: 0000:01:00.0 > supports-statistics: yes > supports-test: no > supports-eeprom-access: no > supports-register-dump: yes > supports-priv-flags: no >=20