From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id AD0353CB35 for ; Mon, 10 Sep 2018 18:40:45 -0400 (EDT) Received: by mail-wr1-x429.google.com with SMTP id u12-v6so23553112wrr.4 for ; Mon, 10 Sep 2018 15:40:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=3uoxFCJ9cgVu7/d5GMQB6xSltA344berZAEekH3CtL4=; b=KBsFuOowW3wWmymkICwpH63r3mjc5yb4+C0svOCmMDOJCB1SDrqcWgZjXpHHJUXWk5 +fJUvBbE+eKGK6vzhv14iLztilkDsVKyRo748zdNUHTJ9WC6pwxQ5MFQNksiHuA0ks3W 3f/cxFgoaSMsC2aZFRaQ2Xa+ou0penUiD8seDjIpbljNtVoOMsylFJJhe+CpUYzxMgji XoHkLVLB+Wu673Dji+rUOrFBN+UGJF6zjmZjMJMrxd81hdi9QF4xaS+ZT9O0wZEQB36W ZRSi7X2hMNpGIb4TUbQaUURWF688UEBnXvdfoK5kAIUyjBKFXyz3o7gboCwt/rPkXzBR 20Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=3uoxFCJ9cgVu7/d5GMQB6xSltA344berZAEekH3CtL4=; b=TVk7nkBUsbnZXkgQiLVqm+ksto1MVBBaQoyvLNlKuHO3kHKzzeXCkd7grAx+M63blb SYCO2ubOujZaxkcwCUkOvDZDeHRK3fPIgMuriKn/gVm5Z4ns9ETuYueyqa9iI+vk+DC0 OjLNxAew8S9Sv3PB+JtAhxDOV0r2vYWPZpyUZ1vwCZ+M7knome3QQeio9jlMTsPCv9+e +yaKnFf+hKIhbJKvRqdv9xYRMT9h3ux+g2ink1NjXeOx0vdVl7vI+PA5nBCeAbem9hST MHA9HCB93QGuM8y+Jtjpo7FKwwpujFXN2gWRCl8aV/INl23rEXMd43upMH3VVVoA+iWB eO4g== X-Gm-Message-State: APzg51CaAOLCFFkxamR8KGqbBf81u4gs4DhvW9yVc3GI5pjs0g0DJ8UN 0YdZYv++IuIldKlixwMsUsgR9w== X-Google-Smtp-Source: ANB0VdYE4/etReFbWbbnebvSo4gi8nkioDU4n2nCn49Myv2M+6j2IqOPWYteqSNttleKwOD43MvxyA== X-Received: by 2002:a05:6000:c:: with SMTP id h12mr17429014wrx.103.1536619244329; Mon, 10 Sep 2018 15:40:44 -0700 (PDT) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id j191-v6sm19996935wmb.37.2018.09.10.15.40.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Sep 2018 15:40:43 -0700 (PDT) From: Pete Heist Message-Id: Content-Type: multipart/mixed; boundary="Apple-Mail=_A831CE65-766F-4B3E-B1E7-4045E6A1AE7B" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Date: Tue, 11 Sep 2018 00:40:42 +0200 In-Reply-To: Cc: Cake List To: Dave Taht References: <87zhwxzh8o.fsf@toke.dk> <139B295B-7371-43DE-B472-DE629C9B8432@heistp.net> <87efe65wol.fsf@toke.dk> <6C556301-015B-4903-AE5A-F22D3517FFCC@heistp.net> X-Mailer: Apple Mail (2.3445.9.1) Subject: [Cake] Cake vs fq_codel and c/burst on an ER-X bridge X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Sep 2018 22:40:45 -0000 --Apple-Mail=_A831CE65-766F-4B3E-B1E7-4045E6A1AE7B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Subject changed from =E2=80=9CCake on elements of a bridge=E2=80=9D... On Sep 10, 2018, at 9:55 PM, Dave Taht wrote: >=20 > On Mon, Sep 10, 2018 at 12:29 PM Pete Heist wrote: >>=20 >> For anyone who followed this, yes, the regular soft bridge (i.e. set = interfaces bridge br0) works fine on the ER-X, as I suspect it would on = most any Linux. A few notes about it: >>=20 >> - Your qdisc must be added to the physical interface (e.g. eth4), not = the bridge interface >> - Unlike the hardware bridge which has its own MAC, the soft bridge = seems to take the MAC of the lowest (or first listed?) interface port >> - On ER-X, bridge-nf-call-iptables=3D1 is the default so nothing = needs to be changed there for firewalling >> - When firewalling the bridged WAN interface, =E2=80=98in=E2=80=99 = corresponds to bridged traffic and =E2=80=98local=E2=80=99 to routed = traffic, which is different from the semantics for ordinary routed = traffic >> - I can do stateful firewalling for bridged hosts with =E2=80=9Caccept = established and related=E2=80=9D, but have to explicitly allow DHCP (UDP = source/dest port 67-68) in the WAN interface=E2=80=99s =E2=80=98in=E2=80=99= rules for DHCP traffic to pass through the bridge >>=20 >> Performance: >>=20 >> Using Cake with this setup, the fun ends at around 110 Mbit with = ksoftirqd thrashing. Unsurprisingly, there=E2=80=99s probably some = overhead here with the soft bridge. For my purposes though (50 Mbit), = it=E2=80=99s enough, barely=E2=80=A6 >=20 > Can I encourage you to give regular ole htb+fq_codel sqm a shot with a > bigger burst and cburst size for htb? Fiddling with the htb quantum > isn't helping much, > but try this, from: https://github.com/tohojo/sqm-scripts/issues/71 >=20 > (I am thinking burst and cburst should be about 1.1ms of buffering in = size) So this has turned info an interesting exercise that produced a result = counter to what the common wisdom has been (that fq_codel is = =E2=80=9Cfaster=E2=80=9D than cake). Because of that, I=E2=80=99m open = to criticism of my methodology and different criteria for a successful = bitrate for the shaper. First, note that these tests still through a bridge as above, but are = for a more typical setup with separate qdisc instances on egress and = ingress, as opposed to my =E2=80=9C110 Mbit=E2=80=9D result from above, = which was for egress and ingress through a common IFB. It occurs to me that what I really want to know is the maximum set = bitrate for the shaper where it still appears to be behaving properly = and consistently, meaning, the actual measured TCP throughput is held = steady, and at the expected percentage less than the set bitrate. I = first find this out by setting a =E2=80=9Ccomfortable=E2=80=9D rate of = 100Mbit and checking TCP throughput with iperf3, which is typically = around 5% less than the set bitrate. Then I increase the shaper=E2=80=99s = bitrate 5Mbit at a time and re-run the test until I find the last = bitrate at which iperf3 reports a stable (within a few percent) and = correct rate over 10 seconds for several runs in a row. See the attached = iperf3 results for sample runs around the threshold rates. qdisc: egress Mbit / ingress Mbit cake nat dual-srchost / cake nat dual-dsthost ingress: 135 / 145 htb+fq_codel: 125 / 125 htb+fq_codel with burst/cburst=3D96000: 155 / 155 So with this testing criteria, I=E2=80=99m actually seeing cake = =E2=80=9Cwin=E2=80=9D (with the exception of setting htb's burst/cburst = to 96000, which shows a clear improvement, probably at the expense of = something). I also see that the ingress rate for cake can be held steady = to a bit higher of a bitrate than egress. I am using the =E2=80=98ingress=E2= =80=99 keyword on ingress. I have to be careful here because from run to = run there can be slight variations in behavior, but having repeated it = several times at each bitrate around the threshold, I=E2=80=99m fairly = certain about the results. In the ER-X manual (https://dl.ubnt.com/guides/edgemax/EdgeOS_UG.pdf), = they give a guideline of 100-250Mbps on the =E2=80=9Cexpected Smart = Queue shaping performance=E2=80=9D (which means fq_codel) for the ER-X. = In reality, 100Mbps is comfortable, and 250Mbps seems impossible. You = might be able to get that rate by setting fq_codel to 300+Mbit (and you = can=E2=80=99t, through a bridge anyway), but is the queue really = controlled? I think I=E2=80=99m applying at least a little more = consistent criteria for =E2=80=9Csuccess" here at a given bitrate than = we have before. I suppose I should repeat this test with different hardware to be surer = of the claim, but I=E2=80=99m not sure when I=E2=80=99ll have the time. = I will say that Cake=E2=80=99s shaper overall produces more satisfyingly = consistent rates, and given its NAT support and host fairness, that=E2=80=99= s why I=E2=80=99m likely to continue to use it when I can. --Apple-Mail=_A831CE65-766F-4B3E-B1E7-4045E6A1AE7B Content-Disposition: attachment; filename=fq_codel_125.txt Content-Type: text/plain; x-unix-mode=0644; name="fq_codel_125.txt" Content-Transfer-Encoding: quoted-printable hostname:~/.ssh:% iperf3 -c a.b.c.230 Connecting to host a.b.c.230, port 5201 [ 5] local 192.168.1.20 port 51310 connected to a.b.c.230 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 14.5 MBytes 122 Mbits/sec =20 [ 5] 1.00-2.00 sec 14.2 MBytes 119 Mbits/sec =20 [ 5] 2.00-3.00 sec 14.3 MBytes 120 Mbits/sec =20 [ 5] 3.00-4.00 sec 14.3 MBytes 120 Mbits/sec =20 [ 5] 4.00-5.00 sec 14.3 MBytes 120 Mbits/sec =20 [ 5] 5.00-6.00 sec 14.2 MBytes 119 Mbits/sec =20 [ 5] 6.00-7.00 sec 14.3 MBytes 120 Mbits/sec =20 [ 5] 7.00-8.00 sec 14.3 MBytes 120 Mbits/sec =20 [ 5] 8.00-9.00 sec 14.2 MBytes 119 Mbits/sec =20 [ 5] 9.00-10.00 sec 14.3 MBytes 120 Mbits/sec =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 143 MBytes 120 Mbits/sec = sender [ 5] 0.00-10.00 sec 143 MBytes 120 Mbits/sec = receiver --Apple-Mail=_A831CE65-766F-4B3E-B1E7-4045E6A1AE7B Content-Disposition: attachment; filename=fq_codel_130.txt Content-Type: text/plain; x-unix-mode=0644; name="fq_codel_130.txt" Content-Transfer-Encoding: quoted-printable hostname:~/Downloads:% iperf3 -c a.b.c.230 Connecting to host a.b.c.230, port 5201 [ 5] local 192.168.1.20 port 51324 connected to a.b.c.230 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 13.3 MBytes 111 Mbits/sec =20 [ 5] 1.00-2.00 sec 14.0 MBytes 117 Mbits/sec =20 [ 5] 2.00-3.00 sec 13.9 MBytes 117 Mbits/sec =20 [ 5] 3.00-4.00 sec 13.9 MBytes 116 Mbits/sec =20 [ 5] 4.00-5.00 sec 13.6 MBytes 114 Mbits/sec =20 [ 5] 5.00-6.00 sec 13.9 MBytes 116 Mbits/sec =20 [ 5] 6.00-7.00 sec 14.0 MBytes 117 Mbits/sec =20 [ 5] 7.00-8.00 sec 13.9 MBytes 116 Mbits/sec =20 [ 5] 8.00-9.00 sec 13.8 MBytes 116 Mbits/sec =20 [ 5] 9.00-10.00 sec 13.8 MBytes 116 Mbits/sec =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 138 MBytes 116 Mbits/sec = sender [ 5] 0.00-10.00 sec 138 MBytes 116 Mbits/sec = receiver --Apple-Mail=_A831CE65-766F-4B3E-B1E7-4045E6A1AE7B Content-Disposition: attachment; filename=cake_135.txt Content-Type: text/plain; x-unix-mode=0644; name="cake_135.txt" Content-Transfer-Encoding: quoted-printable hostname:~/Downloads:% iperf3 -c a.b.c.230 =20 Connecting to host a.b.c.230, port 5201 [ 5] local 192.168.1.20 port 51328 connected to a.b.c.230 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 15.5 MBytes 130 Mbits/sec =20 [ 5] 1.00-2.00 sec 15.4 MBytes 129 Mbits/sec =20 [ 5] 2.00-3.00 sec 15.4 MBytes 129 Mbits/sec =20 [ 5] 3.00-4.00 sec 15.2 MBytes 128 Mbits/sec =20 [ 5] 4.00-5.00 sec 15.4 MBytes 129 Mbits/sec =20 [ 5] 5.00-6.00 sec 15.4 MBytes 129 Mbits/sec =20 [ 5] 6.00-7.00 sec 15.4 MBytes 129 Mbits/sec =20 [ 5] 7.00-8.00 sec 15.4 MBytes 129 Mbits/sec =20 [ 5] 8.00-9.00 sec 15.3 MBytes 128 Mbits/sec =20 [ 5] 9.00-10.00 sec 14.8 MBytes 124 Mbits/sec =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 153 MBytes 129 Mbits/sec = sender [ 5] 0.00-10.00 sec 153 MBytes 128 Mbits/sec = receiver --Apple-Mail=_A831CE65-766F-4B3E-B1E7-4045E6A1AE7B Content-Disposition: attachment; filename=cake_140.txt Content-Type: text/plain; x-unix-mode=0644; name="cake_140.txt" Content-Transfer-Encoding: quoted-printable hostname:~/Downloads:% iperf3 -c a.b.c.230 Connecting to host a.b.c.230, port 5201 [ 5] local 192.168.1.20 port 51332 connected to a.b.c.230 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 16.3 MBytes 136 Mbits/sec =20 [ 5] 1.00-2.00 sec 16.3 MBytes 137 Mbits/sec =20 [ 5] 2.00-3.00 sec 15.6 MBytes 131 Mbits/sec =20 [ 5] 3.00-4.00 sec 15.6 MBytes 131 Mbits/sec =20 [ 5] 4.00-5.00 sec 15.3 MBytes 128 Mbits/sec =20 [ 5] 5.00-6.00 sec 16.5 MBytes 139 Mbits/sec =20 [ 5] 6.00-7.00 sec 15.5 MBytes 130 Mbits/sec =20 [ 5] 7.00-8.00 sec 15.3 MBytes 129 Mbits/sec =20 [ 5] 8.00-9.00 sec 16.6 MBytes 139 Mbits/sec =20 [ 5] 9.00-10.00 sec 15.3 MBytes 128 Mbits/sec =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 158 MBytes 133 Mbits/sec = sender [ 5] 0.00-10.00 sec 158 MBytes 133 Mbits/sec = receiver --Apple-Mail=_A831CE65-766F-4B3E-B1E7-4045E6A1AE7B Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii --Apple-Mail=_A831CE65-766F-4B3E-B1E7-4045E6A1AE7B--