From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lj1-x231.google.com (mail-lj1-x231.google.com [IPv6:2a00:1450:4864:20::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 94C213B29E for ; Sat, 28 Jul 2018 04:06:20 -0400 (EDT) Received: by mail-lj1-x231.google.com with SMTP id l15-v6so6329125lji.6 for ; Sat, 28 Jul 2018 01:06:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=8gtvzZU51QQlS5dhgVPZCf1/W9Oppsgd3w+nPubswF8=; b=hFvN/Sn36ApyG0NhUvHHjCvSf1GUlvB/mRf4qJ8oXAxqfCFn2LMA98MSDjBPrgX7au Tq/6ITfeUdWUwBaxS3SwYoFOMk5I8mkoUAh1tCuWm7QzBiUUHSs37UFaO6z3gwOmIVFu qGu35393kasX1UtGsyFgYPR9rK929tCCn/iVEU6AczCiZfXLDJYCLVSOklhp4MsS0da+ 2fhwaaMK8YBDCL+6u0I8pON4WSxvVWMPCT9FpAdM1+yidbLrbX2ylPPEsxEah/iOtvjV 5gwzHfo8EzLFeDFlN6Rhak833LbFzlpnh5tYmEABgf31g7jSqjcb3HjIqlWRL4LjPtdw zWMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=8gtvzZU51QQlS5dhgVPZCf1/W9Oppsgd3w+nPubswF8=; b=bQC4bmnhv+XL+Q+cym2p+JdyV1rBPGow+ECILIFHgGOz3v+H7kbT2sGQgPwwE9Zycf ZQ2GI0k+vZBlzDpPnqiOoUcxya/YKrkCPAGE4+kK35q/3NcLyhIewrvQKuc4pAG5W3qq caKfl33MvTZDQBXS0tdKRijbQwz+72FANLgLhZ6A6/IwHhnzaVZ4ugjGwdKEPcb/4gC2 142my/7mEVknvtJHgOQ1uXzRxrpz5FeToqIsfbQK3uias8yvtg0idAnkodbEVyius9GG ZCDK6jU3a5HXIT3EojkCEqsgfbE6zLFhKlG7h1jCuKYaRIVbdJgWQFsfv0Bbxvcp5G8G Q53A== X-Gm-Message-State: AOUpUlFSRKn6YweHjMIhhjMJdX0ZWfsAByUkjQEQYXnyrSpc039qDPp7 I8jgo3ZOi6FfJbsnNX9ieA4= X-Google-Smtp-Source: AAOMgpefD5GhodvjGUw2Zc1ycbQvZTZ2S+0umeK18b7NreK/4iXsXsiSVIGJqMm/4PqLwM0SEvQJlw== X-Received: by 2002:a2e:869a:: with SMTP id l26-v6mr7754870lji.48.1532765179531; Sat, 28 Jul 2018 01:06:19 -0700 (PDT) Received: from [192.168.239.216] (83-245-238-149-nat-p.elisa-mobile.fi. [83.245.238.149]) by smtp.gmail.com with ESMTPSA id h90-v6sm1025907lji.66.2018.07.28.01.06.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 28 Jul 2018 01:06:18 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) From: Jonathan Morton In-Reply-To: <8980FA6C-508B-43E1-8C23-6EBC4A10499A@heistp.net> Date: Sat, 28 Jul 2018 11:06:17 +0300 Cc: Cake List Content-Transfer-Encoding: quoted-printable Message-Id: References: <1357421162.31089.1531812291583@webmail.strato.de> <1c323544b3076c0ab31b887d6113f25f572e41ae.camel@coverfire.com> <87woth28rw.fsf@toke.dk> <87tvol1z6h.fsf@toke.dk> <8980FA6C-508B-43E1-8C23-6EBC4A10499A@heistp.net> To: Pete Heist X-Mailer: Apple Mail (2.3445.9.1) Subject: Re: [Cake] =?utf-8?q?Using_cake_to_shape_1000=E2=80=99s_of_users=2E?= X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Jul 2018 08:06:21 -0000 > There are some older backhaul routers still with 2.6.26.8(!) although = those are being phased out so don=E2=80=99t count them. More current = ones use 3.16.7 and there=E2=80=99s some discussion but I=E2=80=99m not = sure what/when the upgrade plan is. I think the Internet router uses a = more modern Debian 9 which is likely to have a 4.9 series, but I don=E2=80= =99t have access to it. >=20 > FreeNet might be unusual in that administration is distributed. = Volunteer members administer backhaul routers which carry some number of = customers, average a few dozen or so. These do = routing/firewalling/monitoring and QoS, when they=E2=80=99re a = bottleneck. They use either an ancient esfq (has per-IP fairness though) = on older kernels and sfq when it=E2=80=99s not available, which is now = more common. There are arguments (heated ones, I=E2=80=99ve heard) about = centralizing administrative functions, including QoS, at or near the = Internet gateway, but I think this would have to depend on = over-provisioning the backhaul. I=E2=80=99ve volunteered to modernize = the QoS when I can. If ISP flavored Cake doesn=E2=80=99t happen, it will = end up being HTB+sfq|fq_codel|cake depending on what the kernel = supports. This sounds like a relatively complex network topology, in which there = are a lot of different potential bottlenecks, depending on the dynamic = state of the network. But it's encouraging to hear that you have *some* = sort of solution in place, even if it's using rather old techniques at = present. I think we should treat wired and wireless backhaul separately here. By = wireless I specifically mean shared-medium links which are, by design, = half-duplex. >> If all users will have the same link-layer technology (with the same = overhead parameters), then these can be set globally - or if not, they = can be set per-tier. >=20 > Afaik almost all members use WiFi, possibly some Ethernet. There are = no tiers and no per-member rate limits. Okay, so straight away we're in a significantly different regime from = CoverFire's situation, where the subscribers each have a defined link = bandwidth (which may or may not be related to the capabilities of the = underlying physical link). You simply need to share some backhaul link = fairly between subscribers using it at any given moment. I haven't yet considered whether this capability might fall naturally = out of the implementation of a shaped-subscriber model set to infinite = rate, or some other sane default. If not, a separate algorithm will be = required for that case. But it's useful to have it on the radar. >> Is the Diffserv support from Cake likely to be useful, and if so how = flexible should the configuration be? =20 >=20 > Currently the root qdisc on backhaul routers is a prio with "bands 3 = priomap 2 2 2 2 2 2 0 0 2 2 2 2 2 2 2 2=E2=80=9D. I don=E2=80=99t think = we=E2=80=99d want DSCP for anything, with the possible exception of = voip, and even then there might just be a special IP/port rule for the = asterisk server. Let's call this one a vote for "diffserv not required", since DRR++ = copes well by prioritising sparse traffic. >> And are there only a few discrete settings for bandwidth per user, or = do we have to be more flexible to handle a BRAS environment? =20 >=20 > Mainly in this case only fairness between members is needed. There=E2=80= =99s a db mapping members to their MAC addresses (usually one to one but = not always). Could you convert that DB to eBPF rules? This would let you use the = same configuration interface as CoverFire's situation. >> Is it also necessary to account per-user traffic accurately, or will = an external tool be used for that? >=20 > It would be better than what is done now (counting per-IP, which when = a customer has multiple MACs makes it harder to interpret). It appears that this can also be done within eBPF. Don't ask me the = details of how; I haven't yet looked into what eBPF is actually capable = of. > Lastly, do you think a better shaper for point-to-point WiFi is within = the scope of this project? This might be more needed for WISPs, but = here=E2=80=99s why: >=20 > - If we still have bottlenecks in the backhaul, we=E2=80=99ll need to = keep doing QoS there, and for FreeNet that means soft rate limiting, = because the WiFi devices have to keep running airOS for its management = tools. > - If you only rate limit on egress, you lose at least half your = available throughput. > - You can run egress and ingress through a common IFB, but in my = testing, TCP RTT (rrul_be with =E2=80=94socket-stats) gets higher. > - I find it works better to use HTB at the root and have HTB+cake as = leaf queues, then TCP RTT is roughly cut in half. > - But it would be even better if the shaper understood an = approximation of airtime, because aggregate throughput changes based on = the balance of up/down traffic in the case where up/down rates are = stable but asymmetric, which FreeNet sometimes has. > - And, I=E2=80=99d rather not have to use HTB, and be able to use the = better deficit mode shaper in Cake. >=20 > I know that WiFi has many complexities where soft rate limiting = can=E2=80=99t ever be perfect without knowledge from the driver, but I = still think it could be useful to have something that does a better = approximation than what we have today. The question is, is that = something that could be part of this project, or not=E2=80=A6 :) In general, I think the make-wifi-fast team has a better handle on the = subtleties of half-duplex links. If there's any way to get their work = into the airOS devices, so much the better. Otherwise, your problem basically boils down to the problem of choosing = a suitable rate limit, which you can dynamically update Cake's = configuration with from userspace (tc qdisc change=E2=80=A6) without = losing any packets or fairness state. Since the out-of-tree version of = "normal" Cake already works with relatively old kernels, this seems like = a good way to deploy a worthwhile improvement. If the traffic load on = backhaul links is complex enough (more than about several hundred = *simultaneous* flows) to overstress Cake's flow-isolation capabilities, = you might try using its "src-host" and "dst-host" modes instead of = "flows" or "dual-src/dst". But I think there's also a use for "ISP-type" Cake in your network, = especially in the wired backhauls where link bandwidth is relatively = predictable, and I suspect most of these will be in the core parts of = your network where the traffic is most complex. As long as you can get = around to running a suitable kernel version, there should also be no = inherent problem with replicating the same dynamic adjustment of global = shaper rate as "normal" Cake has, which will allow it to be used on some = of your wireless links as well. - Jonathan Morton