From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 757D03B29D for ; Fri, 22 Nov 2019 11:40:22 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1574440822; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JormOtFU8nBxAb+OKwyZFrn1ksBq/RjEUAkCLOmlPo0=; b=ZVuQCxPcqSRnMNEk5l80WR6b2KoHgfXJxMOyfaA4W9JegX/gn/d8D7oehE340jFgyhPnOA ez4tGZECgdCGqQ0yrsoZ7+N5kJNSFGxK8NZjbnWAVKvotkq+27S+0epN4x+XW20+OdTShl QdgzGBip9HC9GgoNP8ayiv1MfyB2r7I= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-327-1YBhsjFXNa6EuniPopBr6Q-1; Fri, 22 Nov 2019 11:40:18 -0500 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6CED2800A02; Fri, 22 Nov 2019 16:40:17 +0000 (UTC) Received: from carbon (ovpn-200-17.brq.redhat.com [10.40.200.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3C5451B472; Fri, 22 Nov 2019 16:40:10 +0000 (UTC) Date: Fri, 22 Nov 2019 17:40:08 +0100 From: Jesper Dangaard Brouer To: Toke =?UTF-8?B?SMO4aWxhbmQtSsO4cmdlbnNlbg==?= Cc: Adam Moffett , cake@lists.bufferbloat.net, brouer@redhat.com Message-ID: <20191122174008.3128f5d2@carbon> In-Reply-To: <878so85e2d.fsf@toke.dk> References: <878so85e2d.fsf@toke.dk> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-MC-Unique: 1YBhsjFXNa6EuniPopBr6Q-1 X-Mimecast-Spam-Score: 0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Cake] Cake implementations X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Nov 2019 16:40:22 -0000 On Fri, 22 Nov 2019 12:46:18 +0100 Toke H=C3=B8iland-J=C3=B8rgensen wrote: > "Adam Moffett" writes: >=20 > >>> > >>> Are there any commercial products already using Cake? =20 > >> > >>Evenroute, eero, ubnt top that list. Evenroute's implementation is > >>superb, the first one that used active line measurements to handle > >>"sag". Anything derived from openwrt (somewhere between 10-30% of the > >>home router market). I'm not sure if preseem is using it or not. > >>dd-wrt. Most other things doing "SQM" are doing it via htb + fq_codel. > >> > >> =20 > > An idea which was floated was to experiment with routing ISP > > customer traffic through a Linux server using cake to improve > > customer experience. Basically like Preseem. My colleague has > > toyed with it a bit in small test cases and was impressed with the > > outcomes. > > > > He's looked closer than I have, but I'm trying to picture how his > > idea would scale. I believe I'm seeing a CLI tool for configuring > > policies. It seems like we'd have to create a middle layer to > > create/update policies for customer's IP address based on > > information obtained from our AAA and CRM systems. I can picture > > some shapes that might take, but I think it would ultimately have > > to revolve around scripting the tc command. There would be > > thousands of policies and a policy would be created/updated > > whenever a subscriber reconnects (e.g. when a DHCP lease renews or > > a RADIUS auth event happens or similar). =20 >=20 > I know several ISPs that do this (route traffic through a Linux box and > shape there). This deployment mode has not been the primary focus of > CAKE, though; the "standard" way to do it is with HTB+FQ-CoDel, which > allows you to set up arbitrarily complex configurations on a single > interface.=20 Yes, I worked for an ISP back in 2005, that route traffic through a Linux box and does shaping. There were a number of scalabilities issues, that we fixed (upstream) and also Open Sources components for others to reuse. I did a public talk in 2008 about how we made it scale: http://people.netfilter.org/hawk/presentations/osd2008/osd2008_iptables_ru= les_JesperDangaardBrouer.pdf This was before Codel and even-before Bufferbloat was coined/defined. Our setup was HTB+SFQ, with a shaper per customer. This actually solved the bufferbloat problem in-practice for people, as SFQ gave each end-user 128 queues. Today I would not use iptables for this. Instead I would use BPF or nftables 'set' infrastructure. > This can also be made to scale, but there's a central qdisc > lock in Linux that you have to get around to scale to multiple cores. > Fortunately, Jesper has already solved this particular issue: >=20 > https://github.com/netoptimizer/xdp-cpumap-tc The central qdisc TX lock is a major scalability issue. But I've solved in above link, via XDP and TC. This actually runs in production at an ISP. =20 > > Should we even pursue this idea? =20 >=20 > In my own totally non-biased opinion: Yes! :) It would be great. > > Although most staff who would touch this will have studied programming= =20 > > in college, I would not qualify any of us as "programmers" per se. My= =20 > > biggest concern would be hitting a service affecting problem that we=20 > > can't solve. =20 >=20 > One way to go about this would be to open source the entire solution. > There are already bits and pieces available as open source (such as > Jesper's repository above, and sqm-scripts[0]) that you can build on, > and this way you could enlist community help to solve any issues with > the Linux side. You'd still need to get the data from your internal > systems, of course, but you could define a simple configuration format > that was part of the open source code; then you'll just need to write a > script that grabs customer info from your CRM and outputs the config > format... Yes, this is the advantage of Open Source! :-) If you create this as Open Source, then feel free to reach out to me directly. And I know of several other people (working at ISPs) that would likely be interested in collaborating. As Toke wrote, you can still keep your CRM system proprietary and closed-source, we don't care anyhow ;-) =20 > > Second concern is that many of our equipment vendors already use > > Linux. Even Cisco now in some products. Maybe we'll waste our time > > trying to roll our own solution and then find that a software update > > from a vendor next year gives us everything we needed anyway. =20 I would not hold my breath... and if it does come as a software upgrade, you can expect to pay plenty for it... Maybe I'm the wrong guy to ask, but I really think it is straight forward to implement an ISP Linux router with per customer handling. All the components seems avail as Open Source. (There do seem to be a lack around a DHCP server that can handle this well). =20 > This would be great, of course, and do go and bug your vendors to > solve this problem! Note, however, that just because a system is > running Linux on the control plane, it may be using a > hardware-offloaded data plane that does not have any of the > bufferbloat mitigation features (unless the vendor specifically > implemented them). I'm hoping that *eventually* these things will be > ubiquitous across the industry, but thus far this has seemed to be an > "any decade now" kind of proposition :/ I would prefer, not to waste time on creating bugs for your vendor, but instead actually have the ability to fix the issue myself, or hire some Linux consultant that can... --=20 Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer