[Cake] Cake implementations

Jesper Dangaard Brouer brouer at redhat.com
Fri Nov 22 11:40:08 EST 2019

On Fri, 22 Nov 2019 12:46:18 +0100
Toke Høiland-Jørgensen <toke at redhat.com> wrote:

> "Adam Moffett" <adam at plexicomm.net> writes:
> >>>
> >>>  Are there any commercial products already using Cake?  
> >>
> >>Evenroute, eero, ubnt top that list. Evenroute's implementation is
> >>superb, the first one that used active line measurements to handle
> >>"sag". Anything derived from openwrt (somewhere between 10-30% of the
> >>home router market). I'm not sure if preseem is using it or not.
> >>dd-wrt. Most other things doing "SQM" are doing it via htb + fq_codel.
> >>
> >>  
> > An idea which was floated was to experiment with routing ISP
> > customer traffic through a Linux server using cake to improve
> > customer experience.  Basically like Preseem.  My colleague has
> > toyed with it a bit in small test cases and was impressed with the
> > outcomes.
> >
> > He's looked closer than I have, but I'm trying to picture how his
> > idea would scale.  I believe I'm seeing a CLI tool for configuring
> > policies. It seems like we'd have to create a middle layer to
> > create/update policies for customer's IP address based on
> > information obtained from our AAA and CRM systems.  I can picture
> > some shapes that might take, but I think it would ultimately have
> > to revolve around scripting the tc command.  There would be
> > thousands of policies and a policy would be created/updated
> > whenever a subscriber reconnects (e.g. when a DHCP lease renews or
> > a RADIUS auth event happens or similar).  
> I know several ISPs that do this (route traffic through a Linux box and
> shape there). This deployment mode has not been the primary focus of
> CAKE, though; the "standard" way to do it is with HTB+FQ-CoDel, which
> allows you to set up arbitrarily complex configurations on a single
> interface. 

Yes, I worked for an ISP back in 2005, that route traffic through a
Linux box and does shaping.  There were a number of scalabilities
issues, that we fixed (upstream) and also Open Sources components for
others to reuse.

I did a public talk in 2008 about how we made it scale:

This was before Codel and even-before Bufferbloat was coined/defined.
Our setup was HTB+SFQ, with a shaper per customer.  This actually
solved the bufferbloat problem in-practice for people, as SFQ gave each
end-user 128 queues.

Today I would not use iptables for this.  Instead I would use BPF or
nftables 'set' infrastructure.

> This can also be made to scale, but there's a central qdisc
> lock in Linux that you have to get around to scale to multiple cores.
> Fortunately, Jesper has already solved this particular issue:
> https://github.com/netoptimizer/xdp-cpumap-tc

The central qdisc TX lock is a major scalability issue.  But I've
solved in above link, via XDP and TC.  This actually runs in production
at an ISP.

> > Should we even pursue this idea?  
> In my own totally non-biased opinion: Yes! :)

It would be great.

> > Although most staff who would touch this will have studied programming 
> > in college, I would not qualify any of us as "programmers" per se.  My 
> > biggest concern would be hitting a service affecting problem that we 
> > can't solve.  
> One way to go about this would be to open source the entire solution.
> There are already bits and pieces available as open source (such as
> Jesper's repository above, and sqm-scripts[0]) that you can build on,
> and this way you could enlist community help to solve any issues with
> the Linux side. You'd still need to get the data from your internal
> systems, of course, but you could define a simple configuration format
> that was part of the open source code; then you'll just need to write a
> script that grabs customer info from your CRM and outputs the config
> format...

Yes, this is the advantage of Open Source! :-)

If you create this as Open Source, then feel free to reach out to me
directly.  And I know of several other people (working at ISPs) that
would likely be interested in collaborating.

As Toke wrote, you can still keep your CRM system proprietary and
closed-source, we don't care anyhow ;-)

> > Second concern is that many of our equipment vendors already use
> > Linux. Even Cisco now in some products. Maybe we'll waste our time
> > trying to roll our own solution and then find that a software update
> > from a vendor next year gives us everything we needed anyway.  

I would not hold my breath... and if it does come as a software
upgrade, you can expect to pay plenty for it...

Maybe I'm the wrong guy to ask, but I really think it is straight
forward to implement an ISP Linux router with per customer handling.
All the components seems avail as Open Source.  (There do seem to be a
lack around a DHCP server that can handle this well).

> This would be great, of course, and do go and bug your vendors to
> solve this problem! Note, however, that just because a system is
> running Linux on the control plane, it may be using a
> hardware-offloaded data plane that does not have any of the
> bufferbloat mitigation features (unless the vendor specifically
> implemented them). I'm hoping that *eventually* these things will be
> ubiquitous across the industry, but thus far this has seemed to be an
> "any decade now" kind of proposition :/

I would prefer, not to waste time on creating bugs for your vendor, but
instead actually have the ability to fix the issue myself, or hire some
Linux consultant that can...

Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

More information about the Cake mailing list