[Cake] Using cake to shape 1000’s of users.

[Dan Siemon]

On Fri, 2018-07-27 at 21:58 +0300, Jonathan Morton wrote:
> > 
> > We have some deployments with multiple access technologies (eg
> > DOCSIS,
> > DSL and wireless) behind the same box so per customer overhead
> > would be
> > useful.
> 
> The design I presently have in mind would allow setting the overhead
> per speed tier.  Thus you could have "10Mbit DOCSIS" and "10Mbit
> ADSL" as separate tiers.  That is, there's likely to be far fewer
> unique overhead settings than customers.

The unique overhead configurations will be much smaller than the number
of subscribers but from a provisioning standpoint, this info belongs
with the subscriber. Having to map a subscriber into an 'overhead
config' vs. just setting those values along with rate would be
inconvenient.

> > I am interested in the diffserv class ideas in Cake but need to do
> > some
> > experimentation to see if that helps in this environment. It would
> > be
> > interesting to be able to target sub-classes (not sure what the
> > proper
> > Cake terminology is) based on a eBPF classifier.
> 
> The trick with Diffserv is classifying the traffic correctly, given
> that most traffic in the wild is not properly marked, and some
> actively tries to hide its nature.  As an ISP, applying your own
> rules could be seen as questionable according to some interpretations
> of Net Neutrality.  With Cake that's not an issue by default, since
> it relies entirely on the DSCP on the packet itself, but it does mean
> that a lot of traffic which *should* probably be classified other
> than best-effort is not.
> 
> Cake does cope well with unclassified latency-sensitive traffic, due
> to DRR++, so it's not actually necessary to specifically identify
> such applications.  What it has trouble with, in common with all
> other flow-isolating qdiscs, is applications which open many parallel
> connections for bulk transfer (eg. BitTorrent, which is used by many
> software-update mechanisms as well as file-sharing).  Classifying
> this traffic as Bulk (CS1 DSCP) gets it out of the way of other
> applications while still permitting them to use available capacity.
> 
> I have some small hope that wider deployment of sensible, basic
> Diffserv at bottlenecks will encourage applications to start using it
> as intended, thereby solving a decades-long chicken-egg
> problem.  Presently Cake has such an implementation by default, but
> is not yet widely deployed enough to have any visible impact.
> 
> This is something we should probably discuss in more detail later.
> 
> > Re accounting, we currently count per-IP via eBPF not via qdisc
> > counts.
> 
> Fine, that's a feature I'm happy to leave out.

One problem I haven't solved yet is counting bytes and packets per-IP
post qdisc. This is required when multiple IPs are sent to the same
QDisc but per-IP stats are still needed.

It would be great if there were a post qdisc hook for eBPF programs.
Something like a cls_act that runs after the other qdiscs on Tx instead
of before. It would be even better if the skb could be flagged with an
ID of some kind so that the outbound eBPF program doesn't need to parse
the headers again and could instead just do a map lookup + update.

I started looking at XDP for this but at the time it was all receive
side. I'm not sure if that's still the case.

> > 
> So in summary, the logical flow of a packet should be:
> 
> 1: Map dst or src IP to subscriber (eBPF).
> 2: Map subscriber to speed/overhead tier (eBPF).
> 3: (optional) Classify Diffserv (???).
> 4: Enqueue per flow, handle global queue overflow (rare!) by dropping
> from head of longest queue (like Cake).

Since the encapsulations used in the ISP world are pretty diverse,
being able to generate the flow hash from the eBPF program is also
required. Today we do this by setting the skb->hash field and letting
the outbound FQ-CoDel classify based on that.

> --- enqueue/dequeue split ---
> 5: Wait for shaper on earliest-scheduled subscriber link with waiting
> traffic (borrow sch_fq's rbtree?).
> 6: Wait for shaper on aggregate backhaul link (congestion can happen
> here too).

Yes, multiple layers of bandwidth limiting are required - two at a
minimum. One for the subscriber's plan rate and another for the nearest
access device.

Conceptually, is it possible to build something akin to the HTB qdisc
that uses DRR++ and retains the flexible hierarchy? I don't have a good
sense of the performance impact of this type of setup vs. a single more
featureful qdisc.

> 7: Choose one of subscriber's queues, apply AQM and deliver a packet
> (mostly like Cake does).
> 
> If that seems reasonable, we can treat it as a baseline spec for
> others' input.
> 
>  - Jonathan Morton