[Cake] overhead and mpu

Tue Sep 5 04:01:27 EDT 2017

Hi Dennis,

> On Sep 5, 2017, at 08:00, Dennis Fedtke <dennisfedtke at gmail.com> wrote:
> 
> Hi Ryan,
> 
> Thanks for you answers.
> Lets assume again ethernet over docsis connection at 50 Mbit/s.

	As the other's already mentioned, on DOCSIS it is customary to lie to the customer regarding the provisioned bandwidth; fortunately the "lie" is actually in favor of the customer, so nobody needs to feel bad about this ;)

> So to get 50 Mbit/s ethernet perfomance, the docsis link speed needs to be set at a higher link speed to compensate for the 18 header ethernet overhead, or?
> or
> Assuming:
> The docsis link syncs at exactly 50Mbit/s. When cake is set to exactly 50Mbit/s, it is actually set at a higher speed then the link actually is.

On DOCSIS networks according to Greg White of cablelabs:

For downstream packets, the DOCSIS MAC header consists of the standard 6-byte MAC Header, plus 6 bytes for the DS EHDR and 5 bytes for the Downstream Privacy EH Element. So, 17+14+4=35 bytes of per-packet overhead for a downstream DOCSIS MAC Frame.

For upstream packets, there will typically be only the 6-byte MAC Header, plus the 4 byte Upstream Privacy EH version 2 Element, so 10+14+4=28 bytes of per-packet overhead for an Upstream DOCSIS MAC Frame.

The 14+4 denotes the 14 byte ethernet header (6 byte dest mac, 6 byte src mac, 2 byte ether type) plus the 4 byte ethernet frame check sequence FCS.

Interestingly, the above DICSIS observations seem irrelevant for end customers as the shaper used in DOCSIS systems that limits a users maximal bandwidth does completely ignore DOCSIS overhead and only includes ethernet frames including their frame check sequence (FCS 4 Byte). Tocite the relevant section from the Docsis standard (http://www.cablelabs.com/specification/docsis-3-0-mac-and-upper-layer-protocols-interface-specification/):

"C.2.2.7.2 Maximum Sustained Traffic Rate 632 This parameter is the rate parameter R of a token-bucket-based rate limit for packets. R is expressed in bits per second, and MUST take into account all MAC frame data PDU of the Service Flow from the byte following the MAC header HCS to the end of the CRC, including every PDU in the case of a Concatenated MAC Frame. This parameter is applied after Payload Header Suppression; it does not include the bytes suppressed for PHS. The number of bytes forwarded (in bytes) is limited during any time interval T by Max(T), as described in the expression: Max(T) = T * (R / 8) + B, (1) where the parameter B (in bytes) is the Maximum Traffic Burst Configuration Setting (refer to Annex C.2.2.7.3). NOTE: This parameter does not limit the instantaneous rate of the Service Flow. The specific algorithm for enforcing this parameter is not mandated here. Any implementation which satisfies the above equation is conformant. In particular, the granularity of enforcement and the minimum implemented value of this parameter are vendor specific. The CMTS SHOULD support a granularity of at most 100 kbps. The CM SHOULD support a granularity of at most 100 kbps. NOTE: If this parameter is omitted or set to zero, then there is no explicitly-enforced traffic rate maximum. This field specifies only a bound, not a guarantee that this rate is available."

So in essence DOCSIS users need to only account for 18 Bytes of ethernet overhead in both ingress and egress directions under non-congested conditions.

> 
> For mpu why 64?
> I assume minimum 46bytes ethernet payload + 18bytes header?

	Well your DOCSIS modem will simply grab the ethernet packets including the FCS and those are simply guranteed to be >= 64 bytes. The linux kernel however might see smaller packets if the payload is smaller (but note for TCP/IPv4 with its 40 bytes of header the difference is quite small anyway). Interestingly if you use a single VLAN tag the ethernet MPU still stays at 64 bytes, only the payload section (including padding) shrinks to 42 bytes...

> 
> But why does cake use hard_header_len for the header size which is 14 bytes?

	I believe this is used internally so cake can deduce the size of the automatically added overhead (the linux kernel will add 14 bytes on ethernet interfaces automatically, which while certainly justifiable are not the ideal value for an ethernet shaper); with this cake allows the user to specify absolute overhead (so if you specify overhead 18, cake will give you 14 - 14 + 18 = 18, while tc's stab method will give you 14 + 18  = 32). So cake is doing the right thing here (the only thing it could do even better is to report the size of the automatically corrected in-kernel hard_header_len, but then the audience for that information is probably to small to justify this feature).

> 
> I don't know how packet sheduler system works, but maybe it is not needed to include the ethernet header.

	All you really need to give cake is a precise estimate of the bandwidth and the effective on-the-wire-size of the data packets; for DOCSIS this means overhead of 18 bytes*

> 
> cake does work on ip pakets or? so this is layer3 i think.
> the hole thing with ethernet headers is happening in layer2.
> This would change the minimum packet size to 46 or?

From sch_cake.c:

static inline u32 cake_overhead(struct cake_sched_data *q, u32 in)
{
        u32 out = in + q->rate_overhead;

        if (q->rate_mpu && out < q->rate_mpu) {
                out = q->rate_mpu;
        }

        if (q->rate_flags & CAKE_FLAG_ATM) {
                out += 47;
                out /= 48;
                out *= 53;
        } else if(q->rate_flags & CAKE_FLAG_PTM) {
                // the following adds one byte per 64 bytes or part thereof
                // this is conservative and easier to calculate than the precise value
                out += (out / 64) + !!(out % 64);
        }

        return out;                  
}

As you can see cake first adds the overhead and then makes sure the packet size is >= MPU... This will do the right thing for DOCSIS, even though it is conceivable that there are encapsulations out that can not be fully described with this (think full ethernet frames with additional overhead, where size+full_overhead might be >= mpu but the correct method would calculate size+ethernet_overhead >= mpu and then add outer_layer_overhead on top of this, but this is largely academic until something like this gets rolled-out in large numbers...)

Best Regards
	Sebastian

P.S.: unlike Ryan and Jonathan I would always use "overhead 18 mpu 64" instead of "docsis" because I prefer explicitness over terseness, but that is a matter of taste and "tc -s qdisc" will helpfully expand docsis into "overhead 18 mpu 64" (which is great).

> 
> Thanks.
> 
> _______________________________________________
> Cake mailing list
> Cake at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

*) The 18 bytes DOCSIS overhead are actually a lie, the true per-packet overhead on docsis is far greater. Without congestion all this additional overhead is simply "payed" for the user out of the shared gross bandwidth and everybody is happy. But under congestion this, IMHO, can lead to massive unfairness in that a user's shate of the gross bandwidth will correlate (bandwidth being equal) with the inversre of the packet size; put differently small packets use relatively more of the gross bandwidth than large packets, and I have not found out how docss gurantees fairness to all the CPE in a segment, but I digress. 