[Codel] Per-Paket-Overhead in DOCSIS networks

[Greg White]

The definition in DOCSIS 3.0 is consistent with the way it was specified in DOCSIS 1.0 (which pre-dates my involvement).  My intuition is that the goal was to configure an Ethernet service rate available to the customer.  So, packet overhead that is invisible to the customer is not included in the rate shaping calculation (and your read of the spec is correct, I misstated the inclusion of MAC Header).   In the case of a Minimum Reserved Rate configuration, this could be especially important, since that configuration could then be offered as an SLA (hence the option to specify the assumed packet size).  

If there were no requirement for compatibility with existing equipment and operational practices, I’m sure things would be very different.  But, that is a universe that doesn’t exist.  ;)  

-Greg



On 10/13/16, 1:56 AM, "Moeller, Sebastian" <moeller at caltech.edu> wrote:

    Hi Greg,
    
    I recently had a discussion with a very knowledgeable customer of a German cable ISP who did some throughput measurement and compared those against the reported service rate (he called that “DOCSIS flow control” and I have no clear understanding where he got the numbers from) and those measurements indicated that onlt ethernet header and FCS seemed relevant for the shaper. That lead me to look closer into the documents you refered to. And there I found:
    "C.2.2.7.2 Maximum Sustained Traffic Rate 632
    
    This parameter is the rate parameter R of a token-bucket-based rate limit for packets. R is expressed in bits per second, and MUST take into account all MAC frame data PDU of the Service Flow from the byte following the MAC header HCS to the end of the CRC, including every PDU in the case of a Concatenated MAC Frame. This parameter is applied after Payload Header Suppression; it does not include the bytes suppressed for PHS.”
    
    Which seems to support the measurements of above users, meaning that for both upstream and downstream DOCSIS users probably only need to account for 18 Bytes of overhead. Interestingly that means that depending the packet sizes in a user’s “flows” the resulting “raw DOCSIS rate required to supply the promised user rate” must vary (with smaller packets requiring a larger share of raw bandwidth). The following section from http://www.cablelabs.com/specification/docsis-3-0-mac-and-upper-layer-protocols-interface-specification/) also supports this view:
    
    "C.2.2.7.5 Assumed Minimum Reserved Rate Packet Size
    This parameter is used by the CMTS to make worst-case DOCSIS overhead assumptions. The Minimum Reserved Traffic Rate of a service flow excludes the DOCSIS MAC header and any other DOCSIS overhead (e.g., for completing an upstream mini-slot). Traffic with smaller packets sizes will require a higher proportion of overall channel capacity for DOCSIS overhead than traffic with larger packet sizes. The CMTS assumes that the worst-case DOCSIS overhead for a service flow will be when all traffic is as small as the size specified in this parameter.
    This parameter is defined in bytes and is specified as the bytes following the MAC header HCS to the end of the CRC.
    If this parameter is omitted, then the default value is CMTS implementation dependent."
    
    	Since I know you guys are way too smart to have done this by accident ;), what is the rationale for configuring the shaper in that way, instead in a way where the raw bandwidth is accounted for properly? I understand that fixed overheads like the downstream MPEG headers can simply be accounted for by a reduction of the raw bandwidth, but all those DOCSIS headers that are required for each user data frame seem to cause inaccuracies in that method. In short why does the DOCSIS standard work with “worst-case DOCSIS overhead assumptions” instead of simply accounting for the real overhead?
    
    	From a typical end user’s perspective all of this will be rather moot, I believe, as there is no guarantee that even the “Maximum Sustained Traffic Rate” is reported back to the user and under congested conditions each user will only see a variable fraction of the contracted bandwidth anyway. But I still am curious.
    	Somewhat unrelated, if you did not have the “legacy” of needing to be compatible first with analog cable TV and later older versions of DOCSIS standards, would you radically change your data management and transfer design?
    
    Best Regards
    	Sebastian
    
    
    
    > On Jun 23, 2016, at 20:51 , Greg White <g.white at CableLabs.com> wrote:
    > 
    > No problem. I’m glad to answer.
    > 
    > To answer the specific question at the end of your email, cable modem service rate is configured using the “Maximum Sustained Traffic Rate” parameter (Section C.2.2.7.2 in http://www.cablelabs.com/specification/docsis-3-0-mac-and-upper-layer-protocols-interface-specification/).  That rate includes all MAC Frame data bytes sent in the Service Flow, which means the DOCSIS MAC Header, Ethernet Header, Ethernet PDU and CRC of each MAC Frame are included.  Depending on how the service is configured, the Service Flow may or may not be carrying a very small amount of network management traffic to/from the IP stack in the modem and/or a very small amount of “MAC Management” traffic to/from the modem MAC-layer, in addition to the user traffic.  It is common that cable operators set the Maximum Sustained Traffic Rate to be a value 10%-20% greater than the rate that they advertise to the customer (presumably to both account for the overhead, and to compensate for the occasional congestion events that can happen during peak hours).  There is, unfortunately, no way for the user to directly find out what values of Max Sustained Traffic Rate are configured for their service.
    > 
    > In terms of the MAC Header size that factors into the above rate shaping calculation, 240 bytes is the absolute maximum total bytes for all EHDRs that the DOCSIS MAC frame format could support, but in fact there is no way that you’d ever get anywhere close to that.  For downstream packets, the DOCSIS MAC header consists of the standard 6-byte MAC Header, plus 6 bytes for the DS EHDR and 5 bytes for the Downstream Privacy EH Element.   So, 17+14+4=35 bytes of per-packet overhead for a downstream DOCSIS MAC Frame.  For upstream packets, there will typically be only the 6-byte MAC Header, plus the 4 byte Upstream Privacy EH version 2 Element, so 10+14+4=28 bytes of per-packet overhead for an Upstream DOCSIS MAC Frame.
    > 
    > If you are concerned about the overall shared channel capacity, and what the other overheads are there, things get more complicated than I want to get into, but for downstream, you could look at Sections 7.1, 7.2 and 7.5 in http://www.cablelabs.com/specification/downstream-rf-interface-specification/ as a start.
    > 
    > -Greg
    > 
    > 
    > 
    > On 6/23/16, 9:59 AM, "Moeller, Sebastian" <moeller at caltech.edu> wrote:
    > 
    >> Hi Greg,
    >> 
    >> I hope you do not mind being contacted directly, but you seem to be one of the few persons with the “intimate" knowledge about docsis low-level details. We had a discussion recently in the cake list, IIRC, about the appropriate per-packet overhead in DOCSIS networks, which lead me to do some research, but now I am fully confused. 
    >> 
    >> 	To me it seems that DOCSIS adds at least 6 Bytes worth of DOCSIS headers and includes otherwise ethernet frames including the frame-check sequence is that correct? So this would mean 6+14+4 = 24 bytes of per-paket-overhead. But what about the data link security EHDR according to Data-Over-Cable Service Interface Specifications DOCSIS 3.0 that can be up to 240 bytes per encapsulated ethernet frame? Is this typically used and what size is this typically 240 bytes seems exessive, no?
    >> 
    >> 	Now for DSL (where I have more experience and data) I know what we need is a “raw” bandwidth number and the to be subtracted per packet overhead (or packet independent overhead from say 64/65 encoding or maybe the MPEG 184/188 encoding). But for DOCSIS I am really at a loss, since cable ISPs do not communicate the exact values for bandwidth and overhead that are applicable…
    >> 	Could you maybe shad some light on what overhead typically is accounted already  in the user- reported bandwidths and what overhead still needs to be accounted for? (I realize that this most likely be only relevant in an otherwise “silent” DOCSIS segment where there is no contention, but currently I have no clue for even that rather simple condition).
    >> 
    >> Many Thanks in advance
    >> 	Sebastian
    >