[Cake] overhead and mpu

[Cake] overhead and mpu

[Dennis Fedtke]

Hi!

what values should be used for an Ethernet connection running over 
docsis modem?

I think cake automatically accounts for the Ethernet header.
But how large is the ethernet header? Some sites say 1514 some other 
sites say 1518.
So we have the mac header (14bytes) + the crc checksum (4 bytes) + max 
payload 1500bytes = 1518 bytes?

I looked at the code and there is hard_header_len used.

After some quick googling i found this info:
The value ofhard_header_lenis14(ETH_HLEN) for Ethernet interfaces.

Which "packets size(1500 or 1514 or 1518)" does cake really see and work 
with when running on an Ethernet interface?
Minimum Ethernet payload size is 46 bytes. So should i use mpu 46, mpu 
60 or mpu 64 setting for an Ethernet interface?

Thanks

Re: [Cake] overhead and mpu

[Ryan Mounce]

Hi Dennis,

I am probably qualified to answer your question as I have done
extensive testing for this case with DOCSIS modems, indeed I was
prompted to contribute the 'mpu' option to cake in order to optimise
this case.

Short answer: just use the 'docsis' keyword for cake which configures
the correct overhead and mpu, then it is just a matter of tuning the
bandwidth.

e.g. this is my command for cake on a DOCSIS connection advertised as
2Mbps upstream but actually shaped to 2.5Mbps:
tc qdisc add dev eth0.2 root cake dual-srchost nat oceanic docsis
bandwidth 2496Kbit


The shaper in DOCSIS modems for upstream traffic (and I believe also
in the CMTS for downstream traffic) works with ethernet frames
including the 14-byte MAC header and 4-byte CRC and excluding any
preamble / inter-packet gap, for 18 bytes total. Cake rounds runt
frames up to the MPU after the overhead has been added, thus 64 bytes
is our MPU for DOCSIS.

It doesn't matter what packet size cake 'sees' on an interface, the
overhead specified at the command line (directly or via keywords) is
always interpreted relative to IP packets rather than relative to the
PDU of a given interface.

Regards,
Ryan Mounce

On 5 September 2017 at 12:16, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
> Hi!
>
> what values should be used for an Ethernet connection running over docsis
> modem?
>
> I think cake automatically accounts for the Ethernet header.
> But how large is the ethernet header? Some sites say 1514 some other sites
> say 1518.
> So we have the mac header (14bytes) + the crc checksum (4 bytes) + max
> payload 1500bytes = 1518 bytes?
>
> I looked at the code and there is hard_header_len used.
>
> After some quick googling i found this info:
> The value ofhard_header_lenis14(ETH_HLEN) for Ethernet interfaces.
>
> Which "packets size(1500 or 1514 or 1518)" does cake really see and work
> with when running on an Ethernet interface?
> Minimum Ethernet payload size is 46 bytes. So should i use mpu 46, mpu 60 or
> mpu 64 setting for an Ethernet interface?
>
> Thanks
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] overhead and mpu

[Ryan Mounce]

I will clarify a little further, that cake does make a guess as to the
overhead of an interface based on what it 'sees' however this is
overridden (rather than simply added to) if the overhead is specified
directly or via a keyword.

In my case running cake on a VLAN subinterface, it happens to guess 18
bytes (14 ethernet + 4 802.1q) which is subsequently set to 18 bytes
again by the 'docsis' keyword. 'tc qdisc show dev eth0.2' naively
determines that since my configured overhead is the same as the
overhead it 'sees', that I must have configured cake to use the 'raw'
overhead.

On 5 September 2017 at 12:56, Ryan Mounce <ryan@mounce.com.au> wrote:
> Hi Dennis,
>
> I am probably qualified to answer your question as I have done
> extensive testing for this case with DOCSIS modems, indeed I was
> prompted to contribute the 'mpu' option to cake in order to optimise
> this case.
>
> Short answer: just use the 'docsis' keyword for cake which configures
> the correct overhead and mpu, then it is just a matter of tuning the
> bandwidth.
>
> e.g. this is my command for cake on a DOCSIS connection advertised as
> 2Mbps upstream but actually shaped to 2.5Mbps:
> tc qdisc add dev eth0.2 root cake dual-srchost nat oceanic docsis
> bandwidth 2496Kbit
>
>
> The shaper in DOCSIS modems for upstream traffic (and I believe also
> in the CMTS for downstream traffic) works with ethernet frames
> including the 14-byte MAC header and 4-byte CRC and excluding any
> preamble / inter-packet gap, for 18 bytes total. Cake rounds runt
> frames up to the MPU after the overhead has been added, thus 64 bytes
> is our MPU for DOCSIS.
>
> It doesn't matter what packet size cake 'sees' on an interface, the
> overhead specified at the command line (directly or via keywords) is
> always interpreted relative to IP packets rather than relative to the
> PDU of a given interface.
>
> Regards,
> Ryan Mounce
>
> On 5 September 2017 at 12:16, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
>> Hi!
>>
>> what values should be used for an Ethernet connection running over docsis
>> modem?
>>
>> I think cake automatically accounts for the Ethernet header.
>> But how large is the ethernet header? Some sites say 1514 some other sites
>> say 1518.
>> So we have the mac header (14bytes) + the crc checksum (4 bytes) + max
>> payload 1500bytes = 1518 bytes?
>>
>> I looked at the code and there is hard_header_len used.
>>
>> After some quick googling i found this info:
>> The value ofhard_header_lenis14(ETH_HLEN) for Ethernet interfaces.
>>
>> Which "packets size(1500 or 1514 or 1518)" does cake really see and work
>> with when running on an Ethernet interface?
>> Minimum Ethernet payload size is 46 bytes. So should i use mpu 46, mpu 60 or
>> mpu 64 setting for an Ethernet interface?
>>
>> Thanks
>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] overhead and mpu

[Dennis Fedtke]

Hi Ryan,

Thanks for you answers.
Lets assume again ethernet over docsis connection at 50 Mbit/s.
So to get 50 Mbit/s ethernet perfomance, the docsis link speed needs to be set at a higher link speed to compensate for the 18 header ethernet overhead, or?
or
Assuming:
The docsis link syncs at exactly 50Mbit/s. When cake is set to exactly 50Mbit/s, it is actually set at a higher speed then the link actually is.

For mpu why 64?
I assume minimum 46bytes ethernet payload + 18bytes header?

But why does cake use hard_header_len for the header size which is 14 bytes?

I don't know how packet sheduler system works, but maybe it is not needed to include the ethernet header.

cake does work on ip pakets or? so this is layer3 i think.
the hole thing with ethernet headers is happening in layer2.
This would change the minimum packet size to 46 or?

Thanks.

Re: [Cake] overhead and mpu

[Ryan Mounce]

Hi Dennis,

Firstly to nitpick, DOCSIS doesn't 'sync' in the traditional xDSL
sense and more or less operates at a fixed speed at the physical layer
(DOCSIS 3.1 changes this a bit but it is irrelevant to what you're
trying to achieve here). The modem and CMTS are typically capable of
high speeds however the coax is a shared medium. The actual speed you
see as an end user is typically defined by a shaper in the CM / CMTS
in order to present the illusion of a reliable fixed speed pipe and to
create an artificial model to charge more for higher speeds.

So, it depends where and how that "50Mbps" is defined. It is typical
for a MSO / cable ISP to actually under-sell and over-deliver on the
headline speed, e.g. in my ISP's case advertising "100/2Mbps" on
paper, configuring the shapers at 120/2.5Mbps (which is inclusive of
18 bytes overhead and padding for frames smaller than 64 bytes), which
then results in real world application layer throughput somewhere
between those two values (~114/2.4Mbps on something like speedtest.net
in ideal conditions). As far as cake is concerned, 120/2.5 is the
important figure here.

You want to configure cake with the 'docsis' keyword (equivalent to
overhead of 18 bytes on top of IP, and 64 byte MPU inclusive of
Ethernet header), and 'bandwidth' that is about 99% of whatever your
ISP configures the modem for your plan.

The only problem is that you normally can't view what the modem is
actually configured for, so you need to do some testing which comes
down to trial and error. My technique to configure the upstream rate
was to configure cake for a speed slightly above that of my actual
connection, continuously ping a nearby server on the internet, and
then start a large file upload. The modem's buffer quickly fills up
and latency increases until the buffer is full and packets are being
dropped. From this point I kept re-configuring cake and slightly
reducing the bandwidth until the buffer started to clear and pings
slowly returned to baseline. In my case I observed this at 2499Kbps
(suggesting that my modem's shaper is configured for precisely
2500Kbps), however I use 2496Kbps as the final value so that the
buffer clears slightly faster in case is a 'burp' for whatever reason
that fills it.

Why is the MPU 64? Good question, that is an Ethernet thing defined in
the 70's and inherited by DOCSIS many years later. But yes, this
results in a minimum Ethernet payload of 46 bytes.

Re: hard_header_len, it's been months since I have dug into the source
and I have since forgotten the internal details. I understood it once
upon a time, my advise is to not concern yourself with this.

What I do remember for sure is that from the user configuration
perspective everything is relative to IP packets. It doesn't matter
what layer cake 'sees' internally, this is abstracted away from the
user in order to present a consistent interface.

TL;DR just use the 'docsis' keyword and forget everything you know
about overhead and mpu. That's why it's there, so that nobody else
ever has to go through the pain I did when fine-tuning cake for my
DOCSIS connection.

Regards,
Ryan Mounce


On 5 September 2017 at 15:30, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
> Hi Ryan,
>
> Thanks for you answers.
> Lets assume again ethernet over docsis connection at 50 Mbit/s.
> So to get 50 Mbit/s ethernet perfomance, the docsis link speed needs to be
> set at a higher link speed to compensate for the 18 header ethernet
> overhead, or?
> or
> Assuming:
> The docsis link syncs at exactly 50Mbit/s. When cake is set to exactly
> 50Mbit/s, it is actually set at a higher speed then the link actually is.
>
> For mpu why 64?
> I assume minimum 46bytes ethernet payload + 18bytes header?
>
> But why does cake use hard_header_len for the header size which is 14 bytes?
>
> I don't know how packet sheduler system works, but maybe it is not needed to
> include the ethernet header.
>
> cake does work on ip pakets or? so this is layer3 i think.
> the hole thing with ethernet headers is happening in layer2.
> This would change the minimum packet size to 46 or?
>
> Thanks.
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] overhead and mpu

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 1816 bytes --]

I designed Cake to require as little networking expertise as feasible of
the end user. You only really need to know your network topology. So don't
overthink this - just set "bandwidth 48Mbit docsis ingress" on your inbound
shaper, and "bandwidth (whatever) docsis egress" on the outbound.

Why 48 instead of 50?  Because Cake needs that when downstream of the
bottleneck link, just like any other shaper, so that it controls the
bottleneck queue. If your ISP decided to install Cake themselves, they
wouldn't need that hack, because they have access to the upstream end of
the downlink.

Now if you still want to know the technical details...

Cake's shaper works by calculating the time it takes to get a given packet
through the bottleneck link, measured from the beginning of transmission of
that packet to the beginning of transmission of the next packet.  To do
that, it needs to know the number of bytes per second that the link can
carry, and the actual number of byte times occupied by the packet on the
wire - not all of which might be included in your typical frame header and
payload; we also have to count preamble, checksum, quiet times, etc.

These are layer 2 concerns, and in fact Cake normally sees layer 2 packets
when attached to an Ethernet interface.  Just give it the raw data, and
it'll take care of nearly all the nasty calculations and expert knowledge
you'd need with most other shapers.

The rest of Cake is largely concerned with choosing which packet to deliver
next, given the choice among all the packets in its queue.  This involves
examining layer 3 and layer 4 information within the packet, where
available.  There are layer 3 (IP) packets without a layer 4 (TCP/UDP)
payload, and layer 2 packets without an IP payload, which Cake also has to
deal with sanely.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 1988 bytes --]

Re: [Cake] overhead and mpu

[Sebastian Moeller]

Hi Dennis,


> On Sep 5, 2017, at 08:00, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
> 
> Hi Ryan,
> 
> Thanks for you answers.
> Lets assume again ethernet over docsis connection at 50 Mbit/s.

	As the other's already mentioned, on DOCSIS it is customary to lie to the customer regarding the provisioned bandwidth; fortunately the "lie" is actually in favor of the customer, so nobody needs to feel bad about this ;)


> So to get 50 Mbit/s ethernet perfomance, the docsis link speed needs to be set at a higher link speed to compensate for the 18 header ethernet overhead, or?
> or
> Assuming:
> The docsis link syncs at exactly 50Mbit/s. When cake is set to exactly 50Mbit/s, it is actually set at a higher speed then the link actually is.

On DOCSIS networks according to Greg White of cablelabs:

For downstream packets, the DOCSIS MAC header consists of the standard 6-byte MAC Header, plus 6 bytes for the DS EHDR and 5 bytes for the Downstream Privacy EH Element. So, 17+14+4=35 bytes of per-packet overhead for a downstream DOCSIS MAC Frame.

For upstream packets, there will typically be only the 6-byte MAC Header, plus the 4 byte Upstream Privacy EH version 2 Element, so 10+14+4=28 bytes of per-packet overhead for an Upstream DOCSIS MAC Frame.

The 14+4 denotes the 14 byte ethernet header (6 byte dest mac, 6 byte src mac, 2 byte ether type) plus the 4 byte ethernet frame check sequence FCS.

Interestingly, the above DICSIS observations seem irrelevant for end customers as the shaper used in DOCSIS systems that limits a users maximal bandwidth does completely ignore DOCSIS overhead and only includes ethernet frames including their frame check sequence (FCS 4 Byte). Tocite the relevant section from the Docsis standard (http://www.cablelabs.com/specification/docsis-3-0-mac-and-upper-layer-protocols-interface-specification/):

"C.2.2.7.2 Maximum Sustained Traffic Rate 632 This parameter is the rate parameter R of a token-bucket-based rate limit for packets. R is expressed in bits per second, and MUST take into account all MAC frame data PDU of the Service Flow from the byte following the MAC header HCS to the end of the CRC, including every PDU in the case of a Concatenated MAC Frame. This parameter is applied after Payload Header Suppression; it does not include the bytes suppressed for PHS. The number of bytes forwarded (in bytes) is limited during any time interval T by Max(T), as described in the expression: Max(T) = T * (R / 8) + B, (1) where the parameter B (in bytes) is the Maximum Traffic Burst Configuration Setting (refer to Annex C.2.2.7.3). NOTE: This parameter does not limit the instantaneous rate of the Service Flow. The specific algorithm for enforcing this parameter is not mandated here. Any implementation which satisfies the above equation is conformant. In particular, the granularity of enforcement and the minimum implemented value of this parameter are vendor specific. The CMTS SHOULD support a granularity of at most 100 kbps. The CM SHOULD support a granularity of at most 100 kbps. NOTE: If this parameter is omitted or set to zero, then there is no explicitly-enforced traffic rate maximum. This field specifies only a bound, not a guarantee that this rate is available."

So in essence DOCSIS users need to only account for 18 Bytes of ethernet overhead in both ingress and egress directions under non-congested conditions.



> 
> For mpu why 64?
> I assume minimum 46bytes ethernet payload + 18bytes header?

	Well your DOCSIS modem will simply grab the ethernet packets including the FCS and those are simply guranteed to be >= 64 bytes. The linux kernel however might see smaller packets if the payload is smaller (but note for TCP/IPv4 with its 40 bytes of header the difference is quite small anyway). Interestingly if you use a single VLAN tag the ethernet MPU still stays at 64 bytes, only the payload section (including padding) shrinks to 42 bytes...

> 
> But why does cake use hard_header_len for the header size which is 14 bytes?

	I believe this is used internally so cake can deduce the size of the automatically added overhead (the linux kernel will add 14 bytes on ethernet interfaces automatically, which while certainly justifiable are not the ideal value for an ethernet shaper); with this cake allows the user to specify absolute overhead (so if you specify overhead 18, cake will give you 14 - 14 + 18 = 18, while tc's stab method will give you 14 + 18  = 32). So cake is doing the right thing here (the only thing it could do even better is to report the size of the automatically corrected in-kernel hard_header_len, but then the audience for that information is probably to small to justify this feature).

> 
> I don't know how packet sheduler system works, but maybe it is not needed to include the ethernet header.

	All you really need to give cake is a precise estimate of the bandwidth and the effective on-the-wire-size of the data packets; for DOCSIS this means overhead of 18 bytes*

> 
> cake does work on ip pakets or? so this is layer3 i think.
> the hole thing with ethernet headers is happening in layer2.
> This would change the minimum packet size to 46 or?


From sch_cake.c:

static inline u32 cake_overhead(struct cake_sched_data *q, u32 in)
{
        u32 out = in + q->rate_overhead;
        
        if (q->rate_mpu && out < q->rate_mpu) {
                out = q->rate_mpu;
        }
        
        if (q->rate_flags & CAKE_FLAG_ATM) {
                out += 47;
                out /= 48;
                out *= 53;
        } else if(q->rate_flags & CAKE_FLAG_PTM) {
                // the following adds one byte per 64 bytes or part thereof
                // this is conservative and easier to calculate than the precise value
                out += (out / 64) + !!(out % 64);
        }
        
        return out;                  
}

As you can see cake first adds the overhead and then makes sure the packet size is >= MPU... This will do the right thing for DOCSIS, even though it is conceivable that there are encapsulations out that can not be fully described with this (think full ethernet frames with additional overhead, where size+full_overhead might be >= mpu but the correct method would calculate size+ethernet_overhead >= mpu and then add outer_layer_overhead on top of this, but this is largely academic until something like this gets rolled-out in large numbers...)

Best Regards
	Sebastian

P.S.: unlike Ryan and Jonathan I would always use "overhead 18 mpu 64" instead of "docsis" because I prefer explicitness over terseness, but that is a matter of taste and "tc -s qdisc" will helpfully expand docsis into "overhead 18 mpu 64" (which is great).


> 
> Thanks.
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



*) The 18 bytes DOCSIS overhead are actually a lie, the true per-packet overhead on docsis is far greater. Without congestion all this additional overhead is simply "payed" for the user out of the shared gross bandwidth and everybody is happy. But under congestion this, IMHO, can lead to massive unfairness in that a user's shate of the gross bandwidth will correlate (bandwidth being equal) with the inversre of the packet size; put differently small packets use relatively more of the gross bandwidth than large packets, and I have not found out how docss gurantees fairness to all the CPE in a segment, but I digress. 

Re: [Cake] overhead and mpu

[Kevin Darbyshire-Bryant]

On 05/09/17 09:01, Sebastian Moeller wrote:
<snip>
> 	I believe this is used internally so cake can deduce the size of the automatically added overhead (the linux kernel will add 14 bytes on ethernet interfaces automatically, which while certainly justifiable are not the ideal value for an ethernet shaper); with this cake allows the user to specify absolute overhead (so if you specify overhead 18, cake will give you 14 - 14 + 18 = 18, while tc's stab method will give you 14 + 18  = 32). So cake is doing the right thing here (the only thing it could do even better is to report the size of the automatically corrected in-kernel hard_header_len, but then the audience for that information is probably to small to justify this feature).


Actually for the sake of a one line change in the tc code it is an 
extremely useful confidence boost and info source as to just what linux 
thinks the overhead actually is:

                // This is actually the *amount* of automatic 
compensation, but we only report
                // its presence as a boolean for now.
                if (ethernet)
-                      fprintf(f, "via-ethernet ");
+                      fprintf(f, "via-ethernet(%d) ", ethernet);

I've run that in a local patch to tc on LEDE for AGES.  Very useful.

Kevin

Re: [Cake] overhead and mpu

[Sebastian Moeller]

Hi Kevin,


> On Sep 5, 2017, at 10:35, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
> 
> 
> 
> On 05/09/17 09:01, Sebastian Moeller wrote:
> <snip>
>> 	I believe this is used internally so cake can deduce the size of the automatically added overhead (the linux kernel will add 14 bytes on ethernet interfaces automatically, which while certainly justifiable are not the ideal value for an ethernet shaper); with this cake allows the user to specify absolute overhead (so if you specify overhead 18, cake will give you 14 - 14 + 18 = 18, while tc's stab method will give you 14 + 18  = 32). So cake is doing the right thing here (the only thing it could do even better is to report the size of the automatically corrected in-kernel hard_header_len, but then the audience for that information is probably to small to justify this feature).
> 
> 
> Actually for the sake of a one line change in the tc code it is an extremely useful confidence boost and info source as to just what linux thinks the overhead actually is:
> 
>               // This is actually the *amount* of automatic compensation, but we only report
>               // its presence as a boolean for now.
>               if (ethernet)
> -                      fprintf(f, "via-ethernet ");
> +                      fprintf(f, "via-ethernet(%d) ", ethernet);
> 
> I've run that in a local patch to tc on LEDE for AGES.  Very useful.

	That sounds like it would be a great addition, since the information is already collected... For what it is worth I would vote for adding this... (to the version that should be upstreamed ;) )

Best Regards
	Sebastian


> 
> Kevin
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] overhead and mpu

[Ryan Mounce]

On 5 September 2017 at 18:28, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Kevin,
>
>
>> On Sep 5, 2017, at 10:35, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>>
>>
>>
>> On 05/09/17 09:01, Sebastian Moeller wrote:
>> <snip>
>>>      I believe this is used internally so cake can deduce the size of the automatically added overhead (the linux kernel will add 14 bytes on ethernet interfaces automatically, which while certainly justifiable are not the ideal value for an ethernet shaper); with this cake allows the user to specify absolute overhead (so if you specify overhead 18, cake will give you 14 - 14 + 18 = 18, while tc's stab method will give you 14 + 18  = 32). So cake is doing the right thing here (the only thing it could do even better is to report the size of the automatically corrected in-kernel hard_header_len, but then the audience for that information is probably to small to justify this feature).
>>
>>
>> Actually for the sake of a one line change in the tc code it is an extremely useful confidence boost and info source as to just what linux thinks the overhead actually is:
>>
>>               // This is actually the *amount* of automatic compensation, but we only report
>>               // its presence as a boolean for now.
>>               if (ethernet)
>> -                      fprintf(f, "via-ethernet ");
>> +                      fprintf(f, "via-ethernet(%d) ", ethernet);
>>
>> I've run that in a local patch to tc on LEDE for AGES.  Very useful.
>
>         That sounds like it would be a great addition, since the information is already collected... For what it is worth I would vote for adding this... (to the version that should be upstreamed ;) )

+1

It would also be nice for tc to know whether the overhead has been
explicitly configured and report appropriately. In my case I use cake
on a VLAN sub-interface that happens to have a hard_header_len of 18
after the 802.1q tag, and then use the docsis keyword to more
explicitly configure the overhead in case I move to an untagged
interface in the future. tc -s qdisc can't tell the difference and
reports 'raw mpu 64' rather than 'overhead 18 mpu 64'.

>
> Best Regards
>         Sebastian
>
>
>>
>> Kevin
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] overhead and mpu

[Sebastian Moeller]

Hi Ryan,

> On Sep 5, 2017, at 16:37, Ryan Mounce <ryan@mounce.com.au> wrote:
> 
> On 5 September 2017 at 18:28, Sebastian Moeller <moeller0@gmx.de> wrote:
>> Hi Kevin,
>> 
>> 
>>> On Sep 5, 2017, at 10:35, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>>> 
>>> 
>>> 
>>> On 05/09/17 09:01, Sebastian Moeller wrote:
>>> <snip>
>>>>     I believe this is used internally so cake can deduce the size of the automatically added overhead (the linux kernel will add 14 bytes on ethernet interfaces automatically, which while certainly justifiable are not the ideal value for an ethernet shaper); with this cake allows the user to specify absolute overhead (so if you specify overhead 18, cake will give you 14 - 14 + 18 = 18, while tc's stab method will give you 14 + 18  = 32). So cake is doing the right thing here (the only thing it could do even better is to report the size of the automatically corrected in-kernel hard_header_len, but then the audience for that information is probably to small to justify this feature).
>>> 
>>> 
>>> Actually for the sake of a one line change in the tc code it is an extremely useful confidence boost and info source as to just what linux thinks the overhead actually is:
>>> 
>>>              // This is actually the *amount* of automatic compensation, but we only report
>>>              // its presence as a boolean for now.
>>>              if (ethernet)
>>> -                      fprintf(f, "via-ethernet ");
>>> +                      fprintf(f, "via-ethernet(%d) ", ethernet);
>>> 
>>> I've run that in a local patch to tc on LEDE for AGES.  Very useful.
>> 
>>        That sounds like it would be a great addition, since the information is already collected... For what it is worth I would vote for adding this... (to the version that should be upstreamed ;) )
> 
> +1
> 
> It would also be nice for tc to know whether the overhead has been
> explicitly configured and report appropriately. In my case I use cake
> on a VLAN sub-interface that happens to have a hard_header_len of 18
> after the 802.1q tag, and then use the docsis keyword to more
> explicitly configure the overhead in case I move to an untagged
> interface in the future. tc -s qdisc can't tell the difference and
> reports 'raw mpu 64' rather than 'overhead 18 mpu 64'.

	Ah, I see, when ever the explicitly requested overhead equals hard_header_len, cake does not include it in its output. I believe this would be nice to change to always explicitly report the overhead (so even default to hard_header_len if no overhead was specified otherwise). 
Since the keywords like docsis are really convenience wrappers (hace a look in tc's q_cake.c) I vote for always expanding to the explicit overhead and potentially MPU values instead of reporting the keyword used.
I believe manually specifying "overhead 18 mpu 64" will be just as future proof as the docsis keyword in regards to changing hard_header_len.
It has to, in the end the docsis keyword gets interpreted by tc as:

               /*
                 * DOCSIS cable shapers account for Ethernet frame with FCS,
                 * but not interframe gap nor preamble.
                 */
                } else if (strcmp(*argv, "docsis") == 0) {
                        atm = 0;
                        overhead += 18;
                        overhead_set = true;
                        mpu = 64;
 

Best Regards
	Sebastian



> 
>> 
>> Best Regards
>>        Sebastian
>> 
>> 
>>> 
>>> Kevin
>>> _______________________________________________
>>> Cake mailing list
>>> Cake@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cake
>> 
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] overhead and mpu

[Kevin Darbyshire-Bryant]

On 05/09/17 15:37, Ryan Mounce wrote:
<snip>
> 
> +1
> 
> It would also be nice for tc to know whether the overhead has been
> explicitly configured and report appropriately. In my case I use cake
> on a VLAN sub-interface that happens to have a hard_header_len of 18
> after the 802.1q tag, and then use the docsis keyword to more
> explicitly configure the overhead in case I move to an untagged
> interface in the future. tc -s qdisc can't tell the difference and
> reports 'raw mpu 64' rather than 'overhead 18 mpu 64'.

The almost reality is that you could do whatever you want with the LEDE 
patch that adds cake awareness to iproute2's tc.  The challenge comes in 
keeping it up to date with whatever lands in 'cake master' and 'cake 
aware tc master'

I say almost reality because some on the LEDE committers list question 
things (rightly) if you're not pointing at master. For these reasons 
(and others) I've been reluctant to point LEDE cake at the cobalt branch 
so the 'in test' features of cobalt (and matching tc) are not being 
exercised there (ingress mode, "Voice tin traffic sometimes delayed by 
more than one bulk packet in best effort" 'solution' except it doesn't 
appear to have solved it)

Either the cobalt branch features need to get into master (and someone 
update the LEDE cake & iproute2 packages) OR someone needs to take the 
step to point LEDE cake at cobalt (and update the packages etc etc)

Cake on LEDE is currently without a maintainer and the cake list & repo 
has been incredibly quiet for the past 4+ months.

Re: [Cake] overhead and mpu

[Dennis Fedtke]

Hi!

Thank you for all answers.
But for me this still makes no sense.
Assuming we have an ethnernet connection running over a docsis line.
docsis is able to transmit full 1500byte ethernet packets.
Lets say it is an 50 Mbit/s Line. (I dont know now how exactly docsis works)
So to reach the 50Mbit/s ethernet speed the docsis link rate needs to be 
higher 50,6 Mbit/s (50*1518/1500 ??)
But when running a speedtest it will still not show the full speed. 
because of other overhead from underlying protocols (tcp/ip for example)
So the ISP will set the sync rate even higher to compensate for that.

But does this matter for the end user?
In case of docsis does it make sense to account for 18 overhead?
The user will enter 50mbit and it will work. If the isp has provided a 
sligher higher syncrate.

and the mpu setting. i don't know how cake handles this in detail.
How the overhead gets added.
lets i enter mpu 46.
And cake we set 18 as overhead.
Will this result in mpu 46 or 64?
Can someone debug the code maybe please? :>
I have the feeling with mpu 46 my pages lot a bit snappier. but could be 
placebo.

Thank you.

Re: [Cake] overhead and mpu

[Ryan Mounce]

On 6 September 2017 at 05:49, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
> Hi!
>
> Thank you for all answers.
> But for me this still makes no sense.
> Assuming we have an ethnernet connection running over a docsis line.
> docsis is able to transmit full 1500byte ethernet packets.
> Lets say it is an 50 Mbit/s Line. (I dont know now how exactly docsis works)
> So to reach the 50Mbit/s ethernet speed the docsis link rate needs to be
> higher 50,6 Mbit/s (50*1518/1500 ??)
> But when running a speedtest it will still not show the full speed. because
> of other overhead from underlying protocols (tcp/ip for example)
> So the ISP will set the sync rate even higher to compensate for that.
>
> But does this matter for the end user?

There is only one speed that matters for an end user configuring cake.
You need to know the rate of the shaper defined in the CM config file
for your plan, or otherwise determine this rate. As I have said CMs
don't 'sync' in the way you are used to with xDSL so this terminology
isn't really correct, however in terms of being the significant
bottleneck immediately 'upstream' of cake the configured rate of this
shaper is equivalent to the 'sync' speed for xDSL.

Simply ignore the speed that your ISP advertises, this is a bogus
number for just about every DOCSIS ISP. Also ignore the speed reported
by speedtest.net unless you are confident that there is no other
activity on your link and that you know what TCP options are in use by
your operating system.

If you get a very consistent result on speedtest.net, it is possible
to estimate the rate of the CMTS's shaper. This is a relationship
between the TCP payload size and full Ethernet frame size. For
example, on an OS that uses TCP timestamps and you get a result of
100Mbps on speedtest.net the L2 rate may be 100 * (1518 / 1448) ~=
104.8Mbps.
By now you can see where 1518 comes from (1500 bytes Ethernet payload
(IP packet) + 14 bytes Ethernet header + 4 bytes  Ethernet FCS). 1448
is (1500 bytes IP packet (Ethernet payload) - 20 bytes IPv4 header -
20 bytes TCP header - 12 bytes TCP timestamps option).

You should use Wireshark to confirm which TCP options and MTU your
system is using with the speedtest.net server and use these values
instead of my above example.

Once you have your estimate, round it down a couple of percent. If
your estimate is too high then it will be as if you don't have cake at
all. Too low and cake will still be effective, you will just sacrifice
a small amount of speed.

> In case of docsis does it make sense to account for 18 overhead?

Whatever the speed may be, you must always configure 'overhead 18 mpu
64' or the equivalent 'docsis' keyword to correctly compensate for the
Ethernet framing seen by the shaper in the CM/CMTS, otherwise cake may
underestimate the link utilisation when smaller packets are being sent
and its benefits will be defeated.

> The user will enter 50mbit and it will work. If the isp has provided a
> sligher higher syncrate.

This 50Mbps figure is suspiciously round, where is it actually from?
This is the advertised speed of your plan by the ISP?

> and the mpu setting. i don't know how cake handles this in detail.
> How the overhead gets added.
> lets i enter mpu 46.
> And cake we set 18 as overhead.
> Will this result in mpu 46 or 64?
> Can someone debug the code maybe please? :>

This has already been answered and the relevant snippet of code has
been posted. Overhead is first added (relative to IP), and packets
that are still below the MPU *after* the overhead is added will be
rounded up to the MPU of 64.

> I have the feeling with mpu 46 my pages lot a bit snappier. but could be
> placebo.

Correct, this is a placebo. You need to configure 'overhead 18 mpu 64'
or equivalently 'docsis', no ifs no buts. Focus your attention on
estimating the rate of your shaper. The framing compensation for
DOCSIS is a solved problem whereas the rate of a given link varies
from ISP to ISP, and from plan to plan.

>
> Thank you.
>
>
>
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] overhead and mpu

[Sebastian Moeller]

Hi Dennis,

> On Sep 5, 2017, at 22:19, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
> 
> Hi!
> 
> Thank you for all answers.
> But for me this still makes no sense.
> Assuming we have an ethnernet connection running over a docsis line.
> docsis is able to transmit full 1500byte ethernet packets.
> Lets say it is an 50 Mbit/s Line. (I dont know now how exactly docsis works)
> So to reach the 50Mbit/s ethernet speed the docsis link rate needs to be higher 50,6 Mbit/s (50*1518/1500 ??)

	Exactly, but that is completely hidden from the end-user as DOCSIS per standard offers speedlimits assuming 18 Bytes Ethernet overhead. Now, unlike DSL-ISPs that often sell close to the physical maximal capacity of a link, docsis being a shared medium with segment bandwidth >> single link bandwidth DOCSIS-ISPs get away with this (even though it raises fairness questions under congesting conditions). And as recommended by CISCO (and other CMTS manufacturers?) DOCSIS-ISPs tend to over provision the links so that in the typical case the measurable TCP/IPv4/HTTP goodput is equal or larger than the contracted bandwidth. That has the issue that it misleads typical customers into thinking that the ISP actually guarantees goodput. Why do I say mislead? Well if you look at the gross bandwidth (and let's restrict ourselves to the ethernet overhead, diving into the DOCSIS details will only complicate matters) and the resulting goodput for different goodputs, or better let's look at the required gross bandwidth for 100Mbps for 1500byte packets and 64byte packets:

100 * ((1500+18)/(1500-20-20)) = 103.97
100 * ((46+18)/(46-20-20)) = 1066.67

And the resulting percentual bandwidth cost of the per packet overhead (at equal goodput):
100-100*((1500+18)/(1500-20-20)) = -3.97 %
100-100*((46+18)/(46-20-20)) = -966.67 %

You quickly realize that there is a factor of 10 between those. Now without congestion a DOCSIS ISP might still grant that, but under congestion conditions it seems clearly suboptimal to split the gross bandwidth based on goodput fairness instead of on gross-bandwidth share basis... For a DOCSIS system the numbers still work, since the segment gross bandwidth seems sufficiently hight to allow for 64 byte packets, but for XDSL links where the gross bandwidth << 10 * the contracted bandwidth it is obvious that guaranteeing goodput is a foo's errand.


> But when running a speedtest it will still not show the full speed. because of other overhead from underlying protocols (tcp/ip for example)

	Yes, there is a number of protocols that might or might not be taken into account, like VLAN tags(s), PPPoE, TCP, IPv4/IPv6, TCP-options, IPv6 extension headers, the HTTP header typical for browser based speedtests. Unfortunately not all of those actually are visible for the endpoints of a speedtest; you really need to get the effective overhead for the bottleneck link.

> So the ISP will set the sync rate even higher to compensate for that.

	Maybe, maybe not, some ISPs do overprovisioning some do not. Personally, while I like more for the same price, I am a bit concened if an ISP seems incapable to correctly inform about the to be expected bandwidth...

> 
> But does this matter for the end user?

	As the 1500/64 numbers above hopefully show, it does since the per packet overhead still is data that needs to be transported and will "eat" into the available gross bandwidth pool.

> In case of docsis does it make sense to account for 18 overhead?

	Empirically it does seem so. But please note that under speedtest conditions (or under any condition where the packet size on the loaded link direction stays constant) not accounting for the per-packet overhead can be compensated by setting the shaper bandwidth slightly lower ad so the effect of missing overhead can easily go unnoticed, it will however cause trouble once a new traffic pattern uses more smaller packets...

> The user will enter 50mbit and it will work. If the isp has provided a sligher higher syncrate.

	As I hope I made clear above this will only work for fixed conditions in which the distribution of packet sizes on the link does not change.

> 
> and the mpu setting. i don't know how cake handles this in detail.
> How the overhead gets added.
> lets i enter mpu 46.
> And cake we set 18 as overhead.
> Will this result in mpu 46 or 64?

	This will result in MPU 46 (as that is what you requested), so an ICMP echo request packet with size 18+20+8 = 46 will be accounted as 46 bytes and not like the 64 it really takes up on the link. As shown, cake will first add the overhead and only compare the packet_len+overhead to mpu and do max(mpu, (paket_len+overhead)).

> Can someone debug the code maybe please? :>
> I have the feeling with mpu 46 my pages lot a bit snappier. but could be placebo.

	Under most conditions this difference will not matter at all, since typically packets+overhead are larger than 64 byte, so how did you measure to come to that feeling?

Best Regards
	Sebastian

> 
> Thank you.
> 
> 
> 
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] overhead and mpu

[Sebastian Moeller]

Hi Ryan,

I fully concur ;)


> On Sep 6, 2017, at 03:45, Ryan Mounce <ryan@mounce.com.au> wrote:
> 
> On 6 September 2017 at 05:49, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
>> Hi!
>> 
>> Thank you for all answers.
>> But for me this still makes no sense.
>> Assuming we have an ethnernet connection running over a docsis line.
>> docsis is able to transmit full 1500byte ethernet packets.
>> Lets say it is an 50 Mbit/s Line. (I dont know now how exactly docsis works)
>> So to reach the 50Mbit/s ethernet speed the docsis link rate needs to be
>> higher 50,6 Mbit/s (50*1518/1500 ??)
>> But when running a speedtest it will still not show the full speed. because
>> of other overhead from underlying protocols (tcp/ip for example)
>> So the ISP will set the sync rate even higher to compensate for that.
>> 
>> But does this matter for the end user?
> 
> There is only one speed that matters for an end user configuring cake.
> You need to know the rate of the shaper defined in the CM config file
> for your plan, or otherwise determine this rate. As I have said CMs
> don't 'sync' in the way you are used to with xDSL so this terminology
> isn't really correct, however in terms of being the significant
> bottleneck immediately 'upstream' of cake the configured rate of this
> shaper is equivalent to the 'sync' speed for xDSL.
> 
> Simply ignore the speed that your ISP advertises, this is a bogus
> number for just about every DOCSIS ISP. Also ignore the speed reported
> by speedtest.net unless you are confident that there is no other
> activity on your link and that you know what TCP options are in use by
> your operating system.
> 
> If you get a very consistent result on speedtest.net, it is possible
> to estimate the rate of the CMTS's shaper. This is a relationship
> between the TCP payload size and full Ethernet frame size. For
> example, on an OS that uses TCP timestamps and you get a result of
> 100Mbps on speedtest.net the L2 rate may be 100 * (1518 / 1448) ~=
> 104.8Mbps.
> By now you can see where 1518 comes from (1500 bytes Ethernet payload
> (IP packet) + 14 bytes Ethernet header + 4 bytes  Ethernet FCS). 1448
> is (1500 bytes IP packet (Ethernet payload) - 20 bytes IPv4 header -
> 20 bytes TCP header - 12 bytes TCP timestamps option).
> 
> You should use Wireshark to confirm which TCP options and MTU your
> system is using with the speedtest.net server and use these values
> instead of my above example.

Another quick and dirty method to measure the MTU is to use https://www.speedguide.net/analyzer.php.


> Once you have your estimate, round it down a couple of percent. If
> your estimate is too high then it will be as if you don't have cake at
> all. Too low and cake will still be effective, you will just sacrifice
> a small amount of speed.
> 
>> In case of docsis does it make sense to account for 18 overhead?
> 
> Whatever the speed may be, you must always configure 'overhead 18 mpu
> 64' or the equivalent 'docsis' keyword to correctly compensate for the
> Ethernet framing seen by the shaper in the CM/CMTS, otherwise cake may
> underestimate the link utilisation when smaller packets are being sent
> and its benefits will be defeated.
> 
>> The user will enter 50mbit and it will work. If the isp has provided a
>> sligher higher syncrate.
> 
> This 50Mbps figure is suspiciously round, where is it actually from?
> This is the advertised speed of your plan by the ISP?
> 
>> and the mpu setting. i don't know how cake handles this in detail.
>> How the overhead gets added.
>> lets i enter mpu 46.
>> And cake we set 18 as overhead.
>> Will this result in mpu 46 or 64?
>> Can someone debug the code maybe please? :>
> 
> This has already been answered and the relevant snippet of code has
> been posted. Overhead is first added (relative to IP), and packets
> that are still below the MPU *after* the overhead is added will be
> rounded up to the MPU of 64.
> 
>> I have the feeling with mpu 46 my pages lot a bit snappier. but could be
>> placebo.
> 
> Correct, this is a placebo. You need to configure 'overhead 18 mpu 64'
> or equivalently 'docsis', no ifs no buts. Focus your attention on
> estimating the rate of your shaper. The framing compensation for
> DOCSIS is a solved problem whereas the rate of a given link varies
> from ISP to ISP, and from plan to plan.
> 
>> 
>> Thank you.
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] overhead and mpu

[Dennis Fedtke]

Hi everyone!

thanks for your answers again.

So what about from should i expect from using the docsis keyword?
(This layer cake script + nat docsis set in advanced config)

tc -s qdisc show dev eth1.100:
qdisc cake 8064: root refcnt 2 bandwidth 19Mbit diffserv3 triple-isolate 
nat rtt 100.0ms raw mpu 64
  Sent 246233450 bytes 1316430 pkt (dropped 65, overlimits 126706 
requeues 0)
  backlog 0b 0p requeues 0
  memory used: 2372672b of 4Mb
  capacity estimate: 19Mbit
                  Bulk   Best Effort      Voice
   thresh      1187Kbit      19Mbit    4750Kbit
   target        15.3ms       5.0ms       5.0ms
   interval     110.3ms     100.0ms      10.0ms
   pk_delay         0us       298us        10us
   av_delay         0us        13us         2us
   sp_delay         0us         1us         1us
   pkts               0     1315692         803
   bytes              0   246260830       59314
   way_inds           0         733           0
   way_miss           0        5012          29
   way_cols           0           0           0
   drops              0          65           0
   marks              0           0           0
   sp_flows           0           1           0
   bk_flows           0           0           0
   un_flows           0           0           0
   max_len            0        1514         405

tc -s qdisc show dev ifb4eth1.100
qdisc cake 8065: root refcnt 2 bandwidth 150Mbit diffserv3 
triple-isolate nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64
  Sent 869520280 bytes 2153104 pkt (dropped 30, overlimits 270811 
requeues 0)
  backlog 0b 0p requeues 0
  memory used: 624320b of 7500000b
  capacity estimate: 150Mbit
                  Bulk   Best Effort      Voice
   thresh      9375Kbit     150Mbit   37500Kbit
   target         5.0ms       5.0ms       5.0ms
   interval     100.0ms     100.0ms      10.0ms
   pk_delay       121us       448us         1us
   av_delay        40us       158us         0us
   sp_delay         2us         2us         0us
   pkts           58805     1362429      731900
   bytes       77433974   748216627    43914060
   way_inds           0      118167           0
   way_miss           5        5911           3
   way_cols           0           0           0
   drops              1          29           0
   marks              0           0           0
   sp_flows           0           0           0
   bk_flows           0           0           1
   un_flows           0           0           0
   max_len         1486        1514          90

Why is there no overhead showing for egress?
Why is there overhead18 + via-ethernet showing for ingress?

Thanks again =)

Re: [Cake] overhead and mpu

[Dennis Fedtke]

Hi!

I forgot to ask why max_len shows 1514 and not 1518 in the output of tc?

Thanks.


Am 07.09.2017 um 06:11 schrieb Dennis Fedtke:
> Hi everyone!
>
> thanks for your answers again.
>
> So what about from should i expect from using the docsis keyword?
> (This layer cake script + nat docsis set in advanced config)
>
> tc -s qdisc show dev eth1.100:
> qdisc cake 8064: root refcnt 2 bandwidth 19Mbit diffserv3 
> triple-isolate nat rtt 100.0ms raw mpu 64
>  Sent 246233450 bytes 1316430 pkt (dropped 65, overlimits 126706 
> requeues 0)
>  backlog 0b 0p requeues 0
>  memory used: 2372672b of 4Mb
>  capacity estimate: 19Mbit
>                  Bulk   Best Effort      Voice
>   thresh      1187Kbit      19Mbit    4750Kbit
>   target        15.3ms       5.0ms       5.0ms
>   interval     110.3ms     100.0ms      10.0ms
>   pk_delay         0us       298us        10us
>   av_delay         0us        13us         2us
>   sp_delay         0us         1us         1us
>   pkts               0     1315692         803
>   bytes              0   246260830       59314
>   way_inds           0         733           0
>   way_miss           0        5012          29
>   way_cols           0           0           0
>   drops              0          65           0
>   marks              0           0           0
>   sp_flows           0           1           0
>   bk_flows           0           0           0
>   un_flows           0           0           0
>   max_len            0        1514         405
>
> tc -s qdisc show dev ifb4eth1.100
> qdisc cake 8065: root refcnt 2 bandwidth 150Mbit diffserv3 
> triple-isolate nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64
>  Sent 869520280 bytes 2153104 pkt (dropped 30, overlimits 270811 
> requeues 0)
>  backlog 0b 0p requeues 0
>  memory used: 624320b of 7500000b
>  capacity estimate: 150Mbit
>                  Bulk   Best Effort      Voice
>   thresh      9375Kbit     150Mbit   37500Kbit
>   target         5.0ms       5.0ms       5.0ms
>   interval     100.0ms     100.0ms      10.0ms
>   pk_delay       121us       448us         1us
>   av_delay        40us       158us         0us
>   sp_delay         2us         2us         0us
>   pkts           58805     1362429      731900
>   bytes       77433974   748216627    43914060
>   way_inds           0      118167           0
>   way_miss           5        5911           3
>   way_cols           0           0           0
>   drops              1          29           0
>   marks              0           0           0
>   sp_flows           0           0           0
>   bk_flows           0           0           1
>   un_flows           0           0           0
>   max_len         1486        1514          90
>
> Why is there no overhead showing for egress?
> Why is there overhead18 + via-ethernet showing for ingress?
>
> Thanks again =)
>

Re: [Cake] overhead and mpu

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 195 bytes --]

Because 1514 is the normal maximum size of an Ethernet frame. This can
easily differ from the size actually used on the wire and from the size
used by the provisioning shaper.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 234 bytes --]

Re: [Cake] overhead and mpu

[Dennis Fedtke]

Hi,

tc does not output the current overhead value when the value is the same 
as the default one, correct?

When i enter overhead 18 for my eth1.100, tc doesnt output any overhead.
When i enter overhead 14 for my ifb4eth1.100, tc doesnt output any overhead.

So the default overhead for my eth.100 is 18.
And for my ifb4eth1.100 device it is 14.

Is this expected behavior?

Thanks.

Am 07.09.2017 um 06:25 schrieb Jonathan Morton:
>
> Because 1514 is the normal maximum size of an Ethernet frame. This can 
> easily differ from the size actually used on the wire and from the 
> size used by the provisioning shaper.
>
> - Jonathan Morton
>

Re: [Cake] overhead and mpu

[Sebastian Moeller]

Hi Dennis,

> On Sep 7, 2017, at 06:11, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
> 
> Hi everyone!
> 
> thanks for your answers again.
> 
> So what about from should i expect from using the docsis keyword?
> (This layer cake script + nat docsis set in advanced config)
> 
> tc -s qdisc show dev eth1.100:
> qdisc cake 8064: root refcnt 2 bandwidth 19Mbit diffserv3 triple-isolate nat rtt 100.0ms raw mpu 64
>  Sent 246233450 bytes 1316430 pkt (dropped 65, overlimits 126706 requeues 0)
>  backlog 0b 0p requeues 0
>  memory used: 2372672b of 4Mb
>  capacity estimate: 19Mbit
>                  Bulk   Best Effort      Voice
>   thresh      1187Kbit      19Mbit    4750Kbit
>   target        15.3ms       5.0ms       5.0ms
>   interval     110.3ms     100.0ms      10.0ms
>   pk_delay         0us       298us        10us
>   av_delay         0us        13us         2us
>   sp_delay         0us         1us         1us
>   pkts               0     1315692         803
>   bytes              0   246260830       59314
>   way_inds           0         733           0
>   way_miss           0        5012          29
>   way_cols           0           0           0
>   drops              0          65           0
>   marks              0           0           0
>   sp_flows           0           1           0
>   bk_flows           0           0           0
>   un_flows           0           0           0
>   max_len            0        1514         405
> 
> tc -s qdisc show dev ifb4eth1.100
> qdisc cake 8065: root refcnt 2 bandwidth 150Mbit diffserv3 triple-isolate nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64
>  Sent 869520280 bytes 2153104 pkt (dropped 30, overlimits 270811 requeues 0)
>  backlog 0b 0p requeues 0
>  memory used: 624320b of 7500000b
>  capacity estimate: 150Mbit
>                  Bulk   Best Effort      Voice
>   thresh      9375Kbit     150Mbit   37500Kbit
>   target         5.0ms       5.0ms       5.0ms
>   interval     100.0ms     100.0ms      10.0ms
>   pk_delay       121us       448us         1us
>   av_delay        40us       158us         0us
>   sp_delay         2us         2us         0us
>   pkts           58805     1362429      731900
>   bytes       77433974   748216627    43914060
>   way_inds           0      118167           0
>   way_miss           5        5911           3
>   way_cols           0           0           0
>   drops              1          29           0
>   marks              0           0           0
>   sp_flows           0           0           0
>   bk_flows           0           0           1
>   un_flows           0           0           0
>   max_len         1486        1514          90
> 
> Why is there no overhead showing for egress?

Could you post post the output of "cat /etc/config/sqm" (assuming you use LEDE/OpenWrt); i have a hunch that you might have forgotten to put the docsis keyword into both advanced configuration fields (or there is a typo n one of those words).


> Why is there overhead18 + via-ethernet showing for ingress?

	This tells you that cake is using ethernet_payload + 18 bytes overhead, and that it automatically effectively subtracted the 14 bytes overhead the kernel added before applying its 18 bytes (so that you do not need to worry about how much the kernel added).

BTW, cake reports the maximum of the pkt_len field of all packets as max_len and on ethernet interfaces the kernel automatically adds 14 bytes (already to skb->len), this is why you see max_len 1514 instead of 1500. If you use tc's stab method for overhead accounting things will look different, as stab will add the full header size to pkt_len, so using tc stab you would probably see 1532 for an ecplicit overhead of 18 (as tc stab does not take the already accounted for header into account so if you request 18 on an ethernet interface you will effectively get 18+14 = 32.) It really is  a great feature of cake to undo the kernel's automagic length adjustment so that one can request overhead on top of the ethernet payload/the IP packets.

Best Regards
	Sebastian

> 
> Thanks again =)
> 

Re: [Cake] overhead and mpu

[Dennis Fedtke]

Hi Sebastian,

here is the outout:

config queue
         option debug_logging '0'
         option interface 'eth1.100'
         option qdisc 'cake'
         option enabled '1'
         option qdisc_advanced '1'
         option qdisc_really_really_advanced '1'
         option linklayer 'none'
         option egress_ecn 'NOECN'
         option upload '19000'
         option verbosity '5'
         option ingress_ecn 'ECN'
         option script 'layer_cake.qos'
         option download '150000'
         option squash_ingress '1'
         option squash_dscp '1'
         option iqdisc_opts 'docsis nat'
         option eqdisc_opts 'docsis nat'

But also see my last email please.

it seems that both devices use different default overhead (14 vs 18)

Thanks.


Am 07.09.2017 um 09:47 schrieb Sebastian Moeller:
> Hi Dennis,
>
>> On Sep 7, 2017, at 06:11, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
>>
>> Hi everyone!
>>
>> thanks for your answers again.
>>
>> So what about from should i expect from using the docsis keyword?
>> (This layer cake script + nat docsis set in advanced config)
>>
>> tc -s qdisc show dev eth1.100:
>> qdisc cake 8064: root refcnt 2 bandwidth 19Mbit diffserv3 triple-isolate nat rtt 100.0ms raw mpu 64
>>   Sent 246233450 bytes 1316430 pkt (dropped 65, overlimits 126706 requeues 0)
>>   backlog 0b 0p requeues 0
>>   memory used: 2372672b of 4Mb
>>   capacity estimate: 19Mbit
>>                   Bulk   Best Effort      Voice
>>    thresh      1187Kbit      19Mbit    4750Kbit
>>    target        15.3ms       5.0ms       5.0ms
>>    interval     110.3ms     100.0ms      10.0ms
>>    pk_delay         0us       298us        10us
>>    av_delay         0us        13us         2us
>>    sp_delay         0us         1us         1us
>>    pkts               0     1315692         803
>>    bytes              0   246260830       59314
>>    way_inds           0         733           0
>>    way_miss           0        5012          29
>>    way_cols           0           0           0
>>    drops              0          65           0
>>    marks              0           0           0
>>    sp_flows           0           1           0
>>    bk_flows           0           0           0
>>    un_flows           0           0           0
>>    max_len            0        1514         405
>>
>> tc -s qdisc show dev ifb4eth1.100
>> qdisc cake 8065: root refcnt 2 bandwidth 150Mbit diffserv3 triple-isolate nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64
>>   Sent 869520280 bytes 2153104 pkt (dropped 30, overlimits 270811 requeues 0)
>>   backlog 0b 0p requeues 0
>>   memory used: 624320b of 7500000b
>>   capacity estimate: 150Mbit
>>                   Bulk   Best Effort      Voice
>>    thresh      9375Kbit     150Mbit   37500Kbit
>>    target         5.0ms       5.0ms       5.0ms
>>    interval     100.0ms     100.0ms      10.0ms
>>    pk_delay       121us       448us         1us
>>    av_delay        40us       158us         0us
>>    sp_delay         2us         2us         0us
>>    pkts           58805     1362429      731900
>>    bytes       77433974   748216627    43914060
>>    way_inds           0      118167           0
>>    way_miss           5        5911           3
>>    way_cols           0           0           0
>>    drops              1          29           0
>>    marks              0           0           0
>>    sp_flows           0           0           0
>>    bk_flows           0           0           1
>>    un_flows           0           0           0
>>    max_len         1486        1514          90
>>
>> Why is there no overhead showing for egress?
> Could you post post the output of "cat /etc/config/sqm" (assuming you use LEDE/OpenWrt); i have a hunch that you might have forgotten to put the docsis keyword into both advanced configuration fields (or there is a typo n one of those words).
>
>
>> Why is there overhead18 + via-ethernet showing for ingress?
> 	This tells you that cake is using ethernet_payload + 18 bytes overhead, and that it automatically effectively subtracted the 14 bytes overhead the kernel added before applying its 18 bytes (so that you do not need to worry about how much the kernel added).
>
> BTW, cake reports the maximum of the pkt_len field of all packets as max_len and on ethernet interfaces the kernel automatically adds 14 bytes (already to skb->len), this is why you see max_len 1514 instead of 1500. If you use tc's stab method for overhead accounting things will look different, as stab will add the full header size to pkt_len, so using tc stab you would probably see 1532 for an ecplicit overhead of 18 (as tc stab does not take the already accounted for header into account so if you request 18 on an ethernet interface you will effectively get 18+14 = 32.) It really is  a great feature of cake to undo the kernel's automagic length adjustment so that one can request overhead on top of the ethernet payload/the IP packets.
>
> Best Regards
> 	Sebastian
>
>> Thanks again =)
>>

Re: [Cake] overhead and mpu

[Sebastian Moeller]

Hi Jonathan,

please let me elaborate below.

> On Sep 7, 2017, at 06:25, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> Because 1514 is the normal maximum size of an Ethernet frame.

	This might or might not be true (depending on the defnition of ethernet frame one uses), but it certainly is not overly helpful to novices, as e.g. https://en.wikipedia.org/wiki/Ethernet_frame will show the FCS as being part of the layer2 ethernet frame and 1514 does not contain the FCS nor any potential VLAN tag(s). 
What is probably correct to say is that 1514 is the kernel's internal view of the size of an ethernet frame (most likely the FCS will be added by the NIC, so 1514 is the true size of the data the kernel shuttles around at that abstraction layer, but certainly not a real complete ethernet frame).

> This can easily differ from the size actually used on the wire

	Not easily, but certainly with 1500 bytes payload an ethernet packet will take at least 1518 bytes on the L2-wire and effectively 1518+20 = 1538 bytes on L1 (here I account for the silent inter-frame gap by the 12 bytes worth of data that could be transmitted in the same time)

> and from the size used by the provisioning shaper.

	And that is the thing that makes specifying overhead hard to automate, instead of the actually connected interface one needs to figure out the overhead (and bandwidth) applicable on the slowest part of the internet connection... (Which in theory is variable, but in practice it luckily is often the link to the ISP).

Best Regards
	Sebastian

> 
> - Jonathan Morton
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] overhead and mpu

[Sebastian Moeller]

Hi Dennis,

> On Sep 7, 2017, at 09:58, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
> 
> Hi Sebastian,
> 
> here is the outout:
> 
> config queue
>         option debug_logging '0'
>         option interface 'eth1.100'
>         option qdisc 'cake'
>         option enabled '1'
>         option qdisc_advanced '1'
>         option qdisc_really_really_advanced '1'
>         option linklayer 'none'
>         option egress_ecn 'NOECN'
>         option upload '19000'
>         option verbosity '5'
>         option ingress_ecn 'ECN'
>         option script 'layer_cake.qos'
>         option download '150000'
>         option squash_ingress '1'
>         option squash_dscp '1'
>         option iqdisc_opts 'docsis nat'
>         option eqdisc_opts 'docsis nat'

Ah, there goes my typo or forgotten keyword hypothesis...
	I bet the reason is that on the IFB the vlan tag might not be visible, while on egress it is; unexpectedly cake will not report the overhead if the specified overhead equals the hard_header_len (@Jonathan, what is the rationale for the behavior? could you live with reporting the hard_header_len unconditionally somewhere in the "tc -s qdisc" output?). Since with a vlan tag the kernel accounts for 18 bytes (a vlan tag takes 4 bytes and 14+4=18) cake simply omits this during reporting. Now why the kernel deos not seem to see/account for the VLAN tag on ingress I do not know. But this certainly shows that cake does the right thing by automatically adjusting for hard_header_len... this certainly is more complicated than it should be...


> 
> But also see my last email please.
> 
> it seems that both devices use different default overhead (14 vs 18)

	I agree, but I have no time to dive into the kernel sources to figure out why...

Best Regards
	Sebastian

> 
> Thanks.
> 
> 
> Am 07.09.2017 um 09:47 schrieb Sebastian Moeller:
>> Hi Dennis,
>> 
>>> On Sep 7, 2017, at 06:11, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
>>> 
>>> Hi everyone!
>>> 
>>> thanks for your answers again.
>>> 
>>> So what about from should i expect from using the docsis keyword?
>>> (This layer cake script + nat docsis set in advanced config)
>>> 
>>> tc -s qdisc show dev eth1.100:
>>> qdisc cake 8064: root refcnt 2 bandwidth 19Mbit diffserv3 triple-isolate nat rtt 100.0ms raw mpu 64
>>>  Sent 246233450 bytes 1316430 pkt (dropped 65, overlimits 126706 requeues 0)
>>>  backlog 0b 0p requeues 0
>>>  memory used: 2372672b of 4Mb
>>>  capacity estimate: 19Mbit
>>>                  Bulk   Best Effort      Voice
>>>   thresh      1187Kbit      19Mbit    4750Kbit
>>>   target        15.3ms       5.0ms       5.0ms
>>>   interval     110.3ms     100.0ms      10.0ms
>>>   pk_delay         0us       298us        10us
>>>   av_delay         0us        13us         2us
>>>   sp_delay         0us         1us         1us
>>>   pkts               0     1315692         803
>>>   bytes              0   246260830       59314
>>>   way_inds           0         733           0
>>>   way_miss           0        5012          29
>>>   way_cols           0           0           0
>>>   drops              0          65           0
>>>   marks              0           0           0
>>>   sp_flows           0           1           0
>>>   bk_flows           0           0           0
>>>   un_flows           0           0           0
>>>   max_len            0        1514         405
>>> 
>>> tc -s qdisc show dev ifb4eth1.100
>>> qdisc cake 8065: root refcnt 2 bandwidth 150Mbit diffserv3 triple-isolate nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64
>>>  Sent 869520280 bytes 2153104 pkt (dropped 30, overlimits 270811 requeues 0)
>>>  backlog 0b 0p requeues 0
>>>  memory used: 624320b of 7500000b
>>>  capacity estimate: 150Mbit
>>>                  Bulk   Best Effort      Voice
>>>   thresh      9375Kbit     150Mbit   37500Kbit
>>>   target         5.0ms       5.0ms       5.0ms
>>>   interval     100.0ms     100.0ms      10.0ms
>>>   pk_delay       121us       448us         1us
>>>   av_delay        40us       158us         0us
>>>   sp_delay         2us         2us         0us
>>>   pkts           58805     1362429      731900
>>>   bytes       77433974   748216627    43914060
>>>   way_inds           0      118167           0
>>>   way_miss           5        5911           3
>>>   way_cols           0           0           0
>>>   drops              1          29           0
>>>   marks              0           0           0
>>>   sp_flows           0           0           0
>>>   bk_flows           0           0           1
>>>   un_flows           0           0           0
>>>   max_len         1486        1514          90
>>> 
>>> Why is there no overhead showing for egress?
>> Could you post post the output of "cat /etc/config/sqm" (assuming you use LEDE/OpenWrt); i have a hunch that you might have forgotten to put the docsis keyword into both advanced configuration fields (or there is a typo n one of those words).
>> 
>> 
>>> Why is there overhead18 + via-ethernet showing for ingress?
>> 	This tells you that cake is using ethernet_payload + 18 bytes overhead, and that it automatically effectively subtracted the 14 bytes overhead the kernel added before applying its 18 bytes (so that you do not need to worry about how much the kernel added).
>> 
>> BTW, cake reports the maximum of the pkt_len field of all packets as max_len and on ethernet interfaces the kernel automatically adds 14 bytes (already to skb->len), this is why you see max_len 1514 instead of 1500. If you use tc's stab method for overhead accounting things will look different, as stab will add the full header size to pkt_len, so using tc stab you would probably see 1532 for an ecplicit overhead of 18 (as tc stab does not take the already accounted for header into account so if you request 18 on an ethernet interface you will effectively get 18+14 = 32.) It really is  a great feature of cake to undo the kernel's automagic length adjustment so that one can request overhead on top of the ethernet payload/the IP packets.
>> 
>> Best Regards
>> 	Sebastian
>> 
>>> Thanks again =)
>>> 
>