* Re: [Cake] overhead and mpu
2017-09-05 6:00 ` Dennis Fedtke
@ 2017-09-05 6:49 ` Ryan Mounce
2017-09-05 6:58 ` Jonathan Morton
2017-09-05 8:01 ` Sebastian Moeller
2 siblings, 0 replies; 24+ messages in thread
From: Ryan Mounce @ 2017-09-05 6:49 UTC (permalink / raw)
To: Dennis Fedtke; +Cc: cake
Hi Dennis,
Firstly to nitpick, DOCSIS doesn't 'sync' in the traditional xDSL
sense and more or less operates at a fixed speed at the physical layer
(DOCSIS 3.1 changes this a bit but it is irrelevant to what you're
trying to achieve here). The modem and CMTS are typically capable of
high speeds however the coax is a shared medium. The actual speed you
see as an end user is typically defined by a shaper in the CM / CMTS
in order to present the illusion of a reliable fixed speed pipe and to
create an artificial model to charge more for higher speeds.
So, it depends where and how that "50Mbps" is defined. It is typical
for a MSO / cable ISP to actually under-sell and over-deliver on the
headline speed, e.g. in my ISP's case advertising "100/2Mbps" on
paper, configuring the shapers at 120/2.5Mbps (which is inclusive of
18 bytes overhead and padding for frames smaller than 64 bytes), which
then results in real world application layer throughput somewhere
between those two values (~114/2.4Mbps on something like speedtest.net
in ideal conditions). As far as cake is concerned, 120/2.5 is the
important figure here.
You want to configure cake with the 'docsis' keyword (equivalent to
overhead of 18 bytes on top of IP, and 64 byte MPU inclusive of
Ethernet header), and 'bandwidth' that is about 99% of whatever your
ISP configures the modem for your plan.
The only problem is that you normally can't view what the modem is
actually configured for, so you need to do some testing which comes
down to trial and error. My technique to configure the upstream rate
was to configure cake for a speed slightly above that of my actual
connection, continuously ping a nearby server on the internet, and
then start a large file upload. The modem's buffer quickly fills up
and latency increases until the buffer is full and packets are being
dropped. From this point I kept re-configuring cake and slightly
reducing the bandwidth until the buffer started to clear and pings
slowly returned to baseline. In my case I observed this at 2499Kbps
(suggesting that my modem's shaper is configured for precisely
2500Kbps), however I use 2496Kbps as the final value so that the
buffer clears slightly faster in case is a 'burp' for whatever reason
that fills it.
Why is the MPU 64? Good question, that is an Ethernet thing defined in
the 70's and inherited by DOCSIS many years later. But yes, this
results in a minimum Ethernet payload of 46 bytes.
Re: hard_header_len, it's been months since I have dug into the source
and I have since forgotten the internal details. I understood it once
upon a time, my advise is to not concern yourself with this.
What I do remember for sure is that from the user configuration
perspective everything is relative to IP packets. It doesn't matter
what layer cake 'sees' internally, this is abstracted away from the
user in order to present a consistent interface.
TL;DR just use the 'docsis' keyword and forget everything you know
about overhead and mpu. That's why it's there, so that nobody else
ever has to go through the pain I did when fine-tuning cake for my
DOCSIS connection.
Regards,
Ryan Mounce
On 5 September 2017 at 15:30, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
> Hi Ryan,
>
> Thanks for you answers.
> Lets assume again ethernet over docsis connection at 50 Mbit/s.
> So to get 50 Mbit/s ethernet perfomance, the docsis link speed needs to be
> set at a higher link speed to compensate for the 18 header ethernet
> overhead, or?
> or
> Assuming:
> The docsis link syncs at exactly 50Mbit/s. When cake is set to exactly
> 50Mbit/s, it is actually set at a higher speed then the link actually is.
>
> For mpu why 64?
> I assume minimum 46bytes ethernet payload + 18bytes header?
>
> But why does cake use hard_header_len for the header size which is 14 bytes?
>
> I don't know how packet sheduler system works, but maybe it is not needed to
> include the ethernet header.
>
> cake does work on ip pakets or? so this is layer3 i think.
> the hole thing with ethernet headers is happening in layer2.
> This would change the minimum packet size to 46 or?
>
> Thanks.
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-05 6:00 ` Dennis Fedtke
2017-09-05 6:49 ` Ryan Mounce
@ 2017-09-05 6:58 ` Jonathan Morton
2017-09-05 20:19 ` Dennis Fedtke
2017-09-05 8:01 ` Sebastian Moeller
2 siblings, 1 reply; 24+ messages in thread
From: Jonathan Morton @ 2017-09-05 6:58 UTC (permalink / raw)
To: Dennis Fedtke; +Cc: Cake List
[-- Attachment #1: Type: text/plain, Size: 1816 bytes --]
I designed Cake to require as little networking expertise as feasible of
the end user. You only really need to know your network topology. So don't
overthink this - just set "bandwidth 48Mbit docsis ingress" on your inbound
shaper, and "bandwidth (whatever) docsis egress" on the outbound.
Why 48 instead of 50? Because Cake needs that when downstream of the
bottleneck link, just like any other shaper, so that it controls the
bottleneck queue. If your ISP decided to install Cake themselves, they
wouldn't need that hack, because they have access to the upstream end of
the downlink.
Now if you still want to know the technical details...
Cake's shaper works by calculating the time it takes to get a given packet
through the bottleneck link, measured from the beginning of transmission of
that packet to the beginning of transmission of the next packet. To do
that, it needs to know the number of bytes per second that the link can
carry, and the actual number of byte times occupied by the packet on the
wire - not all of which might be included in your typical frame header and
payload; we also have to count preamble, checksum, quiet times, etc.
These are layer 2 concerns, and in fact Cake normally sees layer 2 packets
when attached to an Ethernet interface. Just give it the raw data, and
it'll take care of nearly all the nasty calculations and expert knowledge
you'd need with most other shapers.
The rest of Cake is largely concerned with choosing which packet to deliver
next, given the choice among all the packets in its queue. This involves
examining layer 3 and layer 4 information within the packet, where
available. There are layer 3 (IP) packets without a layer 4 (TCP/UDP)
payload, and layer 2 packets without an IP payload, which Cake also has to
deal with sanely.
- Jonathan Morton
[-- Attachment #2: Type: text/html, Size: 1988 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-05 6:58 ` Jonathan Morton
@ 2017-09-05 20:19 ` Dennis Fedtke
2017-09-06 1:45 ` Ryan Mounce
2017-09-06 7:22 ` Sebastian Moeller
0 siblings, 2 replies; 24+ messages in thread
From: Dennis Fedtke @ 2017-09-05 20:19 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Cake List
Hi!
Thank you for all answers.
But for me this still makes no sense.
Assuming we have an ethnernet connection running over a docsis line.
docsis is able to transmit full 1500byte ethernet packets.
Lets say it is an 50 Mbit/s Line. (I dont know now how exactly docsis works)
So to reach the 50Mbit/s ethernet speed the docsis link rate needs to be
higher 50,6 Mbit/s (50*1518/1500 ??)
But when running a speedtest it will still not show the full speed.
because of other overhead from underlying protocols (tcp/ip for example)
So the ISP will set the sync rate even higher to compensate for that.
But does this matter for the end user?
In case of docsis does it make sense to account for 18 overhead?
The user will enter 50mbit and it will work. If the isp has provided a
sligher higher syncrate.
and the mpu setting. i don't know how cake handles this in detail.
How the overhead gets added.
lets i enter mpu 46.
And cake we set 18 as overhead.
Will this result in mpu 46 or 64?
Can someone debug the code maybe please? :>
I have the feeling with mpu 46 my pages lot a bit snappier. but could be
placebo.
Thank you.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-05 20:19 ` Dennis Fedtke
@ 2017-09-06 1:45 ` Ryan Mounce
2017-09-06 7:26 ` Sebastian Moeller
2017-09-06 7:22 ` Sebastian Moeller
1 sibling, 1 reply; 24+ messages in thread
From: Ryan Mounce @ 2017-09-06 1:45 UTC (permalink / raw)
To: Dennis Fedtke; +Cc: Jonathan Morton, Cake List
On 6 September 2017 at 05:49, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
> Hi!
>
> Thank you for all answers.
> But for me this still makes no sense.
> Assuming we have an ethnernet connection running over a docsis line.
> docsis is able to transmit full 1500byte ethernet packets.
> Lets say it is an 50 Mbit/s Line. (I dont know now how exactly docsis works)
> So to reach the 50Mbit/s ethernet speed the docsis link rate needs to be
> higher 50,6 Mbit/s (50*1518/1500 ??)
> But when running a speedtest it will still not show the full speed. because
> of other overhead from underlying protocols (tcp/ip for example)
> So the ISP will set the sync rate even higher to compensate for that.
>
> But does this matter for the end user?
There is only one speed that matters for an end user configuring cake.
You need to know the rate of the shaper defined in the CM config file
for your plan, or otherwise determine this rate. As I have said CMs
don't 'sync' in the way you are used to with xDSL so this terminology
isn't really correct, however in terms of being the significant
bottleneck immediately 'upstream' of cake the configured rate of this
shaper is equivalent to the 'sync' speed for xDSL.
Simply ignore the speed that your ISP advertises, this is a bogus
number for just about every DOCSIS ISP. Also ignore the speed reported
by speedtest.net unless you are confident that there is no other
activity on your link and that you know what TCP options are in use by
your operating system.
If you get a very consistent result on speedtest.net, it is possible
to estimate the rate of the CMTS's shaper. This is a relationship
between the TCP payload size and full Ethernet frame size. For
example, on an OS that uses TCP timestamps and you get a result of
100Mbps on speedtest.net the L2 rate may be 100 * (1518 / 1448) ~=
104.8Mbps.
By now you can see where 1518 comes from (1500 bytes Ethernet payload
(IP packet) + 14 bytes Ethernet header + 4 bytes Ethernet FCS). 1448
is (1500 bytes IP packet (Ethernet payload) - 20 bytes IPv4 header -
20 bytes TCP header - 12 bytes TCP timestamps option).
You should use Wireshark to confirm which TCP options and MTU your
system is using with the speedtest.net server and use these values
instead of my above example.
Once you have your estimate, round it down a couple of percent. If
your estimate is too high then it will be as if you don't have cake at
all. Too low and cake will still be effective, you will just sacrifice
a small amount of speed.
> In case of docsis does it make sense to account for 18 overhead?
Whatever the speed may be, you must always configure 'overhead 18 mpu
64' or the equivalent 'docsis' keyword to correctly compensate for the
Ethernet framing seen by the shaper in the CM/CMTS, otherwise cake may
underestimate the link utilisation when smaller packets are being sent
and its benefits will be defeated.
> The user will enter 50mbit and it will work. If the isp has provided a
> sligher higher syncrate.
This 50Mbps figure is suspiciously round, where is it actually from?
This is the advertised speed of your plan by the ISP?
> and the mpu setting. i don't know how cake handles this in detail.
> How the overhead gets added.
> lets i enter mpu 46.
> And cake we set 18 as overhead.
> Will this result in mpu 46 or 64?
> Can someone debug the code maybe please? :>
This has already been answered and the relevant snippet of code has
been posted. Overhead is first added (relative to IP), and packets
that are still below the MPU *after* the overhead is added will be
rounded up to the MPU of 64.
> I have the feeling with mpu 46 my pages lot a bit snappier. but could be
> placebo.
Correct, this is a placebo. You need to configure 'overhead 18 mpu 64'
or equivalently 'docsis', no ifs no buts. Focus your attention on
estimating the rate of your shaper. The framing compensation for
DOCSIS is a solved problem whereas the rate of a given link varies
from ISP to ISP, and from plan to plan.
>
> Thank you.
>
>
>
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-06 1:45 ` Ryan Mounce
@ 2017-09-06 7:26 ` Sebastian Moeller
2017-09-07 4:11 ` Dennis Fedtke
0 siblings, 1 reply; 24+ messages in thread
From: Sebastian Moeller @ 2017-09-06 7:26 UTC (permalink / raw)
To: Ryan Mounce; +Cc: Dennis Fedtke, Cake List
Hi Ryan,
I fully concur ;)
> On Sep 6, 2017, at 03:45, Ryan Mounce <ryan@mounce.com.au> wrote:
>
> On 6 September 2017 at 05:49, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
>> Hi!
>>
>> Thank you for all answers.
>> But for me this still makes no sense.
>> Assuming we have an ethnernet connection running over a docsis line.
>> docsis is able to transmit full 1500byte ethernet packets.
>> Lets say it is an 50 Mbit/s Line. (I dont know now how exactly docsis works)
>> So to reach the 50Mbit/s ethernet speed the docsis link rate needs to be
>> higher 50,6 Mbit/s (50*1518/1500 ??)
>> But when running a speedtest it will still not show the full speed. because
>> of other overhead from underlying protocols (tcp/ip for example)
>> So the ISP will set the sync rate even higher to compensate for that.
>>
>> But does this matter for the end user?
>
> There is only one speed that matters for an end user configuring cake.
> You need to know the rate of the shaper defined in the CM config file
> for your plan, or otherwise determine this rate. As I have said CMs
> don't 'sync' in the way you are used to with xDSL so this terminology
> isn't really correct, however in terms of being the significant
> bottleneck immediately 'upstream' of cake the configured rate of this
> shaper is equivalent to the 'sync' speed for xDSL.
>
> Simply ignore the speed that your ISP advertises, this is a bogus
> number for just about every DOCSIS ISP. Also ignore the speed reported
> by speedtest.net unless you are confident that there is no other
> activity on your link and that you know what TCP options are in use by
> your operating system.
>
> If you get a very consistent result on speedtest.net, it is possible
> to estimate the rate of the CMTS's shaper. This is a relationship
> between the TCP payload size and full Ethernet frame size. For
> example, on an OS that uses TCP timestamps and you get a result of
> 100Mbps on speedtest.net the L2 rate may be 100 * (1518 / 1448) ~=
> 104.8Mbps.
> By now you can see where 1518 comes from (1500 bytes Ethernet payload
> (IP packet) + 14 bytes Ethernet header + 4 bytes Ethernet FCS). 1448
> is (1500 bytes IP packet (Ethernet payload) - 20 bytes IPv4 header -
> 20 bytes TCP header - 12 bytes TCP timestamps option).
>
> You should use Wireshark to confirm which TCP options and MTU your
> system is using with the speedtest.net server and use these values
> instead of my above example.
Another quick and dirty method to measure the MTU is to use https://www.speedguide.net/analyzer.php.
> Once you have your estimate, round it down a couple of percent. If
> your estimate is too high then it will be as if you don't have cake at
> all. Too low and cake will still be effective, you will just sacrifice
> a small amount of speed.
>
>> In case of docsis does it make sense to account for 18 overhead?
>
> Whatever the speed may be, you must always configure 'overhead 18 mpu
> 64' or the equivalent 'docsis' keyword to correctly compensate for the
> Ethernet framing seen by the shaper in the CM/CMTS, otherwise cake may
> underestimate the link utilisation when smaller packets are being sent
> and its benefits will be defeated.
>
>> The user will enter 50mbit and it will work. If the isp has provided a
>> sligher higher syncrate.
>
> This 50Mbps figure is suspiciously round, where is it actually from?
> This is the advertised speed of your plan by the ISP?
>
>> and the mpu setting. i don't know how cake handles this in detail.
>> How the overhead gets added.
>> lets i enter mpu 46.
>> And cake we set 18 as overhead.
>> Will this result in mpu 46 or 64?
>> Can someone debug the code maybe please? :>
>
> This has already been answered and the relevant snippet of code has
> been posted. Overhead is first added (relative to IP), and packets
> that are still below the MPU *after* the overhead is added will be
> rounded up to the MPU of 64.
>
>> I have the feeling with mpu 46 my pages lot a bit snappier. but could be
>> placebo.
>
> Correct, this is a placebo. You need to configure 'overhead 18 mpu 64'
> or equivalently 'docsis', no ifs no buts. Focus your attention on
> estimating the rate of your shaper. The framing compensation for
> DOCSIS is a solved problem whereas the rate of a given link varies
> from ISP to ISP, and from plan to plan.
>
>>
>> Thank you.
>>
>>
>>
>>
>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-06 7:26 ` Sebastian Moeller
@ 2017-09-07 4:11 ` Dennis Fedtke
2017-09-07 4:18 ` Dennis Fedtke
2017-09-07 7:47 ` Sebastian Moeller
0 siblings, 2 replies; 24+ messages in thread
From: Dennis Fedtke @ 2017-09-07 4:11 UTC (permalink / raw)
To: Sebastian Moeller, Ryan Mounce; +Cc: Cake List
Hi everyone!
thanks for your answers again.
So what about from should i expect from using the docsis keyword?
(This layer cake script + nat docsis set in advanced config)
tc -s qdisc show dev eth1.100:
qdisc cake 8064: root refcnt 2 bandwidth 19Mbit diffserv3 triple-isolate
nat rtt 100.0ms raw mpu 64
Sent 246233450 bytes 1316430 pkt (dropped 65, overlimits 126706
requeues 0)
backlog 0b 0p requeues 0
memory used: 2372672b of 4Mb
capacity estimate: 19Mbit
Bulk Best Effort Voice
thresh 1187Kbit 19Mbit 4750Kbit
target 15.3ms 5.0ms 5.0ms
interval 110.3ms 100.0ms 10.0ms
pk_delay 0us 298us 10us
av_delay 0us 13us 2us
sp_delay 0us 1us 1us
pkts 0 1315692 803
bytes 0 246260830 59314
way_inds 0 733 0
way_miss 0 5012 29
way_cols 0 0 0
drops 0 65 0
marks 0 0 0
sp_flows 0 1 0
bk_flows 0 0 0
un_flows 0 0 0
max_len 0 1514 405
tc -s qdisc show dev ifb4eth1.100
qdisc cake 8065: root refcnt 2 bandwidth 150Mbit diffserv3
triple-isolate nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64
Sent 869520280 bytes 2153104 pkt (dropped 30, overlimits 270811
requeues 0)
backlog 0b 0p requeues 0
memory used: 624320b of 7500000b
capacity estimate: 150Mbit
Bulk Best Effort Voice
thresh 9375Kbit 150Mbit 37500Kbit
target 5.0ms 5.0ms 5.0ms
interval 100.0ms 100.0ms 10.0ms
pk_delay 121us 448us 1us
av_delay 40us 158us 0us
sp_delay 2us 2us 0us
pkts 58805 1362429 731900
bytes 77433974 748216627 43914060
way_inds 0 118167 0
way_miss 5 5911 3
way_cols 0 0 0
drops 1 29 0
marks 0 0 0
sp_flows 0 0 0
bk_flows 0 0 1
un_flows 0 0 0
max_len 1486 1514 90
Why is there no overhead showing for egress?
Why is there overhead18 + via-ethernet showing for ingress?
Thanks again =)
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-07 4:11 ` Dennis Fedtke
@ 2017-09-07 4:18 ` Dennis Fedtke
2017-09-07 4:25 ` Jonathan Morton
2017-09-07 7:47 ` Sebastian Moeller
1 sibling, 1 reply; 24+ messages in thread
From: Dennis Fedtke @ 2017-09-07 4:18 UTC (permalink / raw)
To: Cake List
Hi!
I forgot to ask why max_len shows 1514 and not 1518 in the output of tc?
Thanks.
Am 07.09.2017 um 06:11 schrieb Dennis Fedtke:
> Hi everyone!
>
> thanks for your answers again.
>
> So what about from should i expect from using the docsis keyword?
> (This layer cake script + nat docsis set in advanced config)
>
> tc -s qdisc show dev eth1.100:
> qdisc cake 8064: root refcnt 2 bandwidth 19Mbit diffserv3
> triple-isolate nat rtt 100.0ms raw mpu 64
> Sent 246233450 bytes 1316430 pkt (dropped 65, overlimits 126706
> requeues 0)
> backlog 0b 0p requeues 0
> memory used: 2372672b of 4Mb
> capacity estimate: 19Mbit
> Bulk Best Effort Voice
> thresh 1187Kbit 19Mbit 4750Kbit
> target 15.3ms 5.0ms 5.0ms
> interval 110.3ms 100.0ms 10.0ms
> pk_delay 0us 298us 10us
> av_delay 0us 13us 2us
> sp_delay 0us 1us 1us
> pkts 0 1315692 803
> bytes 0 246260830 59314
> way_inds 0 733 0
> way_miss 0 5012 29
> way_cols 0 0 0
> drops 0 65 0
> marks 0 0 0
> sp_flows 0 1 0
> bk_flows 0 0 0
> un_flows 0 0 0
> max_len 0 1514 405
>
> tc -s qdisc show dev ifb4eth1.100
> qdisc cake 8065: root refcnt 2 bandwidth 150Mbit diffserv3
> triple-isolate nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64
> Sent 869520280 bytes 2153104 pkt (dropped 30, overlimits 270811
> requeues 0)
> backlog 0b 0p requeues 0
> memory used: 624320b of 7500000b
> capacity estimate: 150Mbit
> Bulk Best Effort Voice
> thresh 9375Kbit 150Mbit 37500Kbit
> target 5.0ms 5.0ms 5.0ms
> interval 100.0ms 100.0ms 10.0ms
> pk_delay 121us 448us 1us
> av_delay 40us 158us 0us
> sp_delay 2us 2us 0us
> pkts 58805 1362429 731900
> bytes 77433974 748216627 43914060
> way_inds 0 118167 0
> way_miss 5 5911 3
> way_cols 0 0 0
> drops 1 29 0
> marks 0 0 0
> sp_flows 0 0 0
> bk_flows 0 0 1
> un_flows 0 0 0
> max_len 1486 1514 90
>
> Why is there no overhead showing for egress?
> Why is there overhead18 + via-ethernet showing for ingress?
>
> Thanks again =)
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-07 4:18 ` Dennis Fedtke
@ 2017-09-07 4:25 ` Jonathan Morton
2017-09-07 7:41 ` Dennis Fedtke
2017-09-07 7:58 ` Sebastian Moeller
0 siblings, 2 replies; 24+ messages in thread
From: Jonathan Morton @ 2017-09-07 4:25 UTC (permalink / raw)
To: Dennis Fedtke; +Cc: Cake List
[-- Attachment #1: Type: text/plain, Size: 195 bytes --]
Because 1514 is the normal maximum size of an Ethernet frame. This can
easily differ from the size actually used on the wire and from the size
used by the provisioning shaper.
- Jonathan Morton
[-- Attachment #2: Type: text/html, Size: 234 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-07 4:25 ` Jonathan Morton
@ 2017-09-07 7:41 ` Dennis Fedtke
2017-09-07 7:58 ` Sebastian Moeller
1 sibling, 0 replies; 24+ messages in thread
From: Dennis Fedtke @ 2017-09-07 7:41 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Cake List
Hi,
tc does not output the current overhead value when the value is the same
as the default one, correct?
When i enter overhead 18 for my eth1.100, tc doesnt output any overhead.
When i enter overhead 14 for my ifb4eth1.100, tc doesnt output any overhead.
So the default overhead for my eth.100 is 18.
And for my ifb4eth1.100 device it is 14.
Is this expected behavior?
Thanks.
Am 07.09.2017 um 06:25 schrieb Jonathan Morton:
>
> Because 1514 is the normal maximum size of an Ethernet frame. This can
> easily differ from the size actually used on the wire and from the
> size used by the provisioning shaper.
>
> - Jonathan Morton
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-07 4:25 ` Jonathan Morton
2017-09-07 7:41 ` Dennis Fedtke
@ 2017-09-07 7:58 ` Sebastian Moeller
1 sibling, 0 replies; 24+ messages in thread
From: Sebastian Moeller @ 2017-09-07 7:58 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Dennis Fedtke, Cake List
Hi Jonathan,
please let me elaborate below.
> On Sep 7, 2017, at 06:25, Jonathan Morton <chromatix99@gmail.com> wrote:
>
> Because 1514 is the normal maximum size of an Ethernet frame.
This might or might not be true (depending on the defnition of ethernet frame one uses), but it certainly is not overly helpful to novices, as e.g. https://en.wikipedia.org/wiki/Ethernet_frame will show the FCS as being part of the layer2 ethernet frame and 1514 does not contain the FCS nor any potential VLAN tag(s).
What is probably correct to say is that 1514 is the kernel's internal view of the size of an ethernet frame (most likely the FCS will be added by the NIC, so 1514 is the true size of the data the kernel shuttles around at that abstraction layer, but certainly not a real complete ethernet frame).
> This can easily differ from the size actually used on the wire
Not easily, but certainly with 1500 bytes payload an ethernet packet will take at least 1518 bytes on the L2-wire and effectively 1518+20 = 1538 bytes on L1 (here I account for the silent inter-frame gap by the 12 bytes worth of data that could be transmitted in the same time)
> and from the size used by the provisioning shaper.
And that is the thing that makes specifying overhead hard to automate, instead of the actually connected interface one needs to figure out the overhead (and bandwidth) applicable on the slowest part of the internet connection... (Which in theory is variable, but in practice it luckily is often the link to the ISP).
Best Regards
Sebastian
>
> - Jonathan Morton
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-07 4:11 ` Dennis Fedtke
2017-09-07 4:18 ` Dennis Fedtke
@ 2017-09-07 7:47 ` Sebastian Moeller
2017-09-07 7:58 ` Dennis Fedtke
1 sibling, 1 reply; 24+ messages in thread
From: Sebastian Moeller @ 2017-09-07 7:47 UTC (permalink / raw)
To: Dennis Fedtke; +Cc: Ryan Mounce, Cake List
Hi Dennis,
> On Sep 7, 2017, at 06:11, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
>
> Hi everyone!
>
> thanks for your answers again.
>
> So what about from should i expect from using the docsis keyword?
> (This layer cake script + nat docsis set in advanced config)
>
> tc -s qdisc show dev eth1.100:
> qdisc cake 8064: root refcnt 2 bandwidth 19Mbit diffserv3 triple-isolate nat rtt 100.0ms raw mpu 64
> Sent 246233450 bytes 1316430 pkt (dropped 65, overlimits 126706 requeues 0)
> backlog 0b 0p requeues 0
> memory used: 2372672b of 4Mb
> capacity estimate: 19Mbit
> Bulk Best Effort Voice
> thresh 1187Kbit 19Mbit 4750Kbit
> target 15.3ms 5.0ms 5.0ms
> interval 110.3ms 100.0ms 10.0ms
> pk_delay 0us 298us 10us
> av_delay 0us 13us 2us
> sp_delay 0us 1us 1us
> pkts 0 1315692 803
> bytes 0 246260830 59314
> way_inds 0 733 0
> way_miss 0 5012 29
> way_cols 0 0 0
> drops 0 65 0
> marks 0 0 0
> sp_flows 0 1 0
> bk_flows 0 0 0
> un_flows 0 0 0
> max_len 0 1514 405
>
> tc -s qdisc show dev ifb4eth1.100
> qdisc cake 8065: root refcnt 2 bandwidth 150Mbit diffserv3 triple-isolate nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64
> Sent 869520280 bytes 2153104 pkt (dropped 30, overlimits 270811 requeues 0)
> backlog 0b 0p requeues 0
> memory used: 624320b of 7500000b
> capacity estimate: 150Mbit
> Bulk Best Effort Voice
> thresh 9375Kbit 150Mbit 37500Kbit
> target 5.0ms 5.0ms 5.0ms
> interval 100.0ms 100.0ms 10.0ms
> pk_delay 121us 448us 1us
> av_delay 40us 158us 0us
> sp_delay 2us 2us 0us
> pkts 58805 1362429 731900
> bytes 77433974 748216627 43914060
> way_inds 0 118167 0
> way_miss 5 5911 3
> way_cols 0 0 0
> drops 1 29 0
> marks 0 0 0
> sp_flows 0 0 0
> bk_flows 0 0 1
> un_flows 0 0 0
> max_len 1486 1514 90
>
> Why is there no overhead showing for egress?
Could you post post the output of "cat /etc/config/sqm" (assuming you use LEDE/OpenWrt); i have a hunch that you might have forgotten to put the docsis keyword into both advanced configuration fields (or there is a typo n one of those words).
> Why is there overhead18 + via-ethernet showing for ingress?
This tells you that cake is using ethernet_payload + 18 bytes overhead, and that it automatically effectively subtracted the 14 bytes overhead the kernel added before applying its 18 bytes (so that you do not need to worry about how much the kernel added).
BTW, cake reports the maximum of the pkt_len field of all packets as max_len and on ethernet interfaces the kernel automatically adds 14 bytes (already to skb->len), this is why you see max_len 1514 instead of 1500. If you use tc's stab method for overhead accounting things will look different, as stab will add the full header size to pkt_len, so using tc stab you would probably see 1532 for an ecplicit overhead of 18 (as tc stab does not take the already accounted for header into account so if you request 18 on an ethernet interface you will effectively get 18+14 = 32.) It really is a great feature of cake to undo the kernel's automagic length adjustment so that one can request overhead on top of the ethernet payload/the IP packets.
Best Regards
Sebastian
>
> Thanks again =)
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-07 7:47 ` Sebastian Moeller
@ 2017-09-07 7:58 ` Dennis Fedtke
2017-09-07 8:07 ` Sebastian Moeller
0 siblings, 1 reply; 24+ messages in thread
From: Dennis Fedtke @ 2017-09-07 7:58 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: Cake List
Hi Sebastian,
here is the outout:
config queue
option debug_logging '0'
option interface 'eth1.100'
option qdisc 'cake'
option enabled '1'
option qdisc_advanced '1'
option qdisc_really_really_advanced '1'
option linklayer 'none'
option egress_ecn 'NOECN'
option upload '19000'
option verbosity '5'
option ingress_ecn 'ECN'
option script 'layer_cake.qos'
option download '150000'
option squash_ingress '1'
option squash_dscp '1'
option iqdisc_opts 'docsis nat'
option eqdisc_opts 'docsis nat'
But also see my last email please.
it seems that both devices use different default overhead (14 vs 18)
Thanks.
Am 07.09.2017 um 09:47 schrieb Sebastian Moeller:
> Hi Dennis,
>
>> On Sep 7, 2017, at 06:11, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
>>
>> Hi everyone!
>>
>> thanks for your answers again.
>>
>> So what about from should i expect from using the docsis keyword?
>> (This layer cake script + nat docsis set in advanced config)
>>
>> tc -s qdisc show dev eth1.100:
>> qdisc cake 8064: root refcnt 2 bandwidth 19Mbit diffserv3 triple-isolate nat rtt 100.0ms raw mpu 64
>> Sent 246233450 bytes 1316430 pkt (dropped 65, overlimits 126706 requeues 0)
>> backlog 0b 0p requeues 0
>> memory used: 2372672b of 4Mb
>> capacity estimate: 19Mbit
>> Bulk Best Effort Voice
>> thresh 1187Kbit 19Mbit 4750Kbit
>> target 15.3ms 5.0ms 5.0ms
>> interval 110.3ms 100.0ms 10.0ms
>> pk_delay 0us 298us 10us
>> av_delay 0us 13us 2us
>> sp_delay 0us 1us 1us
>> pkts 0 1315692 803
>> bytes 0 246260830 59314
>> way_inds 0 733 0
>> way_miss 0 5012 29
>> way_cols 0 0 0
>> drops 0 65 0
>> marks 0 0 0
>> sp_flows 0 1 0
>> bk_flows 0 0 0
>> un_flows 0 0 0
>> max_len 0 1514 405
>>
>> tc -s qdisc show dev ifb4eth1.100
>> qdisc cake 8065: root refcnt 2 bandwidth 150Mbit diffserv3 triple-isolate nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64
>> Sent 869520280 bytes 2153104 pkt (dropped 30, overlimits 270811 requeues 0)
>> backlog 0b 0p requeues 0
>> memory used: 624320b of 7500000b
>> capacity estimate: 150Mbit
>> Bulk Best Effort Voice
>> thresh 9375Kbit 150Mbit 37500Kbit
>> target 5.0ms 5.0ms 5.0ms
>> interval 100.0ms 100.0ms 10.0ms
>> pk_delay 121us 448us 1us
>> av_delay 40us 158us 0us
>> sp_delay 2us 2us 0us
>> pkts 58805 1362429 731900
>> bytes 77433974 748216627 43914060
>> way_inds 0 118167 0
>> way_miss 5 5911 3
>> way_cols 0 0 0
>> drops 1 29 0
>> marks 0 0 0
>> sp_flows 0 0 0
>> bk_flows 0 0 1
>> un_flows 0 0 0
>> max_len 1486 1514 90
>>
>> Why is there no overhead showing for egress?
> Could you post post the output of "cat /etc/config/sqm" (assuming you use LEDE/OpenWrt); i have a hunch that you might have forgotten to put the docsis keyword into both advanced configuration fields (or there is a typo n one of those words).
>
>
>> Why is there overhead18 + via-ethernet showing for ingress?
> This tells you that cake is using ethernet_payload + 18 bytes overhead, and that it automatically effectively subtracted the 14 bytes overhead the kernel added before applying its 18 bytes (so that you do not need to worry about how much the kernel added).
>
> BTW, cake reports the maximum of the pkt_len field of all packets as max_len and on ethernet interfaces the kernel automatically adds 14 bytes (already to skb->len), this is why you see max_len 1514 instead of 1500. If you use tc's stab method for overhead accounting things will look different, as stab will add the full header size to pkt_len, so using tc stab you would probably see 1532 for an ecplicit overhead of 18 (as tc stab does not take the already accounted for header into account so if you request 18 on an ethernet interface you will effectively get 18+14 = 32.) It really is a great feature of cake to undo the kernel's automagic length adjustment so that one can request overhead on top of the ethernet payload/the IP packets.
>
> Best Regards
> Sebastian
>
>> Thanks again =)
>>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-07 7:58 ` Dennis Fedtke
@ 2017-09-07 8:07 ` Sebastian Moeller
0 siblings, 0 replies; 24+ messages in thread
From: Sebastian Moeller @ 2017-09-07 8:07 UTC (permalink / raw)
To: Dennis Fedtke; +Cc: Cake List
Hi Dennis,
> On Sep 7, 2017, at 09:58, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
>
> Hi Sebastian,
>
> here is the outout:
>
> config queue
> option debug_logging '0'
> option interface 'eth1.100'
> option qdisc 'cake'
> option enabled '1'
> option qdisc_advanced '1'
> option qdisc_really_really_advanced '1'
> option linklayer 'none'
> option egress_ecn 'NOECN'
> option upload '19000'
> option verbosity '5'
> option ingress_ecn 'ECN'
> option script 'layer_cake.qos'
> option download '150000'
> option squash_ingress '1'
> option squash_dscp '1'
> option iqdisc_opts 'docsis nat'
> option eqdisc_opts 'docsis nat'
Ah, there goes my typo or forgotten keyword hypothesis...
I bet the reason is that on the IFB the vlan tag might not be visible, while on egress it is; unexpectedly cake will not report the overhead if the specified overhead equals the hard_header_len (@Jonathan, what is the rationale for the behavior? could you live with reporting the hard_header_len unconditionally somewhere in the "tc -s qdisc" output?). Since with a vlan tag the kernel accounts for 18 bytes (a vlan tag takes 4 bytes and 14+4=18) cake simply omits this during reporting. Now why the kernel deos not seem to see/account for the VLAN tag on ingress I do not know. But this certainly shows that cake does the right thing by automatically adjusting for hard_header_len... this certainly is more complicated than it should be...
>
> But also see my last email please.
>
> it seems that both devices use different default overhead (14 vs 18)
I agree, but I have no time to dive into the kernel sources to figure out why...
Best Regards
Sebastian
>
> Thanks.
>
>
> Am 07.09.2017 um 09:47 schrieb Sebastian Moeller:
>> Hi Dennis,
>>
>>> On Sep 7, 2017, at 06:11, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
>>>
>>> Hi everyone!
>>>
>>> thanks for your answers again.
>>>
>>> So what about from should i expect from using the docsis keyword?
>>> (This layer cake script + nat docsis set in advanced config)
>>>
>>> tc -s qdisc show dev eth1.100:
>>> qdisc cake 8064: root refcnt 2 bandwidth 19Mbit diffserv3 triple-isolate nat rtt 100.0ms raw mpu 64
>>> Sent 246233450 bytes 1316430 pkt (dropped 65, overlimits 126706 requeues 0)
>>> backlog 0b 0p requeues 0
>>> memory used: 2372672b of 4Mb
>>> capacity estimate: 19Mbit
>>> Bulk Best Effort Voice
>>> thresh 1187Kbit 19Mbit 4750Kbit
>>> target 15.3ms 5.0ms 5.0ms
>>> interval 110.3ms 100.0ms 10.0ms
>>> pk_delay 0us 298us 10us
>>> av_delay 0us 13us 2us
>>> sp_delay 0us 1us 1us
>>> pkts 0 1315692 803
>>> bytes 0 246260830 59314
>>> way_inds 0 733 0
>>> way_miss 0 5012 29
>>> way_cols 0 0 0
>>> drops 0 65 0
>>> marks 0 0 0
>>> sp_flows 0 1 0
>>> bk_flows 0 0 0
>>> un_flows 0 0 0
>>> max_len 0 1514 405
>>>
>>> tc -s qdisc show dev ifb4eth1.100
>>> qdisc cake 8065: root refcnt 2 bandwidth 150Mbit diffserv3 triple-isolate nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64
>>> Sent 869520280 bytes 2153104 pkt (dropped 30, overlimits 270811 requeues 0)
>>> backlog 0b 0p requeues 0
>>> memory used: 624320b of 7500000b
>>> capacity estimate: 150Mbit
>>> Bulk Best Effort Voice
>>> thresh 9375Kbit 150Mbit 37500Kbit
>>> target 5.0ms 5.0ms 5.0ms
>>> interval 100.0ms 100.0ms 10.0ms
>>> pk_delay 121us 448us 1us
>>> av_delay 40us 158us 0us
>>> sp_delay 2us 2us 0us
>>> pkts 58805 1362429 731900
>>> bytes 77433974 748216627 43914060
>>> way_inds 0 118167 0
>>> way_miss 5 5911 3
>>> way_cols 0 0 0
>>> drops 1 29 0
>>> marks 0 0 0
>>> sp_flows 0 0 0
>>> bk_flows 0 0 1
>>> un_flows 0 0 0
>>> max_len 1486 1514 90
>>>
>>> Why is there no overhead showing for egress?
>> Could you post post the output of "cat /etc/config/sqm" (assuming you use LEDE/OpenWrt); i have a hunch that you might have forgotten to put the docsis keyword into both advanced configuration fields (or there is a typo n one of those words).
>>
>>
>>> Why is there overhead18 + via-ethernet showing for ingress?
>> This tells you that cake is using ethernet_payload + 18 bytes overhead, and that it automatically effectively subtracted the 14 bytes overhead the kernel added before applying its 18 bytes (so that you do not need to worry about how much the kernel added).
>>
>> BTW, cake reports the maximum of the pkt_len field of all packets as max_len and on ethernet interfaces the kernel automatically adds 14 bytes (already to skb->len), this is why you see max_len 1514 instead of 1500. If you use tc's stab method for overhead accounting things will look different, as stab will add the full header size to pkt_len, so using tc stab you would probably see 1532 for an ecplicit overhead of 18 (as tc stab does not take the already accounted for header into account so if you request 18 on an ethernet interface you will effectively get 18+14 = 32.) It really is a great feature of cake to undo the kernel's automagic length adjustment so that one can request overhead on top of the ethernet payload/the IP packets.
>>
>> Best Regards
>> Sebastian
>>
>>> Thanks again =)
>>>
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-05 20:19 ` Dennis Fedtke
2017-09-06 1:45 ` Ryan Mounce
@ 2017-09-06 7:22 ` Sebastian Moeller
1 sibling, 0 replies; 24+ messages in thread
From: Sebastian Moeller @ 2017-09-06 7:22 UTC (permalink / raw)
To: Dennis Fedtke; +Cc: Jonathan Morton, Cake List
Hi Dennis,
> On Sep 5, 2017, at 22:19, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
>
> Hi!
>
> Thank you for all answers.
> But for me this still makes no sense.
> Assuming we have an ethnernet connection running over a docsis line.
> docsis is able to transmit full 1500byte ethernet packets.
> Lets say it is an 50 Mbit/s Line. (I dont know now how exactly docsis works)
> So to reach the 50Mbit/s ethernet speed the docsis link rate needs to be higher 50,6 Mbit/s (50*1518/1500 ??)
Exactly, but that is completely hidden from the end-user as DOCSIS per standard offers speedlimits assuming 18 Bytes Ethernet overhead. Now, unlike DSL-ISPs that often sell close to the physical maximal capacity of a link, docsis being a shared medium with segment bandwidth >> single link bandwidth DOCSIS-ISPs get away with this (even though it raises fairness questions under congesting conditions). And as recommended by CISCO (and other CMTS manufacturers?) DOCSIS-ISPs tend to over provision the links so that in the typical case the measurable TCP/IPv4/HTTP goodput is equal or larger than the contracted bandwidth. That has the issue that it misleads typical customers into thinking that the ISP actually guarantees goodput. Why do I say mislead? Well if you look at the gross bandwidth (and let's restrict ourselves to the ethernet overhead, diving into the DOCSIS details will only complicate matters) and the resulting goodput for different goodputs, or better let's look at the required gross bandwidth for 100Mbps for 1500byte packets and 64byte packets:
100 * ((1500+18)/(1500-20-20)) = 103.97
100 * ((46+18)/(46-20-20)) = 1066.67
And the resulting percentual bandwidth cost of the per packet overhead (at equal goodput):
100-100*((1500+18)/(1500-20-20)) = -3.97 %
100-100*((46+18)/(46-20-20)) = -966.67 %
You quickly realize that there is a factor of 10 between those. Now without congestion a DOCSIS ISP might still grant that, but under congestion conditions it seems clearly suboptimal to split the gross bandwidth based on goodput fairness instead of on gross-bandwidth share basis... For a DOCSIS system the numbers still work, since the segment gross bandwidth seems sufficiently hight to allow for 64 byte packets, but for XDSL links where the gross bandwidth << 10 * the contracted bandwidth it is obvious that guaranteeing goodput is a foo's errand.
> But when running a speedtest it will still not show the full speed. because of other overhead from underlying protocols (tcp/ip for example)
Yes, there is a number of protocols that might or might not be taken into account, like VLAN tags(s), PPPoE, TCP, IPv4/IPv6, TCP-options, IPv6 extension headers, the HTTP header typical for browser based speedtests. Unfortunately not all of those actually are visible for the endpoints of a speedtest; you really need to get the effective overhead for the bottleneck link.
> So the ISP will set the sync rate even higher to compensate for that.
Maybe, maybe not, some ISPs do overprovisioning some do not. Personally, while I like more for the same price, I am a bit concened if an ISP seems incapable to correctly inform about the to be expected bandwidth...
>
> But does this matter for the end user?
As the 1500/64 numbers above hopefully show, it does since the per packet overhead still is data that needs to be transported and will "eat" into the available gross bandwidth pool.
> In case of docsis does it make sense to account for 18 overhead?
Empirically it does seem so. But please note that under speedtest conditions (or under any condition where the packet size on the loaded link direction stays constant) not accounting for the per-packet overhead can be compensated by setting the shaper bandwidth slightly lower ad so the effect of missing overhead can easily go unnoticed, it will however cause trouble once a new traffic pattern uses more smaller packets...
> The user will enter 50mbit and it will work. If the isp has provided a sligher higher syncrate.
As I hope I made clear above this will only work for fixed conditions in which the distribution of packet sizes on the link does not change.
>
> and the mpu setting. i don't know how cake handles this in detail.
> How the overhead gets added.
> lets i enter mpu 46.
> And cake we set 18 as overhead.
> Will this result in mpu 46 or 64?
This will result in MPU 46 (as that is what you requested), so an ICMP echo request packet with size 18+20+8 = 46 will be accounted as 46 bytes and not like the 64 it really takes up on the link. As shown, cake will first add the overhead and only compare the packet_len+overhead to mpu and do max(mpu, (paket_len+overhead)).
> Can someone debug the code maybe please? :>
> I have the feeling with mpu 46 my pages lot a bit snappier. but could be placebo.
Under most conditions this difference will not matter at all, since typically packets+overhead are larger than 64 byte, so how did you measure to come to that feeling?
Best Regards
Sebastian
>
> Thank you.
>
>
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-05 6:00 ` Dennis Fedtke
2017-09-05 6:49 ` Ryan Mounce
2017-09-05 6:58 ` Jonathan Morton
@ 2017-09-05 8:01 ` Sebastian Moeller
2017-09-05 8:35 ` Kevin Darbyshire-Bryant
2 siblings, 1 reply; 24+ messages in thread
From: Sebastian Moeller @ 2017-09-05 8:01 UTC (permalink / raw)
To: Dennis Fedtke; +Cc: cake
Hi Dennis,
> On Sep 5, 2017, at 08:00, Dennis Fedtke <dennisfedtke@gmail.com> wrote:
>
> Hi Ryan,
>
> Thanks for you answers.
> Lets assume again ethernet over docsis connection at 50 Mbit/s.
As the other's already mentioned, on DOCSIS it is customary to lie to the customer regarding the provisioned bandwidth; fortunately the "lie" is actually in favor of the customer, so nobody needs to feel bad about this ;)
> So to get 50 Mbit/s ethernet perfomance, the docsis link speed needs to be set at a higher link speed to compensate for the 18 header ethernet overhead, or?
> or
> Assuming:
> The docsis link syncs at exactly 50Mbit/s. When cake is set to exactly 50Mbit/s, it is actually set at a higher speed then the link actually is.
On DOCSIS networks according to Greg White of cablelabs:
For downstream packets, the DOCSIS MAC header consists of the standard 6-byte MAC Header, plus 6 bytes for the DS EHDR and 5 bytes for the Downstream Privacy EH Element. So, 17+14+4=35 bytes of per-packet overhead for a downstream DOCSIS MAC Frame.
For upstream packets, there will typically be only the 6-byte MAC Header, plus the 4 byte Upstream Privacy EH version 2 Element, so 10+14+4=28 bytes of per-packet overhead for an Upstream DOCSIS MAC Frame.
The 14+4 denotes the 14 byte ethernet header (6 byte dest mac, 6 byte src mac, 2 byte ether type) plus the 4 byte ethernet frame check sequence FCS.
Interestingly, the above DICSIS observations seem irrelevant for end customers as the shaper used in DOCSIS systems that limits a users maximal bandwidth does completely ignore DOCSIS overhead and only includes ethernet frames including their frame check sequence (FCS 4 Byte). Tocite the relevant section from the Docsis standard (http://www.cablelabs.com/specification/docsis-3-0-mac-and-upper-layer-protocols-interface-specification/):
"C.2.2.7.2 Maximum Sustained Traffic Rate 632 This parameter is the rate parameter R of a token-bucket-based rate limit for packets. R is expressed in bits per second, and MUST take into account all MAC frame data PDU of the Service Flow from the byte following the MAC header HCS to the end of the CRC, including every PDU in the case of a Concatenated MAC Frame. This parameter is applied after Payload Header Suppression; it does not include the bytes suppressed for PHS. The number of bytes forwarded (in bytes) is limited during any time interval T by Max(T), as described in the expression: Max(T) = T * (R / 8) + B, (1) where the parameter B (in bytes) is the Maximum Traffic Burst Configuration Setting (refer to Annex C.2.2.7.3). NOTE: This parameter does not limit the instantaneous rate of the Service Flow. The specific algorithm for enforcing this parameter is not mandated here. Any implementation which satisfies the above equation is conformant. In particular, the granularity of enforcement and the minimum implemented value of this parameter are vendor specific. The CMTS SHOULD support a granularity of at most 100 kbps. The CM SHOULD support a granularity of at most 100 kbps. NOTE: If this parameter is omitted or set to zero, then there is no explicitly-enforced traffic rate maximum. This field specifies only a bound, not a guarantee that this rate is available."
So in essence DOCSIS users need to only account for 18 Bytes of ethernet overhead in both ingress and egress directions under non-congested conditions.
>
> For mpu why 64?
> I assume minimum 46bytes ethernet payload + 18bytes header?
Well your DOCSIS modem will simply grab the ethernet packets including the FCS and those are simply guranteed to be >= 64 bytes. The linux kernel however might see smaller packets if the payload is smaller (but note for TCP/IPv4 with its 40 bytes of header the difference is quite small anyway). Interestingly if you use a single VLAN tag the ethernet MPU still stays at 64 bytes, only the payload section (including padding) shrinks to 42 bytes...
>
> But why does cake use hard_header_len for the header size which is 14 bytes?
I believe this is used internally so cake can deduce the size of the automatically added overhead (the linux kernel will add 14 bytes on ethernet interfaces automatically, which while certainly justifiable are not the ideal value for an ethernet shaper); with this cake allows the user to specify absolute overhead (so if you specify overhead 18, cake will give you 14 - 14 + 18 = 18, while tc's stab method will give you 14 + 18 = 32). So cake is doing the right thing here (the only thing it could do even better is to report the size of the automatically corrected in-kernel hard_header_len, but then the audience for that information is probably to small to justify this feature).
>
> I don't know how packet sheduler system works, but maybe it is not needed to include the ethernet header.
All you really need to give cake is a precise estimate of the bandwidth and the effective on-the-wire-size of the data packets; for DOCSIS this means overhead of 18 bytes*
>
> cake does work on ip pakets or? so this is layer3 i think.
> the hole thing with ethernet headers is happening in layer2.
> This would change the minimum packet size to 46 or?
From sch_cake.c:
static inline u32 cake_overhead(struct cake_sched_data *q, u32 in)
{
u32 out = in + q->rate_overhead;
if (q->rate_mpu && out < q->rate_mpu) {
out = q->rate_mpu;
}
if (q->rate_flags & CAKE_FLAG_ATM) {
out += 47;
out /= 48;
out *= 53;
} else if(q->rate_flags & CAKE_FLAG_PTM) {
// the following adds one byte per 64 bytes or part thereof
// this is conservative and easier to calculate than the precise value
out += (out / 64) + !!(out % 64);
}
return out;
}
As you can see cake first adds the overhead and then makes sure the packet size is >= MPU... This will do the right thing for DOCSIS, even though it is conceivable that there are encapsulations out that can not be fully described with this (think full ethernet frames with additional overhead, where size+full_overhead might be >= mpu but the correct method would calculate size+ethernet_overhead >= mpu and then add outer_layer_overhead on top of this, but this is largely academic until something like this gets rolled-out in large numbers...)
Best Regards
Sebastian
P.S.: unlike Ryan and Jonathan I would always use "overhead 18 mpu 64" instead of "docsis" because I prefer explicitness over terseness, but that is a matter of taste and "tc -s qdisc" will helpfully expand docsis into "overhead 18 mpu 64" (which is great).
>
> Thanks.
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
*) The 18 bytes DOCSIS overhead are actually a lie, the true per-packet overhead on docsis is far greater. Without congestion all this additional overhead is simply "payed" for the user out of the shared gross bandwidth and everybody is happy. But under congestion this, IMHO, can lead to massive unfairness in that a user's shate of the gross bandwidth will correlate (bandwidth being equal) with the inversre of the packet size; put differently small packets use relatively more of the gross bandwidth than large packets, and I have not found out how docss gurantees fairness to all the CPE in a segment, but I digress.
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [Cake] overhead and mpu
2017-09-05 8:01 ` Sebastian Moeller
@ 2017-09-05 8:35 ` Kevin Darbyshire-Bryant
2017-09-05 8:58 ` Sebastian Moeller
0 siblings, 1 reply; 24+ messages in thread
From: Kevin Darbyshire-Bryant @ 2017-09-05 8:35 UTC (permalink / raw)
To: cake
On 05/09/17 09:01, Sebastian Moeller wrote:
<snip>
> I believe this is used internally so cake can deduce the size of the automatically added overhead (the linux kernel will add 14 bytes on ethernet interfaces automatically, which while certainly justifiable are not the ideal value for an ethernet shaper); with this cake allows the user to specify absolute overhead (so if you specify overhead 18, cake will give you 14 - 14 + 18 = 18, while tc's stab method will give you 14 + 18 = 32). So cake is doing the right thing here (the only thing it could do even better is to report the size of the automatically corrected in-kernel hard_header_len, but then the audience for that information is probably to small to justify this feature).
Actually for the sake of a one line change in the tc code it is an
extremely useful confidence boost and info source as to just what linux
thinks the overhead actually is:
// This is actually the *amount* of automatic
compensation, but we only report
// its presence as a boolean for now.
if (ethernet)
- fprintf(f, "via-ethernet ");
+ fprintf(f, "via-ethernet(%d) ", ethernet);
I've run that in a local patch to tc on LEDE for AGES. Very useful.
Kevin
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [Cake] overhead and mpu
2017-09-05 8:35 ` Kevin Darbyshire-Bryant
@ 2017-09-05 8:58 ` Sebastian Moeller
2017-09-05 14:37 ` Ryan Mounce
0 siblings, 1 reply; 24+ messages in thread
From: Sebastian Moeller @ 2017-09-05 8:58 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: cake
Hi Kevin,
> On Sep 5, 2017, at 10:35, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>
>
>
> On 05/09/17 09:01, Sebastian Moeller wrote:
> <snip>
>> I believe this is used internally so cake can deduce the size of the automatically added overhead (the linux kernel will add 14 bytes on ethernet interfaces automatically, which while certainly justifiable are not the ideal value for an ethernet shaper); with this cake allows the user to specify absolute overhead (so if you specify overhead 18, cake will give you 14 - 14 + 18 = 18, while tc's stab method will give you 14 + 18 = 32). So cake is doing the right thing here (the only thing it could do even better is to report the size of the automatically corrected in-kernel hard_header_len, but then the audience for that information is probably to small to justify this feature).
>
>
> Actually for the sake of a one line change in the tc code it is an extremely useful confidence boost and info source as to just what linux thinks the overhead actually is:
>
> // This is actually the *amount* of automatic compensation, but we only report
> // its presence as a boolean for now.
> if (ethernet)
> - fprintf(f, "via-ethernet ");
> + fprintf(f, "via-ethernet(%d) ", ethernet);
>
> I've run that in a local patch to tc on LEDE for AGES. Very useful.
That sounds like it would be a great addition, since the information is already collected... For what it is worth I would vote for adding this... (to the version that should be upstreamed ;) )
Best Regards
Sebastian
>
> Kevin
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-05 8:58 ` Sebastian Moeller
@ 2017-09-05 14:37 ` Ryan Mounce
2017-09-05 15:06 ` Sebastian Moeller
2017-09-05 15:33 ` Kevin Darbyshire-Bryant
0 siblings, 2 replies; 24+ messages in thread
From: Ryan Mounce @ 2017-09-05 14:37 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: Kevin Darbyshire-Bryant, cake
On 5 September 2017 at 18:28, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Kevin,
>
>
>> On Sep 5, 2017, at 10:35, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>>
>>
>>
>> On 05/09/17 09:01, Sebastian Moeller wrote:
>> <snip>
>>> I believe this is used internally so cake can deduce the size of the automatically added overhead (the linux kernel will add 14 bytes on ethernet interfaces automatically, which while certainly justifiable are not the ideal value for an ethernet shaper); with this cake allows the user to specify absolute overhead (so if you specify overhead 18, cake will give you 14 - 14 + 18 = 18, while tc's stab method will give you 14 + 18 = 32). So cake is doing the right thing here (the only thing it could do even better is to report the size of the automatically corrected in-kernel hard_header_len, but then the audience for that information is probably to small to justify this feature).
>>
>>
>> Actually for the sake of a one line change in the tc code it is an extremely useful confidence boost and info source as to just what linux thinks the overhead actually is:
>>
>> // This is actually the *amount* of automatic compensation, but we only report
>> // its presence as a boolean for now.
>> if (ethernet)
>> - fprintf(f, "via-ethernet ");
>> + fprintf(f, "via-ethernet(%d) ", ethernet);
>>
>> I've run that in a local patch to tc on LEDE for AGES. Very useful.
>
> That sounds like it would be a great addition, since the information is already collected... For what it is worth I would vote for adding this... (to the version that should be upstreamed ;) )
+1
It would also be nice for tc to know whether the overhead has been
explicitly configured and report appropriately. In my case I use cake
on a VLAN sub-interface that happens to have a hard_header_len of 18
after the 802.1q tag, and then use the docsis keyword to more
explicitly configure the overhead in case I move to an untagged
interface in the future. tc -s qdisc can't tell the difference and
reports 'raw mpu 64' rather than 'overhead 18 mpu 64'.
>
> Best Regards
> Sebastian
>
>
>>
>> Kevin
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Cake] overhead and mpu
2017-09-05 14:37 ` Ryan Mounce
@ 2017-09-05 15:06 ` Sebastian Moeller
2017-09-05 15:33 ` Kevin Darbyshire-Bryant
1 sibling, 0 replies; 24+ messages in thread
From: Sebastian Moeller @ 2017-09-05 15:06 UTC (permalink / raw)
To: Ryan Mounce; +Cc: Kevin Darbyshire-Bryant, cake
Hi Ryan,
> On Sep 5, 2017, at 16:37, Ryan Mounce <ryan@mounce.com.au> wrote:
>
> On 5 September 2017 at 18:28, Sebastian Moeller <moeller0@gmx.de> wrote:
>> Hi Kevin,
>>
>>
>>> On Sep 5, 2017, at 10:35, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>>>
>>>
>>>
>>> On 05/09/17 09:01, Sebastian Moeller wrote:
>>> <snip>
>>>> I believe this is used internally so cake can deduce the size of the automatically added overhead (the linux kernel will add 14 bytes on ethernet interfaces automatically, which while certainly justifiable are not the ideal value for an ethernet shaper); with this cake allows the user to specify absolute overhead (so if you specify overhead 18, cake will give you 14 - 14 + 18 = 18, while tc's stab method will give you 14 + 18 = 32). So cake is doing the right thing here (the only thing it could do even better is to report the size of the automatically corrected in-kernel hard_header_len, but then the audience for that information is probably to small to justify this feature).
>>>
>>>
>>> Actually for the sake of a one line change in the tc code it is an extremely useful confidence boost and info source as to just what linux thinks the overhead actually is:
>>>
>>> // This is actually the *amount* of automatic compensation, but we only report
>>> // its presence as a boolean for now.
>>> if (ethernet)
>>> - fprintf(f, "via-ethernet ");
>>> + fprintf(f, "via-ethernet(%d) ", ethernet);
>>>
>>> I've run that in a local patch to tc on LEDE for AGES. Very useful.
>>
>> That sounds like it would be a great addition, since the information is already collected... For what it is worth I would vote for adding this... (to the version that should be upstreamed ;) )
>
> +1
>
> It would also be nice for tc to know whether the overhead has been
> explicitly configured and report appropriately. In my case I use cake
> on a VLAN sub-interface that happens to have a hard_header_len of 18
> after the 802.1q tag, and then use the docsis keyword to more
> explicitly configure the overhead in case I move to an untagged
> interface in the future. tc -s qdisc can't tell the difference and
> reports 'raw mpu 64' rather than 'overhead 18 mpu 64'.
Ah, I see, when ever the explicitly requested overhead equals hard_header_len, cake does not include it in its output. I believe this would be nice to change to always explicitly report the overhead (so even default to hard_header_len if no overhead was specified otherwise).
Since the keywords like docsis are really convenience wrappers (hace a look in tc's q_cake.c) I vote for always expanding to the explicit overhead and potentially MPU values instead of reporting the keyword used.
I believe manually specifying "overhead 18 mpu 64" will be just as future proof as the docsis keyword in regards to changing hard_header_len.
It has to, in the end the docsis keyword gets interpreted by tc as:
/*
* DOCSIS cable shapers account for Ethernet frame with FCS,
* but not interframe gap nor preamble.
*/
} else if (strcmp(*argv, "docsis") == 0) {
atm = 0;
overhead += 18;
overhead_set = true;
mpu = 64;
Best Regards
Sebastian
>
>>
>> Best Regards
>> Sebastian
>>
>>
>>>
>>> Kevin
>>> _______________________________________________
>>> Cake mailing list
>>> Cake@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cake
>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [Cake] overhead and mpu
2017-09-05 14:37 ` Ryan Mounce
2017-09-05 15:06 ` Sebastian Moeller
@ 2017-09-05 15:33 ` Kevin Darbyshire-Bryant
1 sibling, 0 replies; 24+ messages in thread
From: Kevin Darbyshire-Bryant @ 2017-09-05 15:33 UTC (permalink / raw)
To: cake
On 05/09/17 15:37, Ryan Mounce wrote:
<snip>
>
> +1
>
> It would also be nice for tc to know whether the overhead has been
> explicitly configured and report appropriately. In my case I use cake
> on a VLAN sub-interface that happens to have a hard_header_len of 18
> after the 802.1q tag, and then use the docsis keyword to more
> explicitly configure the overhead in case I move to an untagged
> interface in the future. tc -s qdisc can't tell the difference and
> reports 'raw mpu 64' rather than 'overhead 18 mpu 64'.
The almost reality is that you could do whatever you want with the LEDE
patch that adds cake awareness to iproute2's tc. The challenge comes in
keeping it up to date with whatever lands in 'cake master' and 'cake
aware tc master'
I say almost reality because some on the LEDE committers list question
things (rightly) if you're not pointing at master. For these reasons
(and others) I've been reluctant to point LEDE cake at the cobalt branch
so the 'in test' features of cobalt (and matching tc) are not being
exercised there (ingress mode, "Voice tin traffic sometimes delayed by
more than one bulk packet in best effort" 'solution' except it doesn't
appear to have solved it)
Either the cobalt branch features need to get into master (and someone
update the LEDE cake & iproute2 packages) OR someone needs to take the
step to point LEDE cake at cobalt (and update the packages etc etc)
Cake on LEDE is currently without a maintainer and the cake list & repo
has been incredibly quiet for the past 4+ months.
^ permalink raw reply [flat|nested] 24+ messages in thread