Re: [Cake] Help untangling CAKE settings for FTTH

Re: [Cake] Help untangling CAKE settings for FTTH

[Sebastian Moeller]

HI T.



> On Nov 16, 2022, at 11:22, Thibaut <hacks@slashdirt.org> wrote:
> 
> Hi,
> 
> A few questions for the CAKE experts here:

	Quick note, this is not necessarily the best place to get advice on cake, both the cake mailing list and the OpenWrt forum tend to be directer paths to the "bakers".


> 
> I’m trying to untangle the information available in the openwrt wiki:
> https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm
> https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details
> and for the latter especially the part here: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details#sqmlink_layer_adaptation_tab
> 
> For ADSL/VDSL, I believe the instructions are clear and the Luci interface is too. However for ethernet/fiber, I’m confused:
> 
> In the first list of this paragraph it is first suggested to use « Ethernet with overhead » and set the per-packet overhead to 44 for FFTH (which seems like a large value for this use case),

That is the point here, 44 is a value that in all likelihood is >= any realistic true overhead. Gently over-estimating the true overhead has a relative small cost in potential throughput, underestimating the overhead however can result in noticeable degradation of responsiveness under working conditions, so the recommendation is to rather err on the side of too large an overhead and not too small an overhead.
Why not simply recommend the worst case scenario `overhead 44 atm` to be on the safe side on all links? Because the ATM/AAL5 encapsulation is only used on ADSL links (so is getting rarer and rarer and the ATM encapsulation results in a >=9% throughput reduction on non-ATM links, which is IMHO too steep a price... plus the ATM/AAL5 encapsulation size in addition also depends on the packet size so not a reasonable default in a world where ATM-usage is on the way out.


> then later in the second list (« the details »), it is suggested to use « None », directly contradicting the above.

Are you referring to:

	• None: Fiber, and direct Ethernet connections generally do not need any kind of link layer adaptation. Well, I am kidding, all shaping below the physical gross-rate requires correct per-packet overhead accounting, but for fiber and ethernet it is much harder to figure out the exact overhead to specify… (the question is typically how is the ISP's upstream traffic shaper configured). For true ethernet shaping without VLANs specify 38 bytes (mpu 84).

I was under the impression that the "Well, I am kiddung" part was clear enough, no?


> The 44 byte overhead for ethernet/FTTH is also mentioned here: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm
> 
> More specifically, there are two (increasingly common I think) use cases I would like to document with your help:
> 
> - FTTH with plain DHCP, no VLAN
> - FTTH with PPPoE, no VLAN
> (And adding a footnote for the extra overhead to consider if the above include a VLAN tag)

	I am with you, I would like to have these settled as well, but alas FTTH is not terribly well defined as a technology. For active optical networks the encapsulation is likely ethernet framing, but for PONs like GPON and XGS-PON it is far less clear what needs to be accounted for. Yes with PON large parts of the ethernet framing overhead are replaced by a smaller GEM header, but how to account for the request-grant traffic and stuff...
	The good thing is that gently over-estimating the per-packet overhead only leads to a relative small reduction in maximal theoretical throughput, so `overhead 44` is still a decent recommendation even for FTTH.


> In the latter case (FTTH with PPPoE), another point that needs to be clarified is: do we apply CAKE to the ethernet interface, or to the PPP interface?

	That is your choice...


> (and I assume that depending on the answer, the overhead settings will have to be adjusted).

	It used to yes, but cake is smart enough to get the size of the IP packet and add the overhead on top of that, so the overhead will not depend on whether you instantiate cake on say eth0 or on pppoe-wan (assuming pppoe-wan uses eth0). HOWEVER for simple.qos and simplest.qos it again matters...

> Also in that case, what about ISPs that allow sending a full 1500 frame over PPPoE (using 1508 MTU on the underlying ethernet interface)?

	SQM with cake does not care about that, it sees the IP packet size and adds the configured overhead. That wat cake also has no issue with GRO/GSO meta packets which can be up to 64KB in size abd still get accounted for correctly. Baby-jumbo frames from that perspective simply result in IP packet sizes > 1492. 


> 
> Thanks for your input,
> T
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel

Re: [Cake] Help untangling CAKE settings for FTTH

[Thibaut]

Hi Sebastian,

Thanks for your reply.

> Le 16 nov. 2022 à 11:49, Sebastian Moeller <moeller0@gmx.de> a écrit :
> 
> HI T.
> 
> 
> 
>> On Nov 16, 2022, at 11:22, Thibaut <hacks@slashdirt.org> wrote:
>> 
>> Hi,
>> 
>> A few questions for the CAKE experts here:
> 
> 	Quick note, this is not necessarily the best place to get advice on cake, both the cake mailing list and the OpenWrt forum tend to be directer paths to the "bakers".

Well since this looked to me like an openwrt-specific question (with links to openwrt wiki), it felt more adequate to ask here, but thanks for redirecting.

>> I’m trying to untangle the information available in the openwrt wiki:
>> https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm
>> https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details
>> and for the latter especially the part here: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details#sqmlink_layer_adaptation_tab
>> 
>> For ADSL/VDSL, I believe the instructions are clear and the Luci interface is too. However for ethernet/fiber, I’m confused:
>> 
>> In the first list of this paragraph it is first suggested to use « Ethernet with overhead » and set the per-packet overhead to 44 for FFTH (which seems like a large value for this use case),
> 
> That is the point here, 44 is a value that in all likelihood is >= any realistic true overhead. Gently over-estimating the true overhead has a relative small cost in potential throughput, underestimating the overhead however can result in noticeable degradation of responsiveness under working conditions, so the recommendation is to rather err on the side of too large an overhead and not too small an overhead.
> Why not simply recommend the worst case scenario `overhead 44 atm` to be on the safe side on all links? Because the ATM/AAL5 encapsulation is only used on ADSL links (so is getting rarer and rarer and the ATM encapsulation results in a >=9% throughput reduction on non-ATM links, which is IMHO too steep a price... plus the ATM/AAL5 encapsulation size in addition also depends on the packet size so not a reasonable default in a world where ATM-usage is on the way out.

I hear your argument, however the point here is that either we offer a « single click » solution, and then we shouldn’t even bother allowing to tweak the 44 setting by default, or we allow customizing this value and then it seems logical to offer precise guidance on how to do so, especially in the « detailed » section of the wiki.

Currently we achieve neither IMHO, hence my initial email :)

>> then later in the second list (« the details »), it is suggested to use « None », directly contradicting the above.
> 
> Are you referring to:
> 
> 	• None: Fiber, and direct Ethernet connections generally do not need any kind of link layer adaptation. Well, I am kidding, all shaping below the physical gross-rate requires correct per-packet overhead accounting, but for fiber and ethernet it is much harder to figure out the exact overhead to specify… (the question is typically how is the ISP's upstream traffic shaper configured). For true ethernet shaping without VLANs specify 38 bytes (mpu 84).
> 
> I was under the impression that the "Well, I am kiddung" part was clear enough, no?

It wasn’t to me. And I don’t think that « jokes » should be part of a technical explanation that may be read by non-native speakers (like myself): it’s likely to cause confusion (as it did for me).

So I’ll try to edit this section to move the relevant information back where it belongs.

>> The 44 byte overhead for ethernet/FTTH is also mentioned here: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm
>> 
>> More specifically, there are two (increasingly common I think) use cases I would like to document with your help:
>> 
>> - FTTH with plain DHCP, no VLAN
>> - FTTH with PPPoE, no VLAN
>> (And adding a footnote for the extra overhead to consider if the above include a VLAN tag)
> 
> 	I am with you, I would like to have these settled as well, but alas FTTH is not terribly well defined as a technology. For active optical networks the encapsulation is likely ethernet framing, but for PONs like GPON and XGS-PON it is far less clear what needs to be accounted for. Yes with PON large parts of the ethernet framing overhead are replaced by a smaller GEM header, but how to account for the request-grant traffic and stuff...
> 	The good thing is that gently over-estimating the per-packet overhead only leads to a relative small reduction in maximal theoretical throughput, so `overhead 44` is still a decent recommendation even for FTTH.

Hmm ok.

>> In the latter case (FTTH with PPPoE), another point that needs to be clarified is: do we apply CAKE to the ethernet interface, or to the PPP interface?
> 
> 	That is your choice...

Well I don’t really care to choose (and I suspect a lot of users don’t either), except for the most relevant/efficient choice if there is one: which would that be? :)
In any case I suspect this is a FAQ that should be addressed, even if with a simple « it doesn’t matter ».

>> (and I assume that depending on the answer, the overhead settings will have to be adjusted).
> 
> 	It used to yes, but cake is smart enough to get the size of the IP packet and add the overhead on top of that, so the overhead will not depend on whether you instantiate cake on say eth0 or on pppoe-wan (assuming pppoe-wan uses eth0). HOWEVER for simple.qos and simplest.qos it again matters...

Could you elaborate? I’ll try to integrate your comments in my update to the wiki, with your permission.

>> Also in that case, what about ISPs that allow sending a full 1500 frame over PPPoE (using 1508 MTU on the underlying ethernet interface)?
> 
> 	SQM with cake does not care about that, it sees the IP packet size and adds the configured overhead. That wat cake also has no issue with GRO/GSO meta packets which can be up to 64KB in size abd still get accounted for correctly. Baby-jumbo frames from that perspective simply result in IP packet sizes > 1492. 

OK, understood.

Thanks again,
Thibaut

Re: [Cake] Help untangling CAKE settings for FTTH

[Sebastian Moeller]

Hi T.

> On Nov 16, 2022, at 12:46, Thibaut <hacks@slashdirt.org> wrote:
> 
> Hi Sebastian,
> 
> Thanks for your reply.
> 
>> Le 16 nov. 2022 à 11:49, Sebastian Moeller <moeller0@gmx.de> a écrit :
>> 
>> HI T.
>> 
>> 
>> 
>>> On Nov 16, 2022, at 11:22, Thibaut <hacks@slashdirt.org> wrote:
>>> 
>>> Hi,
>>> 
>>> A few questions for the CAKE experts here:
>> 
>> 	Quick note, this is not necessarily the best place to get advice on cake, both the cake mailing list and the OpenWrt forum tend to be directer paths to the "bakers".
> 
> Well since this looked to me like an openwrt-specific question (with links to openwrt wiki), it felt more adequate to ask here, but thanks for redirecting.

	Fair enough, however the OpenWrt wiki articles are often not written by the core developers (that use the devel mailing list) but by volunteer forum members.


> 
>>> I’m trying to untangle the information available in the openwrt wiki:
>>> https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm
>>> https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details
>>> and for the latter especially the part here: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details#sqmlink_layer_adaptation_tab
>>> 
>>> For ADSL/VDSL, I believe the instructions are clear and the Luci interface is too. However for ethernet/fiber, I’m confused:
>>> 
>>> In the first list of this paragraph it is first suggested to use « Ethernet with overhead » and set the per-packet overhead to 44 for FFTH (which seems like a large value for this use case),
>> 
>> That is the point here, 44 is a value that in all likelihood is >= any realistic true overhead. Gently over-estimating the true overhead has a relative small cost in potential throughput, underestimating the overhead however can result in noticeable degradation of responsiveness under working conditions, so the recommendation is to rather err on the side of too large an overhead and not too small an overhead.
>> Why not simply recommend the worst case scenario `overhead 44 atm` to be on the safe side on all links? Because the ATM/AAL5 encapsulation is only used on ADSL links (so is getting rarer and rarer and the ATM encapsulation results in a >=9% throughput reduction on non-ATM links, which is IMHO too steep a price... plus the ATM/AAL5 encapsulation size in addition also depends on the packet size so not a reasonable default in a world where ATM-usage is on the way out.
> 
> I hear your argument, however the point here is that either we offer a « single click » solution,

	As I tried to explain, we can not really do this as the only generally save configuration is too ugly.


> and then we shouldn’t even bother allowing to tweak the 44 setting by default,

	Erm, one idea behind SQM was/is to configure sensible defaults, but allow users to override/change these.

I take it you would like to see "overhead 44" as hard default?
 
> or we allow customizing this value and then it seems logical to offer precise guidance on how to do so, especially in the « detailed » section of the wiki.

	Well, it is a wiki, if you have robust and reliable information about a specific encapsulation feel free to add it to the respective page.

> Currently we achieve neither IMHO, hence my initial email :)

	Again I respectfully disagree, https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm is pretty clear in its recommendations. I expect that people following the link to the details page will actually read that page before jumping to conclusions ;) Then again the details page has seen many small additions and changes and apparently is in need for a full editorial pass again.


> 
>>> then later in the second list (« the details »), it is suggested to use « None », directly contradicting the above.
>> 
>> Are you referring to:
>> 
>> 	• None: Fiber, and direct Ethernet connections generally do not need any kind of link layer adaptation. Well, I am kidding, all shaping below the physical gross-rate requires correct per-packet overhead accounting, but for fiber and ethernet it is much harder to figure out the exact overhead to specify… (the question is typically how is the ISP's upstream traffic shaper configured). For true ethernet shaping without VLANs specify 38 bytes (mpu 84).
>> 
>> I was under the impression that the "Well, I am kiddung" part was clear enough, no?
> 
> It wasn’t to me. And I don’t think that « jokes » should be part of a technical explanation that may be read by non-native speakers (like myself): it’s likely to cause confusion (as it did for me).

	Ah, indeed language barriers can be real issues. 

> So I’ll try to edit this section to move the relevant information back where it belongs.

	Yes, that is what a wiki is for.


> 
>>> The 44 byte overhead for ethernet/FTTH is also mentioned here: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm
>>> 
>>> More specifically, there are two (increasingly common I think) use cases I would like to document with your help:
>>> 
>>> - FTTH with plain DHCP, no VLAN
>>> - FTTH with PPPoE, no VLAN
>>> (And adding a footnote for the extra overhead to consider if the above include a VLAN tag)
>> 
>> 	I am with you, I would like to have these settled as well, but alas FTTH is not terribly well defined as a technology. For active optical networks the encapsulation is likely ethernet framing, but for PONs like GPON and XGS-PON it is far less clear what needs to be accounted for. Yes with PON large parts of the ethernet framing overhead are replaced by a smaller GEM header, but how to account for the request-grant traffic and stuff...
>> 	The good thing is that gently over-estimating the per-packet overhead only leads to a relative small reduction in maximal theoretical throughput, so `overhead 44` is still a decent recommendation even for FTTH.
> 
> Hmm ok.
> 
>>> In the latter case (FTTH with PPPoE), another point that needs to be clarified is: do we apply CAKE to the ethernet interface, or to the PPP interface?
>> 
>> 	That is your choice...
> 
> Well I don’t really care to choose (and I suspect a lot of users don’t either), except for the most relevant/efficient choice if there is one: which would that be? :)

	There isn't a correct choice for all situations, it really depends what you expect.

> In any case I suspect this is a FAQ that should be addressed, even if with a simple « it doesn’t matter ».

	Which would be wrong. For most users one is as good as the other, but there are subtle differences that might matter depending on the circumstances. E.g. PPPoE interfaces are often removed and re-established which SQM deal with gracefully due to hooking into OpenWrt's hotplug mechanisms, but say you want need persistent statistics you should instantiate SQM on the L2-device instead, if you want the statistics to be reset on a new connection to the ISP instantiate SQM on pppoe-wan.


> 
>>> (and I assume that depending on the answer, the overhead settings will have to be adjusted).
>> 
>> 	It used to yes, but cake is smart enough to get the size of the IP packet and add the overhead on top of that, so the overhead will not depend on whether you instantiate cake on say eth0 or on pppoe-wan (assuming pppoe-wan uses eth0). HOWEVER for simple.qos and simplest.qos it again matters...
> 
> Could you elaborate? I’ll try to integrate your comments in my update to the wiki, with your permission.

	Only cake is refined enough to figure out the total IP packet size, tc's stab option will take the skb-size (IIRC) which typically for IP over ethernet will contain 14 bytes of the ethernet header if we look at an ethernet device, but on pppoe-wan it would not show these 14 bytes (these do simply not exist for pppoe-wan, the kernel will add these once the pppoe packet gets handled by the L2-interface) and I forgot what happens on IFB interfaces.
	I would always recommend new users to simply use cake/[ieve_of_cake.qos rather than trying to explain all these pointless intricacies.


> 
>>> Also in that case, what about ISPs that allow sending a full 1500 frame over PPPoE (using 1508 MTU on the underlying ethernet interface)?
>> 
>> 	SQM with cake does not care about that, it sees the IP packet size and adds the configured overhead. That wat cake also has no issue with GRO/GSO meta packets which can be up to 64KB in size abd still get accounted for correctly. Baby-jumbo frames from that perspective simply result in IP packet sizes > 1492. 
> 
> OK, understood.

	The same is true for HTB+fq_codel, BTW. 


> 
> Thanks again,
> Thibaut

Re: [Cake] Help untangling CAKE settings for FTTH

[Sebastian Moeller]

Hi T.


so taking your proposa under consideration I canged the section that threw you off course to read:


	• Ethernet with Overhead: SQM can also account for the overhead imposed by VDSL2 links - add 22 bytes of overhead (mpu 68). Cable Modems (DOCSIS) set both up- and downstream overhead to 18 bytes (6 bytes source MAC, 6 bytes destination MAC, 2 bytes ether-type, 4 bytes FCS), to allow for a possible 4 byte VLAN tag it is recommended to set the overhead to 18 + 4 = 22 (mpu 64). For FTTH the answer is less clear cut, since different underlaying technologies have different relevant per-packet-overheads; however underestimating the per-packet-overhead is considerably worse for responsiveness than (gently) overestimating it, so for FTTH set the overhead to 44 (mpu 84) unless there is more detailed information about the true overhead on a link available.
	• None: All shaping below the physical gross-rate of a link requires correct per-packet overhead accounting to be precise, so None is only useful if approximate shaping is sufficient, say if you want to clamp a guest network to at best ~50% of the available capacity or similar tasks, but even then configuring an approximate correct per-packet-overhead is recommended (overhead 44 (mpu 84) is a decent default to pick).


I hope this is explicit enough.

Regards
	Sebastian


> On Nov 16, 2022, at 12:46, Thibaut <hacks@slashdirt.org> wrote:
> 
> Hi Sebastian,
> 
> Thanks for your reply.
> 
>> Le 16 nov. 2022 à 11:49, Sebastian Moeller <moeller0@gmx.de> a écrit :
>> 
>> HI T.
>> 
>> 
>> 
>>> On Nov 16, 2022, at 11:22, Thibaut <hacks@slashdirt.org> wrote:
>>> 
>>> Hi,
>>> 
>>> A few questions for the CAKE experts here:
>> 
>> 	Quick note, this is not necessarily the best place to get advice on cake, both the cake mailing list and the OpenWrt forum tend to be directer paths to the "bakers".
> 
> Well since this looked to me like an openwrt-specific question (with links to openwrt wiki), it felt more adequate to ask here, but thanks for redirecting.
> 
>>> I’m trying to untangle the information available in the openwrt wiki:
>>> https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm
>>> https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details
>>> and for the latter especially the part here: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details#sqmlink_layer_adaptation_tab
>>> 
>>> For ADSL/VDSL, I believe the instructions are clear and the Luci interface is too. However for ethernet/fiber, I’m confused:
>>> 
>>> In the first list of this paragraph it is first suggested to use « Ethernet with overhead » and set the per-packet overhead to 44 for FFTH (which seems like a large value for this use case),
>> 
>> That is the point here, 44 is a value that in all likelihood is >= any realistic true overhead. Gently over-estimating the true overhead has a relative small cost in potential throughput, underestimating the overhead however can result in noticeable degradation of responsiveness under working conditions, so the recommendation is to rather err on the side of too large an overhead and not too small an overhead.
>> Why not simply recommend the worst case scenario `overhead 44 atm` to be on the safe side on all links? Because the ATM/AAL5 encapsulation is only used on ADSL links (so is getting rarer and rarer and the ATM encapsulation results in a >=9% throughput reduction on non-ATM links, which is IMHO too steep a price... plus the ATM/AAL5 encapsulation size in addition also depends on the packet size so not a reasonable default in a world where ATM-usage is on the way out.
> 
> I hear your argument, however the point here is that either we offer a « single click » solution, and then we shouldn’t even bother allowing to tweak the 44 setting by default, or we allow customizing this value and then it seems logical to offer precise guidance on how to do so, especially in the « detailed » section of the wiki.
> 
> Currently we achieve neither IMHO, hence my initial email :)
> 
>>> then later in the second list (« the details »), it is suggested to use « None », directly contradicting the above.
>> 
>> Are you referring to:
>> 
>> 	• None: Fiber, and direct Ethernet connections generally do not need any kind of link layer adaptation. Well, I am kidding, all shaping below the physical gross-rate requires correct per-packet overhead accounting, but for fiber and ethernet it is much harder to figure out the exact overhead to specify… (the question is typically how is the ISP's upstream traffic shaper configured). For true ethernet shaping without VLANs specify 38 bytes (mpu 84).
>> 
>> I was under the impression that the "Well, I am kiddung" part was clear enough, no?
> 
> It wasn’t to me. And I don’t think that « jokes » should be part of a technical explanation that may be read by non-native speakers (like myself): it’s likely to cause confusion (as it did for me).
> 
> So I’ll try to edit this section to move the relevant information back where it belongs.
> 
>>> The 44 byte overhead for ethernet/FTTH is also mentioned here: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm
>>> 
>>> More specifically, there are two (increasingly common I think) use cases I would like to document with your help:
>>> 
>>> - FTTH with plain DHCP, no VLAN
>>> - FTTH with PPPoE, no VLAN
>>> (And adding a footnote for the extra overhead to consider if the above include a VLAN tag)
>> 
>> 	I am with you, I would like to have these settled as well, but alas FTTH is not terribly well defined as a technology. For active optical networks the encapsulation is likely ethernet framing, but for PONs like GPON and XGS-PON it is far less clear what needs to be accounted for. Yes with PON large parts of the ethernet framing overhead are replaced by a smaller GEM header, but how to account for the request-grant traffic and stuff...
>> 	The good thing is that gently over-estimating the per-packet overhead only leads to a relative small reduction in maximal theoretical throughput, so `overhead 44` is still a decent recommendation even for FTTH.
> 
> Hmm ok.
> 
>>> In the latter case (FTTH with PPPoE), another point that needs to be clarified is: do we apply CAKE to the ethernet interface, or to the PPP interface?
>> 
>> 	That is your choice...
> 
> Well I don’t really care to choose (and I suspect a lot of users don’t either), except for the most relevant/efficient choice if there is one: which would that be? :)
> In any case I suspect this is a FAQ that should be addressed, even if with a simple « it doesn’t matter ».
> 
>>> (and I assume that depending on the answer, the overhead settings will have to be adjusted).
>> 
>> 	It used to yes, but cake is smart enough to get the size of the IP packet and add the overhead on top of that, so the overhead will not depend on whether you instantiate cake on say eth0 or on pppoe-wan (assuming pppoe-wan uses eth0). HOWEVER for simple.qos and simplest.qos it again matters...
> 
> Could you elaborate? I’ll try to integrate your comments in my update to the wiki, with your permission.
> 
>>> Also in that case, what about ISPs that allow sending a full 1500 frame over PPPoE (using 1508 MTU on the underlying ethernet interface)?
>> 
>> 	SQM with cake does not care about that, it sees the IP packet size and adds the configured overhead. That wat cake also has no issue with GRO/GSO meta packets which can be up to 64KB in size abd still get accounted for correctly. Baby-jumbo frames from that perspective simply result in IP packet sizes > 1492. 
> 
> OK, understood.
> 
> Thanks again,
> Thibaut

Re: [Cake] Help untangling CAKE settings for FTTH

[Thibaut]

Hi Sebastian,

> Le 17 nov. 2022 à 10:50, Sebastian Moeller <moeller0@gmx.de> a écrit :
> 
> Hi T.
> 
> 
> so taking your proposa under consideration I canged the section that threw you off course to read:
> 
> 
> 	• Ethernet with Overhead: SQM can also account for the overhead imposed by VDSL2 links - add 22 bytes of overhead (mpu 68). Cable Modems (DOCSIS) set both up- and downstream overhead to 18 bytes (6 bytes source MAC, 6 bytes destination MAC, 2 bytes ether-type, 4 bytes FCS), to allow for a possible 4 byte VLAN tag it is recommended to set the overhead to 18 + 4 = 22 (mpu 64). For FTTH the answer is less clear cut, since different underlaying technologies have different relevant per-packet-overheads; however underestimating the per-packet-overhead is considerably worse for responsiveness than (gently) overestimating it, so for FTTH set the overhead to 44 (mpu 84) unless there is more detailed information about the true overhead on a link available.
> 	• None: All shaping below the physical gross-rate of a link requires correct per-packet overhead accounting to be precise, so None is only useful if approximate shaping is sufficient, say if you want to clamp a guest network to at best ~50% of the available capacity or similar tasks, but even then configuring an approximate correct per-packet-overhead is recommended (overhead 44 (mpu 84) is a decent default to pick).
> 
> 
> I hope this is explicit enough.

Yes this looks a lot better, thank you.

Although I must confess that it certainly feels counter-intuitive that for ethernet (and FTTH) we suggest a higher overhead than e.g. VDSL2/cable (which themselves run off an ethernet interface).

I would also like to see some info about ppp vs ethernet interface in there (matching your previous email): unless you beat me to it I will add it.
I also think the « details » page needs to be reformatted a bit, it’s very dense and relevant info is all over the place and not very well organized. I’ll try to get around improving that.

Thanks

Re: [Cake] Help untangling CAKE settings for FTTH

[Sebastian Moeller]

Hi Thibaut,

> On Nov 17, 2022, at 15:22, Thibaut <hacks@slashdirt.org> wrote:
> 
> Hi Sebastian,
> 
>> Le 17 nov. 2022 à 10:50, Sebastian Moeller <moeller0@gmx.de> a écrit :
>> 
>> Hi T.
>> 
>> 
>> so taking your proposa under consideration I canged the section that threw you off course to read:
>> 
>> 
>> 	• Ethernet with Overhead: SQM can also account for the overhead imposed by VDSL2 links - add 22 bytes of overhead (mpu 68). Cable Modems (DOCSIS) set both up- and downstream overhead to 18 bytes (6 bytes source MAC, 6 bytes destination MAC, 2 bytes ether-type, 4 bytes FCS), to allow for a possible 4 byte VLAN tag it is recommended to set the overhead to 18 + 4 = 22 (mpu 64). For FTTH the answer is less clear cut, since different underlaying technologies have different relevant per-packet-overheads; however underestimating the per-packet-overhead is considerably worse for responsiveness than (gently) overestimating it, so for FTTH set the overhead to 44 (mpu 84) unless there is more detailed information about the true overhead on a link available.
>> 	• None: All shaping below the physical gross-rate of a link requires correct per-packet overhead accounting to be precise, so None is only useful if approximate shaping is sufficient, say if you want to clamp a guest network to at best ~50% of the available capacity or similar tasks, but even then configuring an approximate correct per-packet-overhead is recommended (overhead 44 (mpu 84) is a decent default to pick).
>> 
>> 
>> I hope this is explicit enough.
> 
> Yes this looks a lot better, thank you.
> 
> Although I must confess that it certainly feels counter-intuitive that for ethernet (and FTTH) we suggest a higher overhead than e.g. VDSL2/cable (which themselves run off an ethernet interface).

	That is simply because for (ADSL) DOCSIS VDSL2 we have a much clear picture about what we need to account for. And AON can be essentially using true ethernet encapsulation so we can expect an unspecified "FTTH" class to encompass a broad set of different encapsulations, if we want to recommend a single number that should also cover AON (the PONs are much less clear*). Why do you assume that FTTH necessarily would have a smaller per-packet-overhead than other link technologies? 
	Now, if you have reliable information for say GPON-overhead, by all means add it (but to be useful this also should contain an easy identifier for other users to figure out whether they use GPON in the first place).

	The bigger point however is IMHO, from the perspective of minimizing bufferbloat/latency-under-load-increase using a slightly too large per-packet-overhead is fully benign, while specifying to small an overhead can easily result in measurable bufferbloat increase. So IMHO it is fine to err on the side of too large when estimating the per-packet-overhead.

*) The core problem is that we have no straight forward way to empirically deduce the effective per-packet-overhead over an arbitrary network path, so if a link technology defines/documents the overhead well and conclusively (like docsis) we are in luck, but if the details a vague or leave many options to the ISP we have to make an educated guess.


> I would also like to see some info about ppp vs ethernet interface in there (matching your previous email): unless you beat me to it I will add it.

	I will not beat you to it, because for users of cake it does not matter and we do recommend to use cake in the first place... heck even for folks wanting to use HTB+fq_codel I would recommend to start with cake and then look at the output of `tc -s qdisc` to figure out the overhead that is already accounted for on an interface.

> I also think the « details » page needs to be reformatted a bit, it’s very dense and relevant info is all over the place and not very well organized.

	Might well be true (the page evolved over time and might need a full editorial pass), but it covers quite some territory and hence always will be a bit unwieldy.


> I’ll try to get around improving that.

	Please try to keep all correct information around in the document.

Good luck


> 
> Thanks

Re: [Cake] Help untangling CAKE settings for FTTH

[Thibaut]

[-- Attachment #1: Type: text/plain, Size: 4973 bytes --]



> Le 17 nov. 2022 à 15:42, Sebastian Moeller <moeller0@gmx.de> a écrit :
> 
> Hi Thibaut,
> 
>> On Nov 17, 2022, at 15:22, Thibaut <hacks@slashdirt.org> wrote:
>> 
>> Hi Sebastian,
>> 
>>> Le 17 nov. 2022 à 10:50, Sebastian Moeller <moeller0@gmx.de> a écrit :
>>> 
>>> Hi T.
>>> 
>>> 
>>> so taking your proposa under consideration I canged the section that threw you off course to read:
>>> 
>>> 
>>> 	• Ethernet with Overhead: SQM can also account for the overhead imposed by VDSL2 links - add 22 bytes of overhead (mpu 68). Cable Modems (DOCSIS) set both up- and downstream overhead to 18 bytes (6 bytes source MAC, 6 bytes destination MAC, 2 bytes ether-type, 4 bytes FCS), to allow for a possible 4 byte VLAN tag it is recommended to set the overhead to 18 + 4 = 22 (mpu 64). For FTTH the answer is less clear cut, since different underlaying technologies have different relevant per-packet-overheads; however underestimating the per-packet-overhead is considerably worse for responsiveness than (gently) overestimating it, so for FTTH set the overhead to 44 (mpu 84) unless there is more detailed information about the true overhead on a link available.
>>> 	• None: All shaping below the physical gross-rate of a link requires correct per-packet overhead accounting to be precise, so None is only useful if approximate shaping is sufficient, say if you want to clamp a guest network to at best ~50% of the available capacity or similar tasks, but even then configuring an approximate correct per-packet-overhead is recommended (overhead 44 (mpu 84) is a decent default to pick).
>>> 
>>> 
>>> I hope this is explicit enough.
>> 
>> Yes this looks a lot better, thank you.
>> 
>> Although I must confess that it certainly feels counter-intuitive that for ethernet (and FTTH) we suggest a higher overhead than e.g. VDSL2/cable (which themselves run off an ethernet interface).
> 
> 	That is simply because for (ADSL) DOCSIS VDSL2 we have a much clear picture about what we need to account for. And AON can be essentially using true ethernet encapsulation so we can expect an unspecified "FTTH" class to encompass a broad set of different encapsulations, if we want to recommend a single number that should also cover AON (the PONs are much less clear*). Why do you assume that FTTH necessarily would have a smaller per-packet-overhead than other link technologies? 

I’m not (necessarily) making that assumption for FTTH (although I would expect it to be the case, from my limited understanding of FTTH technologies), I am however certainly making that assumption for plain ethernet, which is included in the 44-byte documentation case. I think it’s a reasonable one?

> 	Now, if you have reliable information for say GPON-overhead, by all means add it (but to be useful this also should contain an easy identifier for other users to figure out whether they use GPON in the first place).
> 
> 	The bigger point however is IMHO, from the perspective of minimizing bufferbloat/latency-under-load-increase using a slightly too large per-packet-overhead is fully benign, while specifying to small an overhead can easily result in measurable bufferbloat increase. So IMHO it is fine to err on the side of too large when estimating the per-packet-overhead.

OK. Although I would think people reading the detailed explanation are looking for precise info and explanations, not one-stop-shop approximations. Not to mention the kind of users who want to squeeze the maximum performance out of their setup.

> *) The core problem is that we have no straight forward way to empirically deduce the effective per-packet-overhead over an arbitrary network path, so if a link technology defines/documents the overhead well and conclusively (like docsis) we are in luck, but if the details a vague or leave many options to the ISP we have to make an educated guess.
> 
> 
>> I would also like to see some info about ppp vs ethernet interface in there (matching your previous email): unless you beat me to it I will add it.
> 
> 	I will not beat you to it, because for users of cake it does not matter

You said the exact opposite in your previous email (persistence of statistics) :)

> and we do recommend to use cake in the first place... heck even for folks wanting to use HTB+fq_codel I would recommend to start with cake and then look at the output of `tc -s qdisc` to figure out the overhead that is already accounted for on an interface.
> 
>> I also think the « details » page needs to be reformatted a bit, it’s very dense and relevant info is all over the place and not very well organized.
> 
> 	Might well be true (the page evolved over time and might need a full editorial pass), but it covers quite some territory and hence always will be a bit unwieldy.
> 
> 
>> I’ll try to get around improving that.
> 
> 	Please try to keep all correct information around in the document.

Sure.

Cheers

[-- Attachment #2: Type: text/html, Size: 19817 bytes --]

Re: [Cake] Help untangling CAKE settings for FTTH

[Sebastian Moeller]

Hi Thibaut,


> On Nov 17, 2022, at 16:00, Thibaut <hacks@slashdirt.org> wrote:
> 
> 
> 
>> Le 17 nov. 2022 à 15:42, Sebastian Moeller <moeller0@gmx.de> a écrit :
>> 
>> Hi Thibaut,
>> 
>>> On Nov 17, 2022, at 15:22, Thibaut <hacks@slashdirt.org> wrote:
>>> 
>>> Hi Sebastian,
>>> 
>>>> Le 17 nov. 2022 à 10:50, Sebastian Moeller <moeller0@gmx.de> a écrit :
>>>> 
>>>> Hi T.
>>>> 
>>>> 
>>>> so taking your proposa under consideration I canged the section that threw you off course to read:
>>>> 
>>>> 
>>>> 	• Ethernet with Overhead: SQM can also account for the overhead imposed by VDSL2 links - add 22 bytes of overhead (mpu 68). Cable Modems (DOCSIS) set both up- and downstream overhead to 18 bytes (6 bytes source MAC, 6 bytes destination MAC, 2 bytes ether-type, 4 bytes FCS), to allow for a possible 4 byte VLAN tag it is recommended to set the overhead to 18 + 4 = 22 (mpu 64). For FTTH the answer is less clear cut, since different underlaying technologies have different relevant per-packet-overheads; however underestimating the per-packet-overhead is considerably worse for responsiveness than (gently) overestimating it, so for FTTH set the overhead to 44 (mpu 84) unless there is more detailed information about the true overhead on a link available.
>>>> 	• None: All shaping below the physical gross-rate of a link requires correct per-packet overhead accounting to be precise, so None is only useful if approximate shaping is sufficient, say if you want to clamp a guest network to at best ~50% of the available capacity or similar tasks, but even then configuring an approximate correct per-packet-overhead is recommended (overhead 44 (mpu 84) is a decent default to pick).
>>>> 
>>>> 
>>>> I hope this is explicit enough.
>>> 
>>> Yes this looks a lot better, thank you.
>>> 
>>> Although I must confess that it certainly feels counter-intuitive that for ethernet (and FTTH) we suggest a higher overhead than e.g. VDSL2/cable (which themselves run off an ethernet interface).
>> 
>> 	That is simply because for (ADSL) DOCSIS VDSL2 we have a much clear picture about what we need to account for. And AON can be essentially using true ethernet encapsulation so we can expect an unspecified "FTTH" class to encompass a broad set of different encapsulations, if we want to recommend a single number that should also cover AON (the PONs are much less clear*). Why do you assume that FTTH necessarily would have a smaller per-packet-overhead than other link technologies? 
> 
> I’m not (necessarily) making that assumption for FTTH (although I would expect it to be the case, from my limited understanding of FTTH technologies),

	Well, all I can say about encapsulations and per-packet-overhead is: it's complicated.

> I am however certainly making that assumption for plain ethernet, which is included in the 44-byte documentation case. I think it’s a reasonable one?

	Yes, the goal is to give one number that works everywhere so 44 is what we recommend. As I said overestimating the overhead is benign, underestimating it is not. True ethernet has an effective* per-packet-overhead of:
7 Byte Preamble + 1 Byte start of frame delimiter (SFD) + 12 Byte inter frame gap (IFG) + 4 Byte Frame Check Sequence (FCS) + 6 (dest MAC) + 6 (src MAC) + 2 (ethertype)
7 + 1 + 12 + 4 + 6 + 6 + 2 = 38 bytes

Now if a VLAN tag is added you end up with 38+4 = 42 so both pure ethernet and ethernet+VLAN have a er-packet-overhead close to 44 bytes.

*) The IFG is not real bytes, but transmission silence for the time it would take to transmit 12 octets.

> 
>> 	Now, if you have reliable information for say GPON-overhead, by all means add it (but to be useful this also should contain an easy identifier for other users to figure out whether they use GPON in the first place).
>> 
>> 	The bigger point however is IMHO, from the perspective of minimizing bufferbloat/latency-under-load-increase using a slightly too large per-packet-overhead is fully benign, while specifying to small an overhead can easily result in measurable bufferbloat increase. So IMHO it is fine to err on the side of too large when estimating the per-packet-overhead.
> 
> OK. Although I would think people reading the detailed explanation are looking for precise info and explanations, not one-stop-shop approximations.

	Sure, but the problem is we can not give them that "precise information" them, you , me would like to have, so we do the best we can.

> Not to mention the kind of users who want to squeeze the maximum performance out of their setup.

	Just read the section where I explain how per-packet-overhead and shaper-rate are not orthogonal variables to understand how problematic the whoe thing is, add to it that ISPs often employ traffic shapers with potentially independent overhead-assumptions making the whole problem even harder.

> 
>> *) The core problem is that we have no straight forward way to empirically deduce the effective per-packet-overhead over an arbitrary network path, so if a link technology defines/documents the overhead well and conclusively (like docsis) we are in luck, but if the details a vague or leave many options to the ISP we have to make an educated guess.
>> 
>> 
>>> I would also like to see some info about ppp vs ethernet interface in there (matching your previous email): unless you beat me to it I will add it.
>> 
>> 	I will not beat you to it, because for users of cake it does not matter
> 
> You said the exact opposite in your previous email (persistence of statistics) :)

	I misunderstood your argument here and thought you wanted a section explaining the issues with getting the overhead numbers for pppoe** and ethernet interfaces set for HTB+fq_codel, sorry. If you want to explain differences between cake on pppoe or on ethN go for it.


**) ppp or pppoe somewhat matters, PPPoE has 6 Bytes of overhead on top of PPP's 2 byte resulting in a total of 
PPPoE: 2 Byte PPP + 6 Byte (PPP)oE = 8 Byte

> 
>> and we do recommend to use cake in the first place... heck even for folks wanting to use HTB+fq_codel I would recommend to start with cake and then look at the output of `tc -s qdisc` to figure out the overhead that is already accounted for on an interface.
>> 
>>> I also think the « details » page needs to be reformatted a bit, it’s very dense and relevant info is all over the place and not very well organized.
>> 
>> 	Might well be true (the page evolved over time and might need a full editorial pass), but it covers quite some territory and hence always will be a bit unwieldy.
>> 
>> 
>>> I’ll try to get around improving that.
>> 
>> 	Please try to keep all correct information around in the document.
> 
> Sure.
> 
> Cheers

Regards

Re: [Cake] Help untangling CAKE settings for FTTH

[Thibaut]

Hi Sebastian,

> Le 17 nov. 2022 à 16:19, Sebastian Moeller <moeller0@gmx.de> a écrit :
> 
> Hi Thibaut,

[...]
>>>> Although I must confess that it certainly feels counter-intuitive that for ethernet (and FTTH) we suggest a higher overhead than e.g. VDSL2/cable (which themselves run off an ethernet interface).
>>> 
>>> 	That is simply because for (ADSL) DOCSIS VDSL2 we have a much clear picture about what we need to account for. And AON can be essentially using true ethernet encapsulation so we can expect an unspecified "FTTH" class to encompass a broad set of different encapsulations, if we want to recommend a single number that should also cover AON (the PONs are much less clear*). Why do you assume that FTTH necessarily would have a smaller per-packet-overhead than other link technologies? 
>> 
>> I’m not (necessarily) making that assumption for FTTH (although I would expect it to be the case, from my limited understanding of FTTH technologies),
> 
> 	Well, all I can say about encapsulations and per-packet-overhead is: it's complicated.

I can see that now :)

>> I am however certainly making that assumption for plain ethernet, which is included in the 44-byte documentation case. I think it’s a reasonable one?
> 
> 	Yes, the goal is to give one number that works everywhere so 44 is what we recommend. As I said overestimating the overhead is benign, underestimating it is not. True ethernet has an effective* per-packet-overhead of:
> 7 Byte Preamble + 1 Byte start of frame delimiter (SFD) + 12 Byte inter frame gap (IFG) + 4 Byte Frame Check Sequence (FCS) + 6 (dest MAC) + 6 (src MAC) + 2 (ethertype)
> 7 + 1 + 12 + 4 + 6 + 6 + 2 = 38 bytes

And now I’m even more confused: why do we suggest an overhead of 34/26 for VDSL or 22 for docsis cable then? I assume the respective modems are connected via ethernet, aren’t they?

Another confusing point in the current detailed doc (and possibly the one that got me started):
"	• Link Layer Adaptation calculates the proper overheads for WAN links such as DSL and ADSL. If you use any kind of DSL link, you should review this section."

This seems to imply LLA is only necessary for « exotic » links such as ADSL/VDSL.

This also brought me to the (naïve) comment that it would be nice if CAKE would account for local NIC L2 overhead internally (since that info is readily available to it, is it not?), and one would only have to configure the actual « overhead » of whatever is connected to the local NIC. There would be less margin for error I believe.

Then I saw this:

"Please note that as of middle 2018 cake, and cake only, will try to interpret any given overhead to be applied on top of IP packets, all other qdiscs (and cake if configured with the “raw” keyword) will add the specified overhead on top of the overhead the kernel already accounted for. This seems confusing, because it is ;) so if in doubt stick to cake."

Well yes, it *is* very confusing :P

> Now if a VLAN tag is added you end up with 38+4 = 42 so both pure ethernet and ethernet+VLAN have a er-packet-overhead close to 44 bytes.
> 
> *) The IFG is not real bytes, but transmission silence for the time it would take to transmit 12 octets.
> 
>> 
>>> 	Now, if you have reliable information for say GPON-overhead, by all means add it (but to be useful this also should contain an easy identifier for other users to figure out whether they use GPON in the first place).
>>> 
>>> 	The bigger point however is IMHO, from the perspective of minimizing bufferbloat/latency-under-load-increase using a slightly too large per-packet-overhead is fully benign, while specifying to small an overhead can easily result in measurable bufferbloat increase. So IMHO it is fine to err on the side of too large when estimating the per-packet-overhead.
>> 
>> OK. Although I would think people reading the detailed explanation are looking for precise info and explanations, not one-stop-shop approximations.
> 
> 	Sure, but the problem is we can not give them that "precise information" them, you , me would like to have, so we do the best we can.
> 
>> Not to mention the kind of users who want to squeeze the maximum performance out of their setup.
> 
> 	Just read the section where I explain how per-packet-overhead and shaper-rate are not orthogonal variables to understand how problematic the whoe thing is, add to it that ISPs often employ traffic shapers with potentially independent overhead-assumptions making the whole problem even harder.

Yes, I see it’s a bit of a headache :)

>>> *) The core problem is that we have no straight forward way to empirically deduce the effective per-packet-overhead over an arbitrary network path, so if a link technology defines/documents the overhead well and conclusively (like docsis) we are in luck, but if the details a vague or leave many options to the ISP we have to make an educated guess.
>>> 
>>> 
>>>> I would also like to see some info about ppp vs ethernet interface in there (matching your previous email): unless you beat me to it I will add it.
>>> 
>>> 	I will not beat you to it, because for users of cake it does not matter
>> 
>> You said the exact opposite in your previous email (persistence of statistics) :)
> 
> 	I misunderstood your argument here and thought you wanted a section explaining the issues with getting the overhead numbers for pppoe** and ethernet interfaces set for HTB+fq_codel, sorry. If you want to explain differences between cake on pppoe or on ethN go for it.

OK but I also believe it’s important to clarify that the overhead setting needs not be adjusted whether we shape the pppoe interface or its underlying ethernet one: that certainly was counter-intuitive for me.

Thanks for bearing with my ignorance,
T

Re: [Cake] Help untangling CAKE settings for FTTH

[Sebastian Moeller]

Hi Thibaut,

> On Nov 17, 2022, at 16:53, Thibaut <hacks@slashdirt.org> wrote:
> 
> Hi Sebastian,
> 
>> Le 17 nov. 2022 à 16:19, Sebastian Moeller <moeller0@gmx.de> a écrit :
>> 
>> Hi Thibaut,
> 
> [...]
>>>>> Although I must confess that it certainly feels counter-intuitive that for ethernet (and FTTH) we suggest a higher overhead than e.g. VDSL2/cable (which themselves run off an ethernet interface).
>>>> 
>>>> 	That is simply because for (ADSL) DOCSIS VDSL2 we have a much clear picture about what we need to account for. And AON can be essentially using true ethernet encapsulation so we can expect an unspecified "FTTH" class to encompass a broad set of different encapsulations, if we want to recommend a single number that should also cover AON (the PONs are much less clear*). Why do you assume that FTTH necessarily would have a smaller per-packet-overhead than other link technologies? 
>>> 
>>> I’m not (necessarily) making that assumption for FTTH (although I would expect it to be the case, from my limited understanding of FTTH technologies),
>> 
>> 	Well, all I can say about encapsulations and per-packet-overhead is: it's complicated.
> 
> I can see that now :)
> 
>>> I am however certainly making that assumption for plain ethernet, which is included in the 44-byte documentation case. I think it’s a reasonable one?
>> 
>> 	Yes, the goal is to give one number that works everywhere so 44 is what we recommend. As I said overestimating the overhead is benign, underestimating it is not. True ethernet has an effective* per-packet-overhead of:
>> 7 Byte Preamble + 1 Byte start of frame delimiter (SFD) + 12 Byte inter frame gap (IFG) + 4 Byte Frame Check Sequence (FCS) + 6 (dest MAC) + 6 (src MAC) + 2 (ethertype)
>> 7 + 1 + 12 + 4 + 6 + 6 + 2 = 38 bytes
> 
> And now I’m even more confused: why do we suggest an overhead of 34/26 for VDSL or 22 for docsis cable then? I assume the respective modems are connected via ethernet, aren’t they?

	For DOCSIS you will be correct as there are no DOCSIS modem-routers that run OpenWrt, for DSL we do have some devices (older xrx200 and in development newer vrx518 lantiq/maxlinear SoCs) that actually have DSL interfaces.

This is a bit of a judgement call, we only need to account for those overheads that actually happen on the bottleneck link of a path.

For DSL/PTM the dsl overhead is smaller than the full ethernet overhead as some parts of ethernet encapsulation are not transmitted via the DSL link.
So let's run the numbers for a router with gigabit ethernet port connected to a DSL modem with gigabit ethernet port. At what DSL rate will the added ethernet overhead actually matter and to save some time let's only look at the worst case of small packets (as there the relative overhead cost is larger):

Ethernet max payload rate for minimal packet size: 
1000 * ((64-18)/(64+20)) = 547.619047619 Mbps (true ethernet adds 20 bytes equivalent on top of the L2 ethernet frame)

DOCSIS:
(1000 * ((64-18)/(64+20))) * ((64)/(64-18)) = 761.904761905 Mbps


Now use this payload rate to see how much raw DSL rate we would need to meet/exceed that rate:
(1000 * ((64-18)/(64+20))) * (65/64) * ((64+4)/(64-18)) = 822.172619048 Mbps (PTM adds 4 bytes on top of the L2 ethernet frame https://en.wikipedia.org/wiki/Ethernet_frame)

Currently VDSL2 is limited to ~250 Mbps, so we are far off from the true ethernet overhead becoming relevant for our traffic shaper.

HOWEVER, if the link between router and modem is only 100 Mbps ethernet the picture changes, and indeed when using a 100 Mbps modem with a 100Mbps VDSL2 link I used `rate 95 overhead 50` (why 50 and not 34+20=54 you might ask? because on the link between modem and router I did not use a VLAN tag, but had the modem add this tag later).

As I said, it's complicated.


It is similar for DOCSIS, except there rates in the >= 1 Gbps range are a thing since some time already and unless rputer and modem use a 2.5/5/10/... Gbps connection the full ethernet overhead becomes relevant. Time to change the docsis recommendations for say rates > 500 Mbps.

ADSL/ATM/AAL5 is essentially only used for speeds below 25 Mbps with devices with at least 100Mbps ethernet ports so let's ignore this for now:


> 
> Another confusing point in the current detailed doc (and possibly the one that got me started):
> "	• Link Layer Adaptation calculates the proper overheads for WAN links such as DSL and ADSL. If you use any kind of DSL link, you should review this section."
> 
> This seems to imply LLA is only necessary for « exotic » links such as ADSL/VDSL.

	This is historic, LLA is exceptionally important for ATM/AAL5 as there the effectove overhead depends on the actual packet size, the importance of proper overhead accounting for all link technologies only became obvious later (one naively assumes that this can be fixed by setting the shaper somewhat lower, but as the section about overhead and shaper rate shows it is not as easy).

> This also brought me to the (naïve) comment that it would be nice if CAKE would account for local NIC L2 overhead internally (since that info is readily available to it, is it not?), and one would only have to configure the actual « overhead » of whatever is connected to the local NIC. There would be less margin for error I believe.

	You need to account for the relevant overhead on the bottleneck link, which often is not an L1/L2 interface of the router itself. Cake has a number of technology specific link layer accounting "sets" prepared you can activate with a keyword, but that does not solve the hard question, which overhead is actually to be accounted for (see the example of VDSL connected over ethernet above with the to be accounted for overhead depending on the ratio od DSL rate to ethernet rate).


> Then I saw this:
> 
> "Please note that as of middle 2018 cake, and cake only, will try to interpret any given overhead to be applied on top of IP packets, all other qdiscs (and cake if configured with the “raw” keyword) will add the specified overhead on top of the overhead the kernel already accounted for. This seems confusing, because it is ;) so if in doubt stick to cake."
> 
> Well yes, it *is* very confusing :P

	Yepp, on ethernet interfaces the kernel will typically only account for 14 bytes of the total 38 bytes of ethernet framing, as the kernel maintain these fields (MAC addresses and ethertype) itself while the other parts are mostly handled by the driver of offloaded onto the NIC, so the kernel is not doing anything wrong by not reporting these other parts.


> 
>> Now if a VLAN tag is added you end up with 38+4 = 42 so both pure ethernet and ethernet+VLAN have a er-packet-overhead close to 44 bytes.
>> 
>> *) The IFG is not real bytes, but transmission silence for the time it would take to transmit 12 octets.
>> 
>>> 
>>>> 	Now, if you have reliable information for say GPON-overhead, by all means add it (but to be useful this also should contain an easy identifier for other users to figure out whether they use GPON in the first place).
>>>> 
>>>> 	The bigger point however is IMHO, from the perspective of minimizing bufferbloat/latency-under-load-increase using a slightly too large per-packet-overhead is fully benign, while specifying to small an overhead can easily result in measurable bufferbloat increase. So IMHO it is fine to err on the side of too large when estimating the per-packet-overhead.
>>> 
>>> OK. Although I would think people reading the detailed explanation are looking for precise info and explanations, not one-stop-shop approximations.
>> 
>> 	Sure, but the problem is we can not give them that "precise information" them, you , me would like to have, so we do the best we can.
>> 
>>> Not to mention the kind of users who want to squeeze the maximum performance out of their setup.
>> 
>> 	Just read the section where I explain how per-packet-overhead and shaper-rate are not orthogonal variables to understand how problematic the whoe thing is, add to it that ISPs often employ traffic shapers with potentially independent overhead-assumptions making the whole problem even harder.
> 
> Yes, I see it’s a bit of a headache :)

	+1; the whole wiki would read easier if the subject matter was less complicated ;)


> 
>>>> *) The core problem is that we have no straight forward way to empirically deduce the effective per-packet-overhead over an arbitrary network path, so if a link technology defines/documents the overhead well and conclusively (like docsis) we are in luck, but if the details a vague or leave many options to the ISP we have to make an educated guess.
>>>> 
>>>> 
>>>>> I would also like to see some info about ppp vs ethernet interface in there (matching your previous email): unless you beat me to it I will add it.
>>>> 
>>>> 	I will not beat you to it, because for users of cake it does not matter
>>> 
>>> You said the exact opposite in your previous email (persistence of statistics) :)
>> 
>> 	I misunderstood your argument here and thought you wanted a section explaining the issues with getting the overhead numbers for pppoe** and ethernet interfaces set for HTB+fq_codel, sorry. If you want to explain differences between cake on pppoe or on ethN go for it.
> 
> OK but I also believe it’s important to clarify that the overhead setting needs not be adjusted whether we shape the pppoe interface or its underlying ethernet one: that certainly was counter-intuitive for me.

	+1; go for it!


> 
> Thanks for bearing with my ignorance,

	This is how wikis improve if open questions are discussed and text changed.


> T
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel