Re: [Cake] tc-cake(8) needs to explain a common mistake

Re: [Cake] tc-cake(8) needs to explain a common mistake

[Dave Taht]

so nice to know cake has made it to russia!!!

On Fri, Apr 3, 2020 at 11:46 AM Alexander E. Patrakov
<patrakov@gmail.com> wrote:
>
> Hello,
>
> there is a recurring cargo cult pattern in many forums (e.g. OpenWRT):
> people keep suggesting various overhead compensation parameters to
> tc-cake without checking what's the bottleneck. They just assume that
> it is always related to the link-layer technology of the connection.
>
> This assumption is mostly incorrect, and this needs to be explained in
> the manual page to stop the cargo cult. E.g., here in Russia, in the
> past year, I had a 1Gbit/s link (1000BASE-X) but they shaped my
> connection down to 500 Mbit/s because that's the bandwidth that I paid
> for. I.e. the link from my router to the ISP equipment was not the
> bottleneck, it was the ISP's shaper.
>
> How about the following addition to the tc-cake(8) manual page, just
> before "Manual Overhead Specification"? Feel free to edit.
>
> General considerations
> -------------------------------
>
> Do not blindly set the overhead compensation parameters to match the
> internet connection link type and protocols running on it. Doing so
> makes sense only if that link (and not something further in the path,
> like the ISP's shaper) is indeed the bottleneck.
>
> Example 1: the ADSL modem connects at 18 Mbit/s, but the ISP further
> throttles the speed to 15 Mbit/s because that's what the user pays
> for, and does so with a shaper that has bufferbloat. Then, the "adsl"
> keyword is likely not appropriate, because the ISP's shaper operates
> on the IP level. The bandwidth needs to be set slightly below 15
> Mbit/s.
>
> Example 2: the ADSL modem connects at 18 Mbit/s, and the user pays for
> "as fast as the modem can get" connection. Then, the "adsl" keyword is
> relevant, and the bandwidth needs to be set to 18 Mbit/s.
>
> Example 3: the user has a 100BASE-TX Ethernet connection, and pays for
> the full 100 Mbit/s bandwidth (i.e. there is no shaper further up).
> Then, the "ethernet" keyword is relevant, and the bandwidth needs to
> be set to 100 Mbit/s.
>
> --
> Alexander E. Patrakov
> CV: http://pc.cd/PLz7



-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

Re: [Cake] tc-cake(8) needs to explain a common mistake

[Sebastian Moeller]

> On Apr 3, 2020, at 20:49, Dave Taht <dave.taht@gmail.com> wrote:
> 
> so nice to know cake has made it to russia!!!
> 
> On Fri, Apr 3, 2020 at 11:46 AM Alexander E. Patrakov
> <patrakov@gmail.com> wrote:
>> 
>> Hello,
>> 
>> there is a recurring cargo cult pattern in many forums (e.g. OpenWRT):
>> people keep suggesting various overhead compensation parameters to
>> tc-cake without checking what's the bottleneck. They just assume that
>> it is always related to the link-layer technology of the connection.
>> 
>> This assumption is mostly incorrect, and this needs to be explained in
>> the manual page to stop the cargo cult. E.g., here in Russia, in the
>> past year, I had a 1Gbit/s link (1000BASE-X) but they shaped my
>> connection down to 500 Mbit/s because that's the bandwidth that I paid
>> for. I.e. the link from my router to the ISP equipment was not the
>> bottleneck, it was the ISP's shaper.
>> 
>> How about the following addition to the tc-cake(8) manual page, just
>> before "Manual Overhead Specification"? Feel free to edit.
>> 
>> General considerations
>> -------------------------------
>> 
>> Do not blindly set the overhead compensation parameters to match the
>> internet connection link type and protocols running on it. Doing so
>> makes sense only if that link (and not something further in the path,
>> like the ISP's shaper) is indeed the bottleneck.

	Well, In general yes, but in reality a competent ISP will configure its shapers to account for the link properties, so assuming the access link's overhead/encapsulation gives a reasonable first guess at what values might be optimal.


>> 
>> Example 1: the ADSL modem connects at 18 Mbit/s, but the ISP further
>> throttles the speed to 15 Mbit/s because that's what the user pays
>> for, and does so with a shaper that has bufferbloat. Then, the "adsl"
>> keyword is likely not appropriate, because the ISP's shaper operates
>> on the IP level. The bandwidth needs to be set slightly below 15
>> Mbit/s.

	Let's run the number shall we? I simply make a few assumptions here to get things started, but the exact numbers really do not matter too much. With that said, let's assume TCP/IPv4 and ATM/AAL5, PPPoE, LLC/SNAP, RFC-2684;
Overhead (bytes): PPP (2), PPPoE (6), Ethernet Header (14), Ethernet PAD [8] (0), ATM LLC (3), ATM SNAP (5), ATM pad (2), ATM AAL5 SAR (8) : Total 40

Let's see what the link will be able to deliver for "full" MTU 1500 packets (quotes as the MTU1500 will only carry to the PPPoE endpoint, internet MTU is going to be 1492)
gross-rate * ((payload size) / (on the wire size)) = net speedtest result (let's use this as proxy as this is what people can easily verify/check)

MTU1500: 18.000 * ((1500-20-20-8) / (ceil((1500-8+40)/48)*53)) = 15.410
MTU150: 18.000 * ((150-20-20-8) / (ceil((150-8+40)/48)*53)) = 8.66037735849
MTU75: 18.000 * ((75-20-20-8) / (ceil((75-8+40)/48)*53)) = 3.05660377358


Now the IP-level shaper at ~80% of the link-speed, if it does not account for the ATM/AAL5 "celling" even if it gets the overhead correctly will give the following:

MTU1500: 15.000 * ((1500-20-20-8) / (ceil((1500-8+40)/1)*1)) = 14.2167101828
MTU150: 15.000 * ((150-20-20-8) / (ceil((150-8+40)/1)*1)) = 8.40659340659
MTU75: 15.000 * ((75-20-20-8) / (ceil((75-8+40)/1)*1)) = 3.78504672897

So for large enough packets static accounting for ATM/AAL5 works reasonably well, but for small packets it fails.
That is why most ISP-grade equipment allows not only to configure the per-packet-overhead for end-user links but also can deal with ATM/AAL5. And as far as I understand most competent ISPs actually configure their traffic-shapers for ADSL links to do this, because DSLAMs are really more like L2-switches with fancy media-converters attached and deal not terribly well with overload and queueing into the switch fabric.

That in turn leads to the following situation:
MTU1500: 15.000 * ((1500-20-20-8) / (ceil((1500-8+40)/48)*53)) = 12.842
MTU150: 15.000 * ((150-20-20-8) / (ceil((150-8+40)/48)*53)) = 7.217
MTU75: 15.000 * ((75-20-20-8) / (ceil((75-8+40)/48)*53)) = 2.547

which will obviously not cause packet buffering in the DSLAM for any packet size mix the link might encounter. AND that in turn means that the actual bottleneck link (the ISP's traffic shaper) still behaves like it would employ ATM/AAL5 encapsulation, and hence the end-user's SQM instance should do as well.



>> 
>> Example 2: the ADSL modem connects at 18 Mbit/s, and the user pays for
>> "as fast as the modem can get" connection. Then, the "adsl" keyword is
>> relevant, and the bandwidth needs to be set to 18 Mbit/s.

	IMHO that is simply a less opaque version of example 1, but I agree here ATM/AAL5 accounting needs to be done...


>> 
>> Example 3: the user has a 100BASE-TX Ethernet connection, and pays for
>> the full 100 Mbit/s bandwidth (i.e. there is no shaper further up).
>> Then, the "ethernet" keyword is relevant, and the bandwidth needs to
>> be set to 100 Mbit/s.

	Yes, if the true bottleneck is an ethernet link, then ethernet overhead needs to be accounted for, so overhead 38 bytes (or 42 if VLANs are in use) would be correct. 


I fully concur that properly accounting for the properties of the true bottleneck is the challenge here. And with few exceptions this is hard, as we have two dependent variables to set gross shaper rate and per-packet-overhead. Looking at the numbers, I have reluctantly become convinced, that getting the per-packet-overhead correctly is not that important as long as one does not under-estimate it, so if in doubt, I rather recommend to err on the side of too much (but I value low latency above maximum throughput, like most people starting to look at sqm).

Best Regards
	Sebastian

P.S.: I am of the opinion, that https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details had very sane and un-cargo-culty advice about the overhead topic:
"Getting [overhead and link layer accounting] exactly right is less important than getting it close, and over-estimating by a few bytes is generally better at keeping bufferbloat down than underestimating. With this in mind, to get started, set the Link Layer Adaptation options based on your connection to the Internet. "

I am less sure about the paragraph you added recently, as it does not seem to consider all the applicable subtleties.


>> 
>> --
>> Alexander E. Patrakov
>> CV: http://pc.cd/PLz7
> 
> 
> 
> -- 
> Make Music, Not War
> 
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-435-0729
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] tc-cake(8) needs to explain a common mistake

[Alexander E. Patrakov]

On Sat, Apr 4, 2020 at 1:44 AM Sebastian Moeller <moeller0@gmx.de> wrote:

> >>
> >> Example 1: the ADSL modem connects at 18 Mbit/s, but the ISP further
> >> throttles the speed to 15 Mbit/s because that's what the user pays
> >> for, and does so with a shaper that has bufferbloat. Then, the "adsl"
> >> keyword is likely not appropriate, because the ISP's shaper operates
> >> on the IP level. The bandwidth needs to be set slightly below 15
> >> Mbit/s.
>
>         Let's run the number shall we? I simply make a few assumptions here to get things started, but the exact numbers really do not matter too much. With that said, let's assume TCP/IPv4 and ATM/AAL5, PPPoE, LLC/SNAP, RFC-2684;
> Overhead (bytes): PPP (2), PPPoE (6), Ethernet Header (14), Ethernet PAD [8] (0), ATM LLC (3), ATM SNAP (5), ATM pad (2), ATM AAL5 SAR (8) : Total 40
>
> Let's see what the link will be able to deliver for "full" MTU 1500 packets (quotes as the MTU1500 will only carry to the PPPoE endpoint, internet MTU is going to be 1492)
> gross-rate * ((payload size) / (on the wire size)) = net speedtest result (let's use this as proxy as this is what people can easily verify/check)
>
> MTU1500: 18.000 * ((1500-20-20-8) / (ceil((1500-8+40)/48)*53)) = 15.410
> MTU150: 18.000 * ((150-20-20-8) / (ceil((150-8+40)/48)*53)) = 8.66037735849
> MTU75: 18.000 * ((75-20-20-8) / (ceil((75-8+40)/48)*53)) = 3.05660377358
>
>
> Now the IP-level shaper at ~80% of the link-speed, if it does not account for the ATM/AAL5 "celling" even if it gets the overhead correctly will give the following:
>
> MTU1500: 15.000 * ((1500-20-20-8) / (ceil((1500-8+40)/1)*1)) = 14.2167101828
> MTU150: 15.000 * ((150-20-20-8) / (ceil((150-8+40)/1)*1)) = 8.40659340659
> MTU75: 15.000 * ((75-20-20-8) / (ceil((75-8+40)/1)*1)) = 3.78504672897
>
> So for large enough packets static accounting for ATM/AAL5 works reasonably well, but for small packets it fails.
> That is why most ISP-grade equipment allows not only to configure the per-packet-overhead for end-user links but also can deal with ATM/AAL5. And as far as I understand most competent ISPs actually configure their traffic-shapers for ADSL links to do this, because DSLAMs are really more like L2-switches with fancy media-converters attached and deal not terribly well with overload and queueing into the switch fabric.
>
> That in turn leads to the following situation:
> MTU1500: 15.000 * ((1500-20-20-8) / (ceil((1500-8+40)/48)*53)) = 12.842
> MTU150: 15.000 * ((150-20-20-8) / (ceil((150-8+40)/48)*53)) = 7.217
> MTU75: 15.000 * ((75-20-20-8) / (ceil((75-8+40)/48)*53)) = 2.547
>
> which will obviously not cause packet buffering in the DSLAM for any packet size mix the link might encounter. AND that in turn means that the actual bottleneck link (the ISP's traffic shaper) still behaves like it would employ ATM/AAL5 encapsulation, and hence the end-user's SQM instance should do as well.

OK, bad (marginal) example, let's adjust it so that the user pays for
10 Mbit/s. Or replace with a 100BASE-TX link shaped (badly) to 50
Mbit/s by the ISP. The point is that sometimes the ISP shaper is what
matters, and ISPs like to sell bandwidth in packages with round
numbers.

And, is the accounting for ATM/AAL5 the default on equipment that ISPs
use for ADSL?

P.S. The example actually comes from my experience with the "Globe"
ISP in the Philippines during my trip there - I had to specifically
ask to increase the speed limit, it was initially 10 Mbit/s and then
upgraded to 15 Mbit/s. The modem always connected at 18 - 19 Mbit/s,
and there was a 21 Mbit/s value once (when I connected the modem to
the powerbank during power outage). And I am not sure that we are
always talking about competent ISPs.

<snip>

> P.S.: I am of the opinion, that https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details had very sane and un-cargo-culty advice about the overhead topic:
> "Getting [overhead and link layer accounting] exactly right is less important than getting it close, and over-estimating by a few bytes is generally better at keeping bufferbloat down than underestimating. With this in mind, to get started, set the Link Layer Adaptation options based on your connection to the Internet. "
>
> I am less sure about the paragraph you added recently, as it does not seem to consider all the applicable subtleties.

Should I undo it?

-- 
Alexander E. Patrakov
CV: http://pc.cd/PLz7