[Cerowrt-devel] SQM Question #5: Link Layer Adaptation Overheads

Mon Jan 6 04:28:52 PST 2014

Hi Dave,

On Jan 6, 2014, at 04:46 , Dave Taht <dave.taht at gmail.com> wrote:

> On Sat, Jan 4, 2014 at 12:22 PM, Sebastian Moeller <moeller0 at gmx.de> wrote:
>> Hi Rich,
>> 
>> On Jan 4, 2014, at 19:16 , Rich Brown <richb.hanover at gmail.com> wrote:
>> 
>>> QUESTION #5: I still don’t have any great answers for the Link Layer Adaptation overhead descriptions and recommendations. In an earlier message, (see https://lists.bufferbloat.net/pipermail/cerowrt-devel/2013-December/001914.html and following messages), Fred Stratton described the overheads carried by various options, and Sebastian Moeller also gave some useful advice.
>>> 
>>> After looking at the options, I despair of giving people a clear recommendation that would be optimal for their equipment. Consequently, I believe the best we can do is come up with “good enough” recommendations that are not wrong, and still give decent performance.
>> 
>>        Not wanting to be a spoilsport, but IMHO the issue is complicated hence no simple recommendations. I know that my last word was that 40bytes would be a good default overhead, but today I had the opportunity to measure the overhead on fast ADSL connection in Luxembourg and found that in this double-play situation (television and internet via DSL) that an other wise invisible VLAN was further increasing the overhead (from the 40 expected to 44 bytes). At least on faster links these combo packets (internet, phone and potentially telephone) are becoming more and more common, so maybe the recommendation should be 44 (hopping that FCS are truly rare).
>> 
>>> 
>>> In this spirit, I have changed Draft #3 of the “Setting up SQM” page to reflect this understanding. See http://www.bufferbloat.net/projects/cerowrt/wiki/Setting_up_AQM_for_CeroWrt_310
>>> 
>>>      ADSL/ATM link: Choose “ADSL/ATM", and set Per Packet Overhead to 40
>> 
>>        While I prefer ATM, I think all deployed ADSL is on ATM so these are synonyms for our purpose. I prefer ATM since the most critical part of the link layer adjustments is caused by the impedance mismatch between what ATM offers and what the data transport layer requires. I have the impression that ADSL might still evolve to a different carrier, while ATM is basically in maintenance mode (not much new deployment if any).
>> 
>>>      VDSL2 link: Choose “VDSL”, and set Per Packet Overhead to 8
>> 
>>        There are several issues with this; VDSL is not the direct predecessor of VDSL2 (rather VDSL2 is the successor of ADSL2+ with some similarities to VDSL). Lumping VDSL with VDSL2 will require us figuring out whether both behave the same. From my cursory reading of the standards of both I think VDSL is not unlikely to be using an ATM link layer, VDSL2 is unlikely to do the same, both seem technically able to use ATM.
>> 
>> 
>>>      Other kind of link (e.g., Cable, Fiber, Ethernet, other not listed): Choose “None (default)”, and set Per Packet Overhead to 0
>> 
>>        This is not going to be worse than today, so sounds fine (it would be good to know whether there is truly no overhead on these links in practical useage).
> 
> Well, I just spent a painful hour trying to find an optimum for the
> device on my mom's (cable) network, which I plan to replace with one
> that is ipv6 compatible shortly. (I am settled in one place for a
> while, so may setup a local dhcpv6-pd server, too - but my primary
> mission is to get a test suite going)
> 
> Before: http://snapon.lab.bufferbloat.net/~cero2/taht_mahal/taht_house_comcast_long-3.svg
> 
> (note it took a long time to fully saturate the buffers as the test
> box was 50ms away. I usually test with one 16ms away)
> 
> After:
> 
> http://snapon.lab.bufferbloat.net/~cero2/taht_mahal/fq_codel_30_8_taht_house_comcast-long.svg
> 
> The upload value isn't making that much sense, and I lost a lot of
> throughput. Variety of other tests in that dir.
> 
>>        Quick vote: anyone on this list using ceroWRT on an VDSL/VDSL2 link or cable fiber whatnot that could do some quick testing for us?
> 
> There doesn't appear to be any need for "overhead" related settings on
> cable. There might be some use on docsis 3 in changing the htb burst
> value.

	That is what I thought before as well, nowadays I am not so sure. Our problem is that we need to get a reliable estimate of the IP bandwidth between modem and CTMS, so we can reliably avoiding filling the modems buffers. The catch is that the advertised speed typically is given in some "raw" reference frame not close enough to what a user/router typically can handle out of the box. With cable I am confident that there are some packet headers that make sure only one modem picks up the data (and from the euphemistic bandwidth descriptions on the DSL side I am quite confident that this is in-band not out-of-band with an independent bandwidth supply.)
	Dave, would you be willing to collect a ping sample on your cable link, so I could use this as control for my ATM detector code?

	As an example of this situation where I have a better handle take, drum rolls, a typical DSL connection :) 
A) The bandwidth promised by the ISP typically is inside a corridor often quite wide, and say 6 to 16Mbit/s is not too helpful for setting the shaper. 

B) The modem itself reports the a number of values "Actual Data Rate" (among others even less useful values like "Attainable Data Rate"); this requires active intervention by the user, these values are typically presented via a http. Alternatively advanced users can telnet/ssh into some modems/routers and get more detailed low-level information from the DSL modem (but I digress).

C) This "Actual Data Rate" is the raw synchronization rate, but since forward error correction (FEC) is handled semi out-of-band an unspecified part (given enough information about the FEC parameters this can b calculated though) of that synch rate might not available to the user (I cynically just assume that these bits are taken from the actual data rate without any real proof). With ADSL it seems that an integer number of sub carrier tones are dedicated to FEC. 

D) The next issue is the 48 in 53 ATM cell encapsulation which reduces the useable rate down to ~ 90% (this is handled by link layer ATM quite well). Naively one would expect that just shaping down to 90% should have the same effect, but see E). To properly configure this the user needs to know whether his link is using ATM to the DSLAM or not (this information might or might not be available on the modem/router's web interface or on the ISP's customer web interface).

E) Unfortunately, the method used to wrap data packets into ATM cells requires that each packet fills an integer number of ATM cells, so the left over of the last cell will be padded. This especially dire as this can almost double the wire size of small packets like ACKs, making it really hard for the shaper to work well with small packet traffic like ACKs and to some degree VoIP. Luckily for the user, thanks to Jesper and Russell, the "linklayer ATM" method for tc handles this as well. But note for this to work the kernel needs to know how large the packet including overhead is going to be, which leads to...

F) Per packet overhead, there is a number of possible methods to "encapsulate" IP packets into an ATM carrier, which add different amounts of overhead. This ranges from IP over ATM (IPoA over VC-MUX) which basically wraps IP packets into ATM cell meta packets with minimal overhead of 8 bytes to PPPoE over LLC/SNAP which can add 40bytes per packet (this not only includes 8byte PPPoE overhead, but also a full ethernet header worth 14 bytes). To add insult to injury it seems that the 40+ bytes is the ISP's preferred option (where is the invisible hand of the market if you need it :) ). Again for small packets not taking the overhead into account makes shaping quite impossible; in addition underestimating the overhead will create specific packet sizes for which the shaper underestimates the wire size by one cell.

G) and finally (at least this is the last thing I stumbled over empirically there might be more out there) any additional overhead the carrier adds to the packet. Foe example I stumbled over an combined IPTV Internet access which added a 4 byte ethernet VLAN tag which got removed at the ISP supplied router creating a total of 44 bytes of overhead. At least in Europe these kinds of multi-play services using VLANs seem quite popular. (Now, interestingly both VC-MUX and LLC/SNAP basically are multiplexing headers that would allow to keep different streams logically and bandwidth wise separated, but a) these are ATM specific and b) do not fit into the current ideal of a pure IP "next generation" network of the telcos. But users on ATM are not getting short changed here.)

Overall, there is quite a significant difference between the "advertised" wire sped and the useable IP bandwidth on ATM links the user actually associates with that number, and it seems non-trivial for a typical user to easily collect all the required information to configure SQM correctly. I do not see this changing any time soon, all-fiber networks are not going to come in the (near) future, as much as I want it to.

> 
>>> NB: I have changed the first menu choice to “ADSL/ATM” and the second to “VDSL” in the description.
>> 
>>        I am fine with changing names, just see what the consensus is for the names.
> 
> I can live with whatever is decided here, and sebastian has commit access.

	So we have "ADSL/ATM" as one, what about "VDSL/PTM", (PTM: packet transfer mode) to keep things symmetric? (We still have the issue with possible ATM carrier on VDSL1 but we should deal with this independently).
Or "ATM/ADSL" and "PTM/VDSL", "ethernet"

Awaiting your votes...

Best Regards
	Sebastian

> 
>>> I would ask that we change to GUI to reflect those names as well. This makes it far easier/less confusing to talk about the options.
>> 
>>        Agreed.
> 
> Go for it!
> 
>> 
>> 
>>> As always, I welcome help in setting out clear recommendations that work well for the vast majority of people who try CeroWrt. Thanks.
> 
> I don't mind if cero stays too complex for mom to configure, although
> I appreciate
> the efforts to simplify things.
> 
>> 
>>        I guess, we need a new wiki page detailing the procedure to figure out the link layer (and overhead if on ATM).
>> 
>> Best
>>        Sebastian
>> 
>>> 
>>> Rich
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel at lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel at lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html