[Cerowrt-devel] some kernel updates

Fri Aug 23 06:15:12 EDT 2013

Hi Jesper,

On Aug 23, 2013, at 09:27 , Jesper Dangaard Brouer <jbrouer at redhat.com> wrote:

> On Thu, 22 Aug 2013 22:13:52 -0700
> Dave Taht <dave.taht at gmail.com> wrote:
> 
>> On Thu, Aug 22, 2013 at 5:52 PM, Sebastian Moeller <moeller0 at gmx.de> wrote:
>> 
>>> Hi List, hi Jesper,
>>> 
>>> So I tested 3.10.9-1 to assess the status of the HTB atm link layer
>>> adjustments to see whether the recent changes resurrected this feature.
>>>        Unfortunately the htb_private link layer adjustments still is
>>> broken (RRUL ping RTT against Toke's netperf host in Germany of ~80ms, same
>>> as without link layer adjustments). On the bright side the tc_stab method
>>> still works as well as before (ping RTT around 40ms).
>>>        I would like to humbly propose to use the tc stab method in
>>> cerowrt to perform ATM link layer adjustments as default. To repeat myself,
>>> simply telling the kernel a lie about the packet size seems more robust
>>> than fudging HTB's rate tables.
> 
> After the (regression) commit 56b765b79 ("htb: improved accuracy at
> high rates"), the kernel no-longer uses the rate tables.  

	See, I am quite a layman here, spelunking through the tc and kernel source code made me believe that the rate tables are still used (I might have looked at too old versions of both repositories though).

> 
> My commit 8a8e3d84b1719 (net_sched: restore "linklayer atm" handling),
> does the ATM cell overhead calculation directly on the packet length,
> see psched_l2t_ns() doing (DIV_ROUND_UP(len,48)*53).
> Thus, the cell calc should actually be more precise now.... but see below

	Is there any way to make HTB report which link layer it assumes?

> 
>>> Especially since the kernel already fudges
>>> the packet size to account for the ethernet header and then some, so this
>>> path should receive more scrutiny by virtue of having more users?
> 
> As you mention, the default kernel path (not tc stab) fudges the packet
> size for Ethernet headers, AND I made a mistake (back in approx 2006,
> sorry) that the "overhead" cannot be a negative number.  

	Mmh, does this also apply to stab?

> Meaning that
> some ATM encap overheads simply cannot be configured correctly (as you
> need to subtract the ethernet header).

	Yes, I see, luckily PPPoA and IPoA seem quite rare, and setting the overhead to be larger than it actually is is relatively benign, as it will overestimate packe size.

> (And its quite problematic to
> change the kABI to allow for a negative overhead)

	Again I have no clue but overhead seems to be integer, not unsigned, so why can it not be negative?

> 
> Perhaps we should change to use "tc stab" for this reason.  But I'm not
> sure "stab" does the right thing either, and its accuracy is also
> limited as its actually also table based.

	But why should a table be problematic here? As long as we can assure the table is equal or larger to the largest packet we are golden. So either we do the manly and  stupid thing and go for 9000 byte jumbo packets for the table size. Or we assume that for the most part ATM users will art best use baby jumbo frames (I think BT does this to allow payload MTU 1500 in spite of PPPoE encapsulation overhead) but than we are quite fine with the default size table maxMTU of 2048 bytes, no?

>  We could easily change the
> kernel to perform the ATM cell overhead calc inside "stab", and we
> should also fix the GSO packet overhead problem.
> (for now remember to disable GSO packets when shaping)

	Yeah I stumbled over the fact that the stab mechanism does not honor the kernels earlier adjustments of packet length (but I seem to be unable to find the actual file and line where this initially is handeled). It would seem relatively easy to make stab take the earlier adjustment into account. Regarding GSO, I assumed that GSO will not play nicely with a AQM anyway as a single large packet will hog too much transfer time...

> 
>> It's my hope that the atm code works but is misconfigured. You can output
>> the tc commands by overriding the TC variable with TC="echo tc" and paste
>> here.
> 
> I also hope is a misconfig.  Please show us the config/script.

	Will do this later. I would be delighted if it is just me being stupid.

> 
> I would appreciate a link to the scripts you are using... perhaps a git tree?

	Unfortunately I have no git tree and no experience with git. I do not think I will be able to set something up quickly. But I use a modified version of cerowrt's AQM scripts which I will post later.

> 
> 
>>>        Now, I have been testing this using Dave's most recent cerowrt
>>> alpha version with a 3.10.9 kernel on mips hardware, I think this kernel
>>> should contain all htb fixes including commit 8a8e3d84b17 (net_sched:
>>> restore "linklayer atm" handling) but am not fully sure.
>>> 
>> 
>> It does.
> 
> It have not hit the stable tree yet, but DaveM promised he would pass it along.
> 
> It does seem Dave Taht have my patch applied:
> http://snapon.lab.bufferbloat.net/~cero2/patches/3.10.9-1/685-net_sched-restore-linklayer-atm-handling.patch

	Ah, good so it should have worked.

> 
>>> While I am not able to build kernels, it seems that I am able to quickly
>>> test whether link layer adjustments work or not. SO aim happy to help where
>>> I can :)
> 
> So, what is you setup lab, that allow you to test this quickly?

	Oh, Dave and Toke are the giants on whose shoulders I stand here (thanks guys), all I bring to the table basically is the fact that I have an ATM carried ADSL2+ connection at home. 
	Anyway, my theory is that proper link layer adjustments should only show up if not performing these would make my traffic exceed my link-speed and hence accumulate in the DSL modems bloated buffers leading to measurable increases in latency. So I try to saturate the both up- and down-link while measuring latency und different conditions. SInce the worst case overhead of the ATM encapsulation approaches 50% (with best case being around 10%) I try to test the system while shaping to 95% percent of link rates where do expect to see an effect of the link layer adjustments and while shaping to 50% where do not expect to see an effect. And basically that seems to work.
	Practically, I use Toke's netsurf-wrapper project with the RRUL test from my cerowrt router behind an ADSL2+ modem to a close netperf server in Germany. The link layer adjustments are configured in my cerowrt router, using Dave's simple.qos script (3 band HTB shaper with fq_codel on each leaf, taking my overhead of 40 bytes into account and optionally the link layer).
	It turns out that this test nicely saturates my link with 4 up and 4 down TCP flows ad uses a train ping probes at 0.2 second period to assess the latency induced by saturating the links. Now I shape down to 95% and 50% of line rates and simply look at the ping RTT plot for different conditions. In my rig I see around 30ms ping RTT without load, 80ms with full saturation and no linklayer adjustments, and 40ms with working link layer adjustments (hand in hand with slightly reduced TCP good put just as one would expect). In my testing so far activating the HTB link layer adjustments yielded the same 80ms delay I get without link layer adjustments. If I shape down to 50% of link rates HTB, stab and no link layer adjustments yield a ping RTT of ~40ms. Still with proper link layer adjustments the TCP good-put is reduced even at 50% shaping. As Dave explained with an unloaded swallow ermm, ping RTT and fq_codel's target set to 5ms the best case would be 30ms + 2*5ms or 40ms, so I am pretty close to ideal with proper link layer adjustments.

	I guess it should be possible to simply use the reduction in good-put as an easy indicator whether the link layer adjustments work or not. But to do this properly I would need to be able to control the size of the sent packets which I am not, at least not with RRUL. But I am quite sure real computer scientists could easily set something up to test the good-put through a shaping device at differently sized packet streams of the same bandwidth, but I digress. 

	On the other hand I do not claim to be an expert in this field in any way and my measurement method might be flawed, if you think so please do not hesitate to let me know how I could improve it.

Best Regards
	Sebastian

> 
> 
> -- 
> Best regards,
>  Jesper Dangaard Brouer
>  MSc.CS, Sr. Network Kernel Developer at Red Hat
>  Author of http://www.iptv-analyzer.org
>  LinkedIn: http://www.linkedin.com/in/brouer