[Cerowrt-devel] Correctly calculating overheads on unknown connections

Tue Sep 23 18:16:20 EDT 2014

Hi Andy,

On Sep 23, 2014, at 21:05 , Andy Furniss <adf.lists at gmail.com> wrote:

> Sebastian Moeller wrote:
>> Hi Andy,
>> 
>> On Sep 23, 2014, at 17:10 , Andy Furniss <adf.lists at gmail.com>
>> wrote:
> 
>> BUT if you look at the kernel code, stab does not automatically
>> include the ethernet overhead, so the subtract 14 in the above is
>> actually wrong. See
>> http://lxr.free-electrons.com/source/net/sched/sch_api.c#L538 where
>> “pkt_len = skb->len + stab->szopts.overhead; is used instead of using
>> “qdisc_skb_cb(skb)->pkt_len” that as filled properly in
>> http://lxr.free-electrons.com/source/net/core/dev.c#L2705 . At least
>> to me this clearly looks like the ethernet overhead is not pre-added
>> when using stab, but I could be wrong. And on an ADSL link you can
>> see this quite well, with the proper overhead values sqm-scripts
>> still controls the latency under netperf-wrapper’s RRUL test nicely
>> even if the shaping rate equals the line rate, with the overhead to
>> small latency goes down the drain ;)
> 
> I guess skb->len varies depending on the interface.
> 
> Anyway here's a quick test on my desktop PC running a git kernel and tc.
> 
> I used to shape remotely pppoa/vc mux dsl so know that for me
> 
> ping -s 10 .... = one cell and -s 11 = 2 cells - overhead on IP was 10.
> 
> Paste time -
> 
> ph4[/mnt/sda8/Qos/stab-tests]# cat stab-hfsc
> #set -x
> TC=/sbin/tc
> 
> $TC qdisc del dev eth0 root &>/dev/null
> 
> if [ "$1" = "stop" ]
> then
>        exit
> fi
> 
> $TC qdisc add dev eth0 root handle 1: stab overhead -4 linklayer atm hfsc default ffff
> $TC class add dev eth0 parent 1: classid 1:1 hfsc sc rate 1kbit ul rate 1kbit
> $TC qdisc add dev eth0 parent 1:1 pfifo limit 200
> $TC class add dev eth0 parent 1:0 classid 1:ffff hfsc sc rate 80mbit ul rate 80mbit
> 
> $TC filter add dev eth0 parent 1: protocol ip prio 1 \
>    u32 match ip protocol 1 0xff classid 1:1
> 
> ph4[/mnt/sda8/Qos/stab-tests]# ./stab-hfsc
> ph4[/mnt/sda8/Qos/stab-tests]# ping -s 10 -c 1 noki
> PING noki.andys.lan (192.168.0.1) 10(38) bytes of data.
> 18 bytes from noki.andys.lan (192.168.0.1): icmp_req=1 ttl=64
> 
> --- noki.andys.lan ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> 
> ph4[/mnt/sda8/Qos/stab-tests]# tc -s qdisc ls dev eth0
> qdisc hfsc 1: root refcnt 2 default ffff
> Sent 106 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo 8005: parent 1:1 limit 200p
> Sent 53 bytes 1 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> ph4[/mnt/sda8/Qos/stab-tests]# ./stab-hfsc
> ph4[/mnt/sda8/Qos/stab-tests]# ping -s 11 -c 1 noki
> PING noki.andys.lan (192.168.0.1) 11(39) bytes of data.
> 19 bytes from noki.andys.lan (192.168.0.1): icmp_req=1 ttl=64
> 
> --- noki.andys.lan ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> 
> ph4[/mnt/sda8/Qos/stab-tests]# tc -s qdisc ls dev eth0
> qdisc hfsc 1: root refcnt 2 default ffff
> Sent 106 bytes 1 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo 8006: parent 1:1 limit 200p
> Sent 106 bytes 1 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> ph4[/mnt/sda8/Qos/stab-tests]#

	Thanks for sharing your test case; I can repeat these results exactly on my machines (I also tried htb instead hfsc for fun: same result as to be expected see below).
Looking back at http://lxr.free-electrons.com/ident?i=qdisc_pkt_len_init (line 2731):

qdisc_skb_cb(skb)->pkt_len += (gso_segs - 1) * hdr_len ;

I begin to realize this function is not responsible for adding single  wire packet’s ethernet header, but for figuring out in how many on-the-wire packets to chop down a GSO packet , and add the header overhead for the additional wire packets, I had completely looked over the (gso-segs - 1) part, oops. 

	@cerowrt-devel: everyone using link layer ATM you might want to try to reduce the the per packet overhead by 14… (but please test)

So I stand corrected, you are right, tic’s stab automatically adds the ethernet header. So I am off to repeat my netperf-wrapper tests right now again with overhead of 30 instead of 44, again these tests confirm your observation. Interestingly, it seems netperf-wrapper’s RRUL test really is suited to figure out the overhead: while shaping to 100% of line rate (on ADSL2+ where line rate rate is the net line rate (after FEC)) specifying too small an overhead the ICMP latency plot shows larger deviations from the expected unload RTT plus 10ms. Too large an overhead however just decreases the good put bait while leaving the latency well under control.

> 
> So it seems that overhead -4 is the correct thing to do.

	And thanks to your help I fully agree.

> 
> I also tested backlogged (-i 0.2) with -s 10 and 11 and tcpdump showed the correct deltas -
> 
> ph4[/mnt/sda8/Qos/stab-tests]# tcpdump -nnttti eth0 icmp and dst host noki
> 
> snip
> 
> 00:00:00.424000 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1345, seq 92, length 18
> 00:00:00.424000 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1345, seq 93, length 18
> 00:00:00.424000 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1345, seq 94, length 18
> 00:00:00.424000 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1345, seq 95, length 18
> 00:00:00.424000 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1345, seq 96, length 18
> 00:00:00.424001 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1345, seq 97, length 18
> 00:00:00.423999 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1345, seq 98, length 18
> 00:00:00.424000 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1347, seq 1, length 19
> 00:00:00.848000 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1347, seq 2, length 19
> 00:00:00.848001 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1347, seq 3, length 19
> 00:00:00.847999 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1347, seq 4, length 19
> 00:00:00.848000 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1347, seq 5, length 19
> 00:00:00.847999 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1347, seq 6, length 19
> 00:00:00.848002 IP 192.168.0.3 > 192.168.0.1: ICMP echo request, id 1347, seq 7, length 19
> 
> 

	I really appreciate your test script, thanks for taking the time.

Best Regards
	Sebastian