[Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller]

HI Mikael,


On Jun 29, 2015, at 10:09 , Mikael Abrahamsson <swmike at swm.pp.se> wrote:

> On Mon, 29 Jun 2015, Sebastian Moeller wrote:
> 
>> 	Ah, I see, you are still using tc’s stab mechanism for account of per packet overhead and link layer adjustments instead of cake’s (you need to check the advanced options check box in the link layer adjustments tab, and then select “cake” as the lnk layer adjustment mechanism).
> 
> Ok, so when I changed "stab" to cake, I get the following, but it still only shapes one way. If I change to discipline to "fq_codel" or "pie" I get bidirectonal shaping with otherwise same settings.

	Probabkr breakage I introduced...

> 
> root at OpenWrt:~# cat /etc/config/sqm
> 
> config queue 'eth1'
> 	option interface 'eth0'
> 	option qdisc_advanced '1'
> 	option squash_dscp '0'
> 	option squash_ingress '0'
> 	option ingress_ecn 'ECN'
> 	option egress_ecn 'ECN'
> 	option qdisc_really_really_advanced '0'
> 	option download '50000'
> 	option upload '50000'
> 	option linklayer 'ethernet'
> 	option overhead '42'
> 	option linklayer_advanced '1'
> 	option tcMTU '2047'
> 	option tcTSIZE '128'
> 	option tcMPU ‘0'

	we can ignore the last 3 savely.

> 	option linklayer_adaptation_mechanism 'cake'
> 	option enabled '1'
> 	option qdisc 'cake'
> 	option script 'simple.qos’

	Okay, it really is trying cake’s LLA now.

> 
> root at OpenWrt:~# tc -d qdisc
> qdisc cake 8002: dev eth0 root refcnt 9 bandwidth 50Mbit diffserv4 flows noatm overhead 42

	And that works on egress, excellent so bot sch_cake and tc are recent enough.

> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn

	Oops, ingress is not right, yet.

> 
> root at OpenWrt:~# tc -d class show dev eth0
> class cake 8002:4e1 parent 8002:
> class cake 8002:b89 parent 8002:
> 
> root at OpenWrt:~# tc -d class show dev ifb4eth0
> class fq_codel :12c parent none
> class fq_codel :222 parent none
> 
> root at OpenWrt:~# /etc/init.d/sqm restart
> SQM: Trying to start/stop SQM on all interfaces.
> SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0
> SQM: ifb associated with interface eth0:
> SQM: trying to create new IFB: ifb4eth0
> RTNETLINK answers: File exists
> SQM: /usr/lib/sqm/stop.sh: Stopping eth0
> SQM: ifb associated with interface eth0:
> SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos
> + . /usr/lib/sqm/functions.sh
> + [ -z 50000 ]
> + [ -z 50000 ]
> + [ -z eth0 ]
> + [ -z cake ]
> + [ -z cake ]
> + [ -z ethernet ]
> + [ -z 42 ]
> + [ -z 2047 ]
> + [ -z 0 ]
> + [ -z 128 ]
> + [ -z  ]
> + AUTOFLOW=0
> + [ -z  ]
> + LIMIT=1001
> + [ -z  ]
> + ILIMIT=
> + [ -z  ]
> + ELIMIT=
> + [ -z  ]
> + ITARGET=
> + [ -z  ]
> + ETARGET=
> + [ -z ECN ]
> + [ -z ECN ]
> + [ -z 0 ]
> + [ -z 0 ]
> + [ -z  ]
> + IQDISC_OPTS=
> + [ -z  ]
> + EQDISC_OPTS=
> + [ -z  ]
> + which tc
> + TC=/usr/sbin/tc
> + [ -z  ]
> + which ip
> + IP=/usr/sbin/ip
> + [ -z  ]
> + which insmod
> + INSMOD=/usr/sbin/insmod
> + [ -z  ]
> + TARGET=5ms
> + [ -z  ]
> + IPT_MASK=0xff
> + [ -z  ]
> + IPT_MASK_STRING=/0xff
> + [ -z  ]
> + get_ifb_for_if eth0
> + CUR_IF=eth0
> + get_ifb_associated_with_if eth0
> + CUR_IF=eth0
> + tc+  -p filtergrep -o -e ifb[^)]\+
> show parent ffff: dev eth0
> + CUR_IFB=
> + sqm_logger ifb associated with interface eth0:
> + logger -t SQM -s ifb associated with interface eth0:
> SQM: ifb associated with interface eth0:
> + echo
> + CUR_IFB=
> + [ -z  ]
> + create_new_ifb_for_if eth0
> + CUR_IF=eth0
> + MAX_IF_NAME_LENGTH=15
> + IFB_PREFIX=ifb4
> + NEW_IFB=ifb4eth0
> + IFB_NAME_LENGTH=8
> + [ 8 -gt 15 ]
> + sqm_logger trying to create new IFB: ifb4eth0
> + logger -t SQM -s trying to create new IFB: ifb4eth0
> SQM: trying to create new IFB: ifb4eth0
> + /usr/sbin/ip link add name ifb4eth0 type ifb
> RTNETLINK answers: File exists
> + echo ifb4eth0
> + CUR_IFB=ifb4eth0
> + [ -z ifb4eth0 ]
> + echo ifb4eth0
> + DEV=ifb4eth0
> + do_modules
> + insmod act_ipt
> + lsmod+ grep
> -q ^act_ipt
> + insmod sch_cake
> + + grep -q ^sch_cake
> lsmod
> + insmod sch_ingress
> + + greplsmod
> -q ^sch_ingress
> + insmod act_mirred
> + + greplsmod -q ^act_mirred
> 
> + insmod cls_fw
> + + grep -q ^cls_fw
> lsmod
> + insmod sch_htb
> + + lsmodgrep -q ^sch_htb
> 
> + ipt_setup
> + ipt -t mangle -N QOS_MARK_eth0
> + echo -t mangle -N QOS_MARK_eth0
> + sed s/-A/-D/g
> + d=-t mangle -N QOS_MARK_eth0
> + [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
> + echo -t mangle -N QOS_MARK_eth0
> + sed s/-I/-D/g
> + d=-t mangle -N QOS_MARK_eth0
> + [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
> + iptables -t mangle -N QOS_MARK_eth0
> + ip6tables -t mangle -N QOS_MARK_eth0
> + sqm_logger cake does all the diffserv work - no need for iptables rules
> + logger -t SQM -s cake
> SQM: cake
> + [ 0 = 1 ]
> + sqm_logger Keeping differentiated services code points (DSCP) from ingress.
> + logger -t SQM -s Keeping differentiated services code points (DSCP) from ingress.
> SQM: Keeping differentiated services code points (DSCP) from ingress.
> + CAKE_OPTS=
> + ipt -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-A/-D/g
> -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-I/-D/g
> -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ipt -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-A/-D/g
> -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-I/-D/g
> -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ipt -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + echo+ sed s/-A/-D/g
> -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + d=-t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + [ -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff != -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
> + echo+ sed s/-I/-D/g
> -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + d=-t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + [ -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff != -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
> + iptables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ip6tables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + iptables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ip6tables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ipt -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + echo+ sed s/-A/-D/g
> -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + d=-t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 ]
> + iptables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + ip6tables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + echo+ sed s/-I/-D/g
> -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + d=-t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 ]
> + iptables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + ip6tables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ 50000 -ne 0 ]
> + egress
> + CEIL=50000
> + expr 50000 / 3
> + PRIO_RATE=16666
> + expr 50000 / 6
> + BE_RATE=8333
> + expr 50000 / 6
> + BK_RATE=8333
> + expr 50000 - 16
> + BE_CEIL=49984
> + get_mtu eth0 50000
> + BW=50000
> + cat /sys/class/net/eth0/mtu
> + F=1500
> + [ -z 1500 ]
> + [ 50000 -gt 20000 ]
> + F=3000
> + [ 50000 -gt 30000 ]
> + F=6000
> + [ 50000 -gt 40000 ]
> + F=12000
> + [ 50000 -gt 50000 ]
> + [ 50000 -gt 60000 ]
> + [ 50000 -gt 80000 ]
> + echo 12000
> + LQ=quantum 12000
> + /usr/sbin/tc qdisc del dev eth0 root
> + get_stab_string
> + STABSTRING=
> + [ cake = tc_stab -a ethernet != none ]
> + echo
> + get_cake_lla_string
> + STABSTRING=
> + [ cake = cake -a ethernet != none ]
> + [ ethernet = atm ]
> + STABSTRING= overhead 42
> + sqm_logger cake link layer adjustments:  overhead 42
> + logger -t SQM -s cake link layer adjustments: overhead 42
> SQM: cake link layer adjustments: overhead 42
> + sqm_logger SQM:  overhead 42
> + logger -t SQM -s SQM: overhead 42
> SQM: SQM: overhead 42
> + echo overhead 42
> + /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead 42

	As seen above egress works.

> + sqm_logger egress shaping activated
> + logger -t SQM -s egress shaping activated
> SQM: egress shaping activated
> + [ 50000 -ne 0 ]
> + ingress
> + CEIL=50000
> + expr 50000 / 3
> + PRIO_RATE=16666
> + expr 50000 / 6
> + BE_RATE=8333
> + expr 50000 / 6
> + BK_RATE=8333
> + expr 50000 - 16
> + BE_CEIL=49984
> + get_mtu eth0 50000
> + BW=50000
> + cat /sys/class/net/eth0/mtu
> + F=1500
> + [ -z 1500 ]
> + [ 50000 -gt 20000 ]
> + F=3000
> + [ 50000 -gt 30000 ]
> + F=6000
> + [ 50000 -gt 40000 ]
> + F=12000
> + e 50000 -gt 50000 ]
> + [ 50000 -gt 60000 ]
> + [ 50000 -gt 80000 ]
> + echo 12000
> + LQ=quantum 12000
> + /usr/sbin/tc qdisc del dev eth0 handle ffff: ingress
> + /usr/sbin/tc qdisc add dev eth0 handle ffff: ingress
> + /usr/sbin/tc qdisc del dev ifb4eth0 root
> + [ 0 = 1 ]
> + sqm_logger Perform DSCP based filtering on ingress. (3-tier classification)
> + logger -t SQM -s Perform DSCP based filtering on ingress. (3-tier classification)
> SQM: Perform DSCP based filtering on ingress. (3-tier classification)
> + get_stab_string
> + STABSTRING=
> + [ cake = tc_stab -a ethernet != none ]
> + echo
> + get_cake_lla_string
> + STABSTRING=
> + [ cake = cake -a ethernet != none ]
> + [ ethernet = atm ]
> + STABSTRING= overhead 42
> + sqm_logger cake link layer adjustments:  overhead 42
> + logger -t SQM -s cake link layer adjustments: overhead 42
> SQM: cake link layer adjustments: overhead 42
> + sqm_logger SQM:  overhead 42
> + logger -t SQM -s SQM: overhead 42
> SQM: SQM: overhead 42
> + echo overhead 42
> + /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead 42

	As seen above egress works. What the F. There is an embarrassing copy and paste error in simple.qosline 180, instead of:
$TC qdisc add dev $IFACE root `get_stab_string` $QDISC bandwidth ${DOWNLINK}kbit `get_cake_lla_string` $CAKE_OPTS ${IQDISC_OPTS}

this line should obviously read:
$TC qdisc add dev $DEV root `get_stab_string` $QDISC bandwidth ${DOWNLINK}kbit `get_cake_lla_string` $CAKE_OPTS ${IQDISC_OPTS}

so simple.qos set-up egress twice, once with the egress parameters and once with the ingress parameters, I guess I did not test 3-tier ingress classification on Dave’s test machine.
Also I should rename a few variables. $DEV and $IFACE are not the best names. (Also cake needs its own .qos file(s) instead of simple and simplest becoming rather “complex”…) Thanks for keeping testing this.

Best Regards
	Sebastian

> RTNETLINK answers: File exists
> + ifconfig ifb4eth0 up
> + /usr/sbin/tc filter add dev eth0 parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth0
> + sqm_logger ingress shaping activated
> + logger -t SQM -s ingress shaping activated
> SQM: ingress shaping activated




> 
> -- 
> Mikael Abrahamsson    email: swmike at swm.pp.se