[Bloat] SFB tuning
Otto Solares Cabrera
solca at guug.org
Mon May 30 19:37:24 EDT 2011
On Tue, May 31, 2011 at 12:05:51AM +0200, Juliusz Chroboczek wrote:
> I don't understand what you're doing on eth6, which has both prio and htb.
>
> You're systematically putting sfq below sfb. You should be aware that
> since sfb keeps the queues short, the effect of sfq is reduced somewhat
> -- you may not be getting all the fairness you're expecting.
Basically my University main router runs Linux with 9 GigE NICs to
different networks (we can't afford expensive routers :) ).
My idea was for the Internet interface to use 2 queues (or bands in
prio parlance) one for realtime or very important traffic and the
other for rest of traffic doing SFB (this is upload to Internet).
For every clients network I used 5 queues being the last one the
default, shaped and "fairnessed" (download from the Internet).
The WiFi network is a client network connected to GigE switches which
in turn connect to 125 Linux APs (WRT160NL with OpenWRT) in the entire
campus.
I've attached my "QoS" scripts so you can form an idea why some things
are done that way but I know is too much asking to take a look, my AQM
or QoS setup is a little elaborate.
("unhandled" bands in the prio qdisc are plain pfifo with qlen 10,
ip_qos is the main script which calls ip_qos_lan for every client net).
> Your packet loss rates are
>
> eth4 (Internet): 0.6%
> eth2 (LAN): 0.2%
> eth6 (Wifi): 3.6%
>
> Only eth6 is congested. Three quarters of the eth6 drops are in sfb
> 52:. There's 3.6 times more earlydrop than bucketdrop, which seems okay
> to me. Increasing increment/decrement might reduce the bucketdrop
> somewhat; so would increasing the target, at the cost of increasing the
> amount of queueing.
>
> Thanks again for the data,
Thank you for your time and analysis!
> P.S. Wow ! Guatemala !
You're welcome!
-
Otto
-------------- next part --------------
#!/bin/sh
#
# ip_qos
#
# UG QoS implementation
#
# Copyright (C)2009-2011, Universidad Galileo
# Otto Solares <solca at galileo.edu>
###############
# definitions #
###############
# Import definitions
. /etc/network/ip_defs
/etc/network/ip_qos_lan eth0 stop
/etc/network/ip_qos_lan eth1 stop
/etc/network/ip_qos_lan eth2 stop
/etc/network/ip_qos_lan eth3 stop
/etc/network/ip_qos_lan eth4 stop
/etc/network/ip_qos_lan eth5 stop
/etc/network/ip_qos_lan eth6 stop
/etc/network/ip_qos_lan eth7 stop
/etc/network/ip_qos_lan eth8 stop
if [ "$1" = "stop" ]; then
exit
fi
#######
# QoS #
#######
# LAN networks
${TC} qdisc add dev eth0 root handle 1: prio bands 2 priomap 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
# simple small packets (<128)
${TC} filter add dev eth0 parent 1: protocol ip prio 1 u32 \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xff80 at 2 \
flowid 1:1
# services
${TC} filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip dst 10.0.0.13 match ip dport 24 0xffff flowid 1:1
${TC} filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip protocol 17 0xff match ip dst 10.0.0.6 flowid 1:1
/etc/network/ip_qos_lan eth1 30000
/etc/network/ip_qos_lan eth2 30000
/etc/network/ip_qos_lan eth3 30000
/etc/network/ip_qos_lan eth6 30000
# WAN networks
# claro main
${TC} qdisc add dev eth4 root handle 1: prio bands 3 priomap 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
#${TC} qdisc add dev eth4 parent 1:2 handle 20: red limit 5000000 min 208333 max 625000 avpkt 1000 burst 347 probability 0.02 bandwidth 25000 ecn
${TC} qdisc add dev eth4 parent 1:3 handle 13: sfb hash-type source limit 100 target 10 max 15 penalty_rate 60
${TC} qdisc add dev eth4 parent 13: handle 131: sfq perturb 10
# simple small packets (<128)
${TC} filter add dev eth4 parent 1: protocol ip prio 1 u32 \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xff80 at 2 \
flowid 1:1
# services
${TC} filter add dev eth4 parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip sport 22 0xffff flowid 1:1
${TC} filter add dev eth4 parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip dport 22 0xffff flowid 1:1
${TC} filter add dev eth4 parent 1: protocol ip prio 1 u32 match ip protocol 17 0xff match ip src 200.9.255.13 flowid 1:1
${TC} filter add dev eth4 parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip src 200.9.255.13 match ip sport 24 0xffff flowid 1:1
${TC} filter add dev eth4 parent 1: protocol ip prio 1 u32 match ip protocol 17 0xff match ip src 200.9.255.6 flowid 1:1
${TC} filter add dev eth4 parent 1: protocol ip prio 1 u32 match ip protocol 17 0xff match ip src 200.9.255.69 flowid 1:1
${TC} filter add dev eth4 parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip src 200.9.255.69 match ip sport 554 0xffff flowid 1:1
${TC} filter add dev eth4 parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip src 200.9.255.69 match ip sport 1755 0xffff flowid 1:1
${TC} filter add dev eth4 parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip src 200.9.255.152 match ip sport 3389 0xffff flowid 1:1
# google-claro cache netblock
${TC} filter add dev eth4 parent 1: protocol ip prio 2 u32 match ip dst 200.6.228.0/24 flowid 1:2
# google netblock
${TC} filter add dev eth4 parent 1: protocol ip prio 2 u32 match ip dst 216.239.32.0/19 flowid 1:2
${TC} filter add dev eth4 parent 1: protocol ip prio 2 u32 match ip dst 64.233.160.0/19 flowid 1:2
${TC} filter add dev eth4 parent 1: protocol ip prio 2 u32 match ip dst 66.249.80.0/20 flowid 1:2
${TC} filter add dev eth4 parent 1: protocol ip prio 2 u32 match ip dst 72.14.192.0/18 flowid 1:2
${TC} filter add dev eth4 parent 1: protocol ip prio 2 u32 match ip dst 209.85.128.0/17 flowid 1:2
${TC} filter add dev eth4 parent 1: protocol ip prio 2 u32 match ip dst 66.102.0.0/20 flowid 1:2
${TC} filter add dev eth4 parent 1: protocol ip prio 2 u32 match ip dst 74.125.0.0/16 flowid 1:2
${TC} filter add dev eth4 parent 1: protocol ip prio 2 u32 match ip dst 64.18.0.0/20 flowid 1:2
${TC} filter add dev eth4 parent 1: protocol ip prio 2 u32 match ip dst 207.126.144.0/20 flowid 1:2
${TC} filter add dev eth4 parent 1: protocol ip prio 2 u32 match ip dst 173.194.0.0/16 flowid 1:2
${TC} filter add dev eth4 parent 1: protocol ip prio 2 u32 match ip dst 216.73.93.70/31 flowid 1:2
${TC} filter add dev eth4 parent 1: protocol ip prio 2 u32 match ip dst 216.73.93.72/31 flowid 1:2
# other physical links
${TC} qdisc add dev eth5 root pfifo
${TC} qdisc add dev eth7 root pfifo
${TC} qdisc add dev eth8 root pfifo
exit 0
-------------- next part --------------
#!/bin/sh
#
# ip_qos
#
# UG QoS implementation
# slightly based on WonderShaper
#
# Copyright (C)2009-2011, Universidad Galileo
# Otto Solares <solca at galileo.edu>
#
# Egress queues:
# 1. real-time priorities
# 2. internal LAN to LAN
# 3. external WAN (Internet) to LAN (unshaped)
# 4. google netblocks (unshaped)
# 5. external WAN (Internet) to LAN (shaped)
###############
# definitions #
###############
# Import definitions
. /etc/network/ip_defs
DEV=$1
# bandwidth for queue 5 shaping
BANDWIDTH=$2
if [ -z "$2" ]; then
exit 1
fi
if [ "$2" = "status" ]; then
${TC} -s qdisc ls dev $DEV
echo
${TC} -s class ls dev $DEV
exit
fi
${TC} qdisc del dev $DEV root >/dev/null 2>&1
${TC} qdisc del dev $DEV ingress >/dev/null 2>&1
if [ "$2" = "stop" ]; then
exit
fi
### egress qdiscs ###
# root egress qdisc
${TC} qdisc add dev ${DEV} root handle 1: prio bands 5 priomap 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
# bands:
# 0 real-time
# 1 internal LAN to LAN
# 2 external WAN (Internet) to LAN unshaped
# 3 google netblock
# 4 external WAN (Internet) to LAN shaped
# external WAN (Internet) to LAN shaped
${TC} qdisc add dev ${DEV} parent 1:5 handle 50: htb default 30 r2q 25
${TC} class add dev ${DEV} parent 50: classid 50:1 htb rate ${BANDWIDTH}kbit ceil ${BANDWIDTH}kbit burst 2k
${TC} class add dev ${DEV} parent 50:1 classid 50:10 htb rate ${BANDWIDTH}kbit ceil ${BANDWIDTH}kbit burst 2k prio 1
${TC} class add dev ${DEV} parent 50:1 classid 50:20 htb rate $[9*${BANDWIDTH}/10]kbit ceil ${BANDWIDTH}kbit burst 2k prio 2
${TC} class add dev ${DEV} parent 50:1 classid 50:30 htb rate $[5*${BANDWIDTH}/10]kbit ceil ${BANDWIDTH}kbit burst 2k prio 3
${TC} qdisc add dev ${DEV} parent 50:20 handle 52: sfb hash-type dest limit 100 target 10 max 15 penalty_rate 100
${TC} qdisc add dev ${DEV} parent 52: handle 521: sfq perturb 10
${TC} qdisc add dev ${DEV} parent 50:30 handle 53: sfb hash-type dest limit 100 target 10 max 15 penalty_rate 100
${TC} qdisc add dev ${DEV} parent 53: handle 531: sfq perturb 10
### classify filters ###
# real-time
# ICMP & TCP ACKs & small (<512) UDP/UDPlite
${TC} filter add dev ${DEV} parent 1: protocol ip prio 1 u32 \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xff80 at 2 \
flowid 1:1
# voip.galileo.edu
${TC} filter add dev ${DEV} parent 1: protocol ip prio 1 u32 match ip protocol 17 0xff match ip src 10.0.0.6 flowid 1:1
# medialab.galileo.edu
${TC} filter add dev ${DEV} parent 1: protocol ip prio 1 u32 match ip protocol 17 0xff match ip src 192.168.15.10 flowid 1:1
${TC} filter add dev ${DEV} parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip src 192.168.15.10 match ip sport 554 0xffff flowid 1:1
${TC} filter add dev ${DEV} parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip src 192.168.15.10 match ip sport 1755 0xffff flowid 1:1
${TC} filter add dev ${DEV} parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip src 192.168.15.10 match ip dport 7007 0xffff flowid 1:1
# home.galileo.edu
${TC} filter add dev ${DEV} parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip src 192.168.0.4 match ip sport 22 0xffff flowid 1:1
if [ "${DEV}" == "eth3" ]; then
# medialab.galileo.edu
${TC} filter add dev ${DEV} parent 1: protocol ip prio 1 u32 match ip protocol 17 0xff match ip dst 192.168.15.10 flowid 1:1
${TC} filter add dev ${DEV} parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip dst 192.168.15.10 match ip dport 554 0xffff flowid 1:1
${TC} filter add dev ${DEV} parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip dst 192.168.15.10 match ip dport 1755 0xffff flowid 1:1
${TC} filter add dev ${DEV} parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip dst 192.168.15.10 match ip sport 7007 0xffff flowid 1:1
# home.galileo.edu
${TC} filter add dev ${DEV} parent 1: protocol ip prio 1 u32 match ip protocol 6 0xff match ip dst 192.168.0.4 match ip dport 22 0xffff flowid 1:1
fi
# internal LAN to LAN
${TC} filter add dev ${DEV} parent 1: protocol ip prio 2 u32 match ip src 10.0.0.0/8 flowid 1:2
${TC} filter add dev ${DEV} parent 1: protocol ip prio 2 u32 match ip src 172.16.0.0/12 flowid 1:2
${TC} filter add dev ${DEV} parent 1: protocol ip prio 2 u32 match ip src 192.168.0.0/16 flowid 1:2
${TC} filter add dev ${DEV} parent 1: protocol ip prio 2 u32 match ip src 224.0.0.0/4 flowid 1:2
# external WAN (Internet) to LAN unshaped
# some WAN links must not be shaped
${TC} filter add dev ${DEV} parent 1: protocol ip prio 3 u32 match ip src 0.0.0.0/0 indev eth5 flowid 1:3
${TC} filter add dev ${DEV} parent 1: protocol ip prio 3 u32 match ip src 0.0.0.0/0 indev eth8 flowid 1:3
# netbooks
#if [ "${DEV}" == "eth6" ]; then
# ${TC} filter add dev ${DEV} parent 1: protocol ip prio 3 u32 match ip protocol 17 0xff match ip dst 10.1.1.120 flowid 1:3
# ${TC} filter add dev ${DEV} parent 1: protocol ip prio 3 u32 match ip protocol 6 0xff match ip dst 10.1.1.120 flowid 1:3
#fi
# google-claro cache netblock must not be shaped
${TC} filter add dev ${DEV} parent 1: protocol ip prio 4 u32 match ip src 200.6.228.0/24 flowid 1:4
# google netblocks must not be shaped
${TC} filter add dev ${DEV} parent 1: protocol ip prio 4 u32 match ip src 216.239.32.0/19 flowid 1:4
${TC} filter add dev ${DEV} parent 1: protocol ip prio 4 u32 match ip src 64.233.160.0/19 flowid 1:4
${TC} filter add dev ${DEV} parent 1: protocol ip prio 4 u32 match ip src 66.249.80.0/20 flowid 1:4
${TC} filter add dev ${DEV} parent 1: protocol ip prio 4 u32 match ip src 72.14.192.0/18 flowid 1:4
${TC} filter add dev ${DEV} parent 1: protocol ip prio 4 u32 match ip src 209.85.128.0/17 flowid 1:4
${TC} filter add dev ${DEV} parent 1: protocol ip prio 4 u32 match ip src 66.102.0.0/20 flowid 1:4
${TC} filter add dev ${DEV} parent 1: protocol ip prio 4 u32 match ip src 74.125.0.0/16 flowid 1:4
${TC} filter add dev ${DEV} parent 1: protocol ip prio 4 u32 match ip src 64.18.0.0/20 flowid 1:4
${TC} filter add dev ${DEV} parent 1: protocol ip prio 4 u32 match ip src 207.126.144.0/20 flowid 1:4
${TC} filter add dev ${DEV} parent 1: protocol ip prio 4 u32 match ip src 173.194.0.0/16 flowid 1:4
${TC} filter add dev ${DEV} parent 1: protocol ip prio 4 u32 match ip src 216.73.93.70/31 flowid 1:4
${TC} filter add dev ${DEV} parent 1: protocol ip prio 4 u32 match ip src 216.73.93.72/31 flowid 1:4
# everything else must be shaped
${TC} filter add dev ${DEV} parent 1: protocol ip prio 5 u32 match ip src 0.0.0.0/0 indev eth4 flowid 1:5
# shaped priorities
# IP ToS Minimize-Delay & UDP & UDPlite
${TC} filter add dev ${DEV} parent 50: protocol ip prio 1 u32 match ip tos 0x10 0xff flowid 50:10
${TC} filter add dev ${DEV} parent 50: protocol ip prio 1 u32 match ip protocol 17 0xff flowid 50:10
${TC} filter add dev ${DEV} parent 50: protocol ip prio 1 u32 match ip protocol 136 0xff flowid 50:10
# HTTP
${TC} filter add dev ${DEV} parent 50: protocol ip prio 1 u32 match ip sport 80 0xffff flowid 50:20
${TC} filter add dev ${DEV} parent 50: protocol ip prio 1 u32 match ip dport 80 0xffff flowid 50:20
# HTTPS
${TC} filter add dev ${DEV} parent 50: protocol ip prio 1 u32 match ip sport 443 0xffff flowid 50:20
${TC} filter add dev ${DEV} parent 50: protocol ip prio 1 u32 match ip dport 443 0xffff flowid 50:20
More information about the Bloat
mailing list