From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-gy0-f171.google.com (mail-gy0-f171.google.com [209.85.160.171]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 445A92006AE for ; Sat, 19 Nov 2011 12:33:53 -0800 (PST) Received: by ghbf1 with SMTP id f1so2541176ghb.16 for ; Sat, 19 Nov 2011 12:33:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=HasZtxTD92r1zxII5oO4cgVS/5FEShMpQNaYF4YruXw=; b=hKLOSoE8iIOtHh0hBuV/z6ZwHSqQ1ywNJXx7j+nxRwYtN36IUY9HAC/yQ6E4/TRz4C xZhQrdJV4rAYdxuSTwtFpjuADeZS4zG0nCEhLSXsgJX1m9BOm/Un+cgrEYdsV8LYrdUy 4H7pEqTXikEPWkakKcB1UjagPgy3oRTop/Y1M= MIME-Version: 1.0 Received: by 10.182.207.104 with SMTP id lv8mr1760984obc.36.1321734829551; Sat, 19 Nov 2011 12:33:49 -0800 (PST) Received: by 10.182.193.65 with HTTP; Sat, 19 Nov 2011 12:33:49 -0800 (PST) Date: Sat, 19 Nov 2011 21:33:49 +0100 Message-ID: From: Dave Taht To: Tom Herbert , bloat Content-Type: multipart/mixed; boundary=e89a8f6428f200af2604b21c5f9a Subject: [Bloat] some (very good) preliminary results from fiddling with byte queue limits on 100Mbit ethernet X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Nov 2011 20:33:53 -0000 --e89a8f6428f200af2604b21c5f9a Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Dear Tom (author of the byte queue limits patch) I have finally got out of 'embedded computing' mode and more into a place where I can hack on the kernel. (Not that I'm any good at it) So I extracted the current set of BQL related patches from the debloat-testing kernel and applied them to a recent linus-head (3.2-rc2 + a little) (they are at: http://www.teklibre.com/~d/tnq ) Now, the behavior that I had hoped for was that tx rate would be closely tied to completion rate, and the buffers on the device driver would more rarely fill. (it's a e1000e in my case - tx ring set to 256 by default, only reducable to 64 via ethtool) this is my standard latency under load test for a gigE ethernet card stepped down to 100Mbit. ethtool -s eth0 advertise 0x008 # force the device to 100Mbit ethtool -G eth0 tx 64 # Knock the ring buffer down as far as it can go # Plug the device in (which runs the attached script to configure the interface to ~'good' values) netperf -l 60 -H some_other_server # in my case cerowrt - which has a 4 buffer tx ring and a 8 buffer txqueuelen set at the moment - far too low) and ping some_other_server in another window. AND - YES! YES! YES! SUB 10ms inter-stream latencies. Ranging from 1.3ms to about 6ms, median around 4ms. I haven't seen latencies under load this good on 100Mbit ethernet since the DECchip tulip 21140! This is within the range you'd expect for SFQ's 'typical' bunching of packets. And a tiny fraction of tcp's speed is lost in the general case. I mean, it's so close to to what I'd get without the script as to be statistically insignificant. CPU load, hardly measurable... Now. Look at the script. When a link speed of < 101 mbit is detected: I set the byte queue limit to 3*mtu # lower and latencies get mildly lower and more unstable When without BQL... I tried to use cbq to set a bandwidth limit at 92mbit I added in sfq on top of that# the documentation for which is now wrong, there's no way to set a packet limit Without byte queue limits, latency under load goes to 130ms and stays there. EG - the default buffering in the ethernet driver defeats my attempt at controlling bandwidth with CBQ + SFQ entirely. With byte queue limits alone and the default pfifo fast qdisc... ... at mtu*3, we still end up with 130ms latency under load. :( With byte queue limits at mtu*3 + the SFQ qdisc, latency under load can be hammered down below 6ms when running at a 100Mbit line rate. No CBQ needed. When doing a reverse test (mostly data) - with cerowrt set to the above (insanely low values) I see similar response times to the above. netperf -l 60 -H 172.30.42.1 -t TCP_MAERTS Anyway, script could use improvement, and I'm busily patching BQL into the ag71xx driver as I write. Sorry it's taken me so long to get to this since your bufferbloat talks at linux plumbers. APPLAUSE. It's looking like BQL + SFQ is an effective means of improving fairness and reducing latency on drivers that can support it. Even if they have large tx rings that the hardware demands. More testing on more stuff is needed of course... I'd like to convince QFQ to work... #!/bin/sh # Starving the beast on ethernet v.000001 # Place this file in /etc/network/if-up.d NetworkManager will call it # for you automagically when the interface is brought up. # Today's ethernet device drivers are over-optimized for 1000Mbit # If you are unfortunate enough to run at less than that # you are going to lose on latency. As one example you will # have over 130ms latency under load with the default settings in the e1000= e # driver - common to many laptops. # To force your network device to 100Mbit # (so you can test and then bitch about bloat in your driver) # ethtool -s your_device advertise 0x008 # It will then stay stuck at 100Mbit until you change it back. # It also helps to lower your ring buffer as far as it will go # ethtool -G your_device tx 64 # or lower if you can # And after doing all that you wil be lucky to get 120ms latency under load= . # So I have also built byte queue limits into my kernels at # http://www.teklibre.com/~d/tnq # Adding in the below, without byte queue limits enabled, and cbq, gets you= to # around 12ms. With byte queue limits, I can get to ~4-6 ms latency under l= oad. # However, (less often of late), I sometimes end up at 130ms. # It would be my hope, with some more tuning (QFQ?), better SFQ setup? # to get below 1 ms. debloat_ethernet() { percent=3D92 txqueuelen=3D100 bytelimit=3D64000 speed=3D`cat /sys/class/net/$IFACE/speed` mtu=3D`ip -o link show dev $IFACE | awk '{print $5;}'` bytelimit=3D`expr $mtu '*' 3` [ $speed -lt 1001 ] && { percent=3D94; txqueuelen=3D100; } if [ $speed -lt 101 ] then percent=3D92; txqueuelen=3D50; fi #[ $speed -lt 11 ] && { percent=3D90; txqueuelen=3D20; } newspeed=3D`expr $speed \* $percent / 100` modprobe sch_cbq modprobe sch_sfq modprobe sch_qfq # I can't get QFQ to work # Doing this twice kicks the driver harder. Sometimes it gets stuck otherwi= se ifconfig $IFACE txqueuelen $txqueuelen tc qdisc del dev $IFACE root ifconfig $IFACE txqueuelen $txqueuelen tc qdisc del dev $IFACE root #tc qdisc add dev $IFACE root handle 1 cbq bandwidth ${newspeed}mbit avpkt = 1524 #tc qdisc add dev $IFACE parent 1: handle 10 sfq if [ -e /sys/class/net/$IFACE/queues/tx-0/byte_queue_limits ] then for i in /sys/class/net/$IFACE/queues/tx-*/byte_queue_limits do echo $bytelimit > $i/limit_max done tc qdisc add dev $IFACE handle 1 root sfq else tc qdisc add dev $IFACE root handle 1 cbq bandwidth ${newspeed}mbit avpkt 1= 524 tc qdisc add dev $IFACE parent 1: handle 10 sfq fi } debloat_wireless() { # HAH. Like any of this helps wireless exit percent=3D92 txqueuelen=3D100 speed=3D`cat /sys/class/net/$IFACE/speed` [ $speed -lt 1001 ] && { percent=3D94; txqueuelen=3D100; } [ $speed -lt 101 ] && { percent=3D93; txqueuelen=3D50; } [ $speed -lt 11 ] && { percent=3D90; txqueuelen=3D20; } newspeed=3D`expr $speed \* $percent / 100` #echo $newspeed modprobe sch_cbq modprobe sch_sfq modprobe sch_qfq # Just this much would help. If wireless had a 'speed' ifconfig $IFACE txqueuelen $txqueuelen } if [ -h /sys/class/net/$IFACE/phy80211 ] then debloat_wireless else debloat_ethernet fi --=20 Dave T=E4ht SKYPE: davetaht US Tel: 1-239-829-5608 FR Tel: 0638645374 http://www.bufferbloat.net --e89a8f6428f200af2604b21c5f9a Content-Type: application/octet-stream; name=debloat Content-Disposition: attachment; filename=debloat Content-Transfer-Encoding: base64 X-Attachment-Id: f_gv7293xe0 IyEvYmluL3NoCiMgU3RhcnZpbmcgdGhlIGJlYXN0IG9uIGV0aGVybmV0IHYuMDAwMDAxCiMgUGxh Y2UgdGhpcyBmaWxlIGluIC9ldGMvbmV0d29yay9pZi11cC5kIE5ldHdvcmtNYW5hZ2VyIHdpbGwg Y2FsbCBpdAojIGZvciB5b3UgYXV0b21hZ2ljYWxseSB3aGVuIHRoZSBpbnRlcmZhY2UgaXMgYnJv dWdodCB1cC4KCiMgVG9kYXkncyBldGhlcm5ldCBkZXZpY2UgZHJpdmVycyBhcmUgb3Zlci1vcHRp bWl6ZWQgZm9yIDEwMDBNYml0CiMgSWYgeW91IGFyZSB1bmZvcnR1bmF0ZSBlbm91Z2ggdG8gcnVu IGF0IGxlc3MgdGhhbiB0aGF0CiMgeW91IGFyZSBnb2luZyB0byBsb3NlIG9uIGxhdGVuY3kuIEFz IG9uZSBleGFtcGxlIHlvdSB3aWxsCiMgaGF2ZSBvdmVyIDEzMG1zIGxhdGVuY3kgdW5kZXIgbG9h ZCB3aXRoIHRoZSBkZWZhdWx0IHNldHRpbmdzIGluIHRoZSBlMTAwMGUKIyBkcml2ZXIgLSBjb21t b24gdG8gbWFueSBsYXB0b3BzLiAKCiMgVG8gZm9yY2UgeW91ciBuZXR3b3JrIGRldmljZSB0byAx MDBNYml0IAojIChzbyB5b3UgY2FuIHRlc3QgYW5kIHRoZW4gYml0Y2ggYWJvdXQgYmxvYXQgaW4g eW91ciBkcml2ZXIpCiMgZXRodG9vbCAtcyB5b3VyX2RldmljZSBhZHZlcnRpc2UgMHgwMDgKIyBJ dCB3aWxsIHRoZW4gc3RheSBzdHVjayBhdCAxMDBNYml0IHVudGlsIHlvdSBjaGFuZ2UgaXQgYmFj ay4KIyBJdCBhbHNvIGhlbHBzIHRvIGxvd2VyIHlvdXIgcmluZyBidWZmZXIgYXMgZmFyIGFzIGl0 IHdpbGwgZ28KIyBldGh0b29sIC1HIHlvdXJfZGV2aWNlIHR4IDY0ICMgb3IgbG93ZXIgaWYgeW91 IGNhbgojIEFuZCBhZnRlciBkb2luZyBhbGwgdGhhdCB5b3Ugd2lsIGJlIGx1Y2t5IHRvIGdldCAx MjBtcyBsYXRlbmN5IHVuZGVyIGxvYWQuCgojIFNvIEkgaGF2ZSBhbHNvIGJ1aWx0IGJ5dGUgcXVl dWUgbGltaXRzIGludG8gbXkga2VybmVscyBhdAojIGh0dHA6Ly93d3cudGVrbGlicmUuY29tL35k L3RucQoKIyBBZGRpbmcgaW4gdGhlIGJlbG93LCB3aXRob3V0IGJ5dGUgcXVldWUgbGltaXRzIGVu YWJsZWQsIGFuZCBjYnEsIGdldHMgeW91IHRvCiMgYXJvdW5kIDEybXMuIFdpdGggYnl0ZSBxdWV1 ZSBsaW1pdHMsIEkgY2FuIGdldCB0byB+NC02IG1zIGxhdGVuY3kgdW5kZXIgbG9hZC4KIyBIb3dl dmVyLCAobGVzcyBvZnRlbiBvZiBsYXRlKSwgSSBzb21ldGltZXMgZW5kIHVwIGF0IDEzMG1zLiAK IyBJdCB3b3VsZCBiZSBteSBob3BlLCB3aXRoIHNvbWUgbW9yZSB0dW5pbmcgKFFGUT8pLCBiZXR0 ZXIgU0ZRIHNldHVwPwojIHRvIGdldCBiZWxvdyAxIG1zLgoKZGVibG9hdF9ldGhlcm5ldCgpIHsK cGVyY2VudD05Mgp0eHF1ZXVlbGVuPTEwMApieXRlbGltaXQ9NjQwMDAKCnNwZWVkPWBjYXQgL3N5 cy9jbGFzcy9uZXQvJElGQUNFL3NwZWVkYAptdHU9YGlwIC1vIGxpbmsgc2hvdyBkZXYgJElGQUNF IHwgYXdrICd7cHJpbnQgJDU7fSdgCmJ5dGVsaW1pdD1gZXhwciAkbXR1ICcqJyAzYAoKWyAkc3Bl ZWQgLWx0IDEwMDEgXSAmJiB7IHBlcmNlbnQ9OTQ7IHR4cXVldWVsZW49MTAwOyB9CmlmIFsgJHNw ZWVkIC1sdCAxMDEgXSAKdGhlbgoJcGVyY2VudD05MjsgCgl0eHF1ZXVlbGVuPTUwOyAKCmZpCgoj WyAkc3BlZWQgLWx0IDExIF0gJiYgeyBwZXJjZW50PTkwOyB0eHF1ZXVlbGVuPTIwOyB9CgpuZXdz cGVlZD1gZXhwciAkc3BlZWQgXCogJHBlcmNlbnQgLyAxMDBgCgptb2Rwcm9iZSBzY2hfY2JxCm1v ZHByb2JlIHNjaF9zZnEKbW9kcHJvYmUgc2NoX3FmcSAjIEkgY2FuJ3QgZ2V0IFFGUSB0byB3b3Jr CgojIERvaW5nIHRoaXMgdHdpY2Uga2lja3MgdGhlIGRyaXZlciBoYXJkZXIuIFNvbWV0aW1lcyBp dCBnZXRzIHN0dWNrIG90aGVyd2lzZQoKaWZjb25maWcgJElGQUNFIHR4cXVldWVsZW4gJHR4cXVl dWVsZW4KdGMgcWRpc2MgZGVsIGRldiAkSUZBQ0Ugcm9vdAppZmNvbmZpZyAkSUZBQ0UgdHhxdWV1 ZWxlbiAkdHhxdWV1ZWxlbgoKdGMgcWRpc2MgZGVsIGRldiAkSUZBQ0Ugcm9vdAojdGMgcWRpc2Mg YWRkIGRldiAkSUZBQ0Ugcm9vdCBoYW5kbGUgMSBjYnEgYmFuZHdpZHRoICR7bmV3c3BlZWR9bWJp dCBhdnBrdCAxNTI0CiN0YyBxZGlzYyBhZGQgZGV2ICRJRkFDRSBwYXJlbnQgMTogaGFuZGxlIDEw IHNmcQoKaWYgWyAtZSAvc3lzL2NsYXNzL25ldC8kSUZBQ0UvcXVldWVzL3R4LTAvYnl0ZV9xdWV1 ZV9saW1pdHMgXQp0aGVuCglmb3IgaSBpbiAvc3lzL2NsYXNzL25ldC8kSUZBQ0UvcXVldWVzL3R4 LSovYnl0ZV9xdWV1ZV9saW1pdHMKCWRvCgkJZWNobyAkYnl0ZWxpbWl0ID4gJGkvbGltaXRfbWF4 Cglkb25lCgoJdGMgcWRpc2MgYWRkIGRldiAkSUZBQ0UgaGFuZGxlIDEgcm9vdCBzZnEKZWxzZQoK dGMgcWRpc2MgYWRkIGRldiAkSUZBQ0Ugcm9vdCBoYW5kbGUgMSBjYnEgYmFuZHdpZHRoICR7bmV3 c3BlZWR9bWJpdCBhdnBrdCAxNTI0CnRjIHFkaXNjIGFkZCBkZXYgJElGQUNFIHBhcmVudCAxOiBo YW5kbGUgMTAgc2ZxCgpmaQoKfQoKZGVibG9hdF93aXJlbGVzcygpIHsKCiMgSEFILiBMaWtlIGFu eSBvZiB0aGlzIGhlbHBzIHdpcmVsZXNzCgpleGl0CgpwZXJjZW50PTkyCnR4cXVldWVsZW49MTAw CnNwZWVkPWBjYXQgL3N5cy9jbGFzcy9uZXQvJElGQUNFL3NwZWVkYApbICRzcGVlZCAtbHQgMTAw MSBdICYmIHsgcGVyY2VudD05NDsgdHhxdWV1ZWxlbj0xMDA7IH0KWyAkc3BlZWQgLWx0IDEwMSBd ICYmIHsgcGVyY2VudD05MzsgdHhxdWV1ZWxlbj01MDsgfQpbICRzcGVlZCAtbHQgMTEgXSAmJiB7 IHBlcmNlbnQ9OTA7IHR4cXVldWVsZW49MjA7IH0KCm5ld3NwZWVkPWBleHByICRzcGVlZCBcKiAk cGVyY2VudCAvIDEwMGAKCiNlY2hvICRuZXdzcGVlZAoKbW9kcHJvYmUgc2NoX2NicQptb2Rwcm9i ZSBzY2hfc2ZxCm1vZHByb2JlIHNjaF9xZnEKCiMgSnVzdCB0aGlzIG11Y2ggd291bGQgaGVscC4g SWYgd2lyZWxlc3MgaGFkIGEgJ3NwZWVkJwppZmNvbmZpZyAkSUZBQ0UgdHhxdWV1ZWxlbiAkdHhx dWV1ZWxlbgoKfQoKaWYgWyAtaCAvc3lzL2NsYXNzL25ldC8kSUZBQ0UvcGh5ODAyMTEgXQp0aGVu CglkZWJsb2F0X3dpcmVsZXNzCmVsc2UKCWRlYmxvYXRfZXRoZXJuZXQKZmkKCg== --e89a8f6428f200af2604b21c5f9a--