latency limiting to 4ms?

Mon Feb 28 19:55:00 EST 2011

Hi Dave,

I started to think of what one who doesn't know much about kernel
specifics but has some practical experience (growing an 802.11 WAN
over a decade) would say.

Here is a bit of real-world information maybe related to buffer-bloat
and buffers.  It may be interesting as my connection (unique* rural
broadband** - see below) is from a MacBook Pro via an Airport to
several different "hosts" which I've abbreviated in caps:

LM - the local MacBook that I am testing from
AP - the airport, configured a/n and operating in 802.11n mode on 5.8GHz
RM - a mac mini connected to the GB ethernet side of the airport
VM1 - A virtual ubuntu machine running in Oracle's Virtual Box on RM
VM2 - A virtual debian machine running in VMware Fusion on RM
GW - my internet gateway (a 900MHz tranzeo client)
GG - one of google's machines
AV - an ubuntu amazon cloud tiny instance in Virginia

An additional note is that the AP ethernet (Gb) is connected to the GW
 via a 10/100 switch and from the GW is s 900MHz hop about 300m
through scrub  to a radio which links at 2.4GHz to a radio about 4km
away and then via a 3.5MHz link, another 4km to a length of fibre…
not sure about the exact details, but the E10 on the fibre runs about
60km and then goes about 100km via microwave which is not actually a
straight line, *but an L-shaped link that actually bounces off a
passive reflector (looks like a billboard) on a convenient mountain
peak.  My connection is then rate limited by equipment at that point
to 1Mbps down/512kbps up (not quit IC definition of broadband) and it
then continues on fibre to an "outdated switch" and from there via
fibre to on of the internet exchanges and off to google.  (I could
have a bit wrong, but should let you know the connection is a bit
unique).  **Final note -- when the E10 "saturates" surfing is painful,
utube can't be watched in realtime, and voip gets pretty unusable -
thankfully people stop using it and it gets better :D

Here are the relevant kernels:
~~~~~~~~~~~~~~~~~~~~~~~

LM~  uname -a (10.0.1.4)
 Darwin LM-MacBook-Pro.local 10.4.1 Darwin Kernel Version 10.4.1: Fri
Jul 16 23:04:20 PDT 2010; root:xnu-1504.7.51~1/RELEASE_I386 i386

RM~ uname -a (10.0.1.3)
 Darwin RM-mac-mini.local 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul
15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386 i386

VM1~ ssh -l root 10.0.1.5 uname -a
 Linux vm1 2.6.35-22-generic #33-Ubuntu SMP Sun Sep 19 20:34:50 UTC
2010 i686 GNU/Linux

VM2~ ssh -l root 10.0.1.27 uname -a
 Linux vm2 2.6.26-2-686 #1 SMP Mon Jun 21 05:58:44 UTC 2010 i686 GNU/Linux

AV ~ uname -a
Linux AV 1 2.6.35-24-virtual #42-Ubuntu SMP Thu Dec 2 05:01:52 UTC
2010 i686 GNU/Linux

Here are some latencies.  The method I used to saturate the link was
to scp two files at the same time in each direction between LM and
VM1.  Aggregate average throughput for the scp transfers was 9.8MBps.

Baseline latencies:  (802.11n unloaded)
------------------------------------------

LM ~ --- 10.0.1.4 ping statistics ---
28 packets transmitted, 28 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.047/0.104/0.133/0.023 ms

AP ~ --- 10.0.1.1 ping statistics ---
12 packets transmitted, 12 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.941/2.791/3.087/0.560 ms

RM ~ --- 10.0.1.3 ping statistics ---
20 packets transmitted, 20 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.447/3.759/5.691/0.488 ms

VM1 ~ --- 10.0.1.5 ping statistics ---
16 packets transmitted, 16 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.076/3.582/5.622/0.578 ms

VM2 ~ --- 10.0.1.27 ping statistics ---
14 packets transmitted, 14 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.281/3.426/5.513/0.981 ms

GW ~ --- 192.168.100.1 ping statistics ---
8 packets transmitted, 8 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.215/3.444/3.747/0.186 ms

GG ~ --- www.l.google.com ping statistics ---
6 packets transmitted, 6 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 45.662/50.934/54.759/2.748 ms

AV --- 50.70.250.xx ping statistics ---
9 packets transmitted, 9 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 107.358/116.404/133.579/7.394 ms

Latencies Under Load (802.11n loaded by bi-directional scp):
========================================

LM ~ --- 10.0.1.4 ping statistics ---
33 packets transmitted, 33 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.025/0.081/0.153/0.041 ms

AP ~ --- 10.0.1.1 ping statistics ---
27 packets transmitted, 27 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.797/14.706/36.340/13.125 ms

RM ~ --- 10.0.1.3 ping statistics ---
59 packets transmitted, 59 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.698/20.292/51.019/13.059 ms

VM1 ~ --- 10.0.1.5 ping statistics ---
23 packets transmitted, 23 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.237/11.516/27.664/6.489 ms

VM2 ~ --- 10.0.1.27 ping statistics ---
24 packets transmitted, 24 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.316/16.195/32.407/7.777 ms

GW ~ --- 192.168.100.1 ping statistics ---
28 packets transmitted, 28 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.299/22.965/51.203/13.779 ms

GG ~ --- www.l.google.com ping statistics ---
19 packets transmitted, 19 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 55.531/73.927/91.541/10.198 ms

AV~ --- 50.70.250.xx ping statistics ---
16 packets transmitted, 16 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 117.986/138.770/169.439/13.535 ms

Maybe this is helpful to someone?

Devin

On Mon, Feb 28, 2011 at 5:19 AM, Dave Täht <d at taht.net> wrote:
>
> I'm about to get on a plane but I saw this thread on lkml go by:
>
> http://groups.google.com/group/linux.kernel/browse_thread/thread/b0add1502aafa46f/28dd3798b3d5e4f9?lnk=raot
>
> Where the proposed sched_fifo_ewma.c code in that thread:
>
> "Well, with 10ms buffer timeout latency goes to 10-20ms on 54Mbit wifi
> link (zd1211rw driver) from >500ms (ping rtt when iperf running same
> time). So for that it's good enough. "
>
> And, well, I'd like to go for 4ms across the entire transmit
> range. 10-20ms is *just barely* on the wrong side for voip and audio
> applications, and the above stat is being measured at 54Mbit.
>
> Someone care to leap in over there?
>
>
> --
> Dave Taht
> http://nex-6.taht.net
> _______________________________________________
> Bloat-devel mailing list
> Bloat-devel at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat-devel
>