[Cerowrt-devel] cerowrt 3.3.8-17: nice latency improvements, some issues with bind

Sat Aug 18 02:38:01 PDT 2012

On 08/17/2012 09:05 PM, Dave Taht wrote:
> I'm widening the distribution of this email a little bit in light of
> the benchmark results (somewhat too far) below, and some of the other
> issues raised.
> 
> On Fri, Aug 17, 2012 at 1:52 AM, Török Edwin
>> Onto the good news, here are some measurements (ping time / bandwidth from my laptop connected through WiFi to my desktop connected through GbitE):
>>
>> no fq_codel on laptop, openwrt, wlan0 5Ghz: 0.859/174.859/923.768/198.308 ms; 120 - 140Mbps
>> w/ fq_codel on laptop, openwrt, wlan0 5Ghz: 1.693/ 26.727/ 54.936/ 11.746 ms; 120 - 140Mbps
>> no fq_codel on laptop, cerowrt, wlan0 5Ghz: 2.310/ 15.183/140.495/ 30.337 ms; 75 - 85 Mbps
>> w/ fq_codel on laptop, cerowrt, wlan0 5Ghz: 1.464/  1.981/  2.223/  0.221 ms; 75 - 85 Mbps
>>
>> The latency improvement is awesome, and I don't really mind the sacrificed bandwidth to accomplish it.
> 
> A man after my own heart.
> 
> Thx! The industry as a whole has been focused on "bandwidth at any
> cost, including massive latency", which leads to things like the ~1
> second delays you observed on your fq_codel-less test. (and far worse
> has been observed in the field) We're focused on improving latency,
> because as stuart cheshire says: "once you have latency, you can't get
> it back"

Yeah latency is most annoying thing on wireless (even more so on 3G),
if I really want more LAN bandwidth I can just plug in the Ethernet cable :)
Although I measured nttcp -r too now (see below) and the bandwidth drop is quite significant there,
can only do 39 Mbps.

> 
> We hope that once some other concepts prove out, we can keep the low
> latency and add even more bandwidth back.
> 
> http://www.bufferbloat.net/projects/cerowrt/wiki/Fq_Codel_on_Wireless
> 
> In day-to-day use the lowered latency and jitter in cero currently can
> be really "felt" particularly in applications like skype and google
> hangouts, and web pages (under load) feel much faster, as DNS lookups
> happen really fast...
> 
> and (as another example), things like youtube far more rarely "stall out".
> 
> It's kind of hard to measure "feel", though. I wish we had better
> benchmarks to show what we're accomplishing.
> 
>> Is the bandwidth drop intended though? When enabling fq_codel just on my laptop I didn't notice any bandwidth drop at all.
> 
> The core non-fq_codel change on cerowrt vs openwrt and/or your laptop
> is that the aggregation buffer size at the driver level has been
> severely reduced in cerowrt, from it's default of 128 buffers, to 3.
> This means that the maximum aggregate size has been cut to 3 packets
> from ~42, but more importantly, total outstanding buffers not managed
> by codel to 3, rather than 128....
> 
> The fact that this costs so little bandwidth (40%) in exchange for
> reducing latency and jitter by 25x (or 400x compared to no fq_codel at
> all) suggests that in the long run, once we come up with fixes to the
> mac80211 layer, we will be able to achieve better utilization,
> latency, AND jitter overall than the current hw deployed everywhere.
> 
> IF you'd like to have more bandwidth back, you can jiggle the qlen_*
> variables in the debloat script up, but remember that tcp's reaction
> time is quadratic as to the amount of buffering. I'd be interested in
> you repeating your benchmark after doing that? The difference between
> 3 buffers and 8 is pretty dramatic...

Here are some measurements (50 pings, nttcp -D -n200000)

Setup:
 laptop with Linux 3.5.0, iwl4965 AGN, 5.22 Ghz (channel 44) only radio on 5GHz spectrum.
 desktop with Linux 3.5.1 running nttcp -i, connected through GbE to cerowrt router

Baseline (only ping, no other traffic):            0.806/ 1.323/  8.753/ 1.333 ms
no fq_codel on laptop, cerowrt defaults, nttcp -t: 1.192/16.605/107.351/25.265 ms;  94 Mbps
no fq_codel on laptop, cerowrt qlen_*=4, nttcp -t: 1.285/25.108/105.519/22.607 ms; 107 Mbps
no fq_codel on laptop, cerowrt qlen_*=5, nttcp -t: 1.118/15.538/ 68.262/15.773 ms; 104 Mbps
no fq_codel on laptop, cerowrt qlen_*=6, nttcp -t: 1.353/15.592/116.026/18.782 ms; 113 Mbps
no fq_codel on laptop, cerowrt qlen_*=7, nttcp -t: 1.344/18.719/112.757/23.691 ms; 113 Mbps
no fq_codel on laptop, cerowrt qlen_*=8, nttcp -t: 1.717/20.842/ 59.412/16.120 ms; 127 Mbps
no fq_codel on laptop, cerowrt qlen_*=9, nttcp -t: 1.663/21.406/141.338/26.270 ms; 123 Mbps
no fq_codel on laptop, cerowrt qlen_*=10,nttcp -t: 1.818/21.361/117.023/20.727 ms; 120 Mbps
no fq_codel on laptop, cerowrt qlen_*=12,nttcp -t: 2.195/24.277/131.490/21.161 ms; 127 Mbps

no fq_codel on laptop, cerowrt defaults, nttcp -r: 1.332/ 3.129/ 41.900/ 5.221 ms;  39 Mbps
no fq_codel on laptop, cerowrt qlen_*=4, nttcp -r: 1.514/ 3.205/  8.595/ 1.817 ms;  46 Mbps
no fq_codel on laptop, cerowrt qlen_*=5, nttcp -r: 1.342/ 7.250/114.095/17.591 ms;  52 Mbps
no fq_codel on laptop, cerowrt qlen_*=6, nttcp -r: 1.299/ 3.200/ 12.080/ 1.972 ms;  58 Mbps
no fq_codel on laptop, cerowrt qlen_*=7, nttcp -r: 1.617/ 5.220/ 76.478/10.971 ms;  63 Mbps
no fq_codel on laptop, cerowrt qlen_*=8, nttcp -r: 1.810/ 4.267/ 24.560/ 4.065 ms;  67 Mbps
no fq_codel on laptop, cerowrt qlen_*=9, nttcp -r: 2.015/ 5.090/ 78.256/10.710 ms;  71 Mbps
no fq_codel on laptop, cerowrt qlen_*=10,nttcp -r: 1.931/ 4.107/ 15.141/ 3.049 ms;  75 Mbps
no fq_codel on laptop, cerowrt qlen_*=11,nttcp -r: 1.982/ 3.849/ 12.163/ 2.407 ms;  80 Mbps
no fq_codel on laptop, cerowrt qlen_*=12,nttcp -r: 2.025/ 5.173/ 16.890/ 3.763 ms;  81 Mbps
no fq_codel on laptop, cerowrt qlen_*=20,nttcp -r: 1.525/ 3.492/ 13.737/ 2.465 ms;  82 Mbps
no fq_codel on laptop, cerowrt qlen_*=30,nttcp -r: 1.907/11.243/142.129/24.017 ms; 104 Mbps
no fq_codel on laptop, cerowrt qlen_*=35,nttcp -r: 2.893/ 7.895/130.859/17.621 ms; 119 Mbps
no fq_codel on laptop, cerowrt qlen_*=40,nttcp -r: 2.917/ 7.766/ 98.252/13.105 ms; 120 Mbps
no fq_codel on laptop, cerowrt qlen_*=50,nttcp -r: 0.951/ 7.810/ 47.646/ 6.428 ms; 131 Mbps
no fq_codel on laptop, cerowrt qlen_*=100,nttcp -r:5.149/ 8.766/ 14.371/ 2.191 ms; 128 Mbps

To get twice the speed a qlen=11 is enough already, and to get all the speed back a qlen=35 is needed.

And here are the results with fq_codel on the laptop too (just nttcp -t as thats the one affected):

fq_codel on laptop, cerowrt defaults,  nttcp -t:  1.248/12.960/108.490/16.733 ms; 90 Mbps
fq_codel on laptop, cerowrt qlen_*=4,  nttcp -t:  1.205/10.843/ 76.983/12.460 ms; 105 Mbps
fq_codel on laptop, cerowrt qlen_*=8,  nttcp -t:  4.034/16.088/ 98.611/17.050 ms; 120 Mbps
fq_codel on laptop, cerowrt qlen_*=11, nttcp -t:  3.766/15.687/ 56.684/11.135 ms; 114 Mbps
fq_codel on laptop, cerowrt qlen_*=35, nttcp -t: 11.360/26.742/ 48.051/ 7.489 ms; 113 Mbps

Shouldn't wireless N be able to do 200 - 300 Mbps though? If I enable debugging in iwl4965 I see that it
starts TX aggregation, so not sure whats wrong (router or laptop?). With encryption off I can get at most 160 Mbps.

iw dev sw10 station dump shows:
...
        signal:         -56 [-60, -59] dBm
        signal avg:     -125 [-65, -58] dBm
        tx bitrate:     300.0 MBit/s MCS 15 40Mhz short GI
        rx bitrate:     300.0 MBit/s MCS 15 40Mhz short GI

On laptop:
	tx bitrate:     300.0 Mbit/s MCS 15 40Mhz short GI

> 
> Personally I'd be happy if we could hold wifi jitter below 30ms, and
> typical latency below 10ms, in most (home) scenarios. I think that is
> eminently doable, and a reasonable compromise between cero's all-out
> assault on latency and the marketing need for more bandwidth. fq_codel
> all by itself gets close (the fair queuing part helps a lot)
> 
> 'Course I'd love it if low latency became the subject of all out
> marketing wars between home gateway makers, and between ISPs, with
> 1/100 the technical resources thrown into the problem as has been
> expended on raw bandwidth.
> 
> Possible themes:
> 
> "Hetgear": Frag your friends, faster!

Yeah gamers would probably care about latency, game server admins too.

> "Binksys": Torrents and TV? no problem.
> "Chuckalo": making DNS zippy!
> 
> "Boogle fiber: now with 2ms cross town latency!"
> "Merizon: Coast to coast in under 60ms!"
> "Nomcast: Making webex work better"
> 
> But we're not living in that alternate reality (yet), although I think
> we're beginning to see some light at the end of the tunnel.
> 
> That said, there are infrastructural problems regarding the overuse of
> aggregation everywhere, in gpon, in cable modems and CMTSes, and in
> other backbone technologies, in addition to the queue management
> problem. It's going to be a hard slog to get back to where the
> distance between your couch and the internet is consistently less than
> between here and the moon.
> 
> But worth it, in terms of the global productivity gain, and lowered
> annoyance levels, worldwide.

:)

Best regards,
--Edwin