From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.etorok.net (mail.etorok.net [IPv6:2a01:4f8:160:1223::beef:2]) by huchra.bufferbloat.net (Postfix) with ESMTP id 44851200BE5; Sat, 18 Aug 2012 02:38:06 -0700 (PDT) Received: from [IPv6:2a02:2f02:1022:71a5:1e6f:65ff:fe23:db0d] (unknown [IPv6:2a02:2f02:1022:71a5:1e6f:65ff:fe23:db0d]) by mail.etorok.net (Postfix) with ESMTPSA id CEE7946A8; Sat, 18 Aug 2012 11:38:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=etorok.net; s=MAILOUT; t=1345282684; bh=OoOZAjqr25iKJVGsnUXjUK5kmQOxfaoVnWEr9NWwwf4=; h=Message-ID:Date:From:MIME-Version:To:CC:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=TpTaE5Wz0QqRwKDphP7+vr/PxRBcZZAC1pCy/N2Qf9QS7PF8dGksfFBmUtMFefhjo 1Do5VZYBBll43A/Rv+xYz39IZ4Q4Gb5Ul1h/TYSVJ6ObDrVYZwj9Kgk64udwGgJEln M/0E3E4NjQgjLGzDvl4Dnk1Z0yRuwxznaMV0wujc= Message-ID: <502F6279.1090708@etorok.net> Date: Sat, 18 Aug 2012 12:38:01 +0300 From: =?ISO-8859-1?Q?T=F6r=F6k_Edwin?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.6esrpre) Gecko/20120805 Icedove/10.0.6 MIME-Version: 1.0 To: Dave Taht References: <502E064C.50305@etorok.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 0.97.5 at mail X-Virus-Status: Clean Cc: cerowrt-devel@lists.bufferbloat.net, bloat Subject: Re: [Bloat] [Cerowrt-devel] cerowrt 3.3.8-17: nice latency improvements, some issues with bind X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Aug 2012 09:38:06 -0000 On 08/17/2012 09:05 PM, Dave Taht wrote: > I'm widening the distribution of this email a little bit in light of > the benchmark results (somewhat too far) below, and some of the other > issues raised. > > On Fri, Aug 17, 2012 at 1:52 AM, Török Edwin >> Onto the good news, here are some measurements (ping time / bandwidth from my laptop connected through WiFi to my desktop connected through GbitE): >> >> no fq_codel on laptop, openwrt, wlan0 5Ghz: 0.859/174.859/923.768/198.308 ms; 120 - 140Mbps >> w/ fq_codel on laptop, openwrt, wlan0 5Ghz: 1.693/ 26.727/ 54.936/ 11.746 ms; 120 - 140Mbps >> no fq_codel on laptop, cerowrt, wlan0 5Ghz: 2.310/ 15.183/140.495/ 30.337 ms; 75 - 85 Mbps >> w/ fq_codel on laptop, cerowrt, wlan0 5Ghz: 1.464/ 1.981/ 2.223/ 0.221 ms; 75 - 85 Mbps >> >> The latency improvement is awesome, and I don't really mind the sacrificed bandwidth to accomplish it. > > A man after my own heart. > > Thx! The industry as a whole has been focused on "bandwidth at any > cost, including massive latency", which leads to things like the ~1 > second delays you observed on your fq_codel-less test. (and far worse > has been observed in the field) We're focused on improving latency, > because as stuart cheshire says: "once you have latency, you can't get > it back" Yeah latency is most annoying thing on wireless (even more so on 3G), if I really want more LAN bandwidth I can just plug in the Ethernet cable :) Although I measured nttcp -r too now (see below) and the bandwidth drop is quite significant there, can only do 39 Mbps. > > We hope that once some other concepts prove out, we can keep the low > latency and add even more bandwidth back. > > http://www.bufferbloat.net/projects/cerowrt/wiki/Fq_Codel_on_Wireless > > In day-to-day use the lowered latency and jitter in cero currently can > be really "felt" particularly in applications like skype and google > hangouts, and web pages (under load) feel much faster, as DNS lookups > happen really fast... > > and (as another example), things like youtube far more rarely "stall out". > > It's kind of hard to measure "feel", though. I wish we had better > benchmarks to show what we're accomplishing. > >> Is the bandwidth drop intended though? When enabling fq_codel just on my laptop I didn't notice any bandwidth drop at all. > > The core non-fq_codel change on cerowrt vs openwrt and/or your laptop > is that the aggregation buffer size at the driver level has been > severely reduced in cerowrt, from it's default of 128 buffers, to 3. > This means that the maximum aggregate size has been cut to 3 packets > from ~42, but more importantly, total outstanding buffers not managed > by codel to 3, rather than 128.... > > The fact that this costs so little bandwidth (40%) in exchange for > reducing latency and jitter by 25x (or 400x compared to no fq_codel at > all) suggests that in the long run, once we come up with fixes to the > mac80211 layer, we will be able to achieve better utilization, > latency, AND jitter overall than the current hw deployed everywhere. > > IF you'd like to have more bandwidth back, you can jiggle the qlen_* > variables in the debloat script up, but remember that tcp's reaction > time is quadratic as to the amount of buffering. I'd be interested in > you repeating your benchmark after doing that? The difference between > 3 buffers and 8 is pretty dramatic... Here are some measurements (50 pings, nttcp -D -n200000) Setup: laptop with Linux 3.5.0, iwl4965 AGN, 5.22 Ghz (channel 44) only radio on 5GHz spectrum. desktop with Linux 3.5.1 running nttcp -i, connected through GbE to cerowrt router Baseline (only ping, no other traffic): 0.806/ 1.323/ 8.753/ 1.333 ms no fq_codel on laptop, cerowrt defaults, nttcp -t: 1.192/16.605/107.351/25.265 ms; 94 Mbps no fq_codel on laptop, cerowrt qlen_*=4, nttcp -t: 1.285/25.108/105.519/22.607 ms; 107 Mbps no fq_codel on laptop, cerowrt qlen_*=5, nttcp -t: 1.118/15.538/ 68.262/15.773 ms; 104 Mbps no fq_codel on laptop, cerowrt qlen_*=6, nttcp -t: 1.353/15.592/116.026/18.782 ms; 113 Mbps no fq_codel on laptop, cerowrt qlen_*=7, nttcp -t: 1.344/18.719/112.757/23.691 ms; 113 Mbps no fq_codel on laptop, cerowrt qlen_*=8, nttcp -t: 1.717/20.842/ 59.412/16.120 ms; 127 Mbps no fq_codel on laptop, cerowrt qlen_*=9, nttcp -t: 1.663/21.406/141.338/26.270 ms; 123 Mbps no fq_codel on laptop, cerowrt qlen_*=10,nttcp -t: 1.818/21.361/117.023/20.727 ms; 120 Mbps no fq_codel on laptop, cerowrt qlen_*=12,nttcp -t: 2.195/24.277/131.490/21.161 ms; 127 Mbps no fq_codel on laptop, cerowrt defaults, nttcp -r: 1.332/ 3.129/ 41.900/ 5.221 ms; 39 Mbps no fq_codel on laptop, cerowrt qlen_*=4, nttcp -r: 1.514/ 3.205/ 8.595/ 1.817 ms; 46 Mbps no fq_codel on laptop, cerowrt qlen_*=5, nttcp -r: 1.342/ 7.250/114.095/17.591 ms; 52 Mbps no fq_codel on laptop, cerowrt qlen_*=6, nttcp -r: 1.299/ 3.200/ 12.080/ 1.972 ms; 58 Mbps no fq_codel on laptop, cerowrt qlen_*=7, nttcp -r: 1.617/ 5.220/ 76.478/10.971 ms; 63 Mbps no fq_codel on laptop, cerowrt qlen_*=8, nttcp -r: 1.810/ 4.267/ 24.560/ 4.065 ms; 67 Mbps no fq_codel on laptop, cerowrt qlen_*=9, nttcp -r: 2.015/ 5.090/ 78.256/10.710 ms; 71 Mbps no fq_codel on laptop, cerowrt qlen_*=10,nttcp -r: 1.931/ 4.107/ 15.141/ 3.049 ms; 75 Mbps no fq_codel on laptop, cerowrt qlen_*=11,nttcp -r: 1.982/ 3.849/ 12.163/ 2.407 ms; 80 Mbps no fq_codel on laptop, cerowrt qlen_*=12,nttcp -r: 2.025/ 5.173/ 16.890/ 3.763 ms; 81 Mbps no fq_codel on laptop, cerowrt qlen_*=20,nttcp -r: 1.525/ 3.492/ 13.737/ 2.465 ms; 82 Mbps no fq_codel on laptop, cerowrt qlen_*=30,nttcp -r: 1.907/11.243/142.129/24.017 ms; 104 Mbps no fq_codel on laptop, cerowrt qlen_*=35,nttcp -r: 2.893/ 7.895/130.859/17.621 ms; 119 Mbps no fq_codel on laptop, cerowrt qlen_*=40,nttcp -r: 2.917/ 7.766/ 98.252/13.105 ms; 120 Mbps no fq_codel on laptop, cerowrt qlen_*=50,nttcp -r: 0.951/ 7.810/ 47.646/ 6.428 ms; 131 Mbps no fq_codel on laptop, cerowrt qlen_*=100,nttcp -r:5.149/ 8.766/ 14.371/ 2.191 ms; 128 Mbps To get twice the speed a qlen=11 is enough already, and to get all the speed back a qlen=35 is needed. And here are the results with fq_codel on the laptop too (just nttcp -t as thats the one affected): fq_codel on laptop, cerowrt defaults, nttcp -t: 1.248/12.960/108.490/16.733 ms; 90 Mbps fq_codel on laptop, cerowrt qlen_*=4, nttcp -t: 1.205/10.843/ 76.983/12.460 ms; 105 Mbps fq_codel on laptop, cerowrt qlen_*=8, nttcp -t: 4.034/16.088/ 98.611/17.050 ms; 120 Mbps fq_codel on laptop, cerowrt qlen_*=11, nttcp -t: 3.766/15.687/ 56.684/11.135 ms; 114 Mbps fq_codel on laptop, cerowrt qlen_*=35, nttcp -t: 11.360/26.742/ 48.051/ 7.489 ms; 113 Mbps Shouldn't wireless N be able to do 200 - 300 Mbps though? If I enable debugging in iwl4965 I see that it starts TX aggregation, so not sure whats wrong (router or laptop?). With encryption off I can get at most 160 Mbps. iw dev sw10 station dump shows: ... signal: -56 [-60, -59] dBm signal avg: -125 [-65, -58] dBm tx bitrate: 300.0 MBit/s MCS 15 40Mhz short GI rx bitrate: 300.0 MBit/s MCS 15 40Mhz short GI On laptop: tx bitrate: 300.0 Mbit/s MCS 15 40Mhz short GI > > Personally I'd be happy if we could hold wifi jitter below 30ms, and > typical latency below 10ms, in most (home) scenarios. I think that is > eminently doable, and a reasonable compromise between cero's all-out > assault on latency and the marketing need for more bandwidth. fq_codel > all by itself gets close (the fair queuing part helps a lot) > > 'Course I'd love it if low latency became the subject of all out > marketing wars between home gateway makers, and between ISPs, with > 1/100 the technical resources thrown into the problem as has been > expended on raw bandwidth. > > Possible themes: > > "Hetgear": Frag your friends, faster! Yeah gamers would probably care about latency, game server admins too. > "Binksys": Torrents and TV? no problem. > "Chuckalo": making DNS zippy! > > "Boogle fiber: now with 2ms cross town latency!" > "Merizon: Coast to coast in under 60ms!" > "Nomcast: Making webex work better" > > But we're not living in that alternate reality (yet), although I think > we're beginning to see some light at the end of the tunnel. > > That said, there are infrastructural problems regarding the overuse of > aggregation everywhere, in gpon, in cable modems and CMTSes, and in > other backbone technologies, in addition to the queue management > problem. It's going to be a hard slog to get back to where the > distance between your couch and the internet is consistently less than > between here and the moon. > > But worth it, in terms of the global productivity gain, and lowered > annoyance levels, worldwide. :) Best regards, --Edwin