From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-x234.google.com (mail-wi0-x234.google.com [IPv6:2a00:1450:400c:c05::234]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 26C1021F2EC for ; Sun, 2 Mar 2014 13:51:38 -0800 (PST) Received: by mail-wi0-f180.google.com with SMTP id hm4so2444862wib.13 for ; Sun, 02 Mar 2014 13:51:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:from:subject:date:to; bh=/4pgY6+Kw9BGHl1MbQJFEMcp1k1cqEsWi8E2LmSlSHo=; b=IfMzgFvjRUBYMqnhHQFul6so/kOa5Ezwhwxrhz1PPo+Eh640f/7nNT1/ghCYM6tO2v UtsmwZJZDypEUNgbq6P5oMJ9fnUOmBm9Wb624vhyl6bEk+NN+U2b0NUwAY3DlezTeJsZ 2HEs176V6DoHXEHig9/K5Fin9u92rlP3u7V9IdBqXnag3elMCIYW6Ee1dnl0bQZ9Br0p eSIw8nEWPYxs8AXiAxoSdEokgrez/rxj/CtNG4J5oCIeRmaNdvW92fNlEbAsroM0iXoq OFHEhz170YWRCGKZ2qXNOjS2iG2cOFglJPcWE0/qezw5b4/kgwmadrYiW5A3Tk5gyWGP 5C4w== X-Received: by 10.180.75.105 with SMTP id b9mr11360234wiw.6.1393797096318; Sun, 02 Mar 2014 13:51:36 -0800 (PST) Received: from [172.30.42.84] (gut75-7-78-225-42-24.fbx.proxad.net. [78.225.42.24]) by mx.google.com with ESMTPSA id ga10sm11648494wjb.23.2014.03.02.13.51.34 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 02 Mar 2014 13:51:35 -0800 (PST) References: <5EC471C4-64B9-4D83-AB78-5219E2090886@gmail.com> Mime-Version: 1.0 (1.0) In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-Id: <6D412C07-A429-4811-BE23-279C5B46CDE6@gmail.com> X-Mailer: iPhone Mail (11B651) From: Aaron Wood Date: Sun, 2 Mar 2014 22:51:32 +0100 To: Dave Taht Cc: cerowrt-devel Subject: Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16 X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Mar 2014 21:51:38 -0000 Is there a writeup on each of the fq_codel variants? -Aaron Sent from my iPhone > On Mar 2, 2014, at 22:41, Dave Taht wrote: >=20 > Nice work! >=20 > I have a problem in that I can't remember if target autotuning made it > into that release or not. >=20 > Coulde you do a tc -s qdisc show dev ge00 on your favorite of the > above and paste? I still think > target is still too low on the egress side with the current calculation. >=20 > Secondly, now that you have a setting you like, trying pie, codel, and > ns2_codel also would be interesting. >=20 > efq_codel is currently uninteresting. Wasn't clear if you were using > nfq_codel or fq_codel throughout. >=20 >=20 >> On Sun, Mar 2, 2014 at 1:18 PM, Rich Brown wrot= e: >> I took some time this weekend, and ran careful speed and latency tests on= the CeroWrt 3.10.28-16 build. I have a much better understanding of how all= this works, both in theory and in practice. Here's an executive summary of t= he overall test procedure with lots of details below. >>=20 >> Adjusting CeroWrt's configured up- and download rates in the SQM page aff= ects both the actual data transfer rates as well as the latency. If you set t= he values too low, CeroWrt will enforce that bottleneck, and the transfer ra= tes will be lower than you could attain on your link. If you configure them t= oo high, though, the transfer rates may look better, but the latency can go o= ff the charts. Here's how I arrived at a good balance. >>=20 >> Test Conditions: >>=20 >> - Running tests from my MacBook Pro, 10.9.2. >> - Wi-Fi off; ethernet cable direct to Netgear WNDR3700v2 with CeroWrt 3.1= 0.28-16. >> - DSL service from Fairpoint, nominally "7 Mbps down/768kbps up". >> - DSL Modem sync rate (the actual rate that bits enter/leave my house) is= 7616kbps down; 864kbps up. The line is apparently fairly clean, too. >> - Base ping time to the nearest router at ISP (via traceroute) is 29-30 m= sec. >> - To minimize other traffic, I turned off most of the computers at home, a= nd also quit my mail client (which is surprisingly chatty). >>=20 >> The Tests: >>=20 >> I ran two different tests: netperf-wrapper with the RRUL test, and speedt= est.net. These give very different views of performance. RRUL really stresse= s the line using multiple simultaneous up and download streams. Speedtest.ne= t is a consumer test that only tests one direction at a time, and for a shor= t time. We want to look good with both. >>=20 >> For the RRUL tests, I invoked netperf-wrapper like this: netperf-wrapper r= rul -p all_scaled -l 60 -H atl.richb-hanover.com -t text-shown-in-chart >> For the Speedtest.net tests, I used their web GUI in the obvious way. >>=20 >> For both tests, I used a script (pingstats.sh, see my next message) to co= llect the ping times and give min, max, average, median, and 10th and 90th p= ercentile readings. >>=20 >> Test Procedure: >>=20 >> I ran a series of tests starting with the up/down link rates spelled out b= y Sebastian Moeller's amazingly detailed note last week. See https://lists.b= ufferbloat.net/pipermail/cerowrt-devel/2014-February/002375.html Read it car= efully. There's a lot of insight available there. >>=20 >> The initial configuration was 6089/737 down/up, with the (nearly default)= values for Queue Discipline (nfq_codel, simple.qos, ECN on for ingress; NOE= CN for egress, auto for both ingress and egress latency targets), and ATM li= nk layer with 44 bytes of overhead. >>=20 >> With those initial configuration values, latency was good but the speeds w= ere disappointing. I then re-ran the tests with CeroWrt configured for highe= r up/down link speeds to see where things broke. >>=20 >> Things got better and better with increasing link rates until I hit 7600/= 850 - at that point, latency began to get quite large. (Of course, with SQM d= isabled, the latency got dreadful.) >>=20 >> There was an anomaly at 7000/800 kbps. The 90th percentile and max number= s jumped up quite a lot, but went *down* for the next test in the sequence w= hen I increased the upload speed to 7000/830. I ran the experiment twice to c= onfirm that behavior. >>=20 >> I should also note that in the course of the experiment, I re-ran many of= these tests. Although I did not document each of the runs, the results (spe= edtest.net rates and the pingstats.sh values) were quite consistent and repe= atable. >>=20 >> Conclusion: >>=20 >> I'm running with CeroWrt 3.10.28-16 configured for down/up 7000/830, (nea= rly) default Queue Discipline and ATM+44 bytes of overhead. With these confi= gurations, latency is well in hand and my network is pretty speedy. >>=20 >> We need to figure out how to explain to people what to expect re: the tra= deoff between "faster speeds" that show up in Speedtest.net (with accompanyi= ng crappy performance) and slightly slower speeds with a *way* better experi= ence. >>=20 >> The data follows... >>=20 >> Rich >>=20 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>=20 >> RRUL Tests: The charts associated with these RRUL runs are all available= at http://richb-hanover.com/rrul-tests-cerowrt-3-10-28-16/ >>=20 >> 6089/737: >> Min: 28.936 10pct: 29.094 Avg: 40.529 Median: 37.961 90pct: 52.636= Max: 77.171 Num pings: 77 >>=20 >> 6200/750: >> Min: 28.715 10pct: 29.298 Avg: 41.805 Median: 39.826 90pct: 57.414= Max: 72.363 Num pings: 77 >>=20 >> 6400/800: >> Min: 28.706 10pct: 29.119 Avg: 39.598 Median: 38.428 90pct: 52.351= Max: 69.492 Num pings: 78 >>=20 >> 6600/830: >> Min: 28.485 10pct: 29.114 Avg: 41.708 Median: 39.753 90pct: 57.552= Max: 87.328 Num pings: 77 >>=20 >> 7000/800: >> Min: 28.570 10pct: 29.180 Avg: 46.245 Median: 42.684 90pct: 62.376= Max: 169.991 Num pings: 77 >> Min: 28.775 10pct: 29.226 Avg: 43.628 Median: 40.446 90pct: 60.216= Max: 121.334 Num pings: 76 (2nd run) >>=20 >> 7000/830: >> Min: 28.942 10pct: 29.285 Avg: 44.283 Median: 45.318 90pct: 58.002= Max: 85.035 Num pings: 78 >> Min: 28.951 10pct: 29.479 Avg: 43.182 Median: 41.000 90pct: 57.570= Max: 74.964 Num pings: 76 (2nd run) >>=20 >> 7600/850: >> Min: 28.756 10pct: 29.078 Avg: 55.426 Median: 46.063 90pct: 81.847= Max: 277.807 Num pings: 84 >>=20 >> SQM Disabled: >> Min: 28.665 10pct: 29.062 Avg: 1802.521 Median: 2051.276 90pct: 27= 62.941 Max: 4217.644 Num pings: 78 >>=20 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>=20 >> Speedtest.net: First values are the reported down/up rates in the Speedte= st GUI >>=20 >> 6089/737: >> 5.00/0.58 >> Min: 28.709 10pct: 28.935 Avg: 33.416 Median: 31.619 90pct: 38.608= Max: 49.193 Num pings: 45 >>=20 >> 6200/750: >> 5.08/0.58 >> Min: 28.759 10pct: 29.055 Avg: 33.974 Median: 32.584 90pct: 41.938= Max: 46.605 Num pings: 44 >>=20 >> 6400/800: >> 5.24/0.60 >> Min: 28.447 10pct: 28.826 Avg: 34.675 Median: 31.155 90pct: 41.285= Max: 81.503 Num pings: 43 >>=20 >> 6600/830: >> 5.41/0.65 >> Min: 28.868 10pct: 29.053 Avg: 35.158 Median: 32.928 90pct: 44.099= Max: 51.571 Num pings: 44 >>=20 >> 7000/800: >> 5.73/0.62 >> Min: 28.359 10pct: 28.841 Avg: 35.205 Median: 33.620 90pct: 43.735= Max: 54.812 Num pings: 44 >>=20 >> 7000/830: >> 5.74/0.65 (5.71/0.62 second run) >> Min: 28.605 10pct: 29.055 Avg: 34.945 Median: 31.773 90pct: 42.645= Max: 54.077 Num pings: 44 >> Min: 28.649 10pct: 28.820 Avg: 34.866 Median: 32.398 90pct: 43.533= Max: 69.288 Num pings: 56 (2nd run) >>=20 >> 7600/850: >> 6.20/0.67 >> Min: 28.835 10pct: 28.963 Avg: 36.253 Median: 34.912 90pct: 44.659= Max: 54.023 Num pings: 48 >>=20 >> SQM Disabled: >> 6.46/0.73 >> Min: 28.452 10pct: 28.872 Avg: 303.754 Median: 173.498 90pct: 499.= 678 Max: 1799.814 Num pings: 45 >> _______________________________________________ >> Cerowrt-devel mailing list >> Cerowrt-devel@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/cerowrt-devel >=20 >=20 >=20 > --=20 > Dave T=C3=A4ht >=20 > Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe= .html > _______________________________________________ > Cerowrt-devel mailing list > Cerowrt-devel@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cerowrt-devel