From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id C7BEE21F262 for ; Sat, 28 Mar 2015 18:15:03 -0700 (PDT) Received: from hms-beagle.home.lan ([87.164.165.7]) by mail.gmx.com (mrgmx103) with ESMTPSA (Nemesis) id 0MGzwE-1YftR33iyI-00DlWM; Sun, 29 Mar 2015 03:14:59 +0200 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) From: Sebastian Moeller In-Reply-To: Date: Sun, 29 Mar 2015 03:14:58 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <3615CBEA-5DC1-4722-9EE9-13B3570E8118@gmx.de> References: <08BAF198-87C5-42B8-8899-53F34E47156E@gmail.com> <896FAE61-B45A-4F34-9449-5ADB82194DD9@gmx.de> <48350C2E-C33A-4534-84BC-5D56F4AAF0EA@gmail.com> <8AC58249-8199-405B-997A-E8F7285A34FB@gmx.de> To: Jonathan Morton X-Mailer: Apple Mail (2.1878.6) X-Provags-ID: V03:K0:zUm98m+Sh9vZHDa3PpNX4LTYmUtfJkMmx6PJSOOnvHonEyh3qeq EcrzGxSUdzZS3XOzSp6OL9Ji6L2Sf1+jdWEHcjTAFiN1VZpILqyw1oeDfZcKkDwZU2t7FKu pKP41QJKQtkezwRdN2sp/ZWQ1Gwd3QF5KViuAC+rVrJssdZaOfVPTQDHGFvNY/9gbtDSKYH z2OrmPco85JdkDO0p28MQ== X-UI-Out-Filterresults: notjunk:1; Cc: cerowrt-devel@lists.bufferbloat.net Subject: Re: [Cerowrt-devel] archer c7 v2, policing, hostapd, test openwrt build X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Mar 2015 01:15:32 -0000 Hi Jonathan, TL;DR: I do not think my measurements show that ingress handling via IFB = is that costly (< 5% bandwidth), that avoiding it will help much. I do = not think that conclusion will change much if more data is acquired (and = I do not intend to collect more ;) ) Also the current diffserv = implementation also costs around 5% bandwidth. Bandwidth cost here means = that the total bandwidth at full saturation is reduced by that amount = (with the total under load being ~90Mbps up+down while the sum of max = bandwidth up+down without load ~115Mbps), latency under load seems not = to suffer significantly once the router runs out of CPU though, which is = nice ;) On Mar 24, 2015, at 09:13 , Jonathan Morton = wrote: > What I'm seeing on your first tests is that double egress gives you = slightly more download at the expense of slightly less upload = throughout. The aggregate is higher. >=20 > Your second set of tests tells me almost nothing, because it exercises = the upload more and the download less. Hence why I'm asking for = effectively the opposite test. Since netperf-wrapper currently does not have rrul_down_only and = rrul_up_only tests (and I have not enough tome/skill to code these = tests) I opted for using Rich Brown=92s nice wrapper scripts from the = Ceroscripts repository. There still is decent report of the =93fate=94 = of the concurrent ICMP probe, but no fancy graphs or sparse UDP streams, = but for our question this should be sufficient... Here is the result for dual egress with simplest.qos from a client = connected to cerowrt=92s se00: simplest.qos: IPv6, download and upload sequentially: moeller@happy-horse:~/CODE/CeroWrtScripts> ./betterspeedtest.sh -6 -H = netperf-eu.bufferbloat.net -t 150 -p netperf-eu.bufferbloat.net -n 4 2015-03-29 00:23:27 Testing against netperf-eu.bufferbloat.net (ipv6) = with 4 simultaneous sessions while pinging netperf-eu.bufferbloat.net = (150 seconds in each direction) = ..........................................................................= ..........................................................................= .. Download: 85.35 Mbps Latency: (in msec, 150 pings, 0.00% packet loss) Min: 37.900=20 10pct: 38.600=20 Median: 40.100=20 Avg: 40.589=20 90pct: 43.600=20 Max: 47.500 = ..........................................................................= ..........................................................................= .. Upload: 32.73 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 37.300=20 10pct: 37.800=20 Median: 38.200=20 Avg: 38.513=20 90pct: 39.400=20 Max: 47.400 simplest.qos: IPv6, download and upload simultaneously: moeller@happy-horse:~/CODE/CeroWrtScripts> ./netperfrunner.sh -6 -H = netperf-eu.bufferbloat.net -t 150 -p netperf-eu.bufferbloat.net -n 4 2015-03-29 00:30:40 Testing netperf-eu.bufferbloat.net (ipv6) with 4 = streams down and up while pinging netperf-eu.bufferbloat.net. Takes = about 150 seconds. Download: 81.42 Mbps Upload: 9.33 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 37.500=20 10pct: 38.700=20 Median: 39.500=20 Avg: 39.997=20 90pct: 42.000=20 Max: 45.500 simplest.qos: IPv4, download and upload sequentially: moeller@happy-horse:~/CODE/CeroWrtScripts> ./betterspeedtest.sh -4 -H = netperf-eu.bufferbloat.net -t 150 -p netperf-eu.bufferbloat.net -n 4 ; = ./netperfrunner.sh -4 -H netperf-eu.bufferbloat.net -t 150 -p = netperf-eu.bufferbloat.net -n 4 2015-03-29 00:33:52 Testing against netperf-eu.bufferbloat.net (ipv4) = with 4 simultaneous sessions while pinging netperf-eu.bufferbloat.net = (150 seconds in each direction) = ..........................................................................= ..........................................................................= .. Download: 86.52 Mbps Latency: (in msec, 150 pings, 0.00% packet loss) Min: 49.300=20 10pct: 50.300=20 Median: 51.400=20 Avg: 51.463=20 90pct: 52.700=20 Max: 54.500 = ..........................................................................= ..........................................................................= .. Upload: 33.45 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 49.300=20 10pct: 49.800=20 Median: 50.100=20 Avg: 50.161=20 90pct: 50.600=20 Max: 52.400 simplest.qos: IPv4, download and upload simultaneously: 2015-03-29 00:38:53 Testing netperf-eu.bufferbloat.net (ipv4) with 4 = streams down and up while pinging netperf-eu.bufferbloat.net. Takes = about 150 seconds. Download: 84.21 Mbps Upload: 6.45 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 49.300=20 10pct: 50.000=20 Median: 51.100=20 Avg: 51.302=20 90pct: 52.700=20 Max: 56.100 The IPv6 route to Sweden is still 10ms shorter than the IPv4 one, no = idea why, but I do not complain ;) And again the same with simple.qos (in both directions) to assess the = cost of our diffserv implementation: simple.qos: IPv6, download and upload sequentially: 2015-03-29 00:44:06 Testing against netperf-eu.bufferbloat.net (ipv6) = with 4 simultaneous sessions while pinging netperf-eu.bufferbloat.net = (150 seconds in each direction) = ..........................................................................= ..........................................................................= .. Download: 82.8 Mbps Latency: (in msec, 150 pings, 0.00% packet loss) Min: 37.600=20 10pct: 38.500=20 Median: 39.600=20 Avg: 40.068=20 90pct: 42.600=20 Max: 47.900 = ..........................................................................= ..........................................................................= .. Upload: 32.8 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 37.300=20 10pct: 37.700=20 Median: 38.100=20 Avg: 38.256=20 90pct: 38.700=20 Max: 43.400 Compared to simplest.qos we loose like 2.5Mbs in downlink, not too bad, = and nothing in the uplink tests, but there the wndr3700v2 is not yet out = of oomph... simple.qos: IPv6, download and upload simultaneously: 2015-03-29 00:49:07 Testing netperf-eu.bufferbloat.net (ipv6) with 4 = streams down and up while pinging netperf-eu.bufferbloat.net. Takes = about 150 seconds. Download: 77.2 Mbps Upload: 9.43 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 37.800=20 10pct: 38.500=20 Median: 39.500=20 Avg: 40.133=20 90pct: 42.200=20 Max: 51.500 But here in the full saturating case we already =93pay=94 4Mbps, still = not bad, but it looks like all the small change adds up (and I should = try to find the time to look at all the tc filtering we do) simple.qos: IPv4, download and upload sequentially: 2015-03-29 00:51:37 Testing against netperf-eu.bufferbloat.net (ipv4) = with 4 simultaneous sessions while pinging netperf-eu.bufferbloat.net = (150 seconds in each direction) = ..........................................................................= ..........................................................................= .. Download: 84.28 Mbps Latency: (in msec, 150 pings, 0.00% packet loss) Min: 49.400=20 10pct: 50.100=20 Median: 51.100=20 Avg: 51.253=20 90pct: 52.500=20 Max: 54.900 = ..........................................................................= ..........................................................................= .. Upload: 33.42 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 49.400=20 10pct: 49.800=20 Median: 50.100=20 Avg: 50.170=20 90pct: 50.600=20 Max: 51.800 simple.qos: IPv4, download and upload simultaneously: 2015-03-29 00:56:38 Testing netperf-eu.bufferbloat.net (ipv4) with 4 = streams down and up while pinging netperf-eu.bufferbloat.net. Takes = about 150 seconds. Download: 81.08 Mbps Upload: 6.73 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 49.300=20 10pct: 50.100=20 Median: 51.100=20 Avg: 51.234=20 90pct: 52.500=20 Max: 56.200 Again this hammers the real upload while leaving the download mainly = intact And for fun/completeness=92 sake of it the same for the standard setup = with activated IFB and ingress shaping on pppoe-ge00: simplest.qos: IPv6, download and upload sequentially: 2015-03-29 01:18:13 Testing against netperf-eu.bufferbloat.net (ipv6) = with 4 simultaneous sessions while pinging netperf-eu.bufferbloat.net = (150 seconds in each direction) = ..........................................................................= ..........................................................................= .. Download: 82.76 Mbps Latency: (in msec, 150 pings, 0.00% packet loss) Min: 37.800=20 10pct: 38.900=20 Median: 40.100=20 Avg: 40.590=20 90pct: 43.200=20 Max: 47.000 = ..........................................................................= ..........................................................................= .. Upload: 32.86 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 37.300=20 10pct: 37.700=20 Median: 38.100=20 Avg: 38.273=20 90pct: 38.700=20 Max: 43.200 So 85.35-82.76 =3D 2.59 Mbps cost for the IFB, comparable to the cost = of our diffserv implementation. simplest.qos: IPv6, download and upload simultaneously: 2015-03-29 01:23:14 Testing netperf-eu.bufferbloat.net (ipv6) with 4 = streams down and up while pinging netperf-eu.bufferbloat.net. Takes = about 150 seconds. Download: 78.61 Mbps Upload: 10.53 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 37.700=20 10pct: 39.100=20 Median: 40.200=20 Avg: 40.509=20 90pct: 42.400=20 Max: 46.200 Weird, IFB still costs 81.42-78.61 =3D 2.81Mbs, but the uplink improved = by 10.53-9.33 =3D 1.2 Mbps, reducing the IFB cost to 1.61Mbps... simplest.qos: IPv4, download and upload sequentially: 2015-03-29 01:25:44 Testing against netperf-eu.bufferbloat.net (ipv4) = with 4 simultaneous sessions while pinging netperf-eu.bufferbloat.net = (150 seconds in each direction) = ..........................................................................= ..........................................................................= .. Download: 84.06 Mbps Latency: (in msec, 150 pings, 0.00% packet loss) Min: 49.700=20 10pct: 50.500=20 Median: 51.900=20 Avg: 51.866=20 90pct: 53.100=20 Max: 55.400 = ..........................................................................= ..........................................................................= .. Upload: 33.45 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 49.500=20 10pct: 49.800=20 Median: 50.200=20 Avg: 50.219=20 90pct: 50.600=20 Max: 52.100 Again IFB usage costs us 86.52-84.06 =3D 2.46 Mbps... simplest.qos: IPv4, download and upload simultaneously: 2015-03-29 01:30:45 Testing netperf-eu.bufferbloat.net (ipv4) with 4 = streams down and up while pinging netperf-eu.bufferbloat.net. Takes = about 150 seconds. Download: 78.97 Mbps Upload: 8.14 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 49.300=20 10pct: 50.300=20 Median: 51.500=20 Avg: 51.906=20 90pct: 53.700=20 Max: 71.700 And again download IFB cost 84.21-78.97 =3D 5.24, with upload recovery = 8.14-6.45 =3D 1.69Mbps, so total IFB cost here 5.24-1.69 =3D 3.55 Mbps = or (100*3.55 /(78.97+8.14)) =3D 4.1%, not great, but certainly not = enough if regained to qualify this hardware for higher bandwidth tiers. = Latency Max got noticeable worse, but up to 90pct the increase is just a = few milliseconds...=20 And for simple.qos with IFB-based ingress shaping: simple.qos: IPv6, download and upload sequentially: 2015-03-29 01:49:36 Testing against netperf-eu.bufferbloat.net (ipv6) = with 4 simultaneous sessions while pinging netperf-eu.bufferbloat.net = (150 seconds in each direction) = ..........................................................................= ..........................................................................= .. Download: 80.24 Mbps Latency: (in msec, 150 pings, 0.00% packet loss) Min: 37.700=20 10pct: 38.500=20 Median: 39.700=20 Avg: 40.285=20 90pct: 42.700=20 Max: 46.500 = ..........................................................................= ..........................................................................= .. Upload: 32.66 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 39.700=20 10pct: 40.300=20 Median: 40.600=20 Avg: 40.694=20 90pct: 41.200=20 Max: 43.200 IFB+diffserv cost: 85.35-80.24 =3D 5.11Mbps download only, upload is not = CPU bound and hence not affected... simple.qos: IPv6, download and upload simultaneously: 2015-03-29 01:54:37 Testing netperf-eu.bufferbloat.net (ipv6) with 4 = streams down and up while pinging netperf-eu.bufferbloat.net. Takes = about 150 seconds. Download: 73.68 Mbps Upload: 10.32 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 40.300=20 10pct: 41.300=20 Median: 42.400=20 Avg: 42.904=20 90pct: 45.500=20 Max: 50.800 IFB+diffserv cost =3D (81.42 - 73.68) + ( 9.33 -10.32) =3D 6.75Mbps or = 100*6.75/(73.68+10.32) =3D 8%, still not enough to qualify the wndr3700 = to the nominally 100/40 Mbps tier I am testing, but enough to warrant = looking at improving diffserv (or better yet switching to cake?) simple.qos: IPv4, download and upload sequentially: 2015-03-29 01:57:07 Testing against netperf-eu.bufferbloat.net (ipv4) = with 4 simultaneous sessions while pinging netperf-eu.bufferbloat.net = (150 seconds in each direction) = ..........................................................................= ..........................................................................= .. Download: 82.3 Mbps Latency: (in msec, 150 pings, 0.00% packet loss) Min: 51.900=20 10pct: 52.800=20 Median: 53.700=20 Avg: 53.922=20 90pct: 55.000=20 Max: 60.300 = ..........................................................................= ..........................................................................= .. Upload: 33.43 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 51.800=20 10pct: 52.300=20 Median: 52.600=20 Avg: 52.657=20 90pct: 53.000=20 Max: 54.200 IFB+diffserv cost: 86.52-82.3 =3D 4.22 Mbps download only, upload is not = CPU bound and hence not affected... simple.qos: IPv4, download and upload simultaneously: 2015-03-29 03:02:08 Testing netperf-eu.bufferbloat.net (ipv4) with 4 = streams down and up while pinging netperf-eu.bufferbloat.net. Takes = about 150 seconds. Download: 76.71 Mbps Upload: 7.94 Mbps Latency: (in msec, 151 pings, 0.00% packet loss) Min: 51.700=20 10pct: 52.500=20 Median: 53.900=20 Avg: 54.145=20 90pct: 56.100=20 Max: 59.700 IFB+diffserv cost =3D (81.42 - 76.71) + ( 9.33 -7.94) =3D 6.1 Mbps or = 100*6.1/(76.71+7.94) =3D 7.2%. > The aggregate is still significantly higher with double egress, = though. >=20 > The ping numbers also tell me that there's no significant latency = penalty either way. Even when CPU saturated, it's still effectively = controlling the latency better than leaving the pipe open. Yes, that is a pretty nice degradation mode. Now if the upload = would to have to bear the brunt of the lacking CPU cycles=85 Best Regards Sebastian >=20 > - Jonathan Morton