From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wg0-x230.google.com (mail-wg0-x230.google.com [IPv6:2a00:1450:400c:c00::230]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 1150F21F1B6 for ; Thu, 31 Oct 2013 18:09:55 -0700 (PDT) Received: by mail-wg0-f48.google.com with SMTP id b13so3489714wgh.27 for ; Thu, 31 Oct 2013 18:09:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=lIyd9/B/cE+ZVyQj7LN4ZpCd9Rl36W2NqQVF6/3ZJpU=; b=e4HhJLatodeuyCXg0wrPHx7VOV5yPgnFSv/VhllEzfOP5c2ucfe06AEifulca4x9iJ uh36P+nAcFri1J4vagKK7qNPh/FKePM7o2QaDVlmMyV8Ate+SEAGMt7c/icr/0DpZM4T GgkWRRxroKvtn/31vS3tx8y0yI4pKXGN3w93Bs6nSQlyI+yCT6Moi6B5gICDoYqp70if GxA2nPTiQrgv2rL/WD7DAYFuYSeSQ1s1BN9Z65Z/yphdRmL0Aoi1tYhS+DHhbWrlH4Ck btPzFXze/vUmPDmx2QW7YddtbupqLZ0OMg+IZgD1jnf02JbyRFGdboAA8O0K5BoBSkHG YZfQ== MIME-Version: 1.0 X-Received: by 10.180.83.194 with SMTP id s2mr362959wiy.60.1383268193857; Thu, 31 Oct 2013 18:09:53 -0700 (PDT) Received: by 10.217.51.5 with HTTP; Thu, 31 Oct 2013 18:09:53 -0700 (PDT) In-Reply-To: <52724CA9.4020606@kau.se> References: <20131030162534.29f34ada@nehalam.linuxnetplumber.net> <5271BAA7.2080905@kau.se> <20131031023257.GB3365@lists.bufferbloat.net> <52724CA9.4020606@kau.se> Date: Thu, 31 Oct 2013 18:09:53 -0700 Message-ID: From: Dave Taht To: Stefan Alfredsson Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: bloat Subject: Re: [Bloat] mobile broadband buffer bloat X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Nov 2013 01:09:56 -0000 On Thu, Oct 31, 2013 at 5:27 AM, Stefan Alfredsson wrote: > On 10/31/13 03:32 , Dave Taht wrote: >> >> On Thu, Oct 31, 2013 at 03:04:23AM +0100, Stefan Alfredsson wrote: >>> >>> Hi, Stephen, the list, >>> >>> We're running a 3G/4G measurement site at Karlstad university, with >>> mobile broadband subscriptions from the four major telecom operators >>> in Sweden. I was curious to see how measurements in our network >>> would compare to yours, so I did a measurement run with netalyzer. >> >> Groovy! >> >> Can I ask that you try something that can more likely stress >> out that bus than netanlyzer? The netperf-wrappers tools >> (notably the tcp_bidirectional and rrul test), use C, rather than java. > > > Sure! Actually we've been doing netperf-wrapper rrul measurements 24/4 > during July to September, roughly 3-4 times per day, in total over 3000 > individual tests. > > I think there's a lot of interesting information in there, but I have not > done aggregate analysis yet (eg. looking at time-of-day effects, long-ter= m > trends, or the 10.000-students-arriving-in-august effect :-) ) if you are interested in measuring one way delays, owamp + a gps is working out well. > I packed the rrul-dumps for one day (1 sept 2013) and put it here: > http://www.cs.kau.se/stefalfr/3G4G/rrul-tests-2013-09-01.tgz . > > Here's an overview of this set, showing the per-run mean RTT, download an= d > upload throughput: > > [alfs@sa-pc]:/tmp/n/2013-06> for f in */*/*/rrul*json.gz ; do test=3D$(ec= ho $f > | cut -d / -f 1-2); netperf-wrapper -i $f -f stats|grep -A 4 'avg:'| awk = -v > test=3D$test '/Mean:/ { if (NR=3D=3D3) { rtt=3D$2 } if (NR=3D=3D10) {dl= =3D$2} if (NR=3D=3D16) > {ul=3D$2; printf "%s:\tRTT: %.1f ms DL: %.1f Mbit/s UL: %.1f Mbit/s\n", t= est, > rtt, dl, ul}}'; done > Tele2/35G: RTT: 490.7 ms DL: 1.3 Mbit/s UL: 0.3 Mbit/s > Tele2/35G: RTT: 498.0 ms DL: 6.1 Mbit/s UL: 0.4 Mbit/s > Tele2/35G: RTT: 562.7 ms DL: 4.7 Mbit/s UL: 0.2 Mbit/s > Tele2/35G: RTT: 641.5 ms DL: 3.5 Mbit/s UL: 0.3 Mbit/s > Tele2/4G: RTT: 891.5 ms DL: 2.3 Mbit/s UL: 0.3 Mbit/s > Tele2/4G: RTT: 389.0 ms DL: 20.8 Mbit/s UL: 1.0 Mbit/s > Tele2/4G: RTT: 361.2 ms DL: 14.4 Mbit/s UL: 1.4 Mbit/s > Tele2/4G: RTT: 359.8 ms DL: 17.3 Mbit/s UL: 1.3 Mbit/s > Telenor/35G: RTT: 809.0 ms DL: 2.2 Mbit/s UL: 0.8 Mbit/s > Telenor/35G: RTT: 377.3 ms DL: 3.8 Mbit/s UL: 0.5 Mbit/s > Telenor/35G: RTT: 727.2 ms DL: 2.9 Mbit/s UL: 0.3 Mbit/s > Telenor/35G: RTT: 341.7 ms DL: 0.1 Mbit/s UL: 0.2 Mbit/s > Telenor/4G: RTT: 427.8 ms DL: 16.9 Mbit/s UL: 0.8 Mbit/s > Telenor/4G: RTT: 378.8 ms DL: 17.9 Mbit/s UL: 1.2 Mbit/s > Telenor/4G: RTT: 623.8 ms DL: 8.6 Mbit/s UL: 0.6 Mbit/s > Telenor/4G: RTT: 321.5 ms DL: 18.2 Mbit/s UL: 1.0 Mbit/s > Telia/35G: RTT: 686.9 ms DL: 4.8 Mbit/s UL: 0.2 Mbit/s > Telia/35G: RTT: 709.0 ms DL: 6.1 Mbit/s UL: 0.2 Mbit/s > Telia/35G: RTT: 546.9 ms DL: 3.8 Mbit/s UL: 0.2 Mbit/s > Telia/35G: RTT: 653.0 ms DL: 3.1 Mbit/s UL: 0.3 Mbit/s > Telia/4G: RTT: 233.6 ms DL: 14.1 Mbit/s UL: 1.1 Mbit/s > Telia/4G: RTT: 324.2 ms DL: 14.6 Mbit/s UL: 1.1 Mbit/s > Telia/4G: RTT: 381.5 ms DL: 15.2 Mbit/s UL: 1.2 Mbit/s > Tre/35G: RTT: 312.2 ms DL: 3.1 Mbit/s UL: 0.6 Mbit/s > Tre/35G: RTT: 376.1 ms DL: 4.1 Mbit/s UL: 0.8 Mbit/s > Tre/35G: RTT: 442.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s > Tre/35G: RTT: 294.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s > Tre/4G: RTT: 471.2 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s > Tre/4G: RTT: 396.2 ms DL: 6.8 Mbit/s UL: 0.7 Mbit/s > Tre/4G: RTT: 237.5 ms DL: 7.2 Mbit/s UL: 0.7 Mbit/s > Tre/4G: RTT: 321.5 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s Oh, that is wonderful data thank you! I confess to be more interested in outliers than averages, and induced delay over base delay. However, I'm concerned that you are not getting a correct result with your script above. I just took a quick look at telia-4g, using the 2100-2013-09-01T200038 sample set, and looking at the raw (rrul) data: What I see here http://snapon.lab.bufferbloat.net/~d/telia-4g.svg is 60-80 ms of induced latency on this link, with 40ms of baseline latency. Not a RTT of > 300! >>> To summarize, the measured buffering is well below one second for >>> all tests, although the results are not directly comparable since we >>> are using directly attached USB modems (Huawei E392 with Ubuntu >>> 12.04) >> >> ubuntu has backported fq_codel as far as 12.04. What kernel are you usin= g? > > On the "mobile terminal" we have 3.2.0-54-generic-pae #82-Ubuntu SMP I am delighted you've been collecting this data long term. I do have a problem in that the 3.2 kernel predates all the bufferbloat related fixes made in the 4 years since that release. we've been working our butts off here... :( Any chance you can update to 3.10.x or later for both this (driver and buffering fixes) and your fixed host? (tcp fixes). And it's generally been my hope people would try fq_codel (or sfq or pie or red) on such links, too... There has been very little done to optimize usb-net, although we tried. If this is using the ppp driver, that was improved substantially around 3.6? > The fixed host ("tptest") runs a self-compiled stock kernel 3.4.1 with > web10g patches, since we were interested in extracting TCP variable data = not > available from packet traces. > >>> instead of a separate router box. However, the numbers might >>> be interesting in a more general perspective. >> >> Yes, they are. These are repeatable? >> >> > > For the netalyzer I've only done singular tests, but it can easily be > repeated if needed (probably not). > The mean of all mean RTTs for 4G, all operators, all times, is 334 ms, an= d > for 3.5G it's 573 ms. > This indicates that results are repeatable, and the set from 2013-09-01 i= s > representative. except that my plot shows differently... and you are running an ancient kernel, yes. Something of a major variable, that... and if you could let us know the usb path to the various underlying drivers we can poke into matters there? > > "below one second" was in relation to the >5 second RTT's, ideally it sho= uld > be much lower of course. As a side note, we also do measurements in unloa= ded > networks, and for 4G we get around 40-60 ms RTT (ICMP and TCP packets). > > Also, you might be interested in a paper we published in June, that repor= ts > on measurements of bloat for short flows in relation to different congest= ion > control algorithms, in 3G, 3.5G and 4G (but for a single operator). > See http://urn.kb.se/resolve?urn=3Durn:nbn:se:kau:diva-27779 > with fulltext at http://dx.doi.org/10.1109/WoWMoM.2013.6583408. > I have also put a local copy (for personal use etc) at > http://www.cs.kau.se/stefalfr/3G4G/alfredsson-bufferbloat-wowmom13.pdf > > > Regards, > Stefan > > -- > Stefan Alfredsson, PhD Tel: +46 (0) 54 700 1668 > Datavetenskap Kontor: 21F-425 (Hus Vanern) > Karlstads universitet PGP 0xB19B4B16 > SE-651 88 Karlstad http://www.cs.kau.se/stefalfr/ > > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat --=20 Dave T=E4ht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.= html