[Bloat] mobile broadband buffer bloat
Dave Taht
dave.taht at gmail.com
Thu Oct 31 21:09:53 EDT 2013
On Thu, Oct 31, 2013 at 5:27 AM, Stefan Alfredsson
<Stefan.Alfredsson at kau.se> wrote:
> On 10/31/13 03:32 , Dave Taht wrote:
>>
>> On Thu, Oct 31, 2013 at 03:04:23AM +0100, Stefan Alfredsson wrote:
>>>
>>> Hi, Stephen, the list,
>>>
>>> We're running a 3G/4G measurement site at Karlstad university, with
>>> mobile broadband subscriptions from the four major telecom operators
>>> in Sweden. I was curious to see how measurements in our network
>>> would compare to yours, so I did a measurement run with netalyzer.
>>
>> Groovy!
>>
>> Can I ask that you try something that can more likely stress
>> out that bus than netanlyzer? The netperf-wrappers tools
>> (notably the tcp_bidirectional and rrul test), use C, rather than java.
>
>
> Sure! Actually we've been doing netperf-wrapper rrul measurements 24/4
> during July to September, roughly 3-4 times per day, in total over 3000
> individual tests.
>
> I think there's a lot of interesting information in there, but I have not
> done aggregate analysis yet (eg. looking at time-of-day effects, long-term
> trends, or the 10.000-students-arriving-in-august effect :-) )
if you are interested in measuring one way delays, owamp + a gps is
working out well.
> I packed the rrul-dumps for one day (1 sept 2013) and put it here:
> http://www.cs.kau.se/stefalfr/3G4G/rrul-tests-2013-09-01.tgz .
>
> Here's an overview of this set, showing the per-run mean RTT, download and
> upload throughput:
>
> [alfs at sa-pc]:/tmp/n/2013-06> for f in */*/*/rrul*json.gz ; do test=$(echo $f
> | cut -d / -f 1-2); netperf-wrapper -i $f -f stats|grep -A 4 'avg:'| awk -v
> test=$test '/Mean:/ { if (NR==3) { rtt=$2 } if (NR==10) {dl=$2} if (NR==16)
> {ul=$2; printf "%s:\tRTT: %.1f ms DL: %.1f Mbit/s UL: %.1f Mbit/s\n", test,
> rtt, dl, ul}}'; done
> Tele2/35G: RTT: 490.7 ms DL: 1.3 Mbit/s UL: 0.3 Mbit/s
> Tele2/35G: RTT: 498.0 ms DL: 6.1 Mbit/s UL: 0.4 Mbit/s
> Tele2/35G: RTT: 562.7 ms DL: 4.7 Mbit/s UL: 0.2 Mbit/s
> Tele2/35G: RTT: 641.5 ms DL: 3.5 Mbit/s UL: 0.3 Mbit/s
> Tele2/4G: RTT: 891.5 ms DL: 2.3 Mbit/s UL: 0.3 Mbit/s
> Tele2/4G: RTT: 389.0 ms DL: 20.8 Mbit/s UL: 1.0 Mbit/s
> Tele2/4G: RTT: 361.2 ms DL: 14.4 Mbit/s UL: 1.4 Mbit/s
> Tele2/4G: RTT: 359.8 ms DL: 17.3 Mbit/s UL: 1.3 Mbit/s
> Telenor/35G: RTT: 809.0 ms DL: 2.2 Mbit/s UL: 0.8 Mbit/s
> Telenor/35G: RTT: 377.3 ms DL: 3.8 Mbit/s UL: 0.5 Mbit/s
> Telenor/35G: RTT: 727.2 ms DL: 2.9 Mbit/s UL: 0.3 Mbit/s
> Telenor/35G: RTT: 341.7 ms DL: 0.1 Mbit/s UL: 0.2 Mbit/s
> Telenor/4G: RTT: 427.8 ms DL: 16.9 Mbit/s UL: 0.8 Mbit/s
> Telenor/4G: RTT: 378.8 ms DL: 17.9 Mbit/s UL: 1.2 Mbit/s
> Telenor/4G: RTT: 623.8 ms DL: 8.6 Mbit/s UL: 0.6 Mbit/s
> Telenor/4G: RTT: 321.5 ms DL: 18.2 Mbit/s UL: 1.0 Mbit/s
> Telia/35G: RTT: 686.9 ms DL: 4.8 Mbit/s UL: 0.2 Mbit/s
> Telia/35G: RTT: 709.0 ms DL: 6.1 Mbit/s UL: 0.2 Mbit/s
> Telia/35G: RTT: 546.9 ms DL: 3.8 Mbit/s UL: 0.2 Mbit/s
> Telia/35G: RTT: 653.0 ms DL: 3.1 Mbit/s UL: 0.3 Mbit/s
> Telia/4G: RTT: 233.6 ms DL: 14.1 Mbit/s UL: 1.1 Mbit/s
> Telia/4G: RTT: 324.2 ms DL: 14.6 Mbit/s UL: 1.1 Mbit/s
> Telia/4G: RTT: 381.5 ms DL: 15.2 Mbit/s UL: 1.2 Mbit/s
> Tre/35G: RTT: 312.2 ms DL: 3.1 Mbit/s UL: 0.6 Mbit/s
> Tre/35G: RTT: 376.1 ms DL: 4.1 Mbit/s UL: 0.8 Mbit/s
> Tre/35G: RTT: 442.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
> Tre/35G: RTT: 294.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
> Tre/4G: RTT: 471.2 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s
> Tre/4G: RTT: 396.2 ms DL: 6.8 Mbit/s UL: 0.7 Mbit/s
> Tre/4G: RTT: 237.5 ms DL: 7.2 Mbit/s UL: 0.7 Mbit/s
> Tre/4G: RTT: 321.5 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s
Oh, that is wonderful data thank you! I confess to be more interested
in outliers than averages, and induced delay over base delay.
However, I'm concerned that you are not getting a correct result with
your script above.
I just took a quick look at telia-4g, using the 2100-2013-09-01T200038
sample set, and looking at the raw (rrul) data: What I see here
http://snapon.lab.bufferbloat.net/~d/telia-4g.svg
is 60-80 ms of induced latency on this link, with 40ms of baseline
latency. Not a RTT of > 300!
>>> To summarize, the measured buffering is well below one second for
>>> all tests, although the results are not directly comparable since we
>>> are using directly attached USB modems (Huawei E392 with Ubuntu
>>> 12.04)
>>
>> ubuntu has backported fq_codel as far as 12.04. What kernel are you using?
>
> On the "mobile terminal" we have 3.2.0-54-generic-pae #82-Ubuntu SMP
I am delighted you've been collecting this data long term.
I do have a problem in that the 3.2 kernel predates all the
bufferbloat related fixes made in the 4 years since that release.
we've been working our butts off here... :(
Any chance you can update to 3.10.x or later for both this (driver and
buffering fixes) and your fixed host? (tcp fixes). And it's generally
been my hope people would try fq_codel (or sfq or pie or red) on such
links, too...
There has been very little done to optimize usb-net, although we
tried. If this is using the ppp driver, that was improved
substantially around 3.6?
> The fixed host ("tptest") runs a self-compiled stock kernel 3.4.1 with
> web10g patches, since we were interested in extracting TCP variable data not
> available from packet traces.
>
>>> instead of a separate router box. However, the numbers might
>>> be interesting in a more general perspective.
>>
>> Yes, they are. These are repeatable?
>>
>>
>
> For the netalyzer I've only done singular tests, but it can easily be
> repeated if needed (probably not).
> The mean of all mean RTTs for 4G, all operators, all times, is 334 ms, and
> for 3.5G it's 573 ms.
> This indicates that results are repeatable, and the set from 2013-09-01 is
> representative.
except that my plot shows differently...
and you are running an ancient kernel, yes. Something of a major
variable, that... and if you could let us know the usb path to the
various underlying drivers we can poke into matters there?
>
> "below one second" was in relation to the >5 second RTT's, ideally it should
> be much lower of course. As a side note, we also do measurements in unloaded
> networks, and for 4G we get around 40-60 ms RTT (ICMP and TCP packets).
>
> Also, you might be interested in a paper we published in June, that reports
> on measurements of bloat for short flows in relation to different congestion
> control algorithms, in 3G, 3.5G and 4G (but for a single operator).
> See http://urn.kb.se/resolve?urn=urn:nbn:se:kau:diva-27779
> with fulltext at http://dx.doi.org/10.1109/WoWMoM.2013.6583408.
> I have also put a local copy (for personal use etc) at
> http://www.cs.kau.se/stefalfr/3G4G/alfredsson-bufferbloat-wowmom13.pdf
>
>
> Regards,
> Stefan
>
> --
> Stefan Alfredsson, PhD Tel: +46 (0) 54 700 1668
> Datavetenskap Kontor: 21F-425 (Hus Vanern)
> Karlstads universitet PGP 0xB19B4B16
> SE-651 88 Karlstad http://www.cs.kau.se/stefalfr/
>
>
> _______________________________________________
> Bloat mailing list
> Bloat at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
--
Dave Täht
Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
More information about the Bloat
mailing list