[Bloat] mobile broadband buffer bloat

Thu Oct 31 21:39:27 EDT 2013

oops that was telenor I was commenting on

http://snapon.lab.bufferbloat.net/~d/telenor-4g.svg

On Thu, Oct 31, 2013 at 6:32 PM, Dave Taht <dave.taht at gmail.com> wrote:
> Just did telia. I really want 4G as good as this!!!
>
> http://snapon.lab.bufferbloat.net/~d/telia-4g.svg
>
> So you multiply the black line x4 to get a total for the down or up,
> then insert a fudge factor for the amount of acks that probably went
> into the down relative to the up and vice versa, and you sort of have
> your bandwidth number each way.
>
> I'm really impressed you can get ~72Mbit down and ~4Mbit up. (closer
> to 8 up when you consider acks) Note also the relationship between
> bandwidth and latency at T+22...
>
>
> On Thu, Oct 31, 2013 at 6:09 PM, Dave Taht <dave.taht at gmail.com> wrote:
>> On Thu, Oct 31, 2013 at 5:27 AM, Stefan Alfredsson
>> <Stefan.Alfredsson at kau.se> wrote:
>>> On 10/31/13 03:32 , Dave Taht wrote:
>>>>
>>>> On Thu, Oct 31, 2013 at 03:04:23AM +0100, Stefan Alfredsson wrote:
>>>>>
>>>>> Hi, Stephen, the list,
>>>>>
>>>>> We're running a 3G/4G measurement site at Karlstad university, with
>>>>> mobile broadband subscriptions from the four major telecom operators
>>>>> in Sweden. I was curious to see how measurements in our network
>>>>> would compare to yours, so I did a measurement run with netalyzer.
>>>>
>>>> Groovy!
>>>>
>>>> Can I ask that you try something that can more likely stress
>>>> out that bus than netanlyzer? The netperf-wrappers tools
>>>> (notably the tcp_bidirectional and rrul test), use C, rather than java.
>>>
>>>
>>> Sure! Actually we've been doing netperf-wrapper rrul measurements 24/4
>>> during July to September, roughly 3-4 times per day, in total over 3000
>>> individual tests.
>>>
>>> I think there's a lot of interesting information in there, but I have not
>>> done aggregate analysis yet (eg. looking at time-of-day effects, long-term
>>> trends, or the 10.000-students-arriving-in-august effect :-) )
>>
>> if you are interested in measuring one way delays, owamp + a gps is
>> working out well.
>>
>>> I packed the rrul-dumps for one day (1 sept 2013) and put it here:
>>> http://www.cs.kau.se/stefalfr/3G4G/rrul-tests-2013-09-01.tgz .
>>>
>>> Here's an overview of this set, showing the per-run mean RTT, download and
>>> upload throughput:
>>>
>>> [alfs at sa-pc]:/tmp/n/2013-06> for f in */*/*/rrul*json.gz ; do test=$(echo $f
>>> | cut -d / -f 1-2); netperf-wrapper -i $f -f stats|grep -A 4 'avg:'| awk -v
>>> test=$test '/Mean:/ { if (NR==3) { rtt=$2 } if (NR==10) {dl=$2} if (NR==16)
>>> {ul=$2; printf "%s:\tRTT: %.1f ms DL: %.1f Mbit/s UL: %.1f Mbit/s\n", test,
>>> rtt, dl, ul}}'; done
>>> Tele2/35G:      RTT: 490.7 ms DL: 1.3 Mbit/s UL: 0.3 Mbit/s
>>> Tele2/35G:      RTT: 498.0 ms DL: 6.1 Mbit/s UL: 0.4 Mbit/s
>>> Tele2/35G:      RTT: 562.7 ms DL: 4.7 Mbit/s UL: 0.2 Mbit/s
>>> Tele2/35G:      RTT: 641.5 ms DL: 3.5 Mbit/s UL: 0.3 Mbit/s
>>> Tele2/4G:       RTT: 891.5 ms DL: 2.3 Mbit/s UL: 0.3 Mbit/s
>>> Tele2/4G:       RTT: 389.0 ms DL: 20.8 Mbit/s UL: 1.0 Mbit/s
>>> Tele2/4G:       RTT: 361.2 ms DL: 14.4 Mbit/s UL: 1.4 Mbit/s
>>> Tele2/4G:       RTT: 359.8 ms DL: 17.3 Mbit/s UL: 1.3 Mbit/s
>>> Telenor/35G:    RTT: 809.0 ms DL: 2.2 Mbit/s UL: 0.8 Mbit/s
>>> Telenor/35G:    RTT: 377.3 ms DL: 3.8 Mbit/s UL: 0.5 Mbit/s
>>> Telenor/35G:    RTT: 727.2 ms DL: 2.9 Mbit/s UL: 0.3 Mbit/s
>>> Telenor/35G:    RTT: 341.7 ms DL: 0.1 Mbit/s UL: 0.2 Mbit/s
>>> Telenor/4G:     RTT: 427.8 ms DL: 16.9 Mbit/s UL: 0.8 Mbit/s
>>> Telenor/4G:     RTT: 378.8 ms DL: 17.9 Mbit/s UL: 1.2 Mbit/s
>>> Telenor/4G:     RTT: 623.8 ms DL: 8.6 Mbit/s UL: 0.6 Mbit/s
>>> Telenor/4G:     RTT: 321.5 ms DL: 18.2 Mbit/s UL: 1.0 Mbit/s
>>> Telia/35G:      RTT: 686.9 ms DL: 4.8 Mbit/s UL: 0.2 Mbit/s
>>> Telia/35G:      RTT: 709.0 ms DL: 6.1 Mbit/s UL: 0.2 Mbit/s
>>> Telia/35G:      RTT: 546.9 ms DL: 3.8 Mbit/s UL: 0.2 Mbit/s
>>> Telia/35G:      RTT: 653.0 ms DL: 3.1 Mbit/s UL: 0.3 Mbit/s
>>> Telia/4G:       RTT: 233.6 ms DL: 14.1 Mbit/s UL: 1.1 Mbit/s
>>> Telia/4G:       RTT: 324.2 ms DL: 14.6 Mbit/s UL: 1.1 Mbit/s
>>> Telia/4G:       RTT: 381.5 ms DL: 15.2 Mbit/s UL: 1.2 Mbit/s
>>> Tre/35G:        RTT: 312.2 ms DL: 3.1 Mbit/s UL: 0.6 Mbit/s
>>> Tre/35G:        RTT: 376.1 ms DL: 4.1 Mbit/s UL: 0.8 Mbit/s
>>> Tre/35G:        RTT: 442.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
>>> Tre/35G:        RTT: 294.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
>>> Tre/4G: RTT: 471.2 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s
>>> Tre/4G: RTT: 396.2 ms DL: 6.8 Mbit/s UL: 0.7 Mbit/s
>>> Tre/4G: RTT: 237.5 ms DL: 7.2 Mbit/s UL: 0.7 Mbit/s
>>> Tre/4G: RTT: 321.5 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s
>>
>> Oh, that is wonderful data thank you! I confess to be more interested
>> in outliers than averages, and induced delay over base delay.
>>
>> However, I'm concerned that you are not getting a correct result with
>> your script above.
>>
>> I just took a quick look at telia-4g, using the 2100-2013-09-01T200038
>> sample set, and looking at the raw (rrul) data: What I see here
>>
>> http://snapon.lab.bufferbloat.net/~d/telia-4g.svg
>>
>> is 60-80 ms of induced latency on this link, with 40ms of baseline
>> latency. Not a RTT of > 300!
>>
>>
>>>>> To summarize, the measured buffering is well below one second for
>>>>> all tests, although the results are not directly comparable since we
>>>>> are using directly attached USB modems (Huawei E392 with Ubuntu
>>>>> 12.04)
>>>>
>>>> ubuntu has backported fq_codel as far as 12.04. What kernel are you using?
>>>
>>> On the "mobile terminal" we have 3.2.0-54-generic-pae #82-Ubuntu SMP
>>
>> I am delighted you've been collecting this data long term.
>>
>> I do have a problem in that the 3.2 kernel predates all the
>> bufferbloat related fixes made in the 4 years since that release.
>> we've been working our butts off here... :(
>>
>> Any chance you can update to 3.10.x or later for both this (driver and
>> buffering fixes) and your fixed host? (tcp fixes). And it's generally
>> been my hope people would try fq_codel (or sfq or pie or red) on such
>> links, too...
>>
>> There has been very little done to optimize usb-net, although we
>> tried. If this is using the ppp driver, that was improved
>> substantially around 3.6?
>>
>>> The fixed host ("tptest") runs a self-compiled stock kernel 3.4.1 with
>>> web10g patches, since we were interested in extracting TCP variable data not
>>> available from packet traces.
>>>
>>>>> instead of a separate router box. However, the numbers might
>>>>> be interesting in a more general perspective.
>>>>
>>>> Yes, they are. These are repeatable?
>>>>
>>>>
>>>
>>> For the netalyzer I've only done singular tests, but it can easily be
>>> repeated if needed (probably not).
>>> The mean of all mean RTTs for 4G, all operators, all times, is 334 ms, and
>>> for 3.5G it's 573 ms.
>>> This indicates that results are repeatable, and the set from 2013-09-01 is
>>> representative.
>>
>> except that my plot shows differently...
>>
>> and you are running an ancient kernel, yes. Something of a major
>> variable, that... and if you could let us know the usb path to the
>> various underlying drivers we can poke into matters there?
>>
>>>
>>> "below one second" was in relation to the >5 second RTT's, ideally it should
>>> be much lower of course. As a side note, we also do measurements in unloaded
>>> networks, and for 4G we get around 40-60 ms RTT (ICMP and TCP packets).
>>>
>>> Also, you might be interested in a paper we published in June, that reports
>>> on measurements of bloat for short flows in relation to different congestion
>>> control algorithms, in 3G, 3.5G and 4G (but for a single operator).
>>> See http://urn.kb.se/resolve?urn=urn:nbn:se:kau:diva-27779
>>> with fulltext at http://dx.doi.org/10.1109/WoWMoM.2013.6583408.
>>> I have also put a local copy (for personal use etc) at
>>> http://www.cs.kau.se/stefalfr/3G4G/alfredsson-bufferbloat-wowmom13.pdf
>>>
>>>
>>> Regards,
>>>  Stefan
>>>
>>> --
>>> Stefan Alfredsson, PhD         Tel: +46 (0) 54 700 1668
>>> Datavetenskap                  Kontor: 21F-425 (Hus Vanern)
>>> Karlstads universitet          PGP 0xB19B4B16
>>> SE-651 88 Karlstad             http://www.cs.kau.se/stefalfr/
>>>
>>>
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat at lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>>
>>
>>
>> --
>> Dave Täht
>>
>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
>
>
>
> --
> Dave Täht
>
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html


-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html