[Bloat] mobile broadband buffer bloat

[Stefan Alfredsson]

On 10/31/13 03:32 , Dave Taht wrote:
> On Thu, Oct 31, 2013 at 03:04:23AM +0100, Stefan Alfredsson wrote:
>> Hi, Stephen, the list,
>>
>> We're running a 3G/4G measurement site at Karlstad university, with
>> mobile broadband subscriptions from the four major telecom operators
>> in Sweden. I was curious to see how measurements in our network
>> would compare to yours, so I did a measurement run with netalyzer.
> Groovy!
>
> Can I ask that you try something that can more likely stress
> out that bus than netanlyzer? The netperf-wrappers tools
> (notably the tcp_bidirectional and rrul test), use C, rather than java.

Sure! Actually we've been doing netperf-wrapper rrul measurements 24/4 
during July to September, roughly 3-4 times per day, in total over 3000 
individual tests.

I think there's a lot of interesting information in there, but I have 
not done aggregate analysis yet (eg. looking at time-of-day effects, 
long-term trends, or the 10.000-students-arriving-in-august effect :-) )

I packed the rrul-dumps for one day (1 sept 2013) and put it here: 
http://www.cs.kau.se/stefalfr/3G4G/rrul-tests-2013-09-01.tgz .

Here's an overview of this set, showing the per-run mean RTT, download 
and upload throughput:

[alfs at sa-pc]:/tmp/n/2013-06> for f in */*/*/rrul*json.gz ; do 
test=$(echo $f | cut -d / -f 1-2); netperf-wrapper -i $f -f stats|grep 
-A 4 'avg:'| awk -v test=$test '/Mean:/ { if (NR==3) { rtt=$2 } if 
(NR==10) {dl=$2} if (NR==16) {ul=$2; printf "%s:\tRTT: %.1f ms DL: %.1f 
Mbit/s UL: %.1f Mbit/s\n", test, rtt, dl, ul}}'; done
Tele2/35G:      RTT: 490.7 ms DL: 1.3 Mbit/s UL: 0.3 Mbit/s
Tele2/35G:      RTT: 498.0 ms DL: 6.1 Mbit/s UL: 0.4 Mbit/s
Tele2/35G:      RTT: 562.7 ms DL: 4.7 Mbit/s UL: 0.2 Mbit/s
Tele2/35G:      RTT: 641.5 ms DL: 3.5 Mbit/s UL: 0.3 Mbit/s
Tele2/4G:       RTT: 891.5 ms DL: 2.3 Mbit/s UL: 0.3 Mbit/s
Tele2/4G:       RTT: 389.0 ms DL: 20.8 Mbit/s UL: 1.0 Mbit/s
Tele2/4G:       RTT: 361.2 ms DL: 14.4 Mbit/s UL: 1.4 Mbit/s
Tele2/4G:       RTT: 359.8 ms DL: 17.3 Mbit/s UL: 1.3 Mbit/s
Telenor/35G:    RTT: 809.0 ms DL: 2.2 Mbit/s UL: 0.8 Mbit/s
Telenor/35G:    RTT: 377.3 ms DL: 3.8 Mbit/s UL: 0.5 Mbit/s
Telenor/35G:    RTT: 727.2 ms DL: 2.9 Mbit/s UL: 0.3 Mbit/s
Telenor/35G:    RTT: 341.7 ms DL: 0.1 Mbit/s UL: 0.2 Mbit/s
Telenor/4G:     RTT: 427.8 ms DL: 16.9 Mbit/s UL: 0.8 Mbit/s
Telenor/4G:     RTT: 378.8 ms DL: 17.9 Mbit/s UL: 1.2 Mbit/s
Telenor/4G:     RTT: 623.8 ms DL: 8.6 Mbit/s UL: 0.6 Mbit/s
Telenor/4G:     RTT: 321.5 ms DL: 18.2 Mbit/s UL: 1.0 Mbit/s
Telia/35G:      RTT: 686.9 ms DL: 4.8 Mbit/s UL: 0.2 Mbit/s
Telia/35G:      RTT: 709.0 ms DL: 6.1 Mbit/s UL: 0.2 Mbit/s
Telia/35G:      RTT: 546.9 ms DL: 3.8 Mbit/s UL: 0.2 Mbit/s
Telia/35G:      RTT: 653.0 ms DL: 3.1 Mbit/s UL: 0.3 Mbit/s
Telia/4G:       RTT: 233.6 ms DL: 14.1 Mbit/s UL: 1.1 Mbit/s
Telia/4G:       RTT: 324.2 ms DL: 14.6 Mbit/s UL: 1.1 Mbit/s
Telia/4G:       RTT: 381.5 ms DL: 15.2 Mbit/s UL: 1.2 Mbit/s
Tre/35G:        RTT: 312.2 ms DL: 3.1 Mbit/s UL: 0.6 Mbit/s
Tre/35G:        RTT: 376.1 ms DL: 4.1 Mbit/s UL: 0.8 Mbit/s
Tre/35G:        RTT: 442.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
Tre/35G:        RTT: 294.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
Tre/4G: RTT: 471.2 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s
Tre/4G: RTT: 396.2 ms DL: 6.8 Mbit/s UL: 0.7 Mbit/s
Tre/4G: RTT: 237.5 ms DL: 7.2 Mbit/s UL: 0.7 Mbit/s
Tre/4G: RTT: 321.5 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s

>> To summarize, the measured buffering is well below one second for
>> all tests, although the results are not directly comparable since we
>> are using directly attached USB modems (Huawei E392 with Ubuntu
>> 12.04)
> ubuntu has backported fq_codel as far as 12.04. What kernel are you using?
On the "mobile terminal" we have 3.2.0-54-generic-pae #82-Ubuntu SMP

The fixed host ("tptest") runs a self-compiled stock kernel 3.4.1 with 
web10g patches, since we were interested in extracting TCP variable data 
not available from packet traces.

>> instead of a separate router box. However, the numbers might
>> be interesting in a more general perspective.
> Yes, they are. These are repeatable?
>
>

For the netalyzer I've only done singular tests, but it can easily be 
repeated if needed (probably not).
The mean of all mean RTTs for 4G, all operators, all times, is 334 ms, 
and for 3.5G it's 573 ms.
This indicates that results are repeatable, and the set from 2013-09-01 
is representative.

"below one second" was in relation to the >5 second RTT's, ideally it 
should be much lower of course. As a side note, we also do measurements 
in unloaded networks, and for 4G we get around 40-60 ms RTT (ICMP and 
TCP packets).

Also, you might be interested in a paper we published in June, that 
reports on measurements of bloat for short flows in relation to 
different congestion control algorithms, in 3G, 3.5G and 4G (but for a 
single operator).
See http://urn.kb.se/resolve?urn=urn:nbn:se:kau:diva-27779
with fulltext at http://dx.doi.org/10.1109/WoWMoM.2013.6583408.
I have also put a local copy (for personal use etc) at 
http://www.cs.kau.se/stefalfr/3G4G/alfredsson-bufferbloat-wowmom13.pdf


Regards,
  Stefan

-- 
Stefan Alfredsson, PhD         Tel: +46 (0) 54 700 1668
Datavetenskap                  Kontor: 21F-425 (Hus Vanern)
Karlstads universitet          PGP 0xB19B4B16
SE-651 88 Karlstad             http://www.cs.kau.se/stefalfr/