[Bloat] T-Mobile LTE buffer bloat

[Bloat] T-Mobile LTE buffer bloat

[Stephen Hemminger]

I got one of these Samsung LTE hotspot.
Not surprisingly it has huge bloat and a stupid http proxy
that netalyzer claims rewrites images.
 
Bandwidth: Up 1.6 Mbit/sec Down 4.3Mbit/sec
Latency: 140ms 0% loss
Buffering: Uplink 5100ms Down 1800ms

How can the uplink side be so bad! 5 seconds???

Might even return it as defective. You can't even add a review on their
website. Probably they would end taking down like Apple.

Re: [Bloat] T-Mobile LTE buffer bloat

[Mikael Abrahamsson]

On Wed, 30 Oct 2013, Stephen Hemminger wrote:

> Not surprisingly it has huge bloat and a stupid http proxy that 
> netalyzer claims rewrites images.

This could be done in the provider network, did you try it without the 
thingie?

> How can the uplink side be so bad! 5 seconds???

My personal record for a mobile network is 180 seconds RTT. They *really* 
*really* want to deliver the packets, and if the radio environment go bad 
they'll still buffer 400 packets (or so).

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] T-Mobile LTE buffer bloat

[Dave Taht]

On Wed, Oct 30, 2013 at 4:34 PM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Wed, 30 Oct 2013, Stephen Hemminger wrote:
>
>> Not surprisingly it has huge bloat and a stupid http proxy that netalyzer
>> claims rewrites images.
>
>
> This could be done in the provider network, did you try it without the
> thingie?
>
>
>> How can the uplink side be so bad! 5 seconds???
>
>
> My personal record for a mobile network is 180 seconds RTT. They *really*
> *really* want to deliver the packets, and if the radio environment go bad
> they'll still buffer 400 packets (or so).

Darn. You win. The worst I've seen is 84 seconds.

>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Bloat] T-Mobile LTE buffer bloat

[Srikanth Sundaresan]

>> 
>> My personal record for a mobile network is 180 seconds RTT. They *really*
>> *really* want to deliver the packets, and if the radio environment go bad
>> they'll still buffer 400 packets (or so).
> 
> Darn. You win. The worst I've seen is 84 seconds.
> 

Here's a ping trace from t-mobile when there was a big music festival a block from my apartment. 122 seconds (but zero loss), before I get disconnected.

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=46 time=46656.509 ms
64 bytes from 8.8.8.8: seq=1 ttl=46 time=47046.226 ms
64 bytes from 8.8.8.8: seq=2 ttl=46 time=47210.621 ms
64 bytes from 8.8.8.8: seq=3 ttl=46 time=46300.097 ms
64 bytes from 8.8.8.8: seq=4 ttl=46 time=46490.259 ms
64 bytes from 8.8.8.8: seq=5 ttl=46 time=45543.573 ms
64 bytes from 8.8.8.8: seq=6 ttl=46 time=44610.118 ms
64 bytes from 8.8.8.8: seq=7 ttl=46 time=43659.564 ms
64 bytes from 8.8.8.8: seq=8 ttl=46 time=42677.023 ms
64 bytes from 8.8.8.8: seq=9 ttl=46 time=41747.164 ms
64 bytes from 8.8.8.8: seq=10 ttl=46 time=40834.181 ms
64 bytes from 8.8.8.8: seq=11 ttl=46 time=39834.719 ms
64 bytes from 8.8.8.8: seq=12 ttl=46 time=38898.367 ms
64 bytes from 8.8.8.8: seq=13 ttl=46 time=37966.098 ms
64 bytes from 8.8.8.8: seq=14 ttl=46 time=38126.381 ms
64 bytes from 8.8.8.8: seq=15 ttl=46 time=37173.727 ms
64 bytes from 8.8.8.8: seq=16 ttl=46 time=37393.522 ms
64 bytes from 8.8.8.8: seq=17 ttl=46 time=36405.295 ms
64 bytes from 8.8.8.8: seq=18 ttl=46 time=35495.237 ms
64 bytes from 8.8.8.8: seq=19 ttl=46 time=34522.977 ms
64 bytes from 8.8.8.8: seq=20 ttl=46 time=35750.939 ms
64 bytes from 8.8.8.8: seq=21 ttl=46 time=40660.576 ms
64 bytes from 8.8.8.8: seq=22 ttl=46 time=39730.537 ms
64 bytes from 8.8.8.8: seq=23 ttl=46 time=38730.503 ms
64 bytes from 8.8.8.8: seq=24 ttl=46 time=37804.354 ms
64 bytes from 8.8.8.8: seq=25 ttl=46 time=36819.838 ms
64 bytes from 8.8.8.8: seq=26 ttl=46 time=37153.646 ms
64 bytes from 8.8.8.8: seq=27 ttl=46 time=37413.711 ms
64 bytes from 8.8.8.8: seq=28 ttl=46 time=37581.268 ms
64 bytes from 8.8.8.8: seq=29 ttl=46 time=37733.075 ms
64 bytes from 8.8.8.8: seq=30 ttl=46 time=40190.826 ms
64 bytes from 8.8.8.8: seq=31 ttl=46 time=101158.557 ms
64 bytes from 8.8.8.8: seq=32 ttl=46 time=101264.386 ms
64 bytes from 8.8.8.8: seq=33 ttl=46 time=103640.134 ms
64 bytes from 8.8.8.8: seq=34 ttl=46 time=103819.819 ms
64 bytes from 8.8.8.8: seq=35 ttl=46 time=103958.008 ms
64 bytes from 8.8.8.8: seq=36 ttl=46 time=104057.496 ms
64 bytes from 8.8.8.8: seq=37 ttl=46 time=103081.290 ms
64 bytes from 8.8.8.8: seq=38 ttl=46 time=103191.206 ms
64 bytes from 8.8.8.8: seq=39 ttl=46 time=107700.964 ms
64 bytes from 8.8.8.8: seq=40 ttl=46 time=107818.726 ms
64 bytes from 8.8.8.8: seq=41 ttl=46 time=106833.938 ms
64 bytes from 8.8.8.8: seq=42 ttl=46 time=107042.706 ms
64 bytes from 8.8.8.8: seq=43 ttl=46 time=110530.331 ms
64 bytes from 8.8.8.8: seq=44 ttl=46 time=115255.932 ms
64 bytes from 8.8.8.8: seq=45 ttl=46 time=121191.910 ms
64 bytes from 8.8.8.8: seq=46 ttl=46 time=120217.588 ms
64 bytes from 8.8.8.8: seq=47 ttl=46 time=120653.489 ms
64 bytes from 8.8.8.8: seq=48 ttl=46 time=122025.125 ms
ping: sendto: Network is unreachable

Re: [Bloat] T-Mobile LTE buffer bloat

[Stephen Hemminger]

On Thu, 31 Oct 2013 00:34:12 +0100 (CET)
Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Wed, 30 Oct 2013, Stephen Hemminger wrote:
> 
> > Not surprisingly it has huge bloat and a stupid http proxy that 
> > netalyzer claims rewrites images.
> 
> This could be done in the provider network, did you try it without the 
> thingie?

There is zero configuration of proxy, it is just a LTE <-> wifi hotspot.
I suspect some caching (or commercial interest) at their end.

Re: [Bloat] T-Mobile LTE buffer bloat

[Mikael Abrahamsson]

On Wed, 30 Oct 2013, Stephen Hemminger wrote:

> There is zero configuration of proxy, it is just a LTE <-> wifi hotspot. 
> I suspect some caching (or commercial interest) at their end.

I have no insight in how T-Moble does things, but I know that vendors 
pitch video/image compression stuff and you want to put it *before* it 
goes out the radio network (because that's the expensive resource), so 
that's why my guess it's doing the image-rewriting not in the end user 
box, but in a centrally placed device in their network.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] T-Mobile LTE buffer bloat

[Dirk Kutscher]

Also, it would be important to understand what radio access technology the LTE-WiFi-GW was really using at the time of the measurement. In hybrid GSM/UMTS/LTE networks (like T-Mobile's), mobile phones frequently fall back to GSM (GPRS), so you cannot blame LTE in this case.

There are many myths around bufferbloat in mobile networks. You really need to understand the specifics of the network, its utilization at a given time -- and the deficiencies of the gear you are using.

Having said that, in LTE there are apparently significant differences in latency (variation) depending on the operator and equipment vendor. For example, I have seen LTE measurements from hotspot areas in Tokyo that still looked much better (max 60 ms uplink delay) than what I hear sometimes from friends in the US.

It's about time for structured measurement campaign that helps us to analyze this better.

Best regards,
Dirk



> -----Original Message-----
> From: bloat-bounces@lists.bufferbloat.net [mailto:bloat-
> bounces@lists.bufferbloat.net] On Behalf Of Mikael Abrahamsson
> Sent: Donnerstag, 31. Oktober 2013 01:02
> To: Stephen Hemminger
> Cc: bloat@lists.bufferbloat.net
> Subject: Re: [Bloat] T-Mobile LTE buffer bloat
> 
> On Wed, 30 Oct 2013, Stephen Hemminger wrote:
> 
> > There is zero configuration of proxy, it is just a LTE <-> wifi hotspot.
> > I suspect some caching (or commercial interest) at their end.
> 
> I have no insight in how T-Moble does things, but I know that vendors pitch
> video/image compression stuff and you want to put it *before* it goes out
> the radio network (because that's the expensive resource), so that's why my
> guess it's doing the image-rewriting not in the end user box, but in a centrally
> placed device in their network.
> 
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] T-Mobile LTE buffer bloat

[Michael Richardson]

I agree with others that the image rewriting/compression is likely a network
"service".

What about the other ports and things?  I wonder if the image rewriting
causes additional bloat...

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [

Re: [Bloat] T-Mobile LTE buffer bloat

[Mikael Abrahamsson]

On Wed, 30 Oct 2013, Michael Richardson wrote:

> What about the other ports and things?  I wonder if the image rewriting 
> causes additional bloat...

I don't really see it causing additional bloat. It might cause additional 
delays (the device there needs to fetch the image, recode it, and then 
serve it to the requesting client), but since the data is now hopefully 
smaller, it shouldn't cause additional buffering.

Guess it comes down to how you define "bloat".

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] T-Mobile LTE buffer bloat

[Stefan Alfredsson]

Hi, Stephen, the list,

We're running a 3G/4G measurement site at Karlstad university, with 
mobile broadband subscriptions from the four major telecom operators in 
Sweden. I was curious to see how measurements in our network would 
compare to yours, so I did a measurement run with netalyzer.

To summarize, the measured buffering is well below one second for all 
tests, although the results are not directly comparable since we are 
using directly attached USB modems (Huawei E392 with Ubuntu 12.04) 
instead of a separate router box. However, the numbers might be 
interesting in a more general perspective.

Tele2 LTE: Upload 12 Mbit/s, 240 ms buffering, Download > 20  Mbit/s, 
buffering not measurable
http://n1.netalyzr.icsi.berkeley.edu/summary/id=43ca253f-6593-7f837e62-5321-488e-b480

Tele2 HSPA+: Upload 1.1 Mbit/s, 630 ms buffering, Download > 20 Mbit/s, 
509 ms buffering
http://n2.netalyzr.icsi.berkeley.edu/summary/id=43ca208a-23052-a0e80cbb-9a7c-44e0-ba88

Telenor LTE: Upload 16  Mbit/s, 190  ms buffering, Download >20  Mbit/s, 
470 ms buffering
http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-31617-45bca5e3-53ef-450e-a9bf

Telenor HSPA+: Upload 3.4 Mbit/s,  200 ms buffering, Download > 14 
Mbit/s, 490 ms buffering
http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-31612-c6d0d98d-8bcf-4a03-832d

Telia LTE: Upload 18 Mbit/s, 140 ms buffering, Download > 20 Mbit/s, 
buffering not measurable
http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-31599-1ebc684e-c164-4786-832a

Telia HSPA+: Upload 1.1 Mbit/s, 630  ms buffering, Download > 20 Mbit/s, 
640 ms buffering
http://n2.netalyzr.icsi.berkeley.edu/summary/id=43ca208a-22984-40c59ae2-0c36-4e67-9039
(note similarity with Tele2 HSPA+ - IIUC Tele2 and Telia share the 3G 
network infrastructure)

Tre LTE: Upload 19 Mbit/s, 150 ms buffering, Download > 20 Mbit/s, 98 ms 
buffering
http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-31648-86a273a5-82ae-4929-a634

Tre HSPA+: Upload 3.5 Mbit/s, 190 ms buffering, Download > 8.6 Mbit/s, 
780 ms buffering
http://n1.netalyzr.icsi.berkeley.edu/summary/id=43ca253f-6609-3316da33-9ea6-41ca-a8ac


For reference, I also made a measurement from the same host, but using a 
wired connection over the Swedish university network: Upload was 
measured to > 20 Mbit/s and download to 15 Mbit/s, with no measurable 
buffering.
http://n1.netalyzr.icsi.berkeley.edu/summary/id=43ca253f-6563-078896ce-dc0a-4333-9119 


Regards,
   Stefan Alfredsson



On 10/31/13 00:25 , Stephen Hemminger wrote:
> I got one of these Samsung LTE hotspot.
> Not surprisingly it has huge bloat and a stupid http proxy
> that netalyzer claims rewrites images.
>   
> Bandwidth: Up 1.6 Mbit/sec Down 4.3Mbit/sec
> Latency: 140ms 0% loss
> Buffering: Uplink 5100ms Down 1800ms
>
> How can the uplink side be so bad! 5 seconds???
>
> Might even return it as defective. You can't even add a review on their
> website. Probably they would end taking down like Apple.
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] T-Mobile LTE buffer bloat

[Dave Taht]

On Thu, Oct 31, 2013 at 03:04:23AM +0100, Stefan Alfredsson wrote:
> Hi, Stephen, the list,
> 
> We're running a 3G/4G measurement site at Karlstad university, with
> mobile broadband subscriptions from the four major telecom operators
> in Sweden. I was curious to see how measurements in our network
> would compare to yours, so I did a measurement run with netalyzer.

Groovy!

Can I ask that you try something that can more likely stress
out that bus than netanlyzer? The netperf-wrappers tools 
(notably the tcp_bidirectional and rrul test), use C, rather than java.

There is a reasonably local-to-you netperf rrul server at demo.tohojo.dk. You 
can also easily setup a local one....

netanlyzer doesn't work well above 12Mbits, in particular. Netperf-wrappers
works to 10GigE speeds.

apt-get install python-matplotlib
for that ubuntu version you will need to build netperf 2.6 from scratch,
pulling it down from svn or it's website
compiling with ./configure --enable-demo && make install

git clone https://github.com/tohojo/netperf-wrapper.git
cd netperf-wrapper
sudo python setup.py install

sample command line:

netperf-wrapper -H demo.tohojo.dk -p all_scaled -o mymodem_name.svg \
-t "the modem I'm testing" --disable-log rrul


> 
> To summarize, the measured buffering is well below one second for
> all tests, although the results are not directly comparable since we
> are using directly attached USB modems (Huawei E392 with Ubuntu
> 12.04) 

ubuntu has backported fq_codel as far as 12.04. What kernel are you using?

> instead of a separate router box. However, the numbers might
> be interesting in a more general perspective.

Yes, they are. These are repeatable?

I don't consider "less than 1 second" to be "good". 5ms is good. 
At 250ms seems to be where stuff starts to break...

> 
> Tele2 LTE: Upload 12 Mbit/s, 240 ms buffering, Download > 20
> Mbit/s, buffering not measurable
> http://n1.netalyzr.icsi.berkeley.edu/summary/id=43ca253f-6593-7f837e62-5321-488e-b480
> 
> Tele2 HSPA+: Upload 1.1 Mbit/s, 630 ms buffering, Download > 20
> Mbit/s, 509 ms buffering
> http://n2.netalyzr.icsi.berkeley.edu/summary/id=43ca208a-23052-a0e80cbb-9a7c-44e0-ba88

You'll note here that the buffering goes up as the bandwidth goes down.
This is the inverse of the desired result.

> 
> Telenor LTE: Upload 16  Mbit/s, 190  ms buffering, Download >20
> Mbit/s, 470 ms buffering
> http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-31617-45bca5e3-53ef-450e-a9bf
> 
> Telenor HSPA+: Upload 3.4 Mbit/s,  200 ms buffering, Download > 14
> Mbit/s, 490 ms buffering
> http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-31612-c6d0d98d-8bcf-4a03-832d
> 
> Telia LTE: Upload 18 Mbit/s, 140 ms buffering, Download > 20 Mbit/s,
> buffering not measurable
> http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-31599-1ebc684e-c164-4786-832a
> 
> Telia HSPA+: Upload 1.1 Mbit/s, 630  ms buffering, Download > 20
> Mbit/s, 640 ms buffering
> http://n2.netalyzr.icsi.berkeley.edu/summary/id=43ca208a-22984-40c59ae2-0c36-4e67-9039
> (note similarity with Tele2 HSPA+ - IIUC Tele2 and Telia share the
> 3G network infrastructure)
> 
> Tre LTE: Upload 19 Mbit/s, 150 ms buffering, Download > 20 Mbit/s,
> 98 ms buffering
> http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-31648-86a273a5-82ae-4929-a634
> 
> Tre HSPA+: Upload 3.5 Mbit/s, 190 ms buffering, Download > 8.6
> Mbit/s, 780 ms buffering
> http://n1.netalyzr.icsi.berkeley.edu/summary/id=43ca253f-6609-3316da33-9ea6-41ca-a8ac
> 
> 
> For reference, I also made a measurement from the same host, but
> using a wired connection over the Swedish university network: Upload
> was measured to > 20 Mbit/s and download to 15 Mbit/s, with no
> measurable buffering.
> http://n1.netalyzr.icsi.berkeley.edu/summary/id=43ca253f-6563-078896ce-dc0a-4333-9119

That was kind of my point. Netanyzler has issues at higher speeds.

Please give the rrul test a shot.

> 
> 
> Regards,
>   Stefan Alfredsson
> 
> 
> 
> On 10/31/13 00:25 , Stephen Hemminger wrote:
> >I got one of these Samsung LTE hotspot.
> >Not surprisingly it has huge bloat and a stupid http proxy
> >that netalyzer claims rewrites images.
> >Bandwidth: Up 1.6 Mbit/sec Down 4.3Mbit/sec
> >Latency: 140ms 0% loss
> >Buffering: Uplink 5100ms Down 1800ms
> >
> >How can the uplink side be so bad! 5 seconds???
> >
> >Might even return it as defective. You can't even add a review on their
> >website. Probably they would end taking down like Apple.
> >_______________________________________________
> >Bloat mailing list
> >Bloat@lists.bufferbloat.net
> >https://lists.bufferbloat.net/listinfo/bloat
> 
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] mobile broadband buffer bloat

[Stefan Alfredsson]

On 10/31/13 03:32 , Dave Taht wrote:
> On Thu, Oct 31, 2013 at 03:04:23AM +0100, Stefan Alfredsson wrote:
>> Hi, Stephen, the list,
>>
>> We're running a 3G/4G measurement site at Karlstad university, with
>> mobile broadband subscriptions from the four major telecom operators
>> in Sweden. I was curious to see how measurements in our network
>> would compare to yours, so I did a measurement run with netalyzer.
> Groovy!
>
> Can I ask that you try something that can more likely stress
> out that bus than netanlyzer? The netperf-wrappers tools
> (notably the tcp_bidirectional and rrul test), use C, rather than java.

Sure! Actually we've been doing netperf-wrapper rrul measurements 24/4 
during July to September, roughly 3-4 times per day, in total over 3000 
individual tests.

I think there's a lot of interesting information in there, but I have 
not done aggregate analysis yet (eg. looking at time-of-day effects, 
long-term trends, or the 10.000-students-arriving-in-august effect :-) )

I packed the rrul-dumps for one day (1 sept 2013) and put it here: 
http://www.cs.kau.se/stefalfr/3G4G/rrul-tests-2013-09-01.tgz .

Here's an overview of this set, showing the per-run mean RTT, download 
and upload throughput:

[alfs@sa-pc]:/tmp/n/2013-06> for f in */*/*/rrul*json.gz ; do 
test=$(echo $f | cut -d / -f 1-2); netperf-wrapper -i $f -f stats|grep 
-A 4 'avg:'| awk -v test=$test '/Mean:/ { if (NR==3) { rtt=$2 } if 
(NR==10) {dl=$2} if (NR==16) {ul=$2; printf "%s:\tRTT: %.1f ms DL: %.1f 
Mbit/s UL: %.1f Mbit/s\n", test, rtt, dl, ul}}'; done
Tele2/35G:      RTT: 490.7 ms DL: 1.3 Mbit/s UL: 0.3 Mbit/s
Tele2/35G:      RTT: 498.0 ms DL: 6.1 Mbit/s UL: 0.4 Mbit/s
Tele2/35G:      RTT: 562.7 ms DL: 4.7 Mbit/s UL: 0.2 Mbit/s
Tele2/35G:      RTT: 641.5 ms DL: 3.5 Mbit/s UL: 0.3 Mbit/s
Tele2/4G:       RTT: 891.5 ms DL: 2.3 Mbit/s UL: 0.3 Mbit/s
Tele2/4G:       RTT: 389.0 ms DL: 20.8 Mbit/s UL: 1.0 Mbit/s
Tele2/4G:       RTT: 361.2 ms DL: 14.4 Mbit/s UL: 1.4 Mbit/s
Tele2/4G:       RTT: 359.8 ms DL: 17.3 Mbit/s UL: 1.3 Mbit/s
Telenor/35G:    RTT: 809.0 ms DL: 2.2 Mbit/s UL: 0.8 Mbit/s
Telenor/35G:    RTT: 377.3 ms DL: 3.8 Mbit/s UL: 0.5 Mbit/s
Telenor/35G:    RTT: 727.2 ms DL: 2.9 Mbit/s UL: 0.3 Mbit/s
Telenor/35G:    RTT: 341.7 ms DL: 0.1 Mbit/s UL: 0.2 Mbit/s
Telenor/4G:     RTT: 427.8 ms DL: 16.9 Mbit/s UL: 0.8 Mbit/s
Telenor/4G:     RTT: 378.8 ms DL: 17.9 Mbit/s UL: 1.2 Mbit/s
Telenor/4G:     RTT: 623.8 ms DL: 8.6 Mbit/s UL: 0.6 Mbit/s
Telenor/4G:     RTT: 321.5 ms DL: 18.2 Mbit/s UL: 1.0 Mbit/s
Telia/35G:      RTT: 686.9 ms DL: 4.8 Mbit/s UL: 0.2 Mbit/s
Telia/35G:      RTT: 709.0 ms DL: 6.1 Mbit/s UL: 0.2 Mbit/s
Telia/35G:      RTT: 546.9 ms DL: 3.8 Mbit/s UL: 0.2 Mbit/s
Telia/35G:      RTT: 653.0 ms DL: 3.1 Mbit/s UL: 0.3 Mbit/s
Telia/4G:       RTT: 233.6 ms DL: 14.1 Mbit/s UL: 1.1 Mbit/s
Telia/4G:       RTT: 324.2 ms DL: 14.6 Mbit/s UL: 1.1 Mbit/s
Telia/4G:       RTT: 381.5 ms DL: 15.2 Mbit/s UL: 1.2 Mbit/s
Tre/35G:        RTT: 312.2 ms DL: 3.1 Mbit/s UL: 0.6 Mbit/s
Tre/35G:        RTT: 376.1 ms DL: 4.1 Mbit/s UL: 0.8 Mbit/s
Tre/35G:        RTT: 442.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
Tre/35G:        RTT: 294.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
Tre/4G: RTT: 471.2 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s
Tre/4G: RTT: 396.2 ms DL: 6.8 Mbit/s UL: 0.7 Mbit/s
Tre/4G: RTT: 237.5 ms DL: 7.2 Mbit/s UL: 0.7 Mbit/s
Tre/4G: RTT: 321.5 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s

>> To summarize, the measured buffering is well below one second for
>> all tests, although the results are not directly comparable since we
>> are using directly attached USB modems (Huawei E392 with Ubuntu
>> 12.04)
> ubuntu has backported fq_codel as far as 12.04. What kernel are you using?
On the "mobile terminal" we have 3.2.0-54-generic-pae #82-Ubuntu SMP

The fixed host ("tptest") runs a self-compiled stock kernel 3.4.1 with 
web10g patches, since we were interested in extracting TCP variable data 
not available from packet traces.

>> instead of a separate router box. However, the numbers might
>> be interesting in a more general perspective.
> Yes, they are. These are repeatable?
>
>

For the netalyzer I've only done singular tests, but it can easily be 
repeated if needed (probably not).
The mean of all mean RTTs for 4G, all operators, all times, is 334 ms, 
and for 3.5G it's 573 ms.
This indicates that results are repeatable, and the set from 2013-09-01 
is representative.

"below one second" was in relation to the >5 second RTT's, ideally it 
should be much lower of course. As a side note, we also do measurements 
in unloaded networks, and for 4G we get around 40-60 ms RTT (ICMP and 
TCP packets).

Also, you might be interested in a paper we published in June, that 
reports on measurements of bloat for short flows in relation to 
different congestion control algorithms, in 3G, 3.5G and 4G (but for a 
single operator).
See http://urn.kb.se/resolve?urn=urn:nbn:se:kau:diva-27779
with fulltext at http://dx.doi.org/10.1109/WoWMoM.2013.6583408.
I have also put a local copy (for personal use etc) at 
http://www.cs.kau.se/stefalfr/3G4G/alfredsson-bufferbloat-wowmom13.pdf


Regards,
  Stefan

-- 
Stefan Alfredsson, PhD         Tel: +46 (0) 54 700 1668
Datavetenskap                  Kontor: 21F-425 (Hus Vanern)
Karlstads universitet          PGP 0xB19B4B16
SE-651 88 Karlstad             http://www.cs.kau.se/stefalfr/

Re: [Bloat] mobile broadband buffer bloat

[Dave Taht]

On Thu, Oct 31, 2013 at 5:27 AM, Stefan Alfredsson
<Stefan.Alfredsson@kau.se> wrote:
> On 10/31/13 03:32 , Dave Taht wrote:
>>
>> On Thu, Oct 31, 2013 at 03:04:23AM +0100, Stefan Alfredsson wrote:
>>>
>>> Hi, Stephen, the list,
>>>
>>> We're running a 3G/4G measurement site at Karlstad university, with
>>> mobile broadband subscriptions from the four major telecom operators
>>> in Sweden. I was curious to see how measurements in our network
>>> would compare to yours, so I did a measurement run with netalyzer.
>>
>> Groovy!
>>
>> Can I ask that you try something that can more likely stress
>> out that bus than netanlyzer? The netperf-wrappers tools
>> (notably the tcp_bidirectional and rrul test), use C, rather than java.
>
>
> Sure! Actually we've been doing netperf-wrapper rrul measurements 24/4
> during July to September, roughly 3-4 times per day, in total over 3000
> individual tests.
>
> I think there's a lot of interesting information in there, but I have not
> done aggregate analysis yet (eg. looking at time-of-day effects, long-term
> trends, or the 10.000-students-arriving-in-august effect :-) )

if you are interested in measuring one way delays, owamp + a gps is
working out well.

> I packed the rrul-dumps for one day (1 sept 2013) and put it here:
> http://www.cs.kau.se/stefalfr/3G4G/rrul-tests-2013-09-01.tgz .
>
> Here's an overview of this set, showing the per-run mean RTT, download and
> upload throughput:
>
> [alfs@sa-pc]:/tmp/n/2013-06> for f in */*/*/rrul*json.gz ; do test=$(echo $f
> | cut -d / -f 1-2); netperf-wrapper -i $f -f stats|grep -A 4 'avg:'| awk -v
> test=$test '/Mean:/ { if (NR==3) { rtt=$2 } if (NR==10) {dl=$2} if (NR==16)
> {ul=$2; printf "%s:\tRTT: %.1f ms DL: %.1f Mbit/s UL: %.1f Mbit/s\n", test,
> rtt, dl, ul}}'; done
> Tele2/35G:      RTT: 490.7 ms DL: 1.3 Mbit/s UL: 0.3 Mbit/s
> Tele2/35G:      RTT: 498.0 ms DL: 6.1 Mbit/s UL: 0.4 Mbit/s
> Tele2/35G:      RTT: 562.7 ms DL: 4.7 Mbit/s UL: 0.2 Mbit/s
> Tele2/35G:      RTT: 641.5 ms DL: 3.5 Mbit/s UL: 0.3 Mbit/s
> Tele2/4G:       RTT: 891.5 ms DL: 2.3 Mbit/s UL: 0.3 Mbit/s
> Tele2/4G:       RTT: 389.0 ms DL: 20.8 Mbit/s UL: 1.0 Mbit/s
> Tele2/4G:       RTT: 361.2 ms DL: 14.4 Mbit/s UL: 1.4 Mbit/s
> Tele2/4G:       RTT: 359.8 ms DL: 17.3 Mbit/s UL: 1.3 Mbit/s
> Telenor/35G:    RTT: 809.0 ms DL: 2.2 Mbit/s UL: 0.8 Mbit/s
> Telenor/35G:    RTT: 377.3 ms DL: 3.8 Mbit/s UL: 0.5 Mbit/s
> Telenor/35G:    RTT: 727.2 ms DL: 2.9 Mbit/s UL: 0.3 Mbit/s
> Telenor/35G:    RTT: 341.7 ms DL: 0.1 Mbit/s UL: 0.2 Mbit/s
> Telenor/4G:     RTT: 427.8 ms DL: 16.9 Mbit/s UL: 0.8 Mbit/s
> Telenor/4G:     RTT: 378.8 ms DL: 17.9 Mbit/s UL: 1.2 Mbit/s
> Telenor/4G:     RTT: 623.8 ms DL: 8.6 Mbit/s UL: 0.6 Mbit/s
> Telenor/4G:     RTT: 321.5 ms DL: 18.2 Mbit/s UL: 1.0 Mbit/s
> Telia/35G:      RTT: 686.9 ms DL: 4.8 Mbit/s UL: 0.2 Mbit/s
> Telia/35G:      RTT: 709.0 ms DL: 6.1 Mbit/s UL: 0.2 Mbit/s
> Telia/35G:      RTT: 546.9 ms DL: 3.8 Mbit/s UL: 0.2 Mbit/s
> Telia/35G:      RTT: 653.0 ms DL: 3.1 Mbit/s UL: 0.3 Mbit/s
> Telia/4G:       RTT: 233.6 ms DL: 14.1 Mbit/s UL: 1.1 Mbit/s
> Telia/4G:       RTT: 324.2 ms DL: 14.6 Mbit/s UL: 1.1 Mbit/s
> Telia/4G:       RTT: 381.5 ms DL: 15.2 Mbit/s UL: 1.2 Mbit/s
> Tre/35G:        RTT: 312.2 ms DL: 3.1 Mbit/s UL: 0.6 Mbit/s
> Tre/35G:        RTT: 376.1 ms DL: 4.1 Mbit/s UL: 0.8 Mbit/s
> Tre/35G:        RTT: 442.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
> Tre/35G:        RTT: 294.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
> Tre/4G: RTT: 471.2 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s
> Tre/4G: RTT: 396.2 ms DL: 6.8 Mbit/s UL: 0.7 Mbit/s
> Tre/4G: RTT: 237.5 ms DL: 7.2 Mbit/s UL: 0.7 Mbit/s
> Tre/4G: RTT: 321.5 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s

Oh, that is wonderful data thank you! I confess to be more interested
in outliers than averages, and induced delay over base delay.

However, I'm concerned that you are not getting a correct result with
your script above.

I just took a quick look at telia-4g, using the 2100-2013-09-01T200038
sample set, and looking at the raw (rrul) data: What I see here

http://snapon.lab.bufferbloat.net/~d/telia-4g.svg

is 60-80 ms of induced latency on this link, with 40ms of baseline
latency. Not a RTT of > 300!


>>> To summarize, the measured buffering is well below one second for
>>> all tests, although the results are not directly comparable since we
>>> are using directly attached USB modems (Huawei E392 with Ubuntu
>>> 12.04)
>>
>> ubuntu has backported fq_codel as far as 12.04. What kernel are you using?
>
> On the "mobile terminal" we have 3.2.0-54-generic-pae #82-Ubuntu SMP

I am delighted you've been collecting this data long term.

I do have a problem in that the 3.2 kernel predates all the
bufferbloat related fixes made in the 4 years since that release.
we've been working our butts off here... :(

Any chance you can update to 3.10.x or later for both this (driver and
buffering fixes) and your fixed host? (tcp fixes). And it's generally
been my hope people would try fq_codel (or sfq or pie or red) on such
links, too...

There has been very little done to optimize usb-net, although we
tried. If this is using the ppp driver, that was improved
substantially around 3.6?

> The fixed host ("tptest") runs a self-compiled stock kernel 3.4.1 with
> web10g patches, since we were interested in extracting TCP variable data not
> available from packet traces.
>
>>> instead of a separate router box. However, the numbers might
>>> be interesting in a more general perspective.
>>
>> Yes, they are. These are repeatable?
>>
>>
>
> For the netalyzer I've only done singular tests, but it can easily be
> repeated if needed (probably not).
> The mean of all mean RTTs for 4G, all operators, all times, is 334 ms, and
> for 3.5G it's 573 ms.
> This indicates that results are repeatable, and the set from 2013-09-01 is
> representative.

except that my plot shows differently...

and you are running an ancient kernel, yes. Something of a major
variable, that... and if you could let us know the usb path to the
various underlying drivers we can poke into matters there?

>
> "below one second" was in relation to the >5 second RTT's, ideally it should
> be much lower of course. As a side note, we also do measurements in unloaded
> networks, and for 4G we get around 40-60 ms RTT (ICMP and TCP packets).
>
> Also, you might be interested in a paper we published in June, that reports
> on measurements of bloat for short flows in relation to different congestion
> control algorithms, in 3G, 3.5G and 4G (but for a single operator).
> See http://urn.kb.se/resolve?urn=urn:nbn:se:kau:diva-27779
> with fulltext at http://dx.doi.org/10.1109/WoWMoM.2013.6583408.
> I have also put a local copy (for personal use etc) at
> http://www.cs.kau.se/stefalfr/3G4G/alfredsson-bufferbloat-wowmom13.pdf
>
>
> Regards,
>  Stefan
>
> --
> Stefan Alfredsson, PhD         Tel: +46 (0) 54 700 1668
> Datavetenskap                  Kontor: 21F-425 (Hus Vanern)
> Karlstads universitet          PGP 0xB19B4B16
> SE-651 88 Karlstad             http://www.cs.kau.se/stefalfr/
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Bloat] mobile broadband buffer bloat

[Dave Taht]

Just did telia. I really want 4G as good as this!!!

http://snapon.lab.bufferbloat.net/~d/telia-4g.svg

So you multiply the black line x4 to get a total for the down or up,
then insert a fudge factor for the amount of acks that probably went
into the down relative to the up and vice versa, and you sort of have
your bandwidth number each way.

I'm really impressed you can get ~72Mbit down and ~4Mbit up. (closer
to 8 up when you consider acks) Note also the relationship between
bandwidth and latency at T+22...


On Thu, Oct 31, 2013 at 6:09 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Thu, Oct 31, 2013 at 5:27 AM, Stefan Alfredsson
> <Stefan.Alfredsson@kau.se> wrote:
>> On 10/31/13 03:32 , Dave Taht wrote:
>>>
>>> On Thu, Oct 31, 2013 at 03:04:23AM +0100, Stefan Alfredsson wrote:
>>>>
>>>> Hi, Stephen, the list,
>>>>
>>>> We're running a 3G/4G measurement site at Karlstad university, with
>>>> mobile broadband subscriptions from the four major telecom operators
>>>> in Sweden. I was curious to see how measurements in our network
>>>> would compare to yours, so I did a measurement run with netalyzer.
>>>
>>> Groovy!
>>>
>>> Can I ask that you try something that can more likely stress
>>> out that bus than netanlyzer? The netperf-wrappers tools
>>> (notably the tcp_bidirectional and rrul test), use C, rather than java.
>>
>>
>> Sure! Actually we've been doing netperf-wrapper rrul measurements 24/4
>> during July to September, roughly 3-4 times per day, in total over 3000
>> individual tests.
>>
>> I think there's a lot of interesting information in there, but I have not
>> done aggregate analysis yet (eg. looking at time-of-day effects, long-term
>> trends, or the 10.000-students-arriving-in-august effect :-) )
>
> if you are interested in measuring one way delays, owamp + a gps is
> working out well.
>
>> I packed the rrul-dumps for one day (1 sept 2013) and put it here:
>> http://www.cs.kau.se/stefalfr/3G4G/rrul-tests-2013-09-01.tgz .
>>
>> Here's an overview of this set, showing the per-run mean RTT, download and
>> upload throughput:
>>
>> [alfs@sa-pc]:/tmp/n/2013-06> for f in */*/*/rrul*json.gz ; do test=$(echo $f
>> | cut -d / -f 1-2); netperf-wrapper -i $f -f stats|grep -A 4 'avg:'| awk -v
>> test=$test '/Mean:/ { if (NR==3) { rtt=$2 } if (NR==10) {dl=$2} if (NR==16)
>> {ul=$2; printf "%s:\tRTT: %.1f ms DL: %.1f Mbit/s UL: %.1f Mbit/s\n", test,
>> rtt, dl, ul}}'; done
>> Tele2/35G:      RTT: 490.7 ms DL: 1.3 Mbit/s UL: 0.3 Mbit/s
>> Tele2/35G:      RTT: 498.0 ms DL: 6.1 Mbit/s UL: 0.4 Mbit/s
>> Tele2/35G:      RTT: 562.7 ms DL: 4.7 Mbit/s UL: 0.2 Mbit/s
>> Tele2/35G:      RTT: 641.5 ms DL: 3.5 Mbit/s UL: 0.3 Mbit/s
>> Tele2/4G:       RTT: 891.5 ms DL: 2.3 Mbit/s UL: 0.3 Mbit/s
>> Tele2/4G:       RTT: 389.0 ms DL: 20.8 Mbit/s UL: 1.0 Mbit/s
>> Tele2/4G:       RTT: 361.2 ms DL: 14.4 Mbit/s UL: 1.4 Mbit/s
>> Tele2/4G:       RTT: 359.8 ms DL: 17.3 Mbit/s UL: 1.3 Mbit/s
>> Telenor/35G:    RTT: 809.0 ms DL: 2.2 Mbit/s UL: 0.8 Mbit/s
>> Telenor/35G:    RTT: 377.3 ms DL: 3.8 Mbit/s UL: 0.5 Mbit/s
>> Telenor/35G:    RTT: 727.2 ms DL: 2.9 Mbit/s UL: 0.3 Mbit/s
>> Telenor/35G:    RTT: 341.7 ms DL: 0.1 Mbit/s UL: 0.2 Mbit/s
>> Telenor/4G:     RTT: 427.8 ms DL: 16.9 Mbit/s UL: 0.8 Mbit/s
>> Telenor/4G:     RTT: 378.8 ms DL: 17.9 Mbit/s UL: 1.2 Mbit/s
>> Telenor/4G:     RTT: 623.8 ms DL: 8.6 Mbit/s UL: 0.6 Mbit/s
>> Telenor/4G:     RTT: 321.5 ms DL: 18.2 Mbit/s UL: 1.0 Mbit/s
>> Telia/35G:      RTT: 686.9 ms DL: 4.8 Mbit/s UL: 0.2 Mbit/s
>> Telia/35G:      RTT: 709.0 ms DL: 6.1 Mbit/s UL: 0.2 Mbit/s
>> Telia/35G:      RTT: 546.9 ms DL: 3.8 Mbit/s UL: 0.2 Mbit/s
>> Telia/35G:      RTT: 653.0 ms DL: 3.1 Mbit/s UL: 0.3 Mbit/s
>> Telia/4G:       RTT: 233.6 ms DL: 14.1 Mbit/s UL: 1.1 Mbit/s
>> Telia/4G:       RTT: 324.2 ms DL: 14.6 Mbit/s UL: 1.1 Mbit/s
>> Telia/4G:       RTT: 381.5 ms DL: 15.2 Mbit/s UL: 1.2 Mbit/s
>> Tre/35G:        RTT: 312.2 ms DL: 3.1 Mbit/s UL: 0.6 Mbit/s
>> Tre/35G:        RTT: 376.1 ms DL: 4.1 Mbit/s UL: 0.8 Mbit/s
>> Tre/35G:        RTT: 442.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
>> Tre/35G:        RTT: 294.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
>> Tre/4G: RTT: 471.2 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s
>> Tre/4G: RTT: 396.2 ms DL: 6.8 Mbit/s UL: 0.7 Mbit/s
>> Tre/4G: RTT: 237.5 ms DL: 7.2 Mbit/s UL: 0.7 Mbit/s
>> Tre/4G: RTT: 321.5 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s
>
> Oh, that is wonderful data thank you! I confess to be more interested
> in outliers than averages, and induced delay over base delay.
>
> However, I'm concerned that you are not getting a correct result with
> your script above.
>
> I just took a quick look at telia-4g, using the 2100-2013-09-01T200038
> sample set, and looking at the raw (rrul) data: What I see here
>
> http://snapon.lab.bufferbloat.net/~d/telia-4g.svg
>
> is 60-80 ms of induced latency on this link, with 40ms of baseline
> latency. Not a RTT of > 300!
>
>
>>>> To summarize, the measured buffering is well below one second for
>>>> all tests, although the results are not directly comparable since we
>>>> are using directly attached USB modems (Huawei E392 with Ubuntu
>>>> 12.04)
>>>
>>> ubuntu has backported fq_codel as far as 12.04. What kernel are you using?
>>
>> On the "mobile terminal" we have 3.2.0-54-generic-pae #82-Ubuntu SMP
>
> I am delighted you've been collecting this data long term.
>
> I do have a problem in that the 3.2 kernel predates all the
> bufferbloat related fixes made in the 4 years since that release.
> we've been working our butts off here... :(
>
> Any chance you can update to 3.10.x or later for both this (driver and
> buffering fixes) and your fixed host? (tcp fixes). And it's generally
> been my hope people would try fq_codel (or sfq or pie or red) on such
> links, too...
>
> There has been very little done to optimize usb-net, although we
> tried. If this is using the ppp driver, that was improved
> substantially around 3.6?
>
>> The fixed host ("tptest") runs a self-compiled stock kernel 3.4.1 with
>> web10g patches, since we were interested in extracting TCP variable data not
>> available from packet traces.
>>
>>>> instead of a separate router box. However, the numbers might
>>>> be interesting in a more general perspective.
>>>
>>> Yes, they are. These are repeatable?
>>>
>>>
>>
>> For the netalyzer I've only done singular tests, but it can easily be
>> repeated if needed (probably not).
>> The mean of all mean RTTs for 4G, all operators, all times, is 334 ms, and
>> for 3.5G it's 573 ms.
>> This indicates that results are repeatable, and the set from 2013-09-01 is
>> representative.
>
> except that my plot shows differently...
>
> and you are running an ancient kernel, yes. Something of a major
> variable, that... and if you could let us know the usb path to the
> various underlying drivers we can poke into matters there?
>
>>
>> "below one second" was in relation to the >5 second RTT's, ideally it should
>> be much lower of course. As a side note, we also do measurements in unloaded
>> networks, and for 4G we get around 40-60 ms RTT (ICMP and TCP packets).
>>
>> Also, you might be interested in a paper we published in June, that reports
>> on measurements of bloat for short flows in relation to different congestion
>> control algorithms, in 3G, 3.5G and 4G (but for a single operator).
>> See http://urn.kb.se/resolve?urn=urn:nbn:se:kau:diva-27779
>> with fulltext at http://dx.doi.org/10.1109/WoWMoM.2013.6583408.
>> I have also put a local copy (for personal use etc) at
>> http://www.cs.kau.se/stefalfr/3G4G/alfredsson-bufferbloat-wowmom13.pdf
>>
>>
>> Regards,
>>  Stefan
>>
>> --
>> Stefan Alfredsson, PhD         Tel: +46 (0) 54 700 1668
>> Datavetenskap                  Kontor: 21F-425 (Hus Vanern)
>> Karlstads universitet          PGP 0xB19B4B16
>> SE-651 88 Karlstad             http://www.cs.kau.se/stefalfr/
>>
>>
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
>
>
> --
> Dave Täht
>
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Bloat] mobile broadband buffer bloat

[Dave Taht]

oops that was telenor I was commenting on

http://snapon.lab.bufferbloat.net/~d/telenor-4g.svg

On Thu, Oct 31, 2013 at 6:32 PM, Dave Taht <dave.taht@gmail.com> wrote:
> Just did telia. I really want 4G as good as this!!!
>
> http://snapon.lab.bufferbloat.net/~d/telia-4g.svg
>
> So you multiply the black line x4 to get a total for the down or up,
> then insert a fudge factor for the amount of acks that probably went
> into the down relative to the up and vice versa, and you sort of have
> your bandwidth number each way.
>
> I'm really impressed you can get ~72Mbit down and ~4Mbit up. (closer
> to 8 up when you consider acks) Note also the relationship between
> bandwidth and latency at T+22...
>
>
> On Thu, Oct 31, 2013 at 6:09 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> On Thu, Oct 31, 2013 at 5:27 AM, Stefan Alfredsson
>> <Stefan.Alfredsson@kau.se> wrote:
>>> On 10/31/13 03:32 , Dave Taht wrote:
>>>>
>>>> On Thu, Oct 31, 2013 at 03:04:23AM +0100, Stefan Alfredsson wrote:
>>>>>
>>>>> Hi, Stephen, the list,
>>>>>
>>>>> We're running a 3G/4G measurement site at Karlstad university, with
>>>>> mobile broadband subscriptions from the four major telecom operators
>>>>> in Sweden. I was curious to see how measurements in our network
>>>>> would compare to yours, so I did a measurement run with netalyzer.
>>>>
>>>> Groovy!
>>>>
>>>> Can I ask that you try something that can more likely stress
>>>> out that bus than netanlyzer? The netperf-wrappers tools
>>>> (notably the tcp_bidirectional and rrul test), use C, rather than java.
>>>
>>>
>>> Sure! Actually we've been doing netperf-wrapper rrul measurements 24/4
>>> during July to September, roughly 3-4 times per day, in total over 3000
>>> individual tests.
>>>
>>> I think there's a lot of interesting information in there, but I have not
>>> done aggregate analysis yet (eg. looking at time-of-day effects, long-term
>>> trends, or the 10.000-students-arriving-in-august effect :-) )
>>
>> if you are interested in measuring one way delays, owamp + a gps is
>> working out well.
>>
>>> I packed the rrul-dumps for one day (1 sept 2013) and put it here:
>>> http://www.cs.kau.se/stefalfr/3G4G/rrul-tests-2013-09-01.tgz .
>>>
>>> Here's an overview of this set, showing the per-run mean RTT, download and
>>> upload throughput:
>>>
>>> [alfs@sa-pc]:/tmp/n/2013-06> for f in */*/*/rrul*json.gz ; do test=$(echo $f
>>> | cut -d / -f 1-2); netperf-wrapper -i $f -f stats|grep -A 4 'avg:'| awk -v
>>> test=$test '/Mean:/ { if (NR==3) { rtt=$2 } if (NR==10) {dl=$2} if (NR==16)
>>> {ul=$2; printf "%s:\tRTT: %.1f ms DL: %.1f Mbit/s UL: %.1f Mbit/s\n", test,
>>> rtt, dl, ul}}'; done
>>> Tele2/35G:      RTT: 490.7 ms DL: 1.3 Mbit/s UL: 0.3 Mbit/s
>>> Tele2/35G:      RTT: 498.0 ms DL: 6.1 Mbit/s UL: 0.4 Mbit/s
>>> Tele2/35G:      RTT: 562.7 ms DL: 4.7 Mbit/s UL: 0.2 Mbit/s
>>> Tele2/35G:      RTT: 641.5 ms DL: 3.5 Mbit/s UL: 0.3 Mbit/s
>>> Tele2/4G:       RTT: 891.5 ms DL: 2.3 Mbit/s UL: 0.3 Mbit/s
>>> Tele2/4G:       RTT: 389.0 ms DL: 20.8 Mbit/s UL: 1.0 Mbit/s
>>> Tele2/4G:       RTT: 361.2 ms DL: 14.4 Mbit/s UL: 1.4 Mbit/s
>>> Tele2/4G:       RTT: 359.8 ms DL: 17.3 Mbit/s UL: 1.3 Mbit/s
>>> Telenor/35G:    RTT: 809.0 ms DL: 2.2 Mbit/s UL: 0.8 Mbit/s
>>> Telenor/35G:    RTT: 377.3 ms DL: 3.8 Mbit/s UL: 0.5 Mbit/s
>>> Telenor/35G:    RTT: 727.2 ms DL: 2.9 Mbit/s UL: 0.3 Mbit/s
>>> Telenor/35G:    RTT: 341.7 ms DL: 0.1 Mbit/s UL: 0.2 Mbit/s
>>> Telenor/4G:     RTT: 427.8 ms DL: 16.9 Mbit/s UL: 0.8 Mbit/s
>>> Telenor/4G:     RTT: 378.8 ms DL: 17.9 Mbit/s UL: 1.2 Mbit/s
>>> Telenor/4G:     RTT: 623.8 ms DL: 8.6 Mbit/s UL: 0.6 Mbit/s
>>> Telenor/4G:     RTT: 321.5 ms DL: 18.2 Mbit/s UL: 1.0 Mbit/s
>>> Telia/35G:      RTT: 686.9 ms DL: 4.8 Mbit/s UL: 0.2 Mbit/s
>>> Telia/35G:      RTT: 709.0 ms DL: 6.1 Mbit/s UL: 0.2 Mbit/s
>>> Telia/35G:      RTT: 546.9 ms DL: 3.8 Mbit/s UL: 0.2 Mbit/s
>>> Telia/35G:      RTT: 653.0 ms DL: 3.1 Mbit/s UL: 0.3 Mbit/s
>>> Telia/4G:       RTT: 233.6 ms DL: 14.1 Mbit/s UL: 1.1 Mbit/s
>>> Telia/4G:       RTT: 324.2 ms DL: 14.6 Mbit/s UL: 1.1 Mbit/s
>>> Telia/4G:       RTT: 381.5 ms DL: 15.2 Mbit/s UL: 1.2 Mbit/s
>>> Tre/35G:        RTT: 312.2 ms DL: 3.1 Mbit/s UL: 0.6 Mbit/s
>>> Tre/35G:        RTT: 376.1 ms DL: 4.1 Mbit/s UL: 0.8 Mbit/s
>>> Tre/35G:        RTT: 442.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
>>> Tre/35G:        RTT: 294.0 ms DL: 2.9 Mbit/s UL: 0.6 Mbit/s
>>> Tre/4G: RTT: 471.2 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s
>>> Tre/4G: RTT: 396.2 ms DL: 6.8 Mbit/s UL: 0.7 Mbit/s
>>> Tre/4G: RTT: 237.5 ms DL: 7.2 Mbit/s UL: 0.7 Mbit/s
>>> Tre/4G: RTT: 321.5 ms DL: 6.9 Mbit/s UL: 0.7 Mbit/s
>>
>> Oh, that is wonderful data thank you! I confess to be more interested
>> in outliers than averages, and induced delay over base delay.
>>
>> However, I'm concerned that you are not getting a correct result with
>> your script above.
>>
>> I just took a quick look at telia-4g, using the 2100-2013-09-01T200038
>> sample set, and looking at the raw (rrul) data: What I see here
>>
>> http://snapon.lab.bufferbloat.net/~d/telia-4g.svg
>>
>> is 60-80 ms of induced latency on this link, with 40ms of baseline
>> latency. Not a RTT of > 300!
>>
>>
>>>>> To summarize, the measured buffering is well below one second for
>>>>> all tests, although the results are not directly comparable since we
>>>>> are using directly attached USB modems (Huawei E392 with Ubuntu
>>>>> 12.04)
>>>>
>>>> ubuntu has backported fq_codel as far as 12.04. What kernel are you using?
>>>
>>> On the "mobile terminal" we have 3.2.0-54-generic-pae #82-Ubuntu SMP
>>
>> I am delighted you've been collecting this data long term.
>>
>> I do have a problem in that the 3.2 kernel predates all the
>> bufferbloat related fixes made in the 4 years since that release.
>> we've been working our butts off here... :(
>>
>> Any chance you can update to 3.10.x or later for both this (driver and
>> buffering fixes) and your fixed host? (tcp fixes). And it's generally
>> been my hope people would try fq_codel (or sfq or pie or red) on such
>> links, too...
>>
>> There has been very little done to optimize usb-net, although we
>> tried. If this is using the ppp driver, that was improved
>> substantially around 3.6?
>>
>>> The fixed host ("tptest") runs a self-compiled stock kernel 3.4.1 with
>>> web10g patches, since we were interested in extracting TCP variable data not
>>> available from packet traces.
>>>
>>>>> instead of a separate router box. However, the numbers might
>>>>> be interesting in a more general perspective.
>>>>
>>>> Yes, they are. These are repeatable?
>>>>
>>>>
>>>
>>> For the netalyzer I've only done singular tests, but it can easily be
>>> repeated if needed (probably not).
>>> The mean of all mean RTTs for 4G, all operators, all times, is 334 ms, and
>>> for 3.5G it's 573 ms.
>>> This indicates that results are repeatable, and the set from 2013-09-01 is
>>> representative.
>>
>> except that my plot shows differently...
>>
>> and you are running an ancient kernel, yes. Something of a major
>> variable, that... and if you could let us know the usb path to the
>> various underlying drivers we can poke into matters there?
>>
>>>
>>> "below one second" was in relation to the >5 second RTT's, ideally it should
>>> be much lower of course. As a side note, we also do measurements in unloaded
>>> networks, and for 4G we get around 40-60 ms RTT (ICMP and TCP packets).
>>>
>>> Also, you might be interested in a paper we published in June, that reports
>>> on measurements of bloat for short flows in relation to different congestion
>>> control algorithms, in 3G, 3.5G and 4G (but for a single operator).
>>> See http://urn.kb.se/resolve?urn=urn:nbn:se:kau:diva-27779
>>> with fulltext at http://dx.doi.org/10.1109/WoWMoM.2013.6583408.
>>> I have also put a local copy (for personal use etc) at
>>> http://www.cs.kau.se/stefalfr/3G4G/alfredsson-bufferbloat-wowmom13.pdf
>>>
>>>
>>> Regards,
>>>  Stefan
>>>
>>> --
>>> Stefan Alfredsson, PhD         Tel: +46 (0) 54 700 1668
>>> Datavetenskap                  Kontor: 21F-425 (Hus Vanern)
>>> Karlstads universitet          PGP 0xB19B4B16
>>> SE-651 88 Karlstad             http://www.cs.kau.se/stefalfr/
>>>
>>>
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>>
>>
>>
>> --
>> Dave Täht
>>
>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
>
>
>
> --
> Dave Täht
>
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Bloat] mobile broadband buffer bloat

[Mikael Abrahamsson]

On Thu, 31 Oct 2013, Dave Taht wrote:

> I'm really impressed you can get ~72Mbit down and ~4Mbit up. (closer to 
> 8 up when you consider acks) Note also the relationship between 
> bandwidth and latency at T+22...

http://www.induowireless.com/nu/gsm-3g-4g-frekvensband/

Basically the swedish operators have more frequencies available to them 
than the US ones, and the country is more sparsely populated. Especially 
the 20Mhz available per operator in 2600MHz makes a huge difference. Even 
initial deployment of 4G in Sweden meant one was able to regularily get 80 
megabit/s of download speed. Now with a lot of users this is of course 
less, but sometimes if you're alone in the cell you can still get this. 
I'm actually surprised that they only get 8 up, usually much more up speed 
should be possible.

(I used to work for one of the swedish mobile operators)

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] mobile broadband buffer bloat

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1382 bytes --]

I find the notion of LTE that's faster than DSL somewhat amazing, still.

(jealous)

-Aaron


On Fri, Nov 1, 2013 at 1:48 PM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Thu, 31 Oct 2013, Dave Taht wrote:
>
>  I'm really impressed you can get ~72Mbit down and ~4Mbit up. (closer to 8
>> up when you consider acks) Note also the relationship between bandwidth and
>> latency at T+22...
>>
>
> http://www.induowireless.com/**nu/gsm-3g-4g-frekvensband/<http://www.induowireless.com/nu/gsm-3g-4g-frekvensband/>
>
> Basically the swedish operators have more frequencies available to them
> than the US ones, and the country is more sparsely populated. Especially
> the 20Mhz available per operator in 2600MHz makes a huge difference. Even
> initial deployment of 4G in Sweden meant one was able to regularily get 80
> megabit/s of download speed. Now with a lot of users this is of course
> less, but sometimes if you're alone in the cell you can still get this. I'm
> actually surprised that they only get 8 up, usually much more up speed
> should be possible.
>
> (I used to work for one of the swedish mobile operators)
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
>
> ______________________________**_________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/**listinfo/bloat<https://lists.bufferbloat.net/listinfo/bloat>
>

[-- Attachment #2: Type: text/html, Size: 2210 bytes --]

Re: [Bloat] mobile broadband buffer bloat

[Jonathan Morton]

On 1 Nov, 2013, at 10:22 pm, Aaron Wood wrote:

> I find the notion of LTE that's faster than DSL somewhat amazing, still.

Here in Finland, even 3G can be faster than DSL, especially if the landline exchange isn't as close as it might be.

 - Jonathan Morton

Re: [Bloat] mobile broadband buffer bloat

[Mikael Abrahamsson]

On Fri, 1 Nov 2013, Aaron Wood wrote:

> I find the notion of LTE that's faster than DSL somewhat amazing, still.

Yes, LTE is really fast when the operator has 20MHz to play with.

The difference from DSL however is that the cell is shared, so it varies a 
lot more in speed compared to DSL (at least in the places where the DSL 
providers aren't congesting their uplinks at peak hour).

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se