From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f47.google.com (eu1sys200aog121.obsmtp.com [207.126.144.151]) by huchra.bufferbloat.net (Postfix) with SMTP id 7286E21F5BC for ; Fri, 25 Jul 2014 07:25:39 -0700 (PDT) Received: from mail-pa0-f47.google.com ([209.85.220.47]) (using TLSv1) by eu1sys200aob121.postini.com ([207.126.147.11]) with SMTP ID DSNKU9Jo4sdoYpYJulZr+NkB6fOsH9xz8Vix@postini.com; Fri, 25 Jul 2014 14:25:39 UTC Received: by mail-pa0-f47.google.com with SMTP id kx10so6141236pab.6 for ; Fri, 25 Jul 2014 07:25:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=/h54DxovhKhc7LOs4v0NGiFptjnkIHqi8kaGgvKR6HM=; b=Tl/Mb2zErXIQNAZznDYp1KlniI6bvGvfE2WfSIpOLAWXeTBsFTEqdO/1oBMYryvUWj WmySeIwW0bZFPnzrOpS1dftIiLGgMcY4klv7YGtmo6OuPEEa5nvMsEs30usN1K/ZNSTd NT86P0+ILS4xej3x29dBH2IEpXRDXX30d7ILyFpvDudDHQ0CXjFQoOP4CR7tkiWqcIja Ng/SOl5/vH4/FoX8RQ3oXVVFgmkdLoIgZj9oY6VM6Qi1DEBVTHp7Oe4ckKXiuuU8psB5 FtW8wkL0Z82iEagE0JQYN2HZmt90X4l7oYu1TEKeUImxVO516XUAQP7p7XH+roqYWZoW MYmg== X-Gm-Message-State: ALoCoQn4+eDUJeEdyJxFGMYD5/eAh0gEb6zoeX6Wxje6TPN0o6kc34JUYhEyVU3M+5QcHPBTPkH0dA6tXxXXXASHjGczP0p3RLULtek5GMVkPzPfI7E0jzXaDMWB8Q9Tb7tQCTPmkZwYqUX4Y6e0VO0cZsRXWA6XoA== X-Received: by 10.70.45.97 with SMTP id l1mr3461207pdm.148.1406298337068; Fri, 25 Jul 2014 07:25:37 -0700 (PDT) X-Received: by 10.70.45.97 with SMTP id l1mr3461140pdm.148.1406298336593; Fri, 25 Jul 2014 07:25:36 -0700 (PDT) MIME-Version: 1.0 Received: by 10.70.90.104 with HTTP; Fri, 25 Jul 2014 07:25:16 -0700 (PDT) In-Reply-To: <66FF8435-C8A5-4596-B43A-EC12D537D49E@gmx.de> References: <03292B76-5273-4912-BB18-90E95C16A9F5@pnsol.com> <66FF8435-C8A5-4596-B43A-EC12D537D49E@gmx.de> From: Martin Geddes Date: Fri, 25 Jul 2014 15:25:16 +0100 Message-ID: To: Sebastian Moeller Content-Type: multipart/alternative; boundary=047d7bfeacb0cd362504ff0558ca X-Mailman-Approved-At: Fri, 25 Jul 2014 11:16:23 -0700 Cc: Neil Davies , cerowrt-devel , bloat Subject: Re: [Cerowrt-devel] [Bloat] Check out www.speedof.me - no Flash X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 14:25:41 -0000 --047d7bfeacb0cd362504ff0558ca Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable You may find the following useful background reading on the state of the art in network measurement, and a primer on =CE=94Q (which is the property = we wish to measure). First, start with this presentation: Network performance optimisation using high-fidelity measures Then read this one to decompose =CE=94Q into G, S and V: Fundamentals of ne= twork performance engineering Then read this one to get a bit more sense on what =CE=94Q is about: Introd= uction to =CE=94Q and Network Performance Science (extracts) Then read these essays: Foundation of Network Science How to do network performance chemistry How to X-ray a telecoms network There is no quality in averages: IPX case study Martin *For fresh thinking about telecoms sign up for my free newsletter or visit the Geddes Think Tank .* LinkedIn Twitter Mobile: +44 7957 499219 Skype: mgeddes Martin Geddes Consulting Ltd, Incorporated in Scotland, number SC275827 VAT Number: 859 5634 72 Registered office: 17-19 East London Street, Edinburgh, EH7 4BN On 25 July 2014 15:17, Sebastian Moeller wrote: > Hi Neil, > > > On Jul 25, 2014, at 14:24 , Neil Davies wrote: > > > Rich > > > > I have a deep worry over this style of single point measurement - and > hence speed - as an appropriate measure. > > But how do you propose to measure the (bottleneck) link capacity > then? It turns out for current CPE and CMTS/DSLAM equipment one typically > can not relay on good QoE out of the box, since typically these devices d= o > not use their (largish) buffers wisely. Instead the current remedy is to > take back control over the bottleneck link by shaping the actually sent > traffic to stay below the hardware link capacity thereby avoiding feeling > the consequences of the over-buffering. But to do this is is quite helpfu= l > to get an educated guess what the bottleneck links capacity actually is. > And for that purpose a speediest seems useful. > > > > We know, and have evidence, that throughput/utilisation is not a good > proxy for the network delivering suitable quality of experience. We work > with organisation (Telco=E2=80=99s, large system integrators etc) where w= e spend a > lot of time having to =E2=80=9Cundo=E2=80=9D the consequences of =E2=80= =9Cmaximising speed=E2=80=9D. Just > like there is more to life than work, there is more to QoE than speed. > > > > For more specific comments see inline > > > > On 25 Jul 2014, at 13:09, Rich Brown wrote: > > > >> Neil, > >> > >> Thanks for the note and the observations. My thoughts: > >> > >> 1) I note that speedof.me does seem to overstate the speed results. At > my home, it reports 5.98mbps down, and 638kbps up, while betterspeedtest.= sh > shows 5.49/0.61 mbps. (speedtest.net gives numbers similar to the > betterspeedtest.net script.) > >> > >> 2) I think we're in agreement about the peak upload rate that you poin= t > out is too high. Their measurement code runs in the browser. It seems > likely that the browser pumps out a few big packets before getting flow > control information, thus giving the impression that they can send at a > higher rate. This comports with the obvious decay that ramps toward the > long-term rate. > > > > I think that its simpler than that, it is measuring the rate at which i= t > can push packets out the interface - its real time rate is precisely that= - > it can not be the rate being reported by the far end, it can never exceed > the limiting link. The long term average (if it is like other speed teste= rs > we=E2=80=99ve had to look into) is being measured at the TCP/IP SDU level= by > measuring the difference in time between the first and last timestamps of > data stream and dividing that into the total data sent. Their > =E2=80=9Cover-estimate=E2=80=9D is because there are packets buffered in = the CPE that have > left the machine but not arrived at the far end. > > Testing from an openwrt router located at a > high-symmetric-bandwidth location shows that speedof.me does not scale > higher than ~ 130 Mbps server to client and ~15Mbps client to server (on > the same connection I can get 130Mbps S2C and ~80Mbps C2S, so the asymmet= ry > in the speedof.me results is not caused by my local environment). > @Rich and Dave, this probably means that for the upper end of > fiber and cable and VDSL connections speed of.me is not going to be a > reliable speed measure=E2=80=A6 Side note www.sppedtest.net shows ~100Mbp= s S2C > and ~100Mbps C2S, so might be better suited to high-upload links... > > > > >> > >> 3) But that long-term speed should be at or below the theoretical > long-term rate, not above it. > > > > Agreed, but in this case knowing the sync rate already defines that > maximum. > > I fully agree, but for ADSL the sync rate also contains a lot of > encapsulation, so the maximum achievable TCP rate is at best ~90% of link > rate. Note for cerowrt=E2=80=99s SQM system the link rate is exactly the = right > number to start out with at that system can take the encapsulation into > account. But even then it is somewhat unintuitive to deduce the expected > good-put from the link rate. > > > > >> > >> Two experiments for you to try: > >> > >> a) What does betterspeedtest.sh show? (It's in the latest CeroWrt, in > /usr/lib/CeroWrtScripts, or get it from github: > https://github.com/richb-hanover/CeroWrtScripts ) > >> > >> b) What does www.speedtest.net show? > >> > >> I will add your question (about the inaccuracy) to the note that I wan= t > to send out to speedof.me this weekend. I will also ask that they include > min/max latency measurements to their test, and an option to send for > 1= 0 > seconds to minimize any effect of PowerBoost=E2=80=A6 > > I think they do already, at least for the download bandwidth; the= y > start with 128Kb and keep doubling the file size until a file takes longe= r > than 8 seconds to transfer, they only claim to report the numbers from th= at > last transferred file, so worst case with a stable link and a bandwidth > > 16kbps ;), it has taken at least 12 seconds (4 plus 8) of measuring befor= e > the end of the plot, so the bandwidth of at least the last half of the > download plot should be representative even assuming power boost. Caveat,= I > assume that power boost will not be reset by the transient lack of data > transfer between the differently sized files (but since it should involve > the same IPs and port# why should power boost reset itself?). > > Best Regards > Sebastian > > > > >> > >> Best regards, > >> > >> Rich > >> > >> > >> > >> On Jul 25, 2014, at 5:10 AM, Neil Davies wrote= : > >> > >>> Rich > >>> > >>> You may want to check how accurate they are to start. > >>> > >>> I just ran a =E2=80=9Cspeed test=E2=80=9D on my line (which I have co= mplete control > and visibility over the various network elements) and it reports an avera= ge > =E2=80=9Cspeed=E2=80=9D (in the up direction) that is in excess of the ca= pacity of the > line, it reports the maximum rate at nearly twice the best possible rate = of > the ADSL connection. > >>> > >>> Doesn=E2=80=99t matter how pretty it is, if its not accurate it is of= no use. > This is rather ironic as the web site claims it is the =E2=80=9Csmartest = and most > accurate=E2=80=9D! > >>> > >>> Neil > >>> > >>> > >>> > >>> PS pretty clear to me what mistake they=E2=80=99ve made in the measur= ement > process - its to do with incorrect inference and hence missing the > buffering effects. > >>> > >>> On 20 Jul 2014, at 14:19, Rich Brown wrote: > >>> > >>>> Doc Searls ( > http://blogs.law.harvard.edu/doc/2014/07/20/the-cliff-peronal-clouds-need= -to-climb/) > mentioned in passing that he uses a new speed test website. I checked it > out, and it was very cool=E2=80=A6 > >>>> > >>>> www.speedof.me is an all-HTML5 website that seems to make accurate > measurements of the up and download speeds of your internet connection. > It=E2=80=99s also very attractive, and the real-time plots of the speed s= how > interesting info. (screen shot at: http://richb-hanover.com/speedof-me/) > >>>> > >>>> Now if we could get them to a) allow longer/bigger tests to > circumvent PowerBoost, and b) include a latency measurement so people cou= ld > point out their bufferbloated equipment. > >>>> > >>>> I'm going to send them a note. Anything else I should add? > >>>> > >>>> Rich > >>>> _______________________________________________ > >>>> Bloat mailing list > >>>> Bloat@lists.bufferbloat.net > >>>> https://lists.bufferbloat.net/listinfo/bloat > >>> > >> > > > > _______________________________________________ > > Bloat mailing list > > Bloat@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > --047d7bfeacb0cd362504ff0558ca Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
You may find the following useful background reading on th= e state of the art in network measurement, and a primer on =CE=94Q (which i= s the property we wish to measure).

First, start with this presentation:=C2=A0Network performance optimisation using high-fidelity measu= res
Then read this o= ne to decompose=C2=A0=CE=94Q into G, S and V:=C2=A0Fundamentals of network performance engineering

For fresh thinking about telecoms=C2=A0sign up for my free newsletter=C2=A0or visit th= e Geddes Think Tank= .
LinkedIn=C2=A0Twitter=C2=A0Mobile: +44 7957 4992= 19=C2=A0Sky= pe: mgeddes=C2=A0
Martin Geddes Consulting Ltd,=C2=A0Incorporated in Scotland, number SC275827=C2=A0VAT Number: 859 5634 72=C2=A0Registered office:=C2= =A017-19 East London Street, Edinburgh, EH7 4BN



On 25 July 2014 15:17, Sebastian Moeller= <moeller0@gmx.de> wrote:
Hi Neil,


On Jul 25, 2014, at 14:24 , Neil Davies <Neil.Davies@pnsol.com> wrote:

> Rich
>
> I have a deep worry over this style of single point measurement - and = hence speed - as an appropriate measure.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 But how do you propose to measure the (bo= ttleneck) link capacity then? It turns out for current CPE and CMTS/DSLAM e= quipment one typically can not relay on good QoE out of the box, since typi= cally these devices do not use their (largish) buffers wisely. Instead the = current remedy is to take back control over the bottleneck link by shaping = the actually sent traffic to stay below the hardware link capacity thereby = avoiding feeling the consequences of the over-buffering. But to do this is = is quite helpful to get an educated guess what the bottleneck links capacit= y actually is. And for that purpose a speediest seems useful.


> We know, and have evidence, that throughput/utilisation is not a good = proxy for the network delivering suitable quality of experience. We work wi= th organisation (Telco=E2=80=99s, large system integrators etc) where we sp= end a lot of time having to =E2=80=9Cundo=E2=80=9D the consequences of =E2= =80=9Cmaximising speed=E2=80=9D. Just like there is more to life than work,= there is more to QoE than speed.
>
> For more specific comments see inline
>
> On 25 Jul 2014, at 13:09, Rich Brown <richb.hanover@gmail.com> wrote:
>
>> Neil,
>>
>> Thanks for the note and the observations. My thoughts:
>>
>> 1) I note that spe= edof.me does seem to overstate the speed results. At my home, it report= s 5.98mbps down, and 638kbps up, while betterspeedtest.sh shows 5.49/0.61 m= bps. (speedtest.net = gives numbers similar to the betterspeedtest.net script.)
>>
>> 2) I think we're in agreement about the peak upload rate that = you point out is too high. Their measurement code runs in the browser. It s= eems likely that the browser pumps out a few big packets before getting flo= w control information, thus giving the impression that they can send at a h= igher rate. This comports with the obvious decay that ramps toward the long= -term rate.
>
> I think that its simpler than that, it is measuring the rate at which = it can push packets out the interface - its real time rate is precisely tha= t - it can not be the rate being reported by the far end, it can never exce= ed the limiting link. The long term average (if it is like other speed test= ers we=E2=80=99ve had to look into) is being measured at the TCP/IP SDU lev= el by measuring the difference in time between the first and last timestamp= s of data stream and dividing that into the total data sent. Their =E2=80= =9Cover-estimate=E2=80=9D is because there are packets buffered in the CPE = that have left the machine but not arrived at the far end.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Testing from an openwrt router located at= a high-symmetric-bandwidth location shows that speedof.me does not scale higher than ~ 130 Mbps s= erver to client and ~15Mbps client to server (on the same connection I can = get 130Mbps S2C and ~80Mbps C2S, so the asymmetry in the speedof.me results is not caused by my lo= cal environment).
=C2=A0 =C2=A0 =C2=A0 =C2=A0 @Rich and Dave, this probably means that for th= e upper end of fiber and cable and VDSL connections speed of.me is not going to be a reliable speed mea= sure=E2=80=A6 Side note www.sppedtest.net shows ~100Mbps S2C and ~100Mbps C2S, so might be = better suited to high-upload links...

>
>>
>> 3) But that long-term speed should be at or below the theoretical = long-term rate, not above it.
>
> Agreed, but in this case knowing the sync rate already defines that ma= ximum.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 I fully agree, but for ADSL the sync rate= also contains a lot of encapsulation, so the maximum achievable TCP rate i= s at best ~90% of link rate. Note for cerowrt=E2=80=99s SQM system the link= rate is exactly the right number to start out with at that system can take= the encapsulation into account. But even then it is somewhat unintuitive t= o deduce the expected good-put from the link rate.

>
>>
>> Two experiments for you to try:
>>
>> a) What does betterspeedtest.sh show? (It's in the latest Cero= Wrt, in /usr/lib/CeroWrtScripts, or get it from github: https://github.c= om/richb-hanover/CeroWrtScripts )
>>
>> b) What does www.speedtest.net show?
>>
>> I will add your question (about the inaccuracy) to the note = that I want to send out to = speedof.me this weekend. I will also ask that they include min/max late= ncy measurements to their test, and an option to send for > 10 seconds t= o minimize any effect of PowerBoost=E2=80=A6

=C2=A0 =C2=A0 =C2=A0 =C2=A0 I think they do already, at least for the downl= oad bandwidth; they start with 128Kb and keep doubling the file size until = a file takes longer than 8 seconds to transfer, they only claim to report t= he numbers from that last transferred file, so worst case with a stable lin= k and a bandwidth > 16kbps ;), it has taken at least 12 seconds (4 plus = 8) of measuring before the end of the plot, so the bandwidth of at least th= e last half of the download plot should be representative even assuming pow= er boost. Caveat, I assume that power boost will not be reset by the transi= ent lack of data transfer between the differently sized files (but since it= should involve the same IPs and port# why should power boost reset itself?= ).

Best Regards
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = Sebastian



>>
>> Best regards,
>>
>> Rich
>>
>>
>>
>> On Jul 25, 2014, at 5:10 AM, Neil Davies <neil.davies@pnsol.com> wrote:
>>
>>> Rich
>>>
>>> You may want to check how accurate they are to start.
>>>
>>> I just ran a =E2=80=9Cspeed test=E2=80=9D on my line (which I = have complete control and visibility over the various network elements) and= it reports an average =E2=80=9Cspeed=E2=80=9D (in the up direction) that i= s in excess of the capacity of the line, it reports the maximum rate at nea= rly twice the best possible rate of the ADSL connection.
>>>
>>> Doesn=E2=80=99t matter how pretty it is, if its not accurate i= t is of no use. This is rather ironic as the web site claims it is the =E2= =80=9Csmartest and most accurate=E2=80=9D!
>>>
>>> Neil
>>>
>>> <speedof_me_14-07-25.png>
>>>
>>> PS pretty clear to me what mistake they=E2=80=99ve made in the= measurement process - its to do with incorrect inference and hence missing= the buffering effects.
>>>
>>> On 20 Jul 2014, at 14:19, Rich Brown <richb.hanover@gmail.com> wrote:
>>>
>>>> Doc Searls (http://= blogs.law.harvard.edu/doc/2014/07/20/the-cliff-peronal-clouds-need-to-climb= /) mentioned in passing that he uses a new speed test website. I checke= d it out, and it was very cool=E2=80=A6
>>>>
>>>> www.sp= eedof.me is an all-HTML5 website that seems to make accurate measuremen= ts of the up and download speeds of your internet connection. It=E2=80=99s = also very attractive, and the real-time plots of the speed show interesting= info. (screen shot at: http://richb-hanover.com/speedof-me/)
>>>>
>>>> Now if we could get them to a) allow longer/bigger tests t= o circumvent PowerBoost, and b) include a latency measurement so people cou= ld point out their bufferbloated equipment.
>>>>
>>>> I'm going to send them a note. Anything else I should = add?
>>>>
>>>> Rich
>>>> _______________________________________________
>>>> Bloat mailing list
>>>> Bloat@lists= .bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/bloat
>>>
>>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat= .net
> https://lists.bufferbloat.net/listinfo/bloat

_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<= /a>
= https://lists.bufferbloat.net/listinfo/bloat

--047d7bfeacb0cd362504ff0558ca--