From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from eu1sys200aog106.obsmtp.com (eu1sys200aog106.obsmtp.com [207.126.144.121]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 73C2E21F3AD; Fri, 25 Jul 2014 05:24:51 -0700 (PDT) Received: from mail.la.pnsol.com ([89.145.213.110]) (using TLSv1) by eu1sys200aob106.postini.com ([207.126.147.11]) with SMTP ID DSNKU9JMkfXqCiBAdl/YE2plIPUp5Ld3+GOJ@postini.com; Fri, 25 Jul 2014 12:24:51 UTC Received: from ba6-office.pnsol.com ([172.20.5.199]) by mail.la.pnsol.com with esmtp (Exim 4.76) (envelope-from ) id 1XAeYK-0004iH-MC; Fri, 25 Jul 2014 13:24:48 +0100 Content-Type: multipart/signed; boundary="Apple-Mail=_6195CCDA-5214-488B-ADD8-244CDCACA864"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) From: Neil Davies In-Reply-To: Date: Fri, 25 Jul 2014 13:24:47 +0100 Message-Id: References: <03292B76-5273-4912-BB18-90E95C16A9F5@pnsol.com> To: Rich Brown X-Mailer: Apple Mail (2.1878.6) X-Mailman-Approved-At: Fri, 25 Jul 2014 11:16:23 -0700 Cc: cerowrt-devel , bloat Subject: Re: [Cerowrt-devel] [Bloat] Check out www.speedof.me - no Flash X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 12:24:52 -0000 --Apple-Mail=_6195CCDA-5214-488B-ADD8-244CDCACA864 Content-Type: multipart/alternative; boundary="Apple-Mail=_B68C7162-B437-479C-9172-1899EDC3185C" --Apple-Mail=_B68C7162-B437-479C-9172-1899EDC3185C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Rich I have a deep worry over this style of single point measurement - and = hence speed - as an appropriate measure. We know, and have evidence, = that throughput/utilisation is not a good proxy for the network = delivering suitable quality of experience. We work with organisation = (Telco=92s, large system integrators etc) where we spend a lot of time = having to =93undo=94 the consequences of =93maximising speed=94. Just = like there is more to life than work, there is more to QoE than speed. For more specific comments see inline On 25 Jul 2014, at 13:09, Rich Brown wrote: > Neil, >=20 > Thanks for the note and the observations. My thoughts: >=20 > 1) I note that speedof.me does seem to overstate the speed results. At = my home, it reports 5.98mbps down, and 638kbps up, while = betterspeedtest.sh shows 5.49/0.61 mbps. (speedtest.net gives numbers = similar to the betterspeedtest.net script.) >=20 > 2) I think we're in agreement about the peak upload rate that you = point out is too high. Their measurement code runs in the browser. It = seems likely that the browser pumps out a few big packets before getting = flow control information, thus giving the impression that they can send = at a higher rate. This comports with the obvious decay that ramps toward = the long-term rate.=20 I think that its simpler than that, it is measuring the rate at which it = can push packets out the interface - its real time rate is precisely = that - it can not be the rate being reported by the far end, it can = never exceed the limiting link. The long term average (if it is like = other speed testers we=92ve had to look into) is being measured at the = TCP/IP SDU level by measuring the difference in time between the first = and last timestamps of data stream and dividing that into the total data = sent. Their =93over-estimate=94 is because there are packets buffered in = the CPE that have left the machine but not arrived at the far end. >=20 > 3) But that long-term speed should be at or below the theoretical = long-term rate, not above it.=20 Agreed, but in this case knowing the sync rate already defines that = maximum. >=20 > Two experiments for you to try: >=20 > a) What does betterspeedtest.sh show? (It's in the latest CeroWrt, in = /usr/lib/CeroWrtScripts, or get it from github: = https://github.com/richb-hanover/CeroWrtScripts ) >=20 > b) What does www.speedtest.net show? >=20 > I will add your question (about the inaccuracy) to the note that I = want to send out to speedof.me this weekend. I will also ask that they = include min/max latency measurements to their test, and an option to = send for > 10 seconds to minimize any effect of PowerBoost... >=20 > Best regards, >=20 > Rich >=20 >=20 >=20 > On Jul 25, 2014, at 5:10 AM, Neil Davies = wrote: >=20 >> Rich >>=20 >> You may want to check how accurate they are to start. >>=20 >> I just ran a =93speed test=94 on my line (which I have complete = control and visibility over the various network elements) and it reports = an average =93speed=94 (in the up direction) that is in excess of the = capacity of the line, it reports the maximum rate at nearly twice the = best possible rate of the ADSL connection. >>=20 >> Doesn=92t matter how pretty it is, if its not accurate it is of no = use. This is rather ironic as the web site claims it is the =93smartest = and most accurate=94! >>=20 >> Neil >>=20 >> >>=20 >> PS pretty clear to me what mistake they=92ve made in the measurement = process - its to do with incorrect inference and hence missing the = buffering effects. >>=20 >> On 20 Jul 2014, at 14:19, Rich Brown wrote: >>=20 >>> Doc Searls = (http://blogs.law.harvard.edu/doc/2014/07/20/the-cliff-peronal-clouds-need= -to-climb/) mentioned in passing that he uses a new speed test website. = I checked it out, and it was very cool=85 >>>=20 >>> www.speedof.me is an all-HTML5 website that seems to make accurate = measurements of the up and download speeds of your internet connection. = It=92s also very attractive, and the real-time plots of the speed show = interesting info. (screen shot at: http://richb-hanover.com/speedof-me/) >>>=20 >>> Now if we could get them to a) allow longer/bigger tests to = circumvent PowerBoost, and b) include a latency measurement so people = could point out their bufferbloated equipment.=20 >>>=20 >>> I'm going to send them a note. Anything else I should add? >>>=20 >>> Rich >>> _______________________________________________ >>> Bloat mailing list >>> Bloat@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/bloat >>=20 >=20 --Apple-Mail=_B68C7162-B437-479C-9172-1899EDC3185C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 Rich

I have a deep worry over = this style of single point measurement - and hence speed - as an = appropriate measure. We know, and have evidence, that = throughput/utilisation is not a good proxy for the network delivering = suitable quality of experience. We work with organisation (Telco=92s, = large system integrators etc) where we spend a lot of time having to = =93undo=94 the consequences of =93maximising speed=94. Just like there = is more to life than work, there is more to QoE than = speed.

For more specific comments see = inline

On 25 Jul 2014, at 13:09, Rich Brown = <richb.hanover@gmail.com> = wrote:

Neil,

Thanks for the note and the = observations. My thoughts:

1) I note that speedof.me does seem to overstate the = speed results. At my home, it reports 5.98mbps down, and 638kbps up, = while betterspeedtest.sh shows 5.49/0.61 mbps. (speedtest.net gives numbers similar = to the betterspeedtest.net = script.)

2) I think we're in agreement about = the peak upload rate that you point out is too high. Their measurement = code runs in the browser. It seems likely that the browser pumps out a = few big packets before getting flow control information, thus giving the = impression that they can send at a higher rate. This comports with the = obvious decay that ramps toward the long-term = rate. 

I think that its = simpler than that, it is measuring the rate at which it can push packets = out the interface - its real time rate is precisely that - it can not be = the rate being reported by the far end, it can never exceed the limiting = link. The long term average (if it is like other speed testers we=92ve = had to look into) is being measured at the TCP/IP SDU level by measuring = the difference in time between the first and last timestamps of data = stream and dividing that into the total data sent. Their =93over-estimate=94= is because there are packets buffered in the CPE that have left the = machine but not arrived at the far end.


3) = But that long-term speed should be at or below the theoretical long-term = rate, not above = it. 

Agreed, but in = this case knowing the sync rate already defines that = maximum.


Two experiments for you to = try:

a) What does betterspeedtest.sh show? = (It's in the latest CeroWrt, in /usr/lib/CeroWrtScripts, or get it from = github: https://github.co= m/richb-hanover/CeroWrtScripts )

b) = What does www.speedtest.net = show?

I will add your question (about the = inaccuracy) to the note that I want to send out to speedof.me this weekend. I will also ask = that they include min/max latency measurements to their test, and an = option to send for > 10 seconds to minimize any effect of = PowerBoost...

Best = regards,

Rich


=

On Jul 25, 2014, at 5:10 AM, Neil Davies <neil.davies@pnsol.com> = wrote:

Rich

You may want to check how = accurate they are to start.

I just ran a =93speed= test=94 on my line (which I have complete control and visibility over = the various network elements) and it reports an average =93speed=94 (in = the up direction) that is in excess of the capacity of the line, it = reports the maximum rate at nearly twice the best possible rate of the = ADSL connection.

Doesn=92t matter how pretty it = is, if its not accurate it is of no use. This is rather ironic as the = web site claims it is the =93smartest and most = accurate=94!

Neil

= <speedof_me_14-07-25.png>

PS = pretty clear to me what mistake they=92ve made in the measurement = process - its to do with incorrect inference and hence missing the = buffering effects.

On 20 Jul 2014, at = 14:19, Rich Brown <richb.hanover@gmail.com> = wrote:

Doc Searls (http://blogs.law.harvard.edu/doc/2014/07/20/the-cliff-p= eronal-clouds-need-to-climb/) mentioned in passing that he uses a = new speed test website. I checked it out, and it was very = cool=85

www.speedof.me is = an all-HTML5 website that seems to make accurate measurements of the up = and download speeds of your internet connection. It=92s also very = attractive, and the real-time plots of the speed show interesting info. = (screen shot at: http://richb-hanover.com/spe= edof-me/)

Now if we could get them to a) allow longer/bigger = tests to circumvent PowerBoost, and b) include a latency measurement so = people could point out their bufferbloated equipment.

I'm going = to send them a note. Anything else I should = add?

Rich
_______________________________________________
Blo= at mailing list
Bloat@lists.bufferbloat.net
https://lists.buffer= bloat.net/listinfo/bloat



= --Apple-Mail=_B68C7162-B437-479C-9172-1899EDC3185C-- --Apple-Mail=_6195CCDA-5214-488B-ADD8-244CDCACA864 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iF4EAREIAAYFAlPSTI8ACgkQJLPLuiuo8FJZPwD7Bydy1nPFi/sjonBc0yRQkBPe IFDl6XWSX9a96em2CboA/R3NTAUogxMZ+qHnRtFrl4Gbk6/RIyLfU60Leg1n1K8M =YTS5 -----END PGP SIGNATURE----- --Apple-Mail=_6195CCDA-5214-488B-ADD8-244CDCACA864--