From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id DD2D321F5CB for ; Fri, 25 Jul 2014 08:58:51 -0700 (PDT) Received: from hms-beagle.lan ([134.2.89.70]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0LsgvV-1WVD9K2ts9-012ESY; Fri, 25 Jul 2014 17:58:47 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) From: Sebastian Moeller In-Reply-To: Date: Fri, 25 Jul 2014 17:58:44 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <41DF4003-BAE8-4794-BEDF-EF2385F03685@gmx.de> References: <03292B76-5273-4912-BB18-90E95C16A9F5@pnsol.com> <66FF8435-C8A5-4596-B43A-EC12D537D49E@gmx.de> To: Martin Geddes X-Mailer: Apple Mail (2.1878.6) X-Provags-ID: V03:K0:QdOQR1v5TdjgtndawScTxpad7uEPe0XqRfuA4LSZ6R/IE01l3ws 6H4Oi/I0i2GRbL+uVfililFWseKqFW/ASON4yJHWyixEu+TCrChOhF2YaubmxIYelWZ0MgQ jnRIDe3qwW61wVBHNjbDCrXpRZ/dwzpcGVAnyv+g13lSjn6TmIr2mKVW1VBzZ7g+jExa7pg ONW3PkdsuEHIolbPGMvhA== Cc: Neil Davies , cerowrt-devel , bloat Subject: Re: [Cerowrt-devel] [Bloat] Check out www.speedof.me - no Flash X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 15:58:52 -0000 Hi Martin, thanks for the pointers, On Jul 25, 2014, at 16:25 , Martin Geddes wrote: > You may find the following useful background reading on the state of = the art in network measurement, and a primer on =CE=94Q (which is the = property we wish to measure). >=20 > First, start with this presentation: Network performance optimisation = using high-fidelity measures > Then read this one to decompose =CE=94Q into G, S and V: Fundamentals = of network performance engineering > Then read this one to get a bit more sense on what =CE=94Q is about: = Introduction to =CE=94Q and Network Performance Science (extracts) >=20 > Then read these essays: >=20 > Foundation of Network Science > How to do network performance chemistry > How to X-ray a telecoms network > There is no quality in averages: IPX case study All of this makes intuitively sense, but it is a bit light on = how deltaQ is to be computed ;). As far as I understand it also has not much bearing on my home = network; the only one under my control. Now, following the buffer bloat = discussion for some years, I have internalized the idea that bandwidth = alone does not suffice to describe the quality of my network connection. = I think that the latency increase under load (for unrelated flows) is = the best of all the bad single number measures of network = dynamics/quality. I should be related to what I understood deltaQ to = depend on (as packet loss for non real time flows will cause an increase = in latency). I think that continuous measurements make a to n of sense = for ISPs, backbone-operators, mobile carriers =E2=80=A6 but at home, = basically, I operate as my own network quality monitor ;) (that is I try = to pin point and debug (transient) anomalies). >=20 > Martin >=20 > For fresh thinking about telecoms sign up for my free newsletter or = visit the Geddes Think Tank. > LinkedIn Twitter Mobile: +44 7957 499219 Skype: mgeddes=20 > Martin Geddes Consulting Ltd, Incorporated in Scotland, number = SC275827 VAT Number: 859 5634 72 Registered office: 17-19 East London = Street, Edinburgh, EH7 4BN >=20 >=20 >=20 > On 25 July 2014 15:17, Sebastian Moeller wrote: > Hi Neil, >=20 >=20 > On Jul 25, 2014, at 14:24 , Neil Davies wrote: >=20 > > Rich > > > > I have a deep worry over this style of single point measurement - = and hence speed - as an appropriate measure. >=20 > But how do you propose to measure the (bottleneck) link = capacity then? It turns out for current CPE and CMTS/DSLAM equipment one = typically can not relay on good QoE out of the box, since typically = these devices do not use their (largish) buffers wisely. Instead the = current remedy is to take back control over the bottleneck link by = shaping the actually sent traffic to stay below the hardware link = capacity thereby avoiding feeling the consequences of the = over-buffering. But to do this is is quite helpful to get an educated = guess what the bottleneck links capacity actually is. And for that = purpose a speediest seems useful. >=20 >=20 > > We know, and have evidence, that throughput/utilisation is not a = good proxy for the network delivering suitable quality of experience. We = work with organisation (Telco=E2=80=99s, large system integrators etc) = where we spend a lot of time having to =E2=80=9Cundo=E2=80=9D the = consequences of =E2=80=9Cmaximising speed=E2=80=9D. Just like there is = more to life than work, there is more to QoE than speed. > > > > For more specific comments see inline > > > > On 25 Jul 2014, at 13:09, Rich Brown = wrote: > > > >> Neil, > >> > >> Thanks for the note and the observations. My thoughts: > >> > >> 1) I note that speedof.me does seem to overstate the speed results. = At my home, it reports 5.98mbps down, and 638kbps up, while = betterspeedtest.sh shows 5.49/0.61 mbps. (speedtest.net gives numbers = similar to the betterspeedtest.net script.) > >> > >> 2) I think we're in agreement about the peak upload rate that you = point out is too high. Their measurement code runs in the browser. It = seems likely that the browser pumps out a few big packets before getting = flow control information, thus giving the impression that they can send = at a higher rate. This comports with the obvious decay that ramps toward = the long-term rate. > > > > I think that its simpler than that, it is measuring the rate at = which it can push packets out the interface - its real time rate is = precisely that - it can not be the rate being reported by the far end, = it can never exceed the limiting link. The long term average (if it is = like other speed testers we=E2=80=99ve had to look into) is being = measured at the TCP/IP SDU level by measuring the difference in time = between the first and last timestamps of data stream and dividing that = into the total data sent. Their =E2=80=9Cover-estimate=E2=80=9D is = because there are packets buffered in the CPE that have left the machine = but not arrived at the far end. >=20 > Testing from an openwrt router located at a = high-symmetric-bandwidth location shows that speedof.me does not scale = higher than ~ 130 Mbps server to client and ~15Mbps client to server (on = the same connection I can get 130Mbps S2C and ~80Mbps C2S, so the = asymmetry in the speedof.me results is not caused by my local = environment). > @Rich and Dave, this probably means that for the upper end of = fiber and cable and VDSL connections speed of.me is not going to be a = reliable speed measure=E2=80=A6 Side note www.sppedtest.net shows = ~100Mbps S2C and ~100Mbps C2S, so might be better suited to high-upload = links... >=20 > > > >> > >> 3) But that long-term speed should be at or below the theoretical = long-term rate, not above it. > > > > Agreed, but in this case knowing the sync rate already defines that = maximum. >=20 > I fully agree, but for ADSL the sync rate also contains a lot = of encapsulation, so the maximum achievable TCP rate is at best ~90% of = link rate. Note for cerowrt=E2=80=99s SQM system the link rate is = exactly the right number to start out with at that system can take the = encapsulation into account. But even then it is somewhat unintuitive to = deduce the expected good-put from the link rate. >=20 > > > >> > >> Two experiments for you to try: > >> > >> a) What does betterspeedtest.sh show? (It's in the latest CeroWrt, = in /usr/lib/CeroWrtScripts, or get it from github: = https://github.com/richb-hanover/CeroWrtScripts ) > >> > >> b) What does www.speedtest.net show? > >> > >> I will add your question (about the inaccuracy) to the note that I = want to send out to speedof.me this weekend. I will also ask that they = include min/max latency measurements to their test, and an option to = send for > 10 seconds to minimize any effect of PowerBoost=E2=80=A6 >=20 > I think they do already, at least for the download bandwidth; = they start with 128Kb and keep doubling the file size until a file takes = longer than 8 seconds to transfer, they only claim to report the numbers = from that last transferred file, so worst case with a stable link and a = bandwidth > 16kbps ;), it has taken at least 12 seconds (4 plus 8) of = measuring before the end of the plot, so the bandwidth of at least the = last half of the download plot should be representative even assuming = power boost. Caveat, I assume that power boost will not be reset by the = transient lack of data transfer between the differently sized files (but = since it should involve the same IPs and port# why should power boost = reset itself?). >=20 > Best Regards > Sebastian >=20 >=20 >=20 > >> > >> Best regards, > >> > >> Rich > >> > >> > >> > >> On Jul 25, 2014, at 5:10 AM, Neil Davies = wrote: > >> > >>> Rich > >>> > >>> You may want to check how accurate they are to start. > >>> > >>> I just ran a =E2=80=9Cspeed test=E2=80=9D on my line (which I have = complete control and visibility over the various network elements) and = it reports an average =E2=80=9Cspeed=E2=80=9D (in the up direction) that = is in excess of the capacity of the line, it reports the maximum rate at = nearly twice the best possible rate of the ADSL connection. > >>> > >>> Doesn=E2=80=99t matter how pretty it is, if its not accurate it is = of no use. This is rather ironic as the web site claims it is the = =E2=80=9Csmartest and most accurate=E2=80=9D! > >>> > >>> Neil > >>> > >>> > >>> > >>> PS pretty clear to me what mistake they=E2=80=99ve made in the = measurement process - its to do with incorrect inference and hence = missing the buffering effects. > >>> > >>> On 20 Jul 2014, at 14:19, Rich Brown = wrote: > >>> > >>>> Doc Searls = (http://blogs.law.harvard.edu/doc/2014/07/20/the-cliff-peronal-clouds-need= -to-climb/) mentioned in passing that he uses a new speed test website. = I checked it out, and it was very cool=E2=80=A6 > >>>> > >>>> www.speedof.me is an all-HTML5 website that seems to make = accurate measurements of the up and download speeds of your internet = connection. It=E2=80=99s also very attractive, and the real-time plots = of the speed show interesting info. (screen shot at: = http://richb-hanover.com/speedof-me/) > >>>> > >>>> Now if we could get them to a) allow longer/bigger tests to = circumvent PowerBoost, and b) include a latency measurement so people = could point out their bufferbloated equipment. > >>>> > >>>> I'm going to send them a note. Anything else I should add? > >>>> > >>>> Rich > >>>> _______________________________________________ > >>>> Bloat mailing list > >>>> Bloat@lists.bufferbloat.net > >>>> https://lists.bufferbloat.net/listinfo/bloat > >>> > >> > > > > _______________________________________________ > > Bloat mailing list > > Bloat@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat >=20 > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat >=20