You may find the following useful background reading on the state of the art in network measurement, and a primer on ΔQ (which is the property we wish to measure). First, start with this presentation: Network performance optimisation using high-fidelity measures Then read this one to decompose ΔQ into G, S and V: Fundamentals of network performance engineering Then read this one to get a bit more sense on what ΔQ is about: Introduction to ΔQ and Network Performance Science (extracts) Then read these essays: Foundation of Network Science How to do network performance chemistry How to X-ray a telecoms network There is no quality in averages: IPX case study Martin *For fresh thinking about telecoms sign up for my free newsletter or visit the Geddes Think Tank .* LinkedIn Twitter Mobile: +44 7957 499219 Skype: mgeddes Martin Geddes Consulting Ltd, Incorporated in Scotland, number SC275827 VAT Number: 859 5634 72 Registered office: 17-19 East London Street, Edinburgh, EH7 4BN On 25 July 2014 15:17, Sebastian Moeller wrote: > Hi Neil, > > > On Jul 25, 2014, at 14:24 , Neil Davies wrote: > > > Rich > > > > I have a deep worry over this style of single point measurement - and > hence speed - as an appropriate measure. > > But how do you propose to measure the (bottleneck) link capacity > then? It turns out for current CPE and CMTS/DSLAM equipment one typically > can not relay on good QoE out of the box, since typically these devices do > not use their (largish) buffers wisely. Instead the current remedy is to > take back control over the bottleneck link by shaping the actually sent > traffic to stay below the hardware link capacity thereby avoiding feeling > the consequences of the over-buffering. But to do this is is quite helpful > to get an educated guess what the bottleneck links capacity actually is. > And for that purpose a speediest seems useful. > > > > We know, and have evidence, that throughput/utilisation is not a good > proxy for the network delivering suitable quality of experience. We work > with organisation (Telco’s, large system integrators etc) where we spend a > lot of time having to “undo” the consequences of “maximising speed”. Just > like there is more to life than work, there is more to QoE than speed. > > > > For more specific comments see inline > > > > On 25 Jul 2014, at 13:09, Rich Brown wrote: > > > >> Neil, > >> > >> Thanks for the note and the observations. My thoughts: > >> > >> 1) I note that speedof.me does seem to overstate the speed results. At > my home, it reports 5.98mbps down, and 638kbps up, while betterspeedtest.sh > shows 5.49/0.61 mbps. (speedtest.net gives numbers similar to the > betterspeedtest.net script.) > >> > >> 2) I think we're in agreement about the peak upload rate that you point > out is too high. Their measurement code runs in the browser. It seems > likely that the browser pumps out a few big packets before getting flow > control information, thus giving the impression that they can send at a > higher rate. This comports with the obvious decay that ramps toward the > long-term rate. > > > > I think that its simpler than that, it is measuring the rate at which it > can push packets out the interface - its real time rate is precisely that - > it can not be the rate being reported by the far end, it can never exceed > the limiting link. The long term average (if it is like other speed testers > we’ve had to look into) is being measured at the TCP/IP SDU level by > measuring the difference in time between the first and last timestamps of > data stream and dividing that into the total data sent. Their > “over-estimate” is because there are packets buffered in the CPE that have > left the machine but not arrived at the far end. > > Testing from an openwrt router located at a > high-symmetric-bandwidth location shows that speedof.me does not scale > higher than ~ 130 Mbps server to client and ~15Mbps client to server (on > the same connection I can get 130Mbps S2C and ~80Mbps C2S, so the asymmetry > in the speedof.me results is not caused by my local environment). > @Rich and Dave, this probably means that for the upper end of > fiber and cable and VDSL connections speed of.me is not going to be a > reliable speed measure… Side note www.sppedtest.net shows ~100Mbps S2C > and ~100Mbps C2S, so might be better suited to high-upload links... > > > > >> > >> 3) But that long-term speed should be at or below the theoretical > long-term rate, not above it. > > > > Agreed, but in this case knowing the sync rate already defines that > maximum. > > I fully agree, but for ADSL the sync rate also contains a lot of > encapsulation, so the maximum achievable TCP rate is at best ~90% of link > rate. Note for cerowrt’s SQM system the link rate is exactly the right > number to start out with at that system can take the encapsulation into > account. But even then it is somewhat unintuitive to deduce the expected > good-put from the link rate. > > > > >> > >> Two experiments for you to try: > >> > >> a) What does betterspeedtest.sh show? (It's in the latest CeroWrt, in > /usr/lib/CeroWrtScripts, or get it from github: > https://github.com/richb-hanover/CeroWrtScripts ) > >> > >> b) What does www.speedtest.net show? > >> > >> I will add your question (about the inaccuracy) to the note that I want > to send out to speedof.me this weekend. I will also ask that they include > min/max latency measurements to their test, and an option to send for > 10 > seconds to minimize any effect of PowerBoost… > > I think they do already, at least for the download bandwidth; they > start with 128Kb and keep doubling the file size until a file takes longer > than 8 seconds to transfer, they only claim to report the numbers from that > last transferred file, so worst case with a stable link and a bandwidth > > 16kbps ;), it has taken at least 12 seconds (4 plus 8) of measuring before > the end of the plot, so the bandwidth of at least the last half of the > download plot should be representative even assuming power boost. Caveat, I > assume that power boost will not be reset by the transient lack of data > transfer between the differently sized files (but since it should involve > the same IPs and port# why should power boost reset itself?). > > Best Regards > Sebastian > > > > >> > >> Best regards, > >> > >> Rich > >> > >> > >> > >> On Jul 25, 2014, at 5:10 AM, Neil Davies wrote: > >> > >>> Rich > >>> > >>> You may want to check how accurate they are to start. > >>> > >>> I just ran a “speed test” on my line (which I have complete control > and visibility over the various network elements) and it reports an average > “speed” (in the up direction) that is in excess of the capacity of the > line, it reports the maximum rate at nearly twice the best possible rate of > the ADSL connection. > >>> > >>> Doesn’t matter how pretty it is, if its not accurate it is of no use. > This is rather ironic as the web site claims it is the “smartest and most > accurate”! > >>> > >>> Neil > >>> > >>> > >>> > >>> PS pretty clear to me what mistake they’ve made in the measurement > process - its to do with incorrect inference and hence missing the > buffering effects. > >>> > >>> On 20 Jul 2014, at 14:19, Rich Brown wrote: > >>> > >>>> Doc Searls ( > http://blogs.law.harvard.edu/doc/2014/07/20/the-cliff-peronal-clouds-need-to-climb/) > mentioned in passing that he uses a new speed test website. I checked it > out, and it was very cool… > >>>> > >>>> www.speedof.me is an all-HTML5 website that seems to make accurate > measurements of the up and download speeds of your internet connection. > It’s also very attractive, and the real-time plots of the speed show > interesting info. (screen shot at: http://richb-hanover.com/speedof-me/) > >>>> > >>>> Now if we could get them to a) allow longer/bigger tests to > circumvent PowerBoost, and b) include a latency measurement so people could > point out their bufferbloated equipment. > >>>> > >>>> I'm going to send them a note. Anything else I should add? > >>>> > >>>> Rich > >>>> _______________________________________________ > >>>> Bloat mailing list > >>>> Bloat@lists.bufferbloat.net > >>>> https://lists.bufferbloat.net/listinfo/bloat > >>> > >> > > > > _______________________________________________ > > Bloat mailing list > > Bloat@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat >