From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id B7FF521F4EA for ; Fri, 25 Jul 2014 10:17:23 -0700 (PDT) Received: from hms-beagle.lan ([134.2.89.70]) by mail.gmx.com (mrgmx003) with ESMTPSA (Nemesis) id 0ML7NR-1XB0640F3B-000MtY; Fri, 25 Jul 2014 19:17:05 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) From: Sebastian Moeller In-Reply-To: <68D38551-A3F0-4E48-91C5-9550722A7018@pnsol.com> Date: Fri, 25 Jul 2014 19:17:02 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <1ADACA8B-D461-4CDC-A6BC-02CBF3FE4878@gmx.de> References: <03292B76-5273-4912-BB18-90E95C16A9F5@pnsol.com> <66FF8435-C8A5-4596-B43A-EC12D537D49E@gmx.de> <41DF4003-BAE8-4794-BEDF-EF2385F03685@gmx.de> <68D38551-A3F0-4E48-91C5-9550722A7018@pnsol.com> To: Neil Davies X-Mailer: Apple Mail (2.1878.6) X-Provags-ID: V03:K0:Awh198PkJQ9XRmRo/pIrFagj/cqMg3hpPzXzb5YhMUqVclB8WA4 Lyari1Fh30H5Lmko6ONSptCU8vr4v1L4y3/X6NQWqgE5GRhThLVT2XTh1MZrgbEjowjOlce e4PxOoy7sEA8zhIUGkKyLF8nxCdawFbhBAG1Ac+uk/AyswGIOVz44mtzsg+sEk0ZFN2IWzU wSkR8Pu4T759CvAZUuyTw== Cc: cerowrt-devel , Martin Geddes , bloat Subject: Re: [Cerowrt-devel] [Bloat] Check out www.speedof.me - no Flash X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 17:17:24 -0000 Hi Neil, On Jul 25, 2014, at 19:14 , Neil Davies wrote: > Try this thesis - Lucian used this for work at CERN, the description = is in there.... (see one of the appendicies) Analysis and predictive = modeling of the performance of the ATLAS TDAQ network Sweet, thanks a lot. best regards sebastian >=20 >=20 > On 25 Jul 2014, at 18:12, Sebastian Moeller wrote: >=20 >> Hello Martin, >>=20 >> thanks a lot. >>=20 >> On Jul 25, 2014, at 18:32 , Martin Geddes = wrote: >>=20 >>> So what is =CE=94Q and how do you "compute" it (to the extent it is = a "computed" thing)? >>>=20 >>> Starting point: the only observable effect of a network is to lose = and delay data -- i.e. to "attenuate quality" by adding the toxic = effects of time to distributed computations. =CE=94Q is a morphism that = relates the "quality attenuation" that the network imposes to the = application performance, and describes the trading spaces at all = intermediate layers of abstraction. It is shown in the attached graphic. >>>=20 >>> Critically, it frames quality as something that can only be lost = ("attenuated"), both by the network and the application. Additionally, = it is stochastic, and works with random variables and distributions. >>>=20 >>> At its most concrete level, it is the individual impairment = encountered by every packet when the network in operation. But we don't = want to have to track every packet - 1:1 scale maps are pretty useless. = So we need to abstract that in order to create a model that has value. >>>=20 >>> Next abstraction: an improper random variable. This unifies loss and = delay into a single stochastic object. >>> Next abstraction: received transport, which is a CDF where we are = interested in the properties of the "tail". >>>=20 >>> Next abstraction, that joins network performance and application QoE = (as relates to performance): relate the CDF to the application through a = Quality Transport Agreement. This "stochastic contract" is both = necessary and sufficient to deliver the application outcome. >>>=20 >>> Next concretisation towards QoE: offered load of demand, as a CDF. >>> Next concretisation towards QoE: breach hazard metric, which = abstracts the application performance. Indicates the likelihood of the = QTA contract being broken, and how badly. >>> Final concretisation: the individual application performance = encountered by every user. Again, a 1:1 map that isn't very helpful. >>>=20 >>> So as you can see, it's about as far away from a single point = average metric as you can possibly get. A far richer model is required = in order to achieve robust performance engineering. >>>=20 >>> It is "computed" using multi-point measurements to capture the = distribution. The G/S/V charts you see are based on processing that data = to account for various issues, including clock skew. >>>=20 >>> I hope that helps. We need to document more of this in public, which = is an ongoing process.=20 >>=20 >> You lost me, I think what I should have asked for is a real = example with numbers and the formulas ;) I guess that is deep in = =E2=80=9Csecret sauce=E2=80=9D territory. Alas, if that should be true = it also means that deltaQ is not going to help me understand my network = any better =E2=80=A6 >>=20 >> Best Regards >> Sebastian >>=20 >>=20 >>>=20 >>> Martin >>>=20 >>> On 25 July 2014 16:58, Sebastian Moeller wrote: >>> Hi Martin, >>>=20 >>> thanks for the pointers, >>>=20 >>>=20 >>> On Jul 25, 2014, at 16:25 , Martin Geddes = wrote: >>>=20 >>> > You may find the following useful background reading on the state = of the art in network measurement, and a primer on =CE=94Q (which is the = property we wish to measure). >>> > >>> > First, start with this presentation: Network performance = optimisation using high-fidelity measures >>> > Then read this one to decompose =CE=94Q into G, S and V: = Fundamentals of network performance engineering >>> > Then read this one to get a bit more sense on what =CE=94Q is = about: Introduction to =CE=94Q and Network Performance Science = (extracts) >>> > >>> > Then read these essays: >>> > >>> > Foundation of Network Science >>> > How to do network performance chemistry >>> > How to X-ray a telecoms network >>> > There is no quality in averages: IPX case study >>>=20 >>> All of this makes intuitively sense, but it is a bit light = on how deltaQ is to be computed ;). >>> As far as I understand it also has not much bearing on my = home network; the only one under my control. Now, following the buffer = bloat discussion for some years, I have internalized the idea that = bandwidth alone does not suffice to describe the quality of my network = connection. I think that the latency increase under load (for unrelated = flows) is the best of all the bad single number measures of network = dynamics/quality. I should be related to what I understood deltaQ to = depend on (as packet loss for non real time flows will cause an increase = in latency). I think that continuous measurements make a to n of sense = for ISPs, backbone-operators, mobile carriers =E2=80=A6 but at home, = basically, I operate as my own network quality monitor ;) (that is I try = to pin point and debug (transient) anomalies). >>>=20 >>> > >>> > Martin >>> > >>> > For fresh thinking about telecoms sign up for my free newsletter = or visit the Geddes Think Tank. >>> > LinkedIn Twitter Mobile: +44 7957 499219 Skype: mgeddes >>> > Martin Geddes Consulting Ltd, Incorporated in Scotland, number = SC275827 VAT Number: 859 5634 72 Registered office: 17-19 East London = Street, Edinburgh, EH7 4BN >>> > >>> > >>> > >>> > On 25 July 2014 15:17, Sebastian Moeller wrote: >>> > Hi Neil, >>> > >>> > >>> > On Jul 25, 2014, at 14:24 , Neil Davies = wrote: >>> > >>> > > Rich >>> > > >>> > > I have a deep worry over this style of single point measurement = - and hence speed - as an appropriate measure. >>> > >>> > But how do you propose to measure the (bottleneck) link = capacity then? It turns out for current CPE and CMTS/DSLAM equipment one = typically can not relay on good QoE out of the box, since typically = these devices do not use their (largish) buffers wisely. Instead the = current remedy is to take back control over the bottleneck link by = shaping the actually sent traffic to stay below the hardware link = capacity thereby avoiding feeling the consequences of the = over-buffering. But to do this is is quite helpful to get an educated = guess what the bottleneck links capacity actually is. And for that = purpose a speediest seems useful. >>> > >>> > >>> > > We know, and have evidence, that throughput/utilisation is not a = good proxy for the network delivering suitable quality of experience. We = work with organisation (Telco=E2=80=99s, large system integrators etc) = where we spend a lot of time having to =E2=80=9Cundo=E2=80=9D the = consequences of =E2=80=9Cmaximising speed=E2=80=9D. Just like there is = more to life than work, there is more to QoE than speed. >>> > > >>> > > For more specific comments see inline >>> > > >>> > > On 25 Jul 2014, at 13:09, Rich Brown = wrote: >>> > > >>> > >> Neil, >>> > >> >>> > >> Thanks for the note and the observations. My thoughts: >>> > >> >>> > >> 1) I note that speedof.me does seem to overstate the speed = results. At my home, it reports 5.98mbps down, and 638kbps up, while = betterspeedtest.sh shows 5.49/0.61 mbps. (speedtest.net gives numbers = similar to the betterspeedtest.net script.) >>> > >> >>> > >> 2) I think we're in agreement about the peak upload rate that = you point out is too high. Their measurement code runs in the browser. = It seems likely that the browser pumps out a few big packets before = getting flow control information, thus giving the impression that they = can send at a higher rate. This comports with the obvious decay that = ramps toward the long-term rate. >>> > > >>> > > I think that its simpler than that, it is measuring the rate at = which it can push packets out the interface - its real time rate is = precisely that - it can not be the rate being reported by the far end, = it can never exceed the limiting link. The long term average (if it is = like other speed testers we=E2=80=99ve had to look into) is being = measured at the TCP/IP SDU level by measuring the difference in time = between the first and last timestamps of data stream and dividing that = into the total data sent. Their =E2=80=9Cover-estimate=E2=80=9D is = because there are packets buffered in the CPE that have left the machine = but not arrived at the far end. >>> > >>> > Testing from an openwrt router located at a = high-symmetric-bandwidth location shows that speedof.me does not scale = higher than ~ 130 Mbps server to client and ~15Mbps client to server (on = the same connection I can get 130Mbps S2C and ~80Mbps C2S, so the = asymmetry in the speedof.me results is not caused by my local = environment). >>> > @Rich and Dave, this probably means that for the upper end = of fiber and cable and VDSL connections speed of.me is not going to be a = reliable speed measure=E2=80=A6 Side note www.sppedtest.net shows = ~100Mbps S2C and ~100Mbps C2S, so might be better suited to high-upload = links... >>> > >>> > > >>> > >> >>> > >> 3) But that long-term speed should be at or below the = theoretical long-term rate, not above it. >>> > > >>> > > Agreed, but in this case knowing the sync rate already defines = that maximum. >>> > >>> > I fully agree, but for ADSL the sync rate also contains a = lot of encapsulation, so the maximum achievable TCP rate is at best ~90% = of link rate. Note for cerowrt=E2=80=99s SQM system the link rate is = exactly the right number to start out with at that system can take the = encapsulation into account. But even then it is somewhat unintuitive to = deduce the expected good-put from the link rate. >>> > >>> > > >>> > >> >>> > >> Two experiments for you to try: >>> > >> >>> > >> a) What does betterspeedtest.sh show? (It's in the latest = CeroWrt, in /usr/lib/CeroWrtScripts, or get it from github: = https://github.com/richb-hanover/CeroWrtScripts ) >>> > >> >>> > >> b) What does www.speedtest.net show? >>> > >> >>> > >> I will add your question (about the inaccuracy) to the note = that I want to send out to speedof.me this weekend. I will also ask that = they include min/max latency measurements to their test, and an option = to send for > 10 seconds to minimize any effect of PowerBoost=E2=80=A6 >>> > >>> > I think they do already, at least for the download = bandwidth; they start with 128Kb and keep doubling the file size until a = file takes longer than 8 seconds to transfer, they only claim to report = the numbers from that last transferred file, so worst case with a stable = link and a bandwidth > 16kbps ;), it has taken at least 12 seconds (4 = plus 8) of measuring before the end of the plot, so the bandwidth of at = least the last half of the download plot should be representative even = assuming power boost. Caveat, I assume that power boost will not be = reset by the transient lack of data transfer between the differently = sized files (but since it should involve the same IPs and port# why = should power boost reset itself?). >>> > >>> > Best Regards >>> > Sebastian >>> > >>> > >>> > >>> > >> >>> > >> Best regards, >>> > >> >>> > >> Rich >>> > >> >>> > >> >>> > >> >>> > >> On Jul 25, 2014, at 5:10 AM, Neil Davies = wrote: >>> > >> >>> > >>> Rich >>> > >>> >>> > >>> You may want to check how accurate they are to start. >>> > >>> >>> > >>> I just ran a =E2=80=9Cspeed test=E2=80=9D on my line (which I = have complete control and visibility over the various network elements) = and it reports an average =E2=80=9Cspeed=E2=80=9D (in the up direction) = that is in excess of the capacity of the line, it reports the maximum = rate at nearly twice the best possible rate of the ADSL connection. >>> > >>> >>> > >>> Doesn=E2=80=99t matter how pretty it is, if its not accurate = it is of no use. This is rather ironic as the web site claims it is the = =E2=80=9Csmartest and most accurate=E2=80=9D! >>> > >>> >>> > >>> Neil >>> > >>> >>> > >>> >>> > >>> >>> > >>> PS pretty clear to me what mistake they=E2=80=99ve made in the = measurement process - its to do with incorrect inference and hence = missing the buffering effects. >>> > >>> >>> > >>> On 20 Jul 2014, at 14:19, Rich Brown = wrote: >>> > >>> >>> > >>>> Doc Searls = (http://blogs.law.harvard.edu/doc/2014/07/20/the-cliff-peronal-clouds-need= -to-climb/) mentioned in passing that he uses a new speed test website. = I checked it out, and it was very cool=E2=80=A6 >>> > >>>> >>> > >>>> www.speedof.me is an all-HTML5 website that seems to make = accurate measurements of the up and download speeds of your internet = connection. It=E2=80=99s also very attractive, and the real-time plots = of the speed show interesting info. (screen shot at: = http://richb-hanover.com/speedof-me/) >>> > >>>> >>> > >>>> Now if we could get them to a) allow longer/bigger tests to = circumvent PowerBoost, and b) include a latency measurement so people = could point out their bufferbloated equipment. >>> > >>>> >>> > >>>> I'm going to send them a note. Anything else I should add? >>> > >>>> >>> > >>>> Rich >>> > >>>> _______________________________________________ >>> > >>>> Bloat mailing list >>> > >>>> Bloat@lists.bufferbloat.net >>> > >>>> https://lists.bufferbloat.net/listinfo/bloat >>> > >>> >>> > >> >>> > > >>> > > _______________________________________________ >>> > > Bloat mailing list >>> > > Bloat@lists.bufferbloat.net >>> > > https://lists.bufferbloat.net/listinfo/bloat >>> > >>> > _______________________________________________ >>> > Bloat mailing list >>> > Bloat@lists.bufferbloat.net >>> > https://lists.bufferbloat.net/listinfo/bloat >>> > >>>=20 >>>=20 >>> <=CE=94Q morphism.png> >>=20 >=20