From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp121.iad3a.emailsrvr.com (smtp121.iad3a.emailsrvr.com [173.203.187.121]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id C0E2621F415 for ; Fri, 25 Jul 2014 15:17:06 -0700 (PDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp24.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 6F8221801F0; Fri, 25 Jul 2014 18:17:05 -0400 (EDT) X-Virus-Scanned: OK Received: from app27.wa-webapps.iad3a (relay.iad3a.rsapps.net [172.27.255.110]) by smtp24.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 46B1918021A; Fri, 25 Jul 2014 18:17:05 -0400 (EDT) Received: from reed.com (localhost.localdomain [127.0.0.1]) by app27.wa-webapps.iad3a (Postfix) with ESMTP id 37608382994; Fri, 25 Jul 2014 18:17:05 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com) with HTTP; Fri, 25 Jul 2014 18:17:05 -0400 (EDT) Date: Fri, 25 Jul 2014 18:17:05 -0400 (EDT) From: dpreed@reed.com To: "David Lang" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20140725181705000000_92047" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: <03292B76-5273-4912-BB18-90E95C16A9F5@pnsol.com> <66FF8435-C8A5-4596-B43A-EC12D537D49E@gmx.de> <41DF4003-BAE8-4794-BEDF-EF2385F03685@gmx.de> Message-ID: <1406326625.225312181@apps.rackspace.com> X-Mailer: webmail7.0 Cc: Neil Davies , cerowrt-devel , Martin Geddes , bloat Subject: Re: [Cerowrt-devel] [Bloat] Check out www.speedof.me - no Flash X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 22:17:07 -0000 ------=_20140725181705000000_92047 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0AI think what is being discussed is "how to measure the quality of one en= dpoint's experience of the entire Internet over all time or over a specific= interval of time".=0A =0AYet the systems that are built on top of the Inte= rnet transport do not have any kind of uniform dependence on the underlying= transport behavior in terms of their quality. Even something like VoIP's = quality as experienced by two humans talking over it has a dependency on th= e Internet's behavior, but one that is hardly simple.=0A =0AAs an extreme, = if one endpoint experiences a direct DDoS attack or is indirectly affected = by one somewhere in the path, the quality of the experience might be dramat= ically reduced.=0A =0ASo any attempt to define a delta-Q that has meaning = in terms of user experience appears pointless and even silly - the endpoint= experience is adequate under a very wide variety of conditions, but degrad= es terribly under certain kinds of conditions.=0A =0AAs a different point, = let's assume that the last-mile is 80% utilized, but the latency variation = in that utilization is not larger than 50 msec. This is a feasible-to-imag= ine operating point, but it requires a certain degreee of tight control tha= t may be very hard to achieve over thousands of independent application ser= vices through that point, so its feasibility is contingent on lots of facto= rs. Then if the 20% capacity is far larger than 64 kb/sec we know that toll= -quality audio can be produced with a small endpoint "jitter buffer". Ther= e's no "delta-Q" there at all - quality is great.=0A =0ASo the point is: a = single number or even a single "morphism" (whatever that is) to a specific = algebraic domain element (a mapping to a semi-lattice with non-Abelian oper= ators?) does not allow one to define a "measure" of an endpoint of the Inte= rnet that can be used to compute "quality" of all applications.=0A =0AOr in= purely non-abstract terms: if there were a delta-Q it would be useless for= most network applications, but might be useful for a single network applic= ation.=0A =0ASo I submit that delta-Q is a *metaphor* and not a very useful= one at that. It's probably as useful as providing a "funkiness" measure f= or an Internat access point. We can certainly talk about and make claims a= bout the relative "funkiness" of different connections and different provid= ers. We might even claim that cable providers make funkier network provide= rs than cellular providers.=0A =0ABut to what end?=0A=0A=0AOn Friday, July = 25, 2014 5:13pm, "David Lang" said:=0A=0A=0A=0A> On Fri, 25= Jul 2014, Martin Geddes wrote:=0A> =0A> > So what is =CE=94Q and how do yo= u "compute" it (to the extent it is a=0A> "computed"=0A> > thing)?=0A> =0A>= don't try to reduce it to a single number, we have two numbers that seem t= o=0A> matter=0A> =0A> 1. throughput (each direction)=0A> =0A> 2. latency un= der load=0A> =0A> Currently the speed test sites report throughput in each = direction and ping time=0A> while not under load=0A> =0A> If they could jus= t add a ping time under load measurement, then we could talk=0A> meaningful= ly about either the delta or ratio of the ping times as the=0A> "bufferbloa= t factor"=0A> =0A> no, it wouldn't account for absolutly every nuance, but = it would come pretty=0A> close.=0A> =0A> If a connection has good throughpu= t and a low bufferbloat factor, it should be=0A> good for any type of use.= =0A> =0A> If it has good throughput, but a horrid bufferbloat factor, then = you need to=0A> artifically limit your traffic to stay clear of saturating = the bandwith=0A> (sacraficing throughput)=0A> =0A> David Lang=0A> =0A> > St= arting point: the only observable effect of a network is to lose and=0A> > = delay data -- i.e. to "attenuate quality" by adding the toxic effects of=0A= > > time to distributed computations. =CE=94Q is a *morphism* that relates = the=0A> > "quality attenuation" that the network imposes to the application= =0A> > performance, and describes the trading spaces at all intermediate la= yers of=0A> > abstraction. It is shown in the attached graphic.=0A> >=0A> >= Critically, it frames quality as something that can only be lost=0A> > ("a= ttenuated"), both by the network and the application. Additionally, it=0A> = > is stochastic, and works with random variables and distributions.=0A> >= =0A> > At its most concrete level, it is the individual impairment encounte= red by=0A> > every packet when the network in operation. But we don't want = to have to=0A> > track every packet - 1:1 scale maps are pretty useless. So= we need to=0A> > abstract that in order to create a model that has value.= =0A> >=0A> > Next abstraction: an improper random variable. This unifies lo= ss and delay=0A> > into a single stochastic object.=0A> > Next abstraction:= received transport, which is a CDF where we are=0A> > interested in the pr= operties of the "tail".=0A> >=0A> > Next abstraction, that joins network pe= rformance and application QoE (as=0A> > relates to performance): relate the= CDF to the application through a=0A> > Quality Transport Agreement. This "= stochastic contract" is both necessary=0A> > and sufficient to deliver the = application outcome.=0A> >=0A> > Next concretisation towards QoE: offered l= oad of demand, as a CDF.=0A> > Next concretisation towards QoE: breach haza= rd metric, which abstracts the=0A> > application performance. Indicates the= likelihood of the QTA contract being=0A> > broken, and how badly.=0A> > Fi= nal concretisation: the individual application performance encountered by= =0A> > every user. Again, a 1:1 map that isn't very helpful.=0A> >=0A> > So= as you can see, it's about as far away from a single point average=0A> > m= etric as you can possibly get. A far richer model is required in order to= =0A> > achieve robust performance engineering.=0A> >=0A> > It is "computed"= using multi-point measurements to capture the=0A> > distribution. The G/S/= V charts you see are based on processing that data to=0A> > account for var= ious issues, including clock skew.=0A> >=0A> > I hope that helps. We need t= o document more of this in public, which is an=0A> > ongoing process.=0A> >= =0A> > Martin=0A> >=0A> > On 25 July 2014 16:58, Sebastian Moeller wrote:=0A> >=0A> >> Hi Martin,=0A> >>=0A> >> thanks for the point= ers,=0A> >>=0A> >>=0A> >> On Jul 25, 2014, at 16:25 , Martin Geddes =0A> wrote:=0A> >>=0A> >>> You may find the following usefu= l background reading on the state of=0A> the=0A> >> art in network measurem= ent, and a primer on =CE=94Q (which is the=0A> property we=0A> >> wish to m= easure).=0A> >>>=0A> >>> First, start with this presentation: Network perfo= rmance=0A> optimisation=0A> >> using high-fidelity measures=0A> >>> Then re= ad this one to decompose =CE=94Q into G, S and V:=0A> Fundamentals of=0A> >= > network performance engineering=0A> >>> Then read this one to get a bit m= ore sense on what =CE=94Q is=0A> about:=0A> >> Introduction to =CE=94Q and = Network Performance Science (extracts)=0A> >>>=0A> >>> Then read these essa= ys:=0A> >>>=0A> >>> Foundation of Network Science=0A> >>> How to do network= performance chemistry=0A> >>> How to X-ray a telecoms network=0A> >>> Ther= e is no quality in averages: IPX case study=0A> >>=0A> >> All of this makes= intuitively sense, but it is a bit light on=0A> how=0A> >> deltaQ is to be= computed ;).=0A> >> As far as I understand it also has not much bearing on= my home=0A> >> network; the only one under my control. Now, following the = buffer bloat=0A> >> discussion for some years, I have internalized the idea= that bandwidth=0A> >> alone does not suffice to describe the quality of my= network connection.=0A> I=0A> >> think that the latency increase under loa= d (for unrelated flows) is the=0A> >> best of all the bad single number mea= sures of network dynamics/quality.=0A> I=0A> >> should be related to what I= understood deltaQ to depend on (as packet=0A> loss=0A> >> for non real tim= e flows will cause an increase in latency). I think=0A> that=0A> >> continu= ous measurements make a to n of sense for ISPs,=0A> backbone-operators,=0A>= >> mobile carriers =E2=80=A6 but at home, basically, I operate as my own= =0A> network=0A> >> quality monitor ;) (that is I try to pin point and debu= g (transient)=0A> >> anomalies).=0A> >>=0A> >>>=0A> >>> Martin=0A> >>>=0A> = >>> For fresh thinking about telecoms sign up for my free newsletter or=0A>= >> visit the Geddes Think Tank.=0A> >>> LinkedIn Twitter Mobile: +44 7957 = 499219 Skype: mgeddes=0A> >>> Martin Geddes Consulting Ltd, Incorporated in= Scotland, number=0A> SC275827=0A> >> VAT Number: 859 5634 72 Registered of= fice: 17-19 East London Street,=0A> >> Edinburgh, EH7 4BN=0A> >>>=0A> >>>= =0A> >>>=0A> >>> On 25 July 2014 15:17, Sebastian Moeller = =0A> wrote:=0A> >>> Hi Neil,=0A> >>>=0A> >>>=0A> >>> On Jul 25, 2014, at 14= :24 , Neil Davies =0A> wrote:=0A> >>>=0A> >>>> Rich= =0A> >>>>=0A> >>>> I have a deep worry over this style of single point meas= urement -=0A> and=0A> >> hence speed - as an appropriate measure.=0A> >>>= =0A> >>> But how do you propose to measure the (bottleneck) link=0A> capaci= ty=0A> >> then? It turns out for current CPE and CMTS/DSLAM equipment one= =0A> typically=0A> >> can not relay on good QoE out of the box, since typic= ally these devices=0A> do=0A> >> not use their (largish) buffers wisely. In= stead the current remedy is to=0A> >> take back control over the bottleneck= link by shaping the actually sent=0A> >> traffic to stay below the hardwar= e link capacity thereby avoiding=0A> feeling=0A> >> the consequences of the= over-buffering. But to do this is is quite=0A> helpful=0A> >> to get an ed= ucated guess what the bottleneck links capacity actually is.=0A> >> And for= that purpose a speediest seems useful.=0A> >>>=0A> >>>=0A> >>>> We know, a= nd have evidence, that throughput/utilisation is not a=0A> good=0A> >> prox= y for the network delivering suitable quality of experience. We work=0A> >>= with organisation (Telco=E2=80=99s, large system integrators etc) where we= =0A> spend a=0A> >> lot of time having to =E2=80=9Cundo=E2=80=9D the conseq= uences of=0A> =E2=80=9Cmaximising speed=E2=80=9D. Just=0A> >> like there is= more to life than work, there is more to QoE than speed.=0A> >>>>=0A> >>>>= For more specific comments see inline=0A> >>>>=0A> >>>> On 25 Jul 2014, at= 13:09, Rich Brown=0A> wrote:=0A> >>>>=0A> >>>>> = Neil,=0A> >>>>>=0A> >>>>> Thanks for the note and the observations. My thou= ghts:=0A> >>>>>=0A> >>>>> 1) I note that speedof.me does seem to overstate = the speed=0A> results.=0A> >> At my home, it reports 5.98mbps down, and 638= kbps up, while=0A> >> betterspeedtest.sh shows 5.49/0.61 mbps. (speedtest.n= et gives numbers=0A> >> similar to the betterspeedtest.net script.)=0A> >>>= >>=0A> >>>>> 2) I think we're in agreement about the peak upload rate that= =0A> you=0A> >> point out is too high. Their measurement code runs in the b= rowser. It=0A> seems=0A> >> likely that the browser pumps out a few big pac= kets before getting flow=0A> >> control information, thus giving the impres= sion that they can send at a=0A> >> higher rate. This comports with the obv= ious decay that ramps toward the=0A> >> long-term rate.=0A> >>>>=0A> >>>> I= think that its simpler than that, it is measuring the rate at=0A> which=0A= > >> it can push packets out the interface - its real time rate is precisel= y=0A> >> that - it can not be the rate being reported by the far end, it ca= n=0A> never=0A> >> exceed the limiting link. The long term average (if it i= s like other=0A> speed=0A> >> testers we=E2=80=99ve had to look into) is be= ing measured at the TCP/IP SDU=0A> level=0A> >> by measuring the difference= in time between the first and last=0A> timestamps=0A> >> of data stream an= d dividing that into the total data sent. Their=0A> >> =E2=80=9Cover-estima= te=E2=80=9D is because there are packets buffered in the=0A> CPE that have= =0A> >> left the machine but not arrived at the far end.=0A> >>>=0A> >>> Te= sting from an openwrt router located at a=0A> >> high-symmetric-bandwidth l= ocation shows that speedof.me does not scale=0A> >> higher than ~ 130 Mbps = server to client and ~15Mbps client to server (on=0A> >> the same connectio= n I can get 130Mbps S2C and ~80Mbps C2S, so the=0A> asymmetry=0A> >> in the= speedof.me results is not caused by my local environment).=0A> >>> @Rich a= nd Dave, this probably means that for the upper end=0A> of=0A> >> fiber and= cable and VDSL connections speed of.me is not going to be a=0A> >> reliabl= e speed measure=E2=80=A6 Side note www.sppedtest.net shows ~100Mbps=0A> S2C= =0A> >> and ~100Mbps C2S, so might be better suited to high-upload links...= =0A> >>>=0A> >>>>=0A> >>>>>=0A> >>>>> 3) But that long-term speed should be= at or below the=0A> theoretical=0A> >> long-term rate, not above it.=0A> >= >>>=0A> >>>> Agreed, but in this case knowing the sync rate already defines= =0A> that=0A> >> maximum.=0A> >>>=0A> >>> I fully agree, but for ADSL the s= ync rate also contains a lot=0A> of=0A> >> encapsulation, so the maximum ac= hievable TCP rate is at best ~90% of=0A> link=0A> >> rate. Note for cerowrt= =E2=80=99s SQM system the link rate is exactly the=0A> right=0A> >> number = to start out with at that system can take the encapsulation into=0A> >> acc= ount. But even then it is somewhat unintuitive to deduce the expected=0A> >= > good-put from the link rate.=0A> >>>=0A> >>>>=0A> >>>>>=0A> >>>>> Two exp= eriments for you to try:=0A> >>>>>=0A> >>>>> a) What does betterspeedtest.s= h show? (It's in the latest=0A> CeroWrt, in=0A> >> /usr/lib/CeroWrtScripts,= or get it from github:=0A> >> https://github.com/richb-hanover/CeroWrtScri= pts )=0A> >>>>>=0A> >>>>> b) What does www.speedtest.net show?=0A> >>>>>=0A= > >>>>> I will add your question (about the inaccuracy) to the note=0A> tha= t I=0A> >> want to send out to speedof.me this weekend. I will also ask tha= t they=0A> >> include min/max latency measurements to their test, and an op= tion to=0A> send=0A> >> for > 10 seconds to minimize any effect of PowerBoo= st=E2=80=A6=0A> >>>=0A> >>> I think they do already, at least for the downl= oad=0A> bandwidth;=0A> >> they start with 128Kb and keep doubling the file = size until a file takes=0A> >> longer than 8 seconds to transfer, they only= claim to report the numbers=0A> >> from that last transferred file, so wor= st case with a stable link and a=0A> >> bandwidth > 16kbps ;), it has taken= at least 12 seconds (4 plus 8) of=0A> >> measuring before the end of the p= lot, so the bandwidth of at least the=0A> last=0A> >> half of the download = plot should be representative even assuming power=0A> >> boost. Caveat, I a= ssume that power boost will not be reset by the=0A> transient=0A> >> lack o= f data transfer between the differently sized files (but since it=0A> >> sh= ould involve the same IPs and port# why should power boost reset=0A> itself= ?).=0A> >>>=0A> >>> Best Regards=0A> >>> Sebastian=0A> >>>=0A> >>>=0A> >>>= =0A> >>>>>=0A> >>>>> Best regards,=0A> >>>>>=0A> >>>>> Rich=0A> >>>>>=0A> >= >>>>=0A> >>>>>=0A> >>>>> On Jul 25, 2014, at 5:10 AM, Neil Davies=0A> =0A> >> wrote:=0A> >>>>>=0A> >>>>>> Rich=0A> >>>>>>=0A> >= >>>>> You may want to check how accurate they are to start.=0A> >>>>>>=0A> = >>>>>> I just ran a =E2=80=9Cspeed test=E2=80=9D on my line (which I=0A> ha= ve complete control=0A> >> and visibility over the various network elements= ) and it reports an=0A> average=0A> >> =E2=80=9Cspeed=E2=80=9D (in the up d= irection) that is in excess of the=0A> capacity of the=0A> >> line, it repo= rts the maximum rate at nearly twice the best possible rate=0A> of=0A> >> t= he ADSL connection.=0A> >>>>>>=0A> >>>>>> Doesn=E2=80=99t matter how pretty= it is, if its not=0A> accurate it is of no=0A> >> use. This is rather iron= ic as the web site claims it is the=0A> =E2=80=9Csmartest and=0A> >> most a= ccurate=E2=80=9D!=0A> >>>>>>=0A> >>>>>> Neil=0A> >>>>>>=0A> >>>>>> =0A> >>>>>>=0A> >>>>>> PS pretty clear to me what mistake = they=E2=80=99ve made in=0A> the measurement=0A> >> process - its to do with= incorrect inference and hence missing the=0A> >> buffering effects.=0A> >>= >>>>=0A> >>>>>> On 20 Jul 2014, at 14:19, Rich Brown=0A> =0A> >> wrote:=0A> >>>>>>=0A> >>>>>>> Doc Searls (=0A> >>=0A> http:/= /blogs.law.harvard.edu/doc/2014/07/20/the-cliff-peronal-clouds-need-to-clim= b/)=0A> >> mentioned in passing that he uses a new speed test website. I ch= ecked it=0A> >> out, and it was very cool=E2=80=A6=0A> >>>>>>>=0A> >>>>>>> = www.speedof.me is an all-HTML5 website that seems to=0A> make accurate=0A> = >> measurements of the up and download speeds of your internet connection.= =0A> >> It=E2=80=99s also very attractive, and the real-time plots of the s= peed=0A> show=0A> >> interesting info. (screen shot at: http://richb-hanove= r.com/speedof-me/)=0A> >>>>>>>=0A> >>>>>>> Now if we could get them to a) a= llow longer/bigger=0A> tests to=0A> >> circumvent PowerBoost, and b) includ= e a latency measurement so people=0A> could=0A> >> point out their bufferbl= oated equipment.=0A> >>>>>>>=0A> >>>>>>> I'm going to send them a note. Any= thing else I should=0A> add?=0A> >>>>>>>=0A> >>>>>>> Rich=0A> >>>>>>> _____= __________________________________________=0A> >>>>>>> Bloat mailing list= =0A> >>>>>>> Bloat@lists.bufferbloat.net=0A> >>>>>>> https://lists.bufferbl= oat.net/listinfo/bloat=0A> >>>>>>=0A> >>>>>=0A> >>>>=0A> >>>> _____________= __________________________________=0A> >>>> Bloat mailing list=0A> >>>> Blo= at@lists.bufferbloat.net=0A> >>>> https://lists.bufferbloat.net/listinfo/bl= oat=0A> >>>=0A> >>> _______________________________________________=0A> >>>= Bloat mailing list=0A> >>> Bloat@lists.bufferbloat.net=0A> >>> https://lis= ts.bufferbloat.net/listinfo/bloat=0A> >>>=0A> >>=0A> >>=0A> >______________= _________________________________=0A> Cerowrt-devel mailing list=0A> Cerowr= t-devel@lists.bufferbloat.net=0A> https://lists.bufferbloat.net/listinfo/ce= rowrt-devel=0A> _______________________________________________=0A> Cerowrt= -devel mailing list=0A> Cerowrt-devel@lists.bufferbloat.net=0A> https://lis= ts.bufferbloat.net/listinfo/cerowrt-devel=0A> ------=_20140725181705000000_92047 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

I think what is being disc= ussed is "how to measure the quality of one endpoint's experience of the en= tire Internet over all time or over a specific interval of time".

=0A

 

=0A

Yet the systems that are built = on top of the Internet transport do not have any kind of uniform dependence= on the underlying transport behavior in terms of their quality.  Even= something like VoIP's quality as experienced by two humans talking over it= has a dependency on the Internet's behavior, but one that is hardly simple= .

=0A

 

=0A

As an extreme, if on= e endpoint experiences a direct DDoS attack or is indirectly affected by on= e somewhere in the path, the quality of the experience might be dramaticall= y reduced.

=0A

 

=0A

So any atte= mpt to define a delta-Q that has  meaning in terms of user experience = appears pointless and even silly - the endpoint experience is adequate unde= r a very wide variety of conditions, but degrades terribly under certain ki= nds of conditions.

=0A

 

=0A

As = a different point, let's assume that the last-mile is 80% utilized, but the= latency variation in that utilization is not larger than 50 msec.  Th= is is a feasible-to-imagine operating point, but it requires a certain degr= eee of tight control that may be very hard to achieve over thousands of ind= ependent application services through that point, so its feasibility is con= tingent on lots of factors. Then if the 20% capacity is far larger than 64 = kb/sec we know that toll-quality audio can be produced with a small endpoin= t "jitter buffer".  There's no "delta-Q" there at all - quality is gre= at.

=0A

 

=0A

So the point is: a= single number or even a single "morphism" (whatever that is) to a specific= algebraic domain element (a mapping to a semi-lattice with non-Abelian ope= rators?) does not allow one to define a "measure" of an endpoint of the Int= ernet that can be used to compute "quality" of all applications.

=0A

 

=0A

Or in purely non-abstract terms:= if there were a delta-Q it would be useless for most network applications,= but might be useful for a single network application.

=0A

 

=0A

So I submit that delta-Q is a *metaphor* a= nd not a very useful one at that.  It's probably as useful as providin= g a "funkiness" measure for an Internat access point.  We can certainl= y talk about and make claims about the relative "funkiness" of different co= nnections and different providers.  We might even claim that cable pro= viders make funkier network providers than cellular providers.

=0A

 

=0A

But to what end?

=0A=0A


On Friday, July 25, 2014 5:13pm, "David Lang" <david@lang.hm&g= t; said:

=0A
=0A

> On Fri, 25 Jul 2014, Martin Geddes wrote:
>
> &g= t; So what is =CE=94Q and how do you "compute" it (to the extent it is a> "computed"
> > thing)?
>
> don't try to= reduce it to a single number, we have two numbers that seem to
> m= atter
>
> 1. throughput (each direction)
>
&= gt; 2. latency under load
>
> Currently the speed test sit= es report throughput in each direction and ping time
> while not un= der load
>
> If they could just add a ping time under load= measurement, then we could talk
> meaningfully about either the de= lta or ratio of the ping times as the
> "bufferbloat factor"
&= gt;
> no, it wouldn't account for absolutly every nuance, but it w= ould come pretty
> close.
>
> If a connection has = good throughput and a low bufferbloat factor, it should be
> good f= or any type of use.
>
> If it has good throughput, but a h= orrid bufferbloat factor, then you need to
> artifically limit your= traffic to stay clear of saturating the bandwith
> (sacraficing th= roughput)
>
> David Lang
>
> > Startin= g point: the only observable effect of a network is to lose and
> &= gt; delay data -- i.e. to "attenuate quality" by adding the toxic effects o= f
> > time to distributed computations. =CE=94Q is a *morphism* = that relates the
> > "quality attenuation" that the network impo= ses to the application
> > performance, and describes the tradin= g spaces at all intermediate layers of
> > abstraction. It is sh= own in the attached graphic.
> >
> > Critically, it f= rames quality as something that can only be lost
> > ("attenuate= d"), both by the network and the application. Additionally, it
> &g= t; is stochastic, and works with random variables and distributions.
&= gt; >
> > At its most concrete level, it is the individual im= pairment encountered by
> > every packet when the network in ope= ration. But we don't want to have to
> > track every packet - 1:= 1 scale maps are pretty useless. So we need to
> > abstract that= in order to create a model that has value.
> >
> > N= ext abstraction: an improper random variable. This unifies loss and delay> > into a single stochastic object.
> > Next abstract= ion: received transport, which is a CDF where we are
> > interes= ted in the properties of the "tail".
> >
> > Next abs= traction, that joins network performance and application QoE (as
> = > relates to performance): relate the CDF to the application through a> > Quality Transport Agreement. This "stochastic contract" is bo= th necessary
> > and sufficient to deliver the application outco= me.
> >
> > Next concretisation towards QoE: offered = load of demand, as a CDF.
> > Next concretisation towards QoE: b= reach hazard metric, which abstracts the
> > application perform= ance. Indicates the likelihood of the QTA contract being
> > bro= ken, and how badly.
> > Final concretisation: the individual app= lication performance encountered by
> > every user. Again, a 1:1= map that isn't very helpful.
> >
> > So as you can s= ee, it's about as far away from a single point average
> > metri= c as you can possibly get. A far richer model is required in order to
= > > achieve robust performance engineering.
> >
> = > It is "computed" using multi-point measurements to capture the
&g= t; > distribution. The G/S/V charts you see are based on processing that= data to
> > account for various issues, including clock skew.> >
> > I hope that helps. We need to document more of= this in public, which is an
> > ongoing process.
> >=
> > Martin
> >
> > On 25 July 2014 16:58,= Sebastian Moeller <moeller0@gmx.de> wrote:
> >
> = >> Hi Martin,
> >>
> >> thanks for the po= inters,
> >>
> >>
> >> On Jul 25,= 2014, at 16:25 , Martin Geddes <mail@martingeddes.com>
> wro= te:
> >>
> >>> You may find the following us= eful background reading on the state of
> the
> >> ar= t in network measurement, and a primer on =CE=94Q (which is the
> p= roperty we
> >> wish to measure).
> >>>
> >>> First, start with this presentation: Network performance=
> optimisation
> >> using high-fidelity measures
> >>> Then read this one to decompose =CE=94Q into G, S and V= :
> Fundamentals of
> >> network performance engineer= ing
> >>> Then read this one to get a bit more sense on wh= at =CE=94Q is
> about:
> >> Introduction to =CE=94Q a= nd Network Performance Science (extracts)
> >>>
> = >>> Then read these essays:
> >>>
> >&= gt;> Foundation of Network Science
> >>> How to do netw= ork performance chemistry
> >>> How to X-ray a telecoms ne= twork
> >>> There is no quality in averages: IPX case stud= y
> >>
> >> All of this makes intuitively sense= , but it is a bit light on
> how
> >> deltaQ is to be= computed ;).
> >> As far as I understand it also has not muc= h bearing on my home
> >> network; the only one under my cont= rol. Now, following the buffer bloat
> >> discussion for some= years, I have internalized the idea that bandwidth
> >> alon= e does not suffice to describe the quality of my network connection.
&= gt; I
> >> think that the latency increase under load (for un= related flows) is the
> >> best of all the bad single number = measures of network dynamics/quality.
> I
> >> should= be related to what I understood deltaQ to depend on (as packet
> l= oss
> >> for non real time flows will cause an increase in la= tency). I think
> that
> >> continuous measurements m= ake a to n of sense for ISPs,
> backbone-operators,
> >&= gt; mobile carriers =E2=80=A6 but at home, basically, I operate as my own> network
> >> quality monitor ;) (that is I try to pi= n point and debug (transient)
> >> anomalies).
> >= >
> >>>
> >>> Martin
> >>= ;>
> >>> For fresh thinking about telecoms sign up for = my free newsletter or
> >> visit the Geddes Think Tank.
= > >>> LinkedIn Twitter Mobile: +44 7957 499219 Skype: mgeddes> >>> Martin Geddes Consulting Ltd, Incorporated in Scotlan= d, number
> SC275827
> >> VAT Number: 859 5634 72 Reg= istered office: 17-19 East London Street,
> >> Edinburgh, EH7= 4BN
> >>>
> >>>
> >>>> >>> On 25 July 2014 15:17, Sebastian Moeller <moeller0= @gmx.de>
> wrote:
> >>> Hi Neil,
> >= >>
> >>>
> >>> On Jul 25, 2014, at = 14:24 , Neil Davies <Neil.Davies@pnsol.com>
> wrote:
>= ; >>>
> >>>> Rich
> >>>>> >>>> I have a deep worry over this style of single poi= nt measurement -
> and
> >> hence speed - as an appro= priate measure.
> >>>
> >>> But how do yo= u propose to measure the (bottleneck) link
> capacity
> >= ;> then? It turns out for current CPE and CMTS/DSLAM equipment one
= > typically
> >> can not relay on good QoE out of the box,= since typically these devices
> do
> >> not use thei= r (largish) buffers wisely. Instead the current remedy is to
> >= > take back control over the bottleneck link by shaping the actually sen= t
> >> traffic to stay below the hardware link capacity there= by avoiding
> feeling
> >> the consequences of the ov= er-buffering. But to do this is is quite
> helpful
> >&g= t; to get an educated guess what the bottleneck links capacity actually is.=
> >> And for that purpose a speediest seems useful.
>= ; >>>
> >>>
> >>>> We know, a= nd have evidence, that throughput/utilisation is not a
> good
= > >> proxy for the network delivering suitable quality of experien= ce. We work
> >> with organisation (Telco=E2=80=99s, large sy= stem integrators etc) where we
> spend a
> >> lot of = time having to =E2=80=9Cundo=E2=80=9D the consequences of
> =E2=80= =9Cmaximising speed=E2=80=9D. Just
> >> like there is more to= life than work, there is more to QoE than speed.
> >>>>= ;
> >>>> For more specific comments see inline
>= ; >>>>
> >>>> On 25 Jul 2014, at 13:09, Ric= h Brown
> <richb.hanover@gmail.com> wrote:
> >>= >>
> >>>>> Neil,
> >>>>>= ;
> >>>>> Thanks for the note and the observations. = My thoughts:
> >>>>>
> >>>>> = 1) I note that speedof.me does seem to overstate the speed
> result= s.
> >> At my home, it reports 5.98mbps down, and 638kbps up,= while
> >> betterspeedtest.sh shows 5.49/0.61 mbps. (speedte= st.net gives numbers
> >> similar to the betterspeedtest.net = script.)
> >>>>>
> >>>>> 2) I= think we're in agreement about the peak upload rate that
> you
> >> point out is too high. Their measurement code runs in the b= rowser. It
> seems
> >> likely that the browser pumps= out a few big packets before getting flow
> >> control infor= mation, thus giving the impression that they can send at a
> >&g= t; higher rate. This comports with the obvious decay that ramps toward the<= br />> >> long-term rate.
> >>>>
> >= ;>>> I think that its simpler than that, it is measuring the rate = at
> which
> >> it can push packets out the interface= - its real time rate is precisely
> >> that - it can not be = the rate being reported by the far end, it can
> never
> &g= t;> exceed the limiting link. The long term average (if it is like other=
> speed
> >> testers we=E2=80=99ve had to look into)= is being measured at the TCP/IP SDU
> level
> >> by = measuring the difference in time between the first and last
> times= tamps
> >> of data stream and dividing that into the total da= ta sent. Their
> >> =E2=80=9Cover-estimate=E2=80=9D is becaus= e there are packets buffered in the
> CPE that have
> >&= gt; left the machine but not arrived at the far end.
> >>>=
> >>> Testing from an openwrt router located at a
&g= t; >> high-symmetric-bandwidth location shows that speedof.me does no= t scale
> >> higher than ~ 130 Mbps server to client and ~15M= bps client to server (on
> >> the same connection I can get 1= 30Mbps S2C and ~80Mbps C2S, so the
> asymmetry
> >> i= n the speedof.me results is not caused by my local environment).
> = >>> @Rich and Dave, this probably means that for the upper end
> of
> >> fiber and cable and VDSL connections speed of.= me is not going to be a
> >> reliable speed measure=E2=80=A6 = Side note www.sppedtest.net shows ~100Mbps
> S2C
> >>= and ~100Mbps C2S, so might be better suited to high-upload links...
&= gt; >>>
> >>>>
> >>>>><= br />> >>>>> 3) But that long-term speed should be at or = below the
> theoretical
> >> long-term rate, not abov= e it.
> >>>>
> >>>> Agreed, but in = this case knowing the sync rate already defines
> that
> &g= t;> maximum.
> >>>
> >>> I fully agree= , but for ADSL the sync rate also contains a lot
> of
> >= ;> encapsulation, so the maximum achievable TCP rate is at best ~90% of<= br />> link
> >> rate. Note for cerowrt=E2=80=99s SQM syst= em the link rate is exactly the
> right
> >> number t= o start out with at that system can take the encapsulation into
> &= gt;> account. But even then it is somewhat unintuitive to deduce the exp= ected
> >> good-put from the link rate.
> >>>= ;
> >>>>
> >>>>>
> >&= gt;>>> Two experiments for you to try:
> >>>>&= gt;
> >>>>> a) What does betterspeedtest.sh show? (I= t's in the latest
> CeroWrt, in
> >> /usr/lib/CeroWrt= Scripts, or get it from github:
> >> https://github.com/richb= -hanover/CeroWrtScripts )
> >>>>>
> >>= >>> b) What does www.speedtest.net show?
> >>>>= ;>
> >>>>> I will add your question (about the in= accuracy) to the note
> that I
> >> want to send out = to speedof.me this weekend. I will also ask that they
> >> in= clude min/max latency measurements to their test, and an option to
>= ; send
> >> for > 10 seconds to minimize any effect of Pow= erBoost=E2=80=A6
> >>>
> >>> I think they= do already, at least for the download
> bandwidth;
> >&= gt; they start with 128Kb and keep doubling the file size until a file take= s
> >> longer than 8 seconds to transfer, they only claim to = report the numbers
> >> from that last transferred file, so w= orst case with a stable link and a
> >> bandwidth > 16kbps= ;), it has taken at least 12 seconds (4 plus 8) of
> >> meas= uring before the end of the plot, so the bandwidth of at least the
>= ; last
> >> half of the download plot should be representativ= e even assuming power
> >> boost. Caveat, I assume that power= boost will not be reset by the
> transient
> >> lack= of data transfer between the differently sized files (but since it
&g= t; >> should involve the same IPs and port# why should power boost re= set
> itself?).
> >>>
> >>> Best = Regards
> >>> Sebastian
> >>>
> &= gt;>>
> >>>
> >>>>>
>= >>>>> Best regards,
> >>>>>
>= ; >>>>> Rich
> >>>>>
> >&g= t;>>>
> >>>>>
> >>>>>= ; On Jul 25, 2014, at 5:10 AM, Neil Davies
> <neil.davies@pnsol.= com>
> >> wrote:
> >>>>>
> = >>>>>> Rich
> >>>>>>
> = >>>>>> You may want to check how accurate they are to sta= rt.
> >>>>>>
> >>>>>> I= just ran a =E2=80=9Cspeed test=E2=80=9D on my line (which I
> have= complete control
> >> and visibility over the various networ= k elements) and it reports an
> average
> >> =E2=80= =9Cspeed=E2=80=9D (in the up direction) that is in excess of the
> = capacity of the
> >> line, it reports the maximum rate at nea= rly twice the best possible rate
> of
> >> the ADSL c= onnection.
> >>>>>>
> >>>>>= ;> Doesn=E2=80=99t matter how pretty it is, if its not
> accurat= e it is of no
> >> use. This is rather ironic as the web site= claims it is the
> =E2=80=9Csmartest and
> >> most a= ccurate=E2=80=9D!
> >>>>>>
> >>>= >>> Neil
> >>>>>>
> >>>= >>> <speedof_me_14-07-25.png>
> >>>>>= >
> >>>>>> PS pretty clear to me what mistake = they=E2=80=99ve made in
> the measurement
> >> proces= s - its to do with incorrect inference and hence missing the
> >= > buffering effects.
> >>>>>>
> >&g= t;>>>> On 20 Jul 2014, at 14:19, Rich Brown
> <richb= .hanover@gmail.com>
> >> wrote:
> >>>>= >>
> >>>>>>> Doc Searls (
> >= >
> http://blogs.law.harvard.edu/doc/2014/07/20/the-cliff-perona= l-clouds-need-to-climb/)
> >> mentioned in passing that he us= es a new speed test website. I checked it
> >> out, and it wa= s very cool=E2=80=A6
> >>>>>>>
> >&= gt;>>>>> www.speedof.me is an all-HTML5 website that seems t= o
> make accurate
> >> measurements of the up and dow= nload speeds of your internet connection.
> >> It=E2=80=99s a= lso very attractive, and the real-time plots of the speed
> show> >> interesting info. (screen shot at: http://richb-hanover.co= m/speedof-me/)
> >>>>>>>
> >>>= ;>>>> Now if we could get them to a) allow longer/bigger
&= gt; tests to
> >> circumvent PowerBoost, and b) include a lat= ency measurement so people
> could
> >> point out the= ir bufferbloated equipment.
> >>>>>>>
>= ; >>>>>>> I'm going to send them a note. Anything else= I should
> add?
> >>>>>>>
> &= gt;>>>>>> Rich
> >>>>>>> ___= ____________________________________________
> >>>>>= >> Bloat mailing list
> >>>>>>> Bloat@li= sts.bufferbloat.net
> >>>>>>> https://lists.bu= fferbloat.net/listinfo/bloat
> >>>>>>
> &= gt;>>>>
> >>>>
> >>>> _= ______________________________________________
> >>>> B= loat mailing list
> >>>> Bloat@lists.bufferbloat.net> >>>> https://lists.bufferbloat.net/listinfo/bloat
= > >>>
> >>> __________________________________= _____________
> >>> Bloat mailing list
> >>&= gt; Bloat@lists.bufferbloat.net
> >>> https://lists.buffer= bloat.net/listinfo/bloat
> >>>
> >>
>= ; >>
> >_______________________________________________> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloa= t.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
&= gt; _______________________________________________
> Cerowrt-devel= mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https= ://lists.bufferbloat.net/listinfo/cerowrt-devel
>

=0A
------=_20140725181705000000_92047--