From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from eu1sys200aog120.obsmtp.com (eu1sys200aog120.obsmtp.com [207.126.144.149]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 1CFB321F643; Sat, 26 Jul 2014 13:16:19 -0700 (PDT) Received: from mail.la.pnsol.com ([89.145.213.110]) (using TLSv1) by eu1sys200aob120.postini.com ([207.126.147.11]) with SMTP ID DSNKU9QMggOUvMIqWZynjy8x17dVm81I1atG@postini.com; Sat, 26 Jul 2014 20:16:20 UTC Received: from git.pnsol.com ([172.20.5.238] helo=roam.smtp.pnsol.com) by mail.la.pnsol.com with esmtp (Exim 4.76) (envelope-from ) id 1XB8Nt-0007Hh-Ua; Sat, 26 Jul 2014 21:16:01 +0100 Received: from [172.20.5.109] by roam.smtp.pnsol.com with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from ) id 1XB8Nt-0000Ix-ND; Sat, 26 Jul 2014 20:16:01 +0000 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Content-Type: multipart/alternative; boundary="Apple-Mail=_D855CB1A-D28A-4BC0-AB20-20BCB349A2C4" From: Neil Davies X-Priority: 3 (Normal) In-Reply-To: <1406326625.225312181@apps.rackspace.com> Date: Sat, 26 Jul 2014 21:16:00 +0100 Message-Id: <1A50E4FD-8C6C-42C0-9269-F2676413E523@pnsol.com> References: <03292B76-5273-4912-BB18-90E95C16A9F5@pnsol.com> <66FF8435-C8A5-4596-B43A-EC12D537D49E@gmx.de> <41DF4003-BAE8-4794-BEDF-EF2385F03685@gmx.de> <1406326625.225312181@apps.rackspace.com> To: dpreed@reed.com X-Mailer: Apple Mail (2.1878.6) Cc: cerowrt-devel , Martin Geddes , bloat Subject: Re: [Cerowrt-devel] [Bloat] Check out www.speedof.me - no Flash X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Jul 2014 20:16:22 -0000 --Apple-Mail=_D855CB1A-D28A-4BC0-AB20-20BCB349A2C4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 David, can I suggest that you take a look at Lucian's thesis = https://cds.cern.ch/record/1504817 - it is a nice read and it uses =E2=88=86= Q as the approach to analyse the performance of the ATLAS = detector/trigger system at the LHC in CERN. What Lucian did was to use the insights from the multiple measurement of = distributions to decompose the issues so that he could apply a series of = tools that identified a potential order of magnitude of improvement in = the peak performance the triggers system - that's the practical outcome. = The wider implication lies in the approach taken. It was able to do the = sort of decompositional analysis that broke the problem down to clear = logical components that could then be synthesised into a new = understanding of the problem. Not only is it a good read, it is an excellent exposition of scientific = method applied to a complex, state of the art collection of problems. Neil=20 On 25 Jul 2014, at 23:17, dpreed@reed.com wrote: > I think what is being discussed is "how to measure the quality of one = endpoint's experience of the entire Internet over all time or over a = specific interval of time". > =20 > Yet the systems that are built on top of the Internet transport do not = have any kind of uniform dependence on the underlying transport behavior = in terms of their quality. Even something like VoIP's quality as = experienced by two humans talking over it has a dependency on the = Internet's behavior, but one that is hardly simple. > =20 > As an extreme, if one endpoint experiences a direct DDoS attack or is = indirectly affected by one somewhere in the path, the quality of the = experience might be dramatically reduced. > =20 > So any attempt to define a delta-Q that has meaning in terms of user = experience appears pointless and even silly - the endpoint experience is = adequate under a very wide variety of conditions, but degrades terribly = under certain kinds of conditions. > =20 > As a different point, let's assume that the last-mile is 80% utilized, = but the latency variation in that utilization is not larger than 50 = msec. This is a feasible-to-imagine operating point, but it requires a = certain degreee of tight control that may be very hard to achieve over = thousands of independent application services through that point, so its = feasibility is contingent on lots of factors. Then if the 20% capacity = is far larger than 64 kb/sec we know that toll-quality audio can be = produced with a small endpoint "jitter buffer". There's no "delta-Q" = there at all - quality is great. > =20 > So the point is: a single number or even a single "morphism" (whatever = that is) to a specific algebraic domain element (a mapping to a = semi-lattice with non-Abelian operators?) does not allow one to define a = "measure" of an endpoint of the Internet that can be used to compute = "quality" of all applications. > =20 > Or in purely non-abstract terms: if there were a delta-Q it would be = useless for most network applications, but might be useful for a single = network application. > =20 > So I submit that delta-Q is a *metaphor* and not a very useful one at = that. It's probably as useful as providing a "funkiness" measure for an = Internat access point. We can certainly talk about and make claims = about the relative "funkiness" of different connections and different = providers. We might even claim that cable providers make funkier = network providers than cellular providers. > =20 > But to what end? >=20 >=20 > On Friday, July 25, 2014 5:13pm, "David Lang" said: >=20 > > On Fri, 25 Jul 2014, Martin Geddes wrote: > >=20 > > > So what is =CE=94Q and how do you "compute" it (to the extent it = is a > > "computed" > > > thing)? > >=20 > > don't try to reduce it to a single number, we have two numbers that = seem to > > matter > >=20 > > 1. throughput (each direction) > >=20 > > 2. latency under load > >=20 > > Currently the speed test sites report throughput in each direction = and ping time > > while not under load > >=20 > > If they could just add a ping time under load measurement, then we = could talk > > meaningfully about either the delta or ratio of the ping times as = the > > "bufferbloat factor" > >=20 > > no, it wouldn't account for absolutly every nuance, but it would = come pretty > > close. > >=20 > > If a connection has good throughput and a low bufferbloat factor, it = should be > > good for any type of use. > >=20 > > If it has good throughput, but a horrid bufferbloat factor, then you = need to > > artifically limit your traffic to stay clear of saturating the = bandwith > > (sacraficing throughput) > >=20 > > David Lang > >=20 > > > Starting point: the only observable effect of a network is to lose = and > > > delay data -- i.e. to "attenuate quality" by adding the toxic = effects of > > > time to distributed computations. =CE=94Q is a *morphism* that = relates the > > > "quality attenuation" that the network imposes to the application > > > performance, and describes the trading spaces at all intermediate = layers of > > > abstraction. It is shown in the attached graphic. > > > > > > Critically, it frames quality as something that can only be lost > > > ("attenuated"), both by the network and the application. = Additionally, it > > > is stochastic, and works with random variables and distributions. > > > > > > At its most concrete level, it is the individual impairment = encountered by > > > every packet when the network in operation. But we don't want to = have to > > > track every packet - 1:1 scale maps are pretty useless. So we need = to > > > abstract that in order to create a model that has value. > > > > > > Next abstraction: an improper random variable. This unifies loss = and delay > > > into a single stochastic object. > > > Next abstraction: received transport, which is a CDF where we are > > > interested in the properties of the "tail". > > > > > > Next abstraction, that joins network performance and application = QoE (as > > > relates to performance): relate the CDF to the application through = a > > > Quality Transport Agreement. This "stochastic contract" is both = necessary > > > and sufficient to deliver the application outcome. > > > > > > Next concretisation towards QoE: offered load of demand, as a CDF. > > > Next concretisation towards QoE: breach hazard metric, which = abstracts the > > > application performance. Indicates the likelihood of the QTA = contract being > > > broken, and how badly. > > > Final concretisation: the individual application performance = encountered by > > > every user. Again, a 1:1 map that isn't very helpful. > > > > > > So as you can see, it's about as far away from a single point = average > > > metric as you can possibly get. A far richer model is required in = order to > > > achieve robust performance engineering. > > > > > > It is "computed" using multi-point measurements to capture the > > > distribution. The G/S/V charts you see are based on processing = that data to > > > account for various issues, including clock skew. > > > > > > I hope that helps. We need to document more of this in public, = which is an > > > ongoing process. > > > > > > Martin > > > > > > On 25 July 2014 16:58, Sebastian Moeller wrote: > > > > > >> Hi Martin, > > >> > > >> thanks for the pointers, > > >> > > >> > > >> On Jul 25, 2014, at 16:25 , Martin Geddes > > wrote: > > >> > > >>> You may find the following useful background reading on the = state of > > the > > >> art in network measurement, and a primer on =CE=94Q (which is the > > property we > > >> wish to measure). > > >>> > > >>> First, start with this presentation: Network performance > > optimisation > > >> using high-fidelity measures > > >>> Then read this one to decompose =CE=94Q into G, S and V: > > Fundamentals of > > >> network performance engineering > > >>> Then read this one to get a bit more sense on what =CE=94Q is > > about: > > >> Introduction to =CE=94Q and Network Performance Science = (extracts) > > >>> > > >>> Then read these essays: > > >>> > > >>> Foundation of Network Science > > >>> How to do network performance chemistry > > >>> How to X-ray a telecoms network > > >>> There is no quality in averages: IPX case study > > >> > > >> All of this makes intuitively sense, but it is a bit light on > > how > > >> deltaQ is to be computed ;). > > >> As far as I understand it also has not much bearing on my home > > >> network; the only one under my control. Now, following the buffer = bloat > > >> discussion for some years, I have internalized the idea that = bandwidth > > >> alone does not suffice to describe the quality of my network = connection. > > I > > >> think that the latency increase under load (for unrelated flows) = is the > > >> best of all the bad single number measures of network = dynamics/quality. > > I > > >> should be related to what I understood deltaQ to depend on (as = packet > > loss > > >> for non real time flows will cause an increase in latency). I = think > > that > > >> continuous measurements make a to n of sense for ISPs, > > backbone-operators, > > >> mobile carriers =E2=80=A6 but at home, basically, I operate as my = own > > network > > >> quality monitor ;) (that is I try to pin point and debug = (transient) > > >> anomalies). > > >> > > >>> > > >>> Martin > > >>> > > >>> For fresh thinking about telecoms sign up for my free newsletter = or > > >> visit the Geddes Think Tank. > > >>> LinkedIn Twitter Mobile: +44 7957 499219 Skype: mgeddes > > >>> Martin Geddes Consulting Ltd, Incorporated in Scotland, number > > SC275827 > > >> VAT Number: 859 5634 72 Registered office: 17-19 East London = Street, > > >> Edinburgh, EH7 4BN > > >>> > > >>> > > >>> > > >>> On 25 July 2014 15:17, Sebastian Moeller > > wrote: > > >>> Hi Neil, > > >>> > > >>> > > >>> On Jul 25, 2014, at 14:24 , Neil Davies > > wrote: > > >>> > > >>>> Rich > > >>>> > > >>>> I have a deep worry over this style of single point measurement = - > > and > > >> hence speed - as an appropriate measure. > > >>> > > >>> But how do you propose to measure the (bottleneck) link > > capacity > > >> then? It turns out for current CPE and CMTS/DSLAM equipment one > > typically > > >> can not relay on good QoE out of the box, since typically these = devices > > do > > >> not use their (largish) buffers wisely. Instead the current = remedy is to > > >> take back control over the bottleneck link by shaping the = actually sent > > >> traffic to stay below the hardware link capacity thereby avoiding > > feeling > > >> the consequences of the over-buffering. But to do this is is = quite > > helpful > > >> to get an educated guess what the bottleneck links capacity = actually is. > > >> And for that purpose a speediest seems useful. > > >>> > > >>> > > >>>> We know, and have evidence, that throughput/utilisation is not = a > > good > > >> proxy for the network delivering suitable quality of experience. = We work > > >> with organisation (Telco=E2=80=99s, large system integrators etc) = where we > > spend a > > >> lot of time having to =E2=80=9Cundo=E2=80=9D the consequences of > > =E2=80=9Cmaximising speed=E2=80=9D. Just > > >> like there is more to life than work, there is more to QoE than = speed. > > >>>> > > >>>> For more specific comments see inline > > >>>> > > >>>> On 25 Jul 2014, at 13:09, Rich Brown > > wrote: > > >>>> > > >>>>> Neil, > > >>>>> > > >>>>> Thanks for the note and the observations. My thoughts: > > >>>>> > > >>>>> 1) I note that speedof.me does seem to overstate the speed > > results. > > >> At my home, it reports 5.98mbps down, and 638kbps up, while > > >> betterspeedtest.sh shows 5.49/0.61 mbps. (speedtest.net gives = numbers > > >> similar to the betterspeedtest.net script.) > > >>>>> > > >>>>> 2) I think we're in agreement about the peak upload rate that > > you > > >> point out is too high. Their measurement code runs in the = browser. It > > seems > > >> likely that the browser pumps out a few big packets before = getting flow > > >> control information, thus giving the impression that they can = send at a > > >> higher rate. This comports with the obvious decay that ramps = toward the > > >> long-term rate. > > >>>> > > >>>> I think that its simpler than that, it is measuring the rate at > > which > > >> it can push packets out the interface - its real time rate is = precisely > > >> that - it can not be the rate being reported by the far end, it = can > > never > > >> exceed the limiting link. The long term average (if it is like = other > > speed > > >> testers we=E2=80=99ve had to look into) is being measured at the = TCP/IP SDU > > level > > >> by measuring the difference in time between the first and last > > timestamps > > >> of data stream and dividing that into the total data sent. Their > > >> =E2=80=9Cover-estimate=E2=80=9D is because there are packets = buffered in the > > CPE that have > > >> left the machine but not arrived at the far end. > > >>> > > >>> Testing from an openwrt router located at a > > >> high-symmetric-bandwidth location shows that speedof.me does not = scale > > >> higher than ~ 130 Mbps server to client and ~15Mbps client to = server (on > > >> the same connection I can get 130Mbps S2C and ~80Mbps C2S, so the > > asymmetry > > >> in the speedof.me results is not caused by my local environment). > > >>> @Rich and Dave, this probably means that for the upper end > > of > > >> fiber and cable and VDSL connections speed of.me is not going to = be a > > >> reliable speed measure=E2=80=A6 Side note www.sppedtest.net shows = ~100Mbps > > S2C > > >> and ~100Mbps C2S, so might be better suited to high-upload = links... > > >>> > > >>>> > > >>>>> > > >>>>> 3) But that long-term speed should be at or below the > > theoretical > > >> long-term rate, not above it. > > >>>> > > >>>> Agreed, but in this case knowing the sync rate already defines > > that > > >> maximum. > > >>> > > >>> I fully agree, but for ADSL the sync rate also contains a lot > > of > > >> encapsulation, so the maximum achievable TCP rate is at best ~90% = of > > link > > >> rate. Note for cerowrt=E2=80=99s SQM system the link rate is = exactly the > > right > > >> number to start out with at that system can take the = encapsulation into > > >> account. But even then it is somewhat unintuitive to deduce the = expected > > >> good-put from the link rate. > > >>> > > >>>> > > >>>>> > > >>>>> Two experiments for you to try: > > >>>>> > > >>>>> a) What does betterspeedtest.sh show? (It's in the latest > > CeroWrt, in > > >> /usr/lib/CeroWrtScripts, or get it from github: > > >> https://github.com/richb-hanover/CeroWrtScripts ) > > >>>>> > > >>>>> b) What does www.speedtest.net show? > > >>>>> > > >>>>> I will add your question (about the inaccuracy) to the note > > that I > > >> want to send out to speedof.me this weekend. I will also ask that = they > > >> include min/max latency measurements to their test, and an option = to > > send > > >> for > 10 seconds to minimize any effect of PowerBoost=E2=80=A6 > > >>> > > >>> I think they do already, at least for the download > > bandwidth; > > >> they start with 128Kb and keep doubling the file size until a = file takes > > >> longer than 8 seconds to transfer, they only claim to report the = numbers > > >> from that last transferred file, so worst case with a stable link = and a > > >> bandwidth > 16kbps ;), it has taken at least 12 seconds (4 plus = 8) of > > >> measuring before the end of the plot, so the bandwidth of at = least the > > last > > >> half of the download plot should be representative even assuming = power > > >> boost. Caveat, I assume that power boost will not be reset by the > > transient > > >> lack of data transfer between the differently sized files (but = since it > > >> should involve the same IPs and port# why should power boost = reset > > itself?). > > >>> > > >>> Best Regards > > >>> Sebastian > > >>> > > >>> > > >>> > > >>>>> > > >>>>> Best regards, > > >>>>> > > >>>>> Rich > > >>>>> > > >>>>> > > >>>>> > > >>>>> On Jul 25, 2014, at 5:10 AM, Neil Davies > > > > >> wrote: > > >>>>> > > >>>>>> Rich > > >>>>>> > > >>>>>> You may want to check how accurate they are to start. > > >>>>>> > > >>>>>> I just ran a =E2=80=9Cspeed test=E2=80=9D on my line (which I > > have complete control > > >> and visibility over the various network elements) and it reports = an > > average > > >> =E2=80=9Cspeed=E2=80=9D (in the up direction) that is in excess = of the > > capacity of the > > >> line, it reports the maximum rate at nearly twice the best = possible rate > > of > > >> the ADSL connection. > > >>>>>> > > >>>>>> Doesn=E2=80=99t matter how pretty it is, if its not > > accurate it is of no > > >> use. This is rather ironic as the web site claims it is the > > =E2=80=9Csmartest and > > >> most accurate=E2=80=9D! > > >>>>>> > > >>>>>> Neil > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> PS pretty clear to me what mistake they=E2=80=99ve made in > > the measurement > > >> process - its to do with incorrect inference and hence missing = the > > >> buffering effects. > > >>>>>> > > >>>>>> On 20 Jul 2014, at 14:19, Rich Brown > > > > >> wrote: > > >>>>>> > > >>>>>>> Doc Searls ( > > >> > > = http://blogs.law.harvard.edu/doc/2014/07/20/the-cliff-peronal-clouds-need-= to-climb/) > > >> mentioned in passing that he uses a new speed test website. I = checked it > > >> out, and it was very cool=E2=80=A6 > > >>>>>>> > > >>>>>>> www.speedof.me is an all-HTML5 website that seems to > > make accurate > > >> measurements of the up and download speeds of your internet = connection. > > >> It=E2=80=99s also very attractive, and the real-time plots of the = speed > > show > > >> interesting info. (screen shot at: = http://richb-hanover.com/speedof-me/) > > >>>>>>> > > >>>>>>> Now if we could get them to a) allow longer/bigger > > tests to > > >> circumvent PowerBoost, and b) include a latency measurement so = people > > could > > >> point out their bufferbloated equipment. > > >>>>>>> > > >>>>>>> I'm going to send them a note. Anything else I should > > add? > > >>>>>>> > > >>>>>>> Rich > > >>>>>>> _______________________________________________ > > >>>>>>> Bloat mailing list > > >>>>>>> Bloat@lists.bufferbloat.net > > >>>>>>> https://lists.bufferbloat.net/listinfo/bloat > > >>>>>> > > >>>>> > > >>>> > > >>>> _______________________________________________ > > >>>> Bloat mailing list > > >>>> Bloat@lists.bufferbloat.net > > >>>> https://lists.bufferbloat.net/listinfo/bloat > > >>> > > >>> _______________________________________________ > > >>> Bloat mailing list > > >>> Bloat@lists.bufferbloat.net > > >>> https://lists.bufferbloat.net/listinfo/bloat > > >>> > > >> > > >> > > >_______________________________________________ > > Cerowrt-devel mailing list > > Cerowrt-devel@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/cerowrt-devel > > _______________________________________________ > > Cerowrt-devel mailing list > > Cerowrt-devel@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/cerowrt-devel > > --Apple-Mail=_D855CB1A-D28A-4BC0-AB20-20BCB349A2C4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 David, = can I suggest that you take a look at Lucian's thesis https://cds.cern.ch/record/150= 4817 - it is a nice read and it uses =E2=88=86Q as the approach to = analyse the performance of the ATLAS detector/trigger system at the LHC = in CERN.

What Lucian did was to use the insights from = the multiple measurement of distributions to decompose the issues so = that he could apply a series of tools that identified a potential order = of magnitude of improvement in the peak performance the triggers system = - that's the practical outcome. The wider implication lies in the = approach taken. It was able to do the sort of decompositional analysis = that broke the problem down to clear logical components that could then = be synthesised into a new understanding of the = problem.

Not only is it a good read, it is an = excellent exposition of scientific method applied to a complex, state of = the art collection of = problems.

Neil 

On = 25 Jul 2014, at 23:17, dpreed@reed.com wrote:

I think = what is being discussed is "how to measure the quality of one endpoint's = experience of the entire Internet over all time or over a specific = interval of time".

 

Yet the systems that are built on top of the = Internet transport do not have any kind of uniform dependence on the = underlying transport behavior in terms of their quality.  Even = something like VoIP's quality as experienced by two humans talking over = it has a dependency on the Internet's behavior, but one that is hardly = simple.

 

As an extreme, if one endpoint experiences a direct DDoS = attack or is indirectly affected by one somewhere in the path, the = quality of the experience might be dramatically reduced.

 

So any = attempt to define a delta-Q that has  meaning in terms of user = experience appears pointless and even silly - the endpoint experience is = adequate under a very wide variety of conditions, but degrades terribly = under certain kinds of conditions.

 

As a = different point, let's assume that the last-mile is 80% utilized, but = the latency variation in that utilization is not larger than 50 msec. =  This is a feasible-to-imagine operating point, but it requires a = certain degreee of tight control that may be very hard to achieve over = thousands of independent application services through that point, so its = feasibility is contingent on lots of factors. Then if the 20% capacity = is far larger than 64 kb/sec we know that toll-quality audio can be = produced with a small endpoint "jitter buffer".  There's no = "delta-Q" there at all - quality is great.

 

So the = point is: a single number or even a single "morphism" (whatever that is) = to a specific algebraic domain element (a mapping to a semi-lattice with = non-Abelian operators?) does not allow one to define a "measure" of an = endpoint of the Internet that can be used to compute "quality" of all = applications.

 

Or in purely non-abstract terms: if there were a delta-Q it = would be useless for most network applications, but might be useful for = a single network application.

 

So I = submit that delta-Q is a *metaphor* and not a very useful one at that. =  It's probably as useful as providing a "funkiness" measure for an = Internat access point.  We can certainly talk about and make claims = about the relative "funkiness" of different connections and different = providers.  We might even claim that cable providers make funkier = network providers than cellular providers.

 

But to = what end?


On Friday, July 25, 2014 5:13pm, "David = Lang" <david@lang.hm> = said:

> = On Fri, 25 Jul 2014, Martin Geddes wrote:
>
> > So what = is =CE=94Q and how do you "compute" it (to the extent it is a
> = "computed"
> > thing)?
>
> don't try to reduce it = to a single number, we have two numbers that seem to
> = matter
>
> 1. throughput (each direction)
>
> = 2. latency under load
>
> Currently the speed test sites = report throughput in each direction and ping time
> while not = under load
>
> If they could just add a ping time under = load measurement, then we could talk
> meaningfully about either = the delta or ratio of the ping times as the
> "bufferbloat = factor"
>
> no, it wouldn't account for absolutly every = nuance, but it would come pretty
> close.
>
> If a = connection has good throughput and a low bufferbloat factor, it should = be
> good for any type of use.
>
> If it has good = throughput, but a horrid bufferbloat factor, then you need to
> = artifically limit your traffic to stay clear of saturating the = bandwith
> (sacraficing throughput)
>
> David = Lang
>
> > Starting point: the only observable effect of = a network is to lose and
> > delay data -- i.e. to "attenuate = quality" by adding the toxic effects of
> > time to distributed = computations. =CE=94Q is a *morphism* that relates the
> > = "quality attenuation" that the network imposes to the = application
> > performance, and describes the trading spaces = at all intermediate layers of
> > abstraction. It is shown in = the attached graphic.
> >
> > Critically, it frames = quality as something that can only be lost
> > ("attenuated"), = both by the network and the application. Additionally, it
> > = is stochastic, and works with random variables and = distributions.
> >
> > At its most concrete level, it = is the individual impairment encountered by
> > every packet = when the network in operation. But we don't want to have to
> > = track every packet - 1:1 scale maps are pretty useless. So we need = to
> > abstract that in order to create a model that has = value.
> >
> > Next abstraction: an improper random = variable. This unifies loss and delay
> > into a single = stochastic object.
> > Next abstraction: received transport, = which is a CDF where we are
> > interested in the properties of = the "tail".
> >
> > Next abstraction, that joins = network performance and application QoE (as
> > relates to = performance): relate the CDF to the application through a
> > = Quality Transport Agreement. This "stochastic contract" is both = necessary
> > and sufficient to deliver the application = outcome.
> >
> > Next concretisation towards QoE: = offered load of demand, as a CDF.
> > Next concretisation = towards QoE: breach hazard metric, which abstracts the
> > = application performance. Indicates the likelihood of the QTA contract = being
> > broken, and how badly.
> > Final = concretisation: the individual application performance encountered = by
> > every user. Again, a 1:1 map that isn't very = helpful.
> >
> > So as you can see, it's about as far = away from a single point average
> > metric as you can possibly = get. A far richer model is required in order to
> > achieve = robust performance engineering.
> >
> > It is = "computed" using multi-point measurements to capture the
> > = distribution. The G/S/V charts you see are based on processing that data = to
> > account for various issues, including clock = skew.
> >
> > I hope that helps. We need to document = more of this in public, which is an
> > ongoing = process.
> >
> > Martin
> >
> > On = 25 July 2014 16:58, Sebastian Moeller <moeller0@gmx.de> wrote:
> = >
> >> Hi Martin,
> >>
> >> = thanks for the pointers,
> >>
> >>
> = >> On Jul 25, 2014, at 16:25 , Martin Geddes <mail@martingeddes.com>
>= ; wrote:
> >>
> >>> You may find the = following useful background reading on the state of
> the
> = >> art in network measurement, and a primer on =CE=94Q (which is = the
> property we
> >> wish to measure).
> = >>>
> >>> First, start with this presentation: = Network performance
> optimisation
> >> using = high-fidelity measures
> >>> Then read this one to = decompose =CE=94Q into G, S and V:
> Fundamentals of
> = >> network performance engineering
> >>> Then read = this one to get a bit more sense on what =CE=94Q is
> = about:
> >> Introduction to =CE=94Q and Network Performance = Science (extracts)
> >>>
> >>> Then read = these essays:
> >>>
> >>> Foundation of = Network Science
> >>> How to do network performance = chemistry
> >>> How to X-ray a telecoms network
> = >>> There is no quality in averages: IPX case study
> = >>
> >> All of this makes intuitively sense, but it is = a bit light on
> how
> >> deltaQ is to be computed = ;).
> >> As far as I understand it also has not much bearing = on my home
> >> network; the only one under my control. Now, = following the buffer bloat
> >> discussion for some years, I = have internalized the idea that bandwidth
> >> alone does = not suffice to describe the quality of my network connection.
> = I
> >> think that the latency increase under load (for = unrelated flows) is the
> >> best of all the bad single = number measures of network dynamics/quality.
> I
> >> = should be related to what I understood deltaQ to depend on (as = packet
> loss
> >> for non real time flows will cause = an increase in latency). I think
> that
> >> = continuous measurements make a to n of sense for ISPs,
> = backbone-operators,
> >> mobile carriers =E2=80=A6 but at = home, basically, I operate as my own
> network
> >> = quality monitor ;) (that is I try to pin point and debug = (transient)
> >> anomalies).
> >>
> = >>>
> >>> Martin
> >>>
> = >>> For fresh thinking about telecoms sign up for my free = newsletter or
> >> visit the Geddes Think Tank.
> = >>> LinkedIn Twitter Mobile: +44 7957 499219 Skype: = mgeddes
> >>> Martin Geddes Consulting Ltd, Incorporated = in Scotland, number
> SC275827
> >> VAT Number: 859 = 5634 72 Registered office: 17-19 East London Street,
> >> = Edinburgh, EH7 4BN
> >>>
> >>>
> = >>>
> >>> On 25 July 2014 15:17, Sebastian = Moeller <moeller0@gmx.de>
> = wrote:
> >>> Hi Neil,
> >>>
> = >>>
> >>> On Jul 25, 2014, at 14:24 , Neil = Davies <Neil.Davies@pnsol.com>
>= ; wrote:
> >>>
> >>>> Rich
> = >>>>
> >>>> I have a deep worry over this = style of single point measurement -
> and
> >> hence = speed - as an appropriate measure.
> >>>
> = >>> But how do you propose to measure the (bottleneck) = link
> capacity
> >> then? It turns out for current = CPE and CMTS/DSLAM equipment one
> typically
> >> can = not relay on good QoE out of the box, since typically these = devices
> do
> >> not use their (largish) buffers = wisely. Instead the current remedy is to
> >> take back = control over the bottleneck link by shaping the actually sent
> = >> traffic to stay below the hardware link capacity thereby = avoiding
> feeling
> >> the consequences of the = over-buffering. But to do this is is quite
> helpful
> = >> to get an educated guess what the bottleneck links capacity = actually is.
> >> And for that purpose a speediest seems = useful.
> >>>
> >>>
> = >>>> We know, and have evidence, that throughput/utilisation = is not a
> good
> >> proxy for the network delivering = suitable quality of experience. We work
> >> with = organisation (Telco=E2=80=99s, large system integrators etc) where = we
> spend a
> >> lot of time having to =E2=80=9Cundo=E2= =80=9D the consequences of
> =E2=80=9Cmaximising speed=E2=80=9D. = Just
> >> like there is more to life than work, there is = more to QoE than speed.
> >>>>
> = >>>> For more specific comments see inline
> = >>>>
> >>>> On 25 Jul 2014, at 13:09, Rich = Brown
> <richb.hanover@gmail.com> = wrote:
> >>>>
> >>>>> = Neil,
> >>>>>
> >>>>> Thanks = for the note and the observations. My thoughts:
> = >>>>>
> >>>>> 1) I note that speedof.me does seem to overstate the = speed
> results.
> >> At my home, it reports 5.98mbps = down, and 638kbps up, while
> >> betterspeedtest.sh shows = 5.49/0.61 mbps. (speedtest.net = gives numbers
> >> similar to the betterspeedtest.net = script.)
> >>>>>
> >>>>> 2) I = think we're in agreement about the peak upload rate that
> = you
> >> point out is too high. Their measurement code runs = in the browser. It
> seems
> >> likely that the = browser pumps out a few big packets before getting flow
> >> = control information, thus giving the impression that they can send at = a
> >> higher rate. This comports with the obvious decay = that ramps toward the
> >> long-term rate.
> = >>>>
> >>>> I think that its simpler than = that, it is measuring the rate at
> which
> >> it can = push packets out the interface - its real time rate is precisely
> = >> that - it can not be the rate being reported by the far end, it = can
> never
> >> exceed the limiting link. The long = term average (if it is like other
> speed
> >> testers = we=E2=80=99ve had to look into) is being measured at the TCP/IP = SDU
> level
> >> by measuring the difference in time = between the first and last
> timestamps
> >> of data = stream and dividing that into the total data sent. Their
> = >> =E2=80=9Cover-estimate=E2=80=9D is because there are packets = buffered in the
> CPE that have
> >> left the machine = but not arrived at the far end.
> >>>
> = >>> Testing from an openwrt router located at a
> = >> high-symmetric-bandwidth location shows that speedof.me does not scale
> = >> higher than ~ 130 Mbps server to client and ~15Mbps client to = server (on
> >> the same connection I can get 130Mbps S2C = and ~80Mbps C2S, so the
> asymmetry
> >> in the speedof.me results is not caused by my = local environment).
> >>> @Rich and Dave, this probably = means that for the upper end
> of
> >> fiber and cable = and VDSL connections speed of.me is not = going to be a
> >> reliable speed measure=E2=80=A6 Side note = www.sppedtest.net shows = ~100Mbps
> S2C
> >> and ~100Mbps C2S, so might be = better suited to high-upload links...
> >>>
> = >>>>
> >>>>>
> = >>>>> 3) But that long-term speed should be at or below = the
> theoretical
> >> long-term rate, not above = it.
> >>>>
> >>>> Agreed, but in = this case knowing the sync rate already defines
> that
> = >> maximum.
> >>>
> >>> I fully = agree, but for ADSL the sync rate also contains a lot
> of
> = >> encapsulation, so the maximum achievable TCP rate is at best = ~90% of
> link
> >> rate. Note for cerowrt=E2=80=99s = SQM system the link rate is exactly the
> right
> >> = number to start out with at that system can take the encapsulation = into
> >> account. But even then it is somewhat unintuitive = to deduce the expected
> >> good-put from the link = rate.
> >>>
> >>>>
> = >>>>>
> >>>>> Two experiments for = you to try:
> >>>>>
> >>>>> a) = What does betterspeedtest.sh show? (It's in the latest
> CeroWrt, = in
> >> /usr/lib/CeroWrtScripts, or get it from = github:
> >> https://github.co= m/richb-hanover/CeroWrtScripts )
> = >>>>>
> >>>>> b) What does www.speedtest.net show?
> = >>>>>
> >>>>> I will add your = question (about the inaccuracy) to the note
> that I
> = >> want to send out to speedof.me = this weekend. I will also ask that they
> >> include min/max = latency measurements to their test, and an option to
> = send
> >> for > 10 seconds to minimize any effect of = PowerBoost=E2=80=A6
> >>>
> >>> I think = they do already, at least for the download
> bandwidth;
> = >> they start with 128Kb and keep doubling the file size until a = file takes
> >> longer than 8 seconds to transfer, they only = claim to report the numbers
> >> from that last transferred = file, so worst case with a stable link and a
> >> bandwidth = > 16kbps ;), it has taken at least 12 seconds (4 plus 8) of
> = >> measuring before the end of the plot, so the bandwidth of at = least the
> last
> >> half of the download plot should = be representative even assuming power
> >> boost. Caveat, I = assume that power boost will not be reset by the
> = transient
> >> lack of data transfer between the differently = sized files (but since it
> >> should involve the same IPs = and port# why should power boost reset
> itself?).
> = >>>
> >>> Best Regards
> >>> = Sebastian
> >>>
> >>>
> = >>>
> >>>>>
> >>>>> = Best regards,
> >>>>>
> >>>>> = Rich
> >>>>>
> >>>>>
> = >>>>>
> >>>>> On Jul 25, 2014, at = 5:10 AM, Neil Davies
> <neil.davies@pnsol.com>
>= ; >> wrote:
> >>>>>
> = >>>>>> Rich
> >>>>>>
> = >>>>>> You may want to check how accurate they are to = start.
> >>>>>>
> >>>>>> = I just ran a =E2=80=9Cspeed test=E2=80=9D on my line (which I
> = have complete control
> >> and visibility over the various = network elements) and it reports an
> average
> >> = =E2=80=9Cspeed=E2=80=9D (in the up direction) that is in excess of = the
> capacity of the
> >> line, it reports the = maximum rate at nearly twice the best possible rate
> of
> = >> the ADSL connection.
> >>>>>>
> = >>>>>> Doesn=E2=80=99t matter how pretty it is, if its = not
> accurate it is of no
> >> use. This is rather = ironic as the web site claims it is the
> =E2=80=9Csmartest = and
> >> most accurate=E2=80=9D!
> = >>>>>>
> >>>>>> Neil
> = >>>>>>
> >>>>>> = <speedof_me_14-07-25.png>
> >>>>>>
> = >>>>>> PS pretty clear to me what mistake they=E2=80=99v= e made in
> the measurement
> >> process - its to do = with incorrect inference and hence missing the
> >> = buffering effects.
> >>>>>>
> = >>>>>> On 20 Jul 2014, at 14:19, Rich Brown
> = <richb.hanover@gmail.com>> >> wrote:
> >>>>>>
> = >>>>>>> Doc Searls (
> >>
> http://blogs.law.harvard.edu/doc/2014/07/20/the-cliff-p= eronal-clouds-need-to-climb/)
> >> mentioned in passing = that he uses a new speed test website. I checked it
> >> = out, and it was very cool=E2=80=A6
> = >>>>>>>
> >>>>>>> www.speedof.me is an all-HTML5 = website that seems to
> make accurate
> >> = measurements of the up and download speeds of your internet = connection.
> >> It=E2=80=99s also very attractive, and the = real-time plots of the speed
> show
> >> interesting = info. (screen shot at: http://richb-hanover.com/spe= edof-me/)
> >>>>>>>
> = >>>>>>> Now if we could get them to a) allow = longer/bigger
> tests to
> >> circumvent PowerBoost, = and b) include a latency measurement so people
> could
> = >> point out their bufferbloated equipment.
> = >>>>>>>
> >>>>>>> I'm = going to send them a note. Anything else I should
> add?
> = >>>>>>>
> >>>>>>> = Rich
> >>>>>>> = _______________________________________________
> = >>>>>>> Bloat mailing list
> = >>>>>>> Bloat@lists.bufferbloat.net
> >>>>>>> https://lists.buffer= bloat.net/listinfo/bloat
> >>>>>>
> = >>>>>
> >>>>
> >>>> = _______________________________________________
> >>>> = Bloat mailing list
> >>>> Bloat@lists.bufferbloat.net
> >>>> https://lists.buffer= bloat.net/listinfo/bloat
> >>>
> >>> = _______________________________________________
> >>> = Bloat mailing list
> >>> Bloat@lists.bufferbloat.net
> >>> https://lists.buffer= bloat.net/listinfo/bloat
> >>>
> = >>
> >>
> = >_______________________________________________
> = Cerowrt-devel mailing list
> Cerowrt-devel@lists.bu= fferbloat.net
> https://list= s.bufferbloat.net/listinfo/cerowrt-devel
> = _______________________________________________
> Cerowrt-devel = mailing list
> Cerowrt-devel@lists.bu= fferbloat.net
> https://list= s.bufferbloat.net/listinfo/cerowrt-devel
>

= --Apple-Mail=_D855CB1A-D28A-4BC0-AB20-20BCB349A2C4--