<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">David, can I suggest that you take a look at Lucian's thesis <a href="https://cds.cern.ch/record/1504817">https://cds.cern.ch/record/1504817</a> - it is a nice read and it uses ∆Q as the approach to analyse the performance of the ATLAS detector/trigger system at the LHC in CERN.<div><br></div><div>What Lucian did was to use the insights from the multiple measurement of distributions to decompose the issues so that he could apply a series of tools that identified a potential order of magnitude of improvement in the peak performance the triggers system - that's the practical outcome. The wider implication lies in the approach taken. It was able to do the sort of decompositional analysis that broke the problem down to clear logical components that could then be synthesised into a new understanding of the problem.</div><div><br></div><div>Not only is it a good read, it is an excellent exposition of scientific method applied to a complex, state of the art collection of problems.</div><div><br></div><div>Neil </div><div><br><div><div>On 25 Jul 2014, at 23:17, <a href="mailto:dpreed@reed.com">dpreed@reed.com</a> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><font face="arial" size="2"><div style="margin: 0px; padding: 0px; font-family: arial; font-size: 10pt; word-wrap: break-word;">I think what is being discussed is "how to measure the quality of one endpoint's experience of the entire Internet over all time or over a specific interval of time".</div><p style="margin:0;padding:0;font-family: arial; font-size: 10pt; word-wrap: break-word;"> </p><div style="margin: 0px; padding: 0px; font-family: arial; font-size: 10pt; word-wrap: break-word;">Yet the systems that are built on top of the Internet transport do not have any kind of uniform dependence on the underlying transport behavior in terms of their quality. Even something like VoIP's quality as experienced by two humans talking over it has a dependency on the Internet's behavior, but one that is hardly simple.</div><p style="margin:0;padding:0;font-family: arial; font-size: 10pt; word-wrap: break-word;"> </p><div style="margin: 0px; padding: 0px; font-family: arial; font-size: 10pt; word-wrap: break-word;">As an extreme, if one endpoint experiences a direct DDoS attack or is indirectly affected by one somewhere in the path, the quality of the experience might be dramatically reduced.</div><p style="margin:0;padding:0;font-family: arial; font-size: 10pt; word-wrap: break-word;"> </p><div style="margin: 0px; padding: 0px; font-family: arial; font-size: 10pt; word-wrap: break-word;">So any attempt to define a delta-Q that has meaning in terms of user experience appears pointless and even silly - the endpoint experience is adequate under a very wide variety of conditions, but degrades terribly under certain kinds of conditions.</div><p style="margin:0;padding:0;font-family: arial; font-size: 10pt; word-wrap: break-word;"> </p><div style="margin: 0px; padding: 0px; font-family: arial; font-size: 10pt; word-wrap: break-word;">As a different point, let's assume that the last-mile is 80% utilized, but the latency variation in that utilization is not larger than 50 msec. This is a feasible-to-imagine operating point, but it requires a certain degreee of tight control that may be very hard to achieve over thousands of independent application services through that point, so its feasibility is contingent on lots of factors. Then if the 20% capacity is far larger than 64 kb/sec we know that toll-quality audio can be produced with a small endpoint "jitter buffer". There's no "delta-Q" there at all - quality is great.</div><p style="margin:0;padding:0;font-family: arial; font-size: 10pt; word-wrap: break-word;"> </p><div style="margin: 0px; padding: 0px; font-family: arial; font-size: 10pt; word-wrap: break-word;">So the point is: a single number or even a single "morphism" (whatever that is) to a specific algebraic domain element (a mapping to a semi-lattice with non-Abelian operators?) does not allow one to define a "measure" of an endpoint of the Internet that can be used to compute "quality" of all applications.</div><p style="margin:0;padding:0;font-family: arial; font-size: 10pt; word-wrap: break-word;"> </p><div style="margin: 0px; padding: 0px; font-family: arial; font-size: 10pt; word-wrap: break-word;">Or in purely non-abstract terms: if there were a delta-Q it would be useless for most network applications, but might be useful for a single network application.</div><p style="margin:0;padding:0;font-family: arial; font-size: 10pt; word-wrap: break-word;"> </p><div style="margin: 0px; padding: 0px; font-family: arial; font-size: 10pt; word-wrap: break-word;">So I submit that delta-Q is a *metaphor* and not a very useful one at that. It's probably as useful as providing a "funkiness" measure for an Internat access point. We can certainly talk about and make claims about the relative "funkiness" of different connections and different providers. We might even claim that cable providers make funkier network providers than cellular providers.</div><p style="margin:0;padding:0;font-family: arial; font-size: 10pt; word-wrap: break-word;"> </p><div style="margin: 0px; padding: 0px; font-family: arial; font-size: 10pt; word-wrap: break-word;">But to what end?</div>
<!--WM_COMPOSE_SIGNATURE_START--><!--WM_COMPOSE_SIGNATURE_END--><div style="margin: 0px; padding: 0px; font-family: arial; font-size: 10pt; word-wrap: break-word;"><br><br>On Friday, July 25, 2014 5:13pm, "David Lang" <<a href="mailto:david@lang.hm">david@lang.hm</a>> said:<br><br></div>
<div id="SafeStyles1406325441"><div style="margin: 0px; padding: 0px; font-family: arial; font-size: 10pt; word-wrap: break-word;">> On Fri, 25 Jul 2014, Martin Geddes wrote:<br>> <br>> > So what is ΔQ and how do you "compute" it (to the extent it is a<br>> "computed"<br>> > thing)?<br>> <br>> don't try to reduce it to a single number, we have two numbers that seem to<br>> matter<br>> <br>> 1. throughput (each direction)<br>> <br>> 2. latency under load<br>> <br>> Currently the speed test sites report throughput in each direction and ping time<br>> while not under load<br>> <br>> If they could just add a ping time under load measurement, then we could talk<br>> meaningfully about either the delta or ratio of the ping times as the<br>> "bufferbloat factor"<br>> <br>> no, it wouldn't account for absolutly every nuance, but it would come pretty<br>> close.<br>> <br>> If a connection has good throughput and a low bufferbloat factor, it should be<br>> good for any type of use.<br>> <br>> If it has good throughput, but a horrid bufferbloat factor, then you need to<br>> artifically limit your traffic to stay clear of saturating the bandwith<br>> (sacraficing throughput)<br>> <br>> David Lang<br>> <br>> > Starting point: the only observable effect of a network is to lose and<br>> > delay data -- i.e. to "attenuate quality" by adding the toxic effects of<br>> > time to distributed computations. ΔQ is a *morphism* that relates the<br>> > "quality attenuation" that the network imposes to the application<br>> > performance, and describes the trading spaces at all intermediate layers of<br>> > abstraction. It is shown in the attached graphic.<br>> ><br>> > Critically, it frames quality as something that can only be lost<br>> > ("attenuated"), both by the network and the application. Additionally, it<br>> > is stochastic, and works with random variables and distributions.<br>> ><br>> > At its most concrete level, it is the individual impairment encountered by<br>> > every packet when the network in operation. But we don't want to have to<br>> > track every packet - 1:1 scale maps are pretty useless. So we need to<br>> > abstract that in order to create a model that has value.<br>> ><br>> > Next abstraction: an improper random variable. This unifies loss and delay<br>> > into a single stochastic object.<br>> > Next abstraction: received transport, which is a CDF where we are<br>> > interested in the properties of the "tail".<br>> ><br>> > Next abstraction, that joins network performance and application QoE (as<br>> > relates to performance): relate the CDF to the application through a<br>> > Quality Transport Agreement. This "stochastic contract" is both necessary<br>> > and sufficient to deliver the application outcome.<br>> ><br>> > Next concretisation towards QoE: offered load of demand, as a CDF.<br>> > Next concretisation towards QoE: breach hazard metric, which abstracts the<br>> > application performance. Indicates the likelihood of the QTA contract being<br>> > broken, and how badly.<br>> > Final concretisation: the individual application performance encountered by<br>> > every user. Again, a 1:1 map that isn't very helpful.<br>> ><br>> > So as you can see, it's about as far away from a single point average<br>> > metric as you can possibly get. A far richer model is required in order to<br>> > achieve robust performance engineering.<br>> ><br>> > It is "computed" using multi-point measurements to capture the<br>> > distribution. The G/S/V charts you see are based on processing that data to<br>> > account for various issues, including clock skew.<br>> ><br>> > I hope that helps. We need to document more of this in public, which is an<br>> > ongoing process.<br>> ><br>> > Martin<br>> ><br>> > On 25 July 2014 16:58, Sebastian Moeller <<a href="mailto:moeller0@gmx.de">moeller0@gmx.de</a>> wrote:<br>> ><br>> >> Hi Martin,<br>> >><br>> >> thanks for the pointers,<br>> >><br>> >><br>> >> On Jul 25, 2014, at 16:25 , Martin Geddes <<a href="mailto:mail@martingeddes.com">mail@martingeddes.com</a>><br>> wrote:<br>> >><br>> >>> You may find the following useful background reading on the state of<br>> the<br>> >> art in network measurement, and a primer on ΔQ (which is the<br>> property we<br>> >> wish to measure).<br>> >>><br>> >>> First, start with this presentation: Network performance<br>> optimisation<br>> >> using high-fidelity measures<br>> >>> Then read this one to decompose ΔQ into G, S and V:<br>> Fundamentals of<br>> >> network performance engineering<br>> >>> Then read this one to get a bit more sense on what ΔQ is<br>> about:<br>> >> Introduction to ΔQ and Network Performance Science (extracts)<br>> >>><br>> >>> Then read these essays:<br>> >>><br>> >>> Foundation of Network Science<br>> >>> How to do network performance chemistry<br>> >>> How to X-ray a telecoms network<br>> >>> There is no quality in averages: IPX case study<br>> >><br>> >> All of this makes intuitively sense, but it is a bit light on<br>> how<br>> >> deltaQ is to be computed ;).<br>> >> As far as I understand it also has not much bearing on my home<br>> >> network; the only one under my control. Now, following the buffer bloat<br>> >> discussion for some years, I have internalized the idea that bandwidth<br>> >> alone does not suffice to describe the quality of my network connection.<br>> I<br>> >> think that the latency increase under load (for unrelated flows) is the<br>> >> best of all the bad single number measures of network dynamics/quality.<br>> I<br>> >> should be related to what I understood deltaQ to depend on (as packet<br>> loss<br>> >> for non real time flows will cause an increase in latency). I think<br>> that<br>> >> continuous measurements make a to n of sense for ISPs,<br>> backbone-operators,<br>> >> mobile carriers … but at home, basically, I operate as my own<br>> network<br>> >> quality monitor ;) (that is I try to pin point and debug (transient)<br>> >> anomalies).<br>> >><br>> >>><br>> >>> Martin<br>> >>><br>> >>> For fresh thinking about telecoms sign up for my free newsletter or<br>> >> visit the Geddes Think Tank.<br>> >>> LinkedIn Twitter Mobile: +44 7957 499219 Skype: mgeddes<br>> >>> Martin Geddes Consulting Ltd, Incorporated in Scotland, number<br>> SC275827<br>> >> VAT Number: 859 5634 72 Registered office: 17-19 East London Street,<br>> >> Edinburgh, EH7 4BN<br>> >>><br>> >>><br>> >>><br>> >>> On 25 July 2014 15:17, Sebastian Moeller <<a href="mailto:moeller0@gmx.de">moeller0@gmx.de</a>><br>> wrote:<br>> >>> Hi Neil,<br>> >>><br>> >>><br>> >>> On Jul 25, 2014, at 14:24 , Neil Davies <<a href="mailto:Neil.Davies@pnsol.com">Neil.Davies@pnsol.com</a>><br>> wrote:<br>> >>><br>> >>>> Rich<br>> >>>><br>> >>>> I have a deep worry over this style of single point measurement -<br>> and<br>> >> hence speed - as an appropriate measure.<br>> >>><br>> >>> But how do you propose to measure the (bottleneck) link<br>> capacity<br>> >> then? It turns out for current CPE and CMTS/DSLAM equipment one<br>> typically<br>> >> can not relay on good QoE out of the box, since typically these devices<br>> do<br>> >> not use their (largish) buffers wisely. Instead the current remedy is to<br>> >> take back control over the bottleneck link by shaping the actually sent<br>> >> traffic to stay below the hardware link capacity thereby avoiding<br>> feeling<br>> >> the consequences of the over-buffering. But to do this is is quite<br>> helpful<br>> >> to get an educated guess what the bottleneck links capacity actually is.<br>> >> And for that purpose a speediest seems useful.<br>> >>><br>> >>><br>> >>>> We know, and have evidence, that throughput/utilisation is not a<br>> good<br>> >> proxy for the network delivering suitable quality of experience. We work<br>> >> with organisation (Telco’s, large system integrators etc) where we<br>> spend a<br>> >> lot of time having to “undo” the consequences of<br>> “maximising speed”. Just<br>> >> like there is more to life than work, there is more to QoE than speed.<br>> >>>><br>> >>>> For more specific comments see inline<br>> >>>><br>> >>>> On 25 Jul 2014, at 13:09, Rich Brown<br>> <<a href="mailto:richb.hanover@gmail.com">richb.hanover@gmail.com</a>> wrote:<br>> >>>><br>> >>>>> Neil,<br>> >>>>><br>> >>>>> Thanks for the note and the observations. My thoughts:<br>> >>>>><br>> >>>>> 1) I note that <a href="http://speedof.me">speedof.me</a> does seem to overstate the speed<br>> results.<br>> >> At my home, it reports 5.98mbps down, and 638kbps up, while<br>> >> betterspeedtest.sh shows 5.49/0.61 mbps. (<a href="http://speedtest.net">speedtest.net</a> gives numbers<br>> >> similar to the <a href="http://betterspeedtest.net">betterspeedtest.net</a> script.)<br>> >>>>><br>> >>>>> 2) I think we're in agreement about the peak upload rate that<br>> you<br>> >> point out is too high. Their measurement code runs in the browser. It<br>> seems<br>> >> likely that the browser pumps out a few big packets before getting flow<br>> >> control information, thus giving the impression that they can send at a<br>> >> higher rate. This comports with the obvious decay that ramps toward the<br>> >> long-term rate.<br>> >>>><br>> >>>> I think that its simpler than that, it is measuring the rate at<br>> which<br>> >> it can push packets out the interface - its real time rate is precisely<br>> >> that - it can not be the rate being reported by the far end, it can<br>> never<br>> >> exceed the limiting link. The long term average (if it is like other<br>> speed<br>> >> testers we’ve had to look into) is being measured at the TCP/IP SDU<br>> level<br>> >> by measuring the difference in time between the first and last<br>> timestamps<br>> >> of data stream and dividing that into the total data sent. Their<br>> >> “over-estimate” is because there are packets buffered in the<br>> CPE that have<br>> >> left the machine but not arrived at the far end.<br>> >>><br>> >>> Testing from an openwrt router located at a<br>> >> high-symmetric-bandwidth location shows that <a href="http://speedof.me">speedof.me</a> does not scale<br>> >> higher than ~ 130 Mbps server to client and ~15Mbps client to server (on<br>> >> the same connection I can get 130Mbps S2C and ~80Mbps C2S, so the<br>> asymmetry<br>> >> in the <a href="http://speedof.me">speedof.me</a> results is not caused by my local environment).<br>> >>> @Rich and Dave, this probably means that for the upper end<br>> of<br>> >> fiber and cable and VDSL connections speed <a href="http://of.me">of.me</a> is not going to be a<br>> >> reliable speed measure… Side note <a href="http://www.sppedtest.net">www.sppedtest.net</a> shows ~100Mbps<br>> S2C<br>> >> and ~100Mbps C2S, so might be better suited to high-upload links...<br>> >>><br>> >>>><br>> >>>>><br>> >>>>> 3) But that long-term speed should be at or below the<br>> theoretical<br>> >> long-term rate, not above it.<br>> >>>><br>> >>>> Agreed, but in this case knowing the sync rate already defines<br>> that<br>> >> maximum.<br>> >>><br>> >>> I fully agree, but for ADSL the sync rate also contains a lot<br>> of<br>> >> encapsulation, so the maximum achievable TCP rate is at best ~90% of<br>> link<br>> >> rate. Note for cerowrt’s SQM system the link rate is exactly the<br>> right<br>> >> number to start out with at that system can take the encapsulation into<br>> >> account. But even then it is somewhat unintuitive to deduce the expected<br>> >> good-put from the link rate.<br>> >>><br>> >>>><br>> >>>>><br>> >>>>> Two experiments for you to try:<br>> >>>>><br>> >>>>> a) What does betterspeedtest.sh show? (It's in the latest<br>> CeroWrt, in<br>> >> /usr/lib/CeroWrtScripts, or get it from github:<br>> >> <a href="https://github.com/richb-hanover/CeroWrtScripts">https://github.com/richb-hanover/CeroWrtScripts</a> )<br>> >>>>><br>> >>>>> b) What does <a href="http://www.speedtest.net">www.speedtest.net</a> show?<br>> >>>>><br>> >>>>> I will add your question (about the inaccuracy) to the note<br>> that I<br>> >> want to send out to <a href="http://speedof.me">speedof.me</a> this weekend. I will also ask that they<br>> >> include min/max latency measurements to their test, and an option to<br>> send<br>> >> for > 10 seconds to minimize any effect of PowerBoost…<br>> >>><br>> >>> I think they do already, at least for the download<br>> bandwidth;<br>> >> they start with 128Kb and keep doubling the file size until a file takes<br>> >> longer than 8 seconds to transfer, they only claim to report the numbers<br>> >> from that last transferred file, so worst case with a stable link and a<br>> >> bandwidth > 16kbps ;), it has taken at least 12 seconds (4 plus 8) of<br>> >> measuring before the end of the plot, so the bandwidth of at least the<br>> last<br>> >> half of the download plot should be representative even assuming power<br>> >> boost. Caveat, I assume that power boost will not be reset by the<br>> transient<br>> >> lack of data transfer between the differently sized files (but since it<br>> >> should involve the same IPs and port# why should power boost reset<br>> itself?).<br>> >>><br>> >>> Best Regards<br>> >>> Sebastian<br>> >>><br>> >>><br>> >>><br>> >>>>><br>> >>>>> Best regards,<br>> >>>>><br>> >>>>> Rich<br>> >>>>><br>> >>>>><br>> >>>>><br>> >>>>> On Jul 25, 2014, at 5:10 AM, Neil Davies<br>> <<a href="mailto:neil.davies@pnsol.com">neil.davies@pnsol.com</a>><br>> >> wrote:<br>> >>>>><br>> >>>>>> Rich<br>> >>>>>><br>> >>>>>> You may want to check how accurate they are to start.<br>> >>>>>><br>> >>>>>> I just ran a “speed test” on my line (which I<br>> have complete control<br>> >> and visibility over the various network elements) and it reports an<br>> average<br>> >> “speed” (in the up direction) that is in excess of the<br>> capacity of the<br>> >> line, it reports the maximum rate at nearly twice the best possible rate<br>> of<br>> >> the ADSL connection.<br>> >>>>>><br>> >>>>>> Doesn’t matter how pretty it is, if its not<br>> accurate it is of no<br>> >> use. This is rather ironic as the web site claims it is the<br>> “smartest and<br>> >> most accurate”!<br>> >>>>>><br>> >>>>>> Neil<br>> >>>>>><br>> >>>>>> <speedof_me_14-07-25.png><br>> >>>>>><br>> >>>>>> PS pretty clear to me what mistake they’ve made in<br>> the measurement<br>> >> process - its to do with incorrect inference and hence missing the<br>> >> buffering effects.<br>> >>>>>><br>> >>>>>> On 20 Jul 2014, at 14:19, Rich Brown<br>> <<a href="mailto:richb.hanover@gmail.com">richb.hanover@gmail.com</a>><br>> >> wrote:<br>> >>>>>><br>> >>>>>>> Doc Searls (<br>> >><br>> <a href="http://blogs.law.harvard.edu/doc/2014/07/20/the-cliff-peronal-clouds-need-to-climb/">http://blogs.law.harvard.edu/doc/2014/07/20/the-cliff-peronal-clouds-need-to-climb/</a>)<br>> >> mentioned in passing that he uses a new speed test website. I checked it<br>> >> out, and it was very cool…<br>> >>>>>>><br>> >>>>>>> <a href="http://www.speedof.me">www.speedof.me</a> is an all-HTML5 website that seems to<br>> make accurate<br>> >> measurements of the up and download speeds of your internet connection.<br>> >> It’s also very attractive, and the real-time plots of the speed<br>> show<br>> >> interesting info. (screen shot at: <a href="http://richb-hanover.com/speedof-me/">http://richb-hanover.com/speedof-me/</a>)<br>> >>>>>>><br>> >>>>>>> Now if we could get them to a) allow longer/bigger<br>> tests to<br>> >> circumvent PowerBoost, and b) include a latency measurement so people<br>> could<br>> >> point out their bufferbloated equipment.<br>> >>>>>>><br>> >>>>>>> I'm going to send them a note. Anything else I should<br>> add?<br>> >>>>>>><br>> >>>>>>> Rich<br>> >>>>>>> _______________________________________________<br>> >>>>>>> Bloat mailing list<br>> >>>>>>> <a href="mailto:Bloat@lists.bufferbloat.net">Bloat@lists.bufferbloat.net</a><br>> >>>>>>> <a href="https://lists.bufferbloat.net/listinfo/bloat">https://lists.bufferbloat.net/listinfo/bloat</a><br>> >>>>>><br>> >>>>><br>> >>>><br>> >>>> _______________________________________________<br>> >>>> Bloat mailing list<br>> >>>> <a href="mailto:Bloat@lists.bufferbloat.net">Bloat@lists.bufferbloat.net</a><br>> >>>> <a href="https://lists.bufferbloat.net/listinfo/bloat">https://lists.bufferbloat.net/listinfo/bloat</a><br>> >>><br>> >>> _______________________________________________<br>> >>> Bloat mailing list<br>> >>> <a href="mailto:Bloat@lists.bufferbloat.net">Bloat@lists.bufferbloat.net</a><br>> >>> <a href="https://lists.bufferbloat.net/listinfo/bloat">https://lists.bufferbloat.net/listinfo/bloat</a><br>> >>><br>> >><br>> >><br>> >_______________________________________________<br>> Cerowrt-devel mailing list<br>> <a href="mailto:Cerowrt-devel@lists.bufferbloat.net">Cerowrt-devel@lists.bufferbloat.net</a><br>> <a href="https://lists.bufferbloat.net/listinfo/cerowrt-devel">https://lists.bufferbloat.net/listinfo/cerowrt-devel</a><br>> _______________________________________________<br>> Cerowrt-devel mailing list<br>> <a href="mailto:Cerowrt-devel@lists.bufferbloat.net">Cerowrt-devel@lists.bufferbloat.net</a><br>> <a href="https://lists.bufferbloat.net/listinfo/cerowrt-devel">https://lists.bufferbloat.net/listinfo/cerowrt-devel</a><br>> </div>
</div></font></blockquote></div><br></div></body></html>