From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 3907521F5FC for ; Sat, 26 Jul 2014 06:03:06 -0700 (PDT) Received: from hms-beagle.lan ([134.2.89.70]) by mail.gmx.com (mrgmx102) with ESMTPSA (Nemesis) id 0Ldspr-1Wjd141vNr-00j15z; Sat, 26 Jul 2014 15:02:45 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) From: Sebastian Moeller In-Reply-To: Date: Sat, 26 Jul 2014 15:02:44 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <03292B76-5273-4912-BB18-90E95C16A9F5@pnsol.com> <66FF8435-C8A5-4596-B43A-EC12D537D49E@gmx.de> <41DF4003-BAE8-4794-BEDF-EF2385F03685@gmx.de> <1406326625.225312181@apps.rackspace.com> To: David Lang X-Mailer: Apple Mail (2.1878.6) X-Provags-ID: V03:K0:Lq95JT6B/X4OojFeot7ovUP93F+9uEsvQ4On8ryUdAOztZd2ayf Z8uAsgUcd13avXLWbtx0aa2FRd2iCcScMOFaxueFAqioxrYlpxYVbxYwjDhYCVF8mDPWXMC ewJBDtSK07x6NzB7CF67GS9A5V+Zsa5+4cTYFRxmj9L3Rjg1Pq6JAUsqy+G7HXIXFyd7u28 aePcrRAG4KCLc3yotH5rw== Cc: cerowrt-devel , bloat Subject: Re: [Cerowrt-devel] [Bloat] Check out www.speedof.me - no Flash X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Jul 2014 13:03:07 -0000 Hi David, On Jul 26, 2014, at 01:26 , David Lang wrote: > But I think that what we are seeing from teh results of the = bufferbloat work is that a properly configured network doesn't degrade = badly as it gets busy. >=20 > Individual services will degrade as they need more bandwidth than is = available, but that sort of degredation is easy for the user to = understand. >=20 > The current status-quo is where good throughput at 80% utilization may = be 80Mb, at 90% utilization it may be 85Mb, at 95% utilization it is = 60Mb, and at 100% utilization it pulses between 10Mb and 80Mb averaging = around 20Mb and latency goes from 10ms to multiple seconds over this = range. >=20 > With BQL and fw_codel, 80% utilization would still be 80Mb, 90% = utilization would be 89Mb, 95% utilization would be 93Mb with latency = only going to 20ms >=20 > so there is a real problem to solve in the current status-quo, and the = question is if there is a way to quantify the problem and test for it in = ways that are repeatable, meaningful and understandable. >=20 > This is a place to avoid letting perfect be the enemy of good enough. >=20 > If you ask even relatively technical people about the quality of a = network connection, they will talk to you about bandwidth and latency. >=20 > But if you talk to a networking expert, they don't even mention that, = they talk about signal strength, waveform distortion, bit error rates, = error correction mechanisms, signal regeneration, and probably many = other things that I don't know enough to even mention :-) >=20 >=20 > Everyone is already measuring peak bandwidth today, and that is always = going to be an important factor, so it will stay around. >=20 > So we need to show the degredation of the network, and I think that = either ping(loaded)-ping(unloaded) or ping(loaded)/ping(unloaded) will = give us meaningful numbers that people can understand and talk about, = while still being meaningful in the real world. Maybe we should follow Neil and Martin=E2=80=99s lead and = consider either ping(unloaded)-ping(loaded) or = ping(unloaded)/ping(loaded) and call the whole thing quality estimator = or factor (as negative quality or a factor < 0 intuitively shows a = degradation). Also my bet is on the difference not on the ratio, why = should people with bad latency to begin with (satellite?) be more = tolerant to further degradation? I would assume that on a high latency = link if at all the =E2=80=9Cbudget=E2=80=9D for further degradation = might be smaller than on a low latency link (reasoning: there might be a = fixed latency budget for acceptable latency for voip). >=20 > which of the two is more useful is something that we would need to get = a bunch of people with different speed lines to report and see which is = affected less by line differences and distance to target. Or make sure we always measure against the closest target (which = with satellite might still be far away)? Best Regards sebastian >=20 > David Lang >=20 > On Fri, 25 Jul 2014, dpreed@reed.com wrote: >=20 >> I think what is being discussed is "how to measure the quality of one = endpoint's experience of the entire Internet over all time or over a = specific interval of time". >> Yet the systems that are built on top of the Internet transport do = not have any kind of uniform dependence on the underlying transport = behavior in terms of their quality. Even something like VoIP's quality = as experienced by two humans talking over it has a dependency on the = Internet's behavior, but one that is hardly simple. >> As an extreme, if one endpoint experiences a direct DDoS attack or is = indirectly affected by one somewhere in the path, the quality of the = experience might be dramatically reduced. >> So any attempt to define a delta-Q that has meaning in terms of user = experience appears pointless and even silly - the endpoint experience is = adequate under a very wide variety of conditions, but degrades terribly = under certain kinds of conditions. >> As a different point, let's assume that the last-mile is 80% = utilized, but the latency variation in that utilization is not larger = than 50 msec. This is a feasible-to-imagine operating point, but it = requires a certain degreee of tight control that may be very hard to = achieve over thousands of independent application services through that = point, so its feasibility is contingent on lots of factors. Then if the = 20% capacity is far larger than 64 kb/sec we know that toll-quality = audio can be produced with a small endpoint "jitter buffer". There's no = "delta-Q" there at all - quality is great. >> So the point is: a single number or even a single "morphism" = (whatever that is) to a specific algebraic domain element (a mapping to = a semi-lattice with non-Abelian operators?) does not allow one to define = a "measure" of an endpoint of the Internet that can be used to compute = "quality" of all applications. >> Or in purely non-abstract terms: if there were a delta-Q it would be = useless for most network applications, but might be useful for a single = network application. >> So I submit that delta-Q is a *metaphor* and not a very useful one at = that. It's probably as useful as providing a "funkiness" measure for an = Internat access point. We can certainly talk about and make claims = about the relative "funkiness" of different connections and different = providers. We might even claim that cable providers make funkier = network providers than cellular providers. >> But to what end? >>=20 >>=20 >> On Friday, July 25, 2014 5:13pm, "David Lang" said: >>=20 >>=20 >>=20 >>> On Fri, 25 Jul 2014, Martin Geddes wrote: >>> > So what is =CE=94Q and how do you "compute" it (to the extent it = is a >>> "computed" >>> > thing)? >>> don't try to reduce it to a single number, we have two numbers that = seem to >>> matter >>> 1. throughput (each direction) >>> 2. latency under load >>> Currently the speed test sites report throughput in each direction = and ping time >>> while not under load >>> If they could just add a ping time under load measurement, then we = could talk >>> meaningfully about either the delta or ratio of the ping times as = the >>> "bufferbloat factor" >>> no, it wouldn't account for absolutly every nuance, but it would = come pretty >>> close. >>> If a connection has good throughput and a low bufferbloat factor, it = should be >>> good for any type of use. >>> If it has good throughput, but a horrid bufferbloat factor, then you = need to >>> artifically limit your traffic to stay clear of saturating the = bandwith >>> (sacraficing throughput) >>> David Lang >>> > Starting point: the only observable effect of a network is to lose = and >>> > delay data -- i.e. to "attenuate quality" by adding the toxic = effects of >>> > time to distributed computations. =CE=94Q is a *morphism* that = relates the >>> > "quality attenuation" that the network imposes to the application >>> > performance, and describes the trading spaces at all intermediate = layers of >>> > abstraction. It is shown in the attached graphic. >>> > >>> > Critically, it frames quality as something that can only be lost >>> > ("attenuated"), both by the network and the application. = Additionally, it >>> > is stochastic, and works with random variables and distributions. >>> > >>> > At its most concrete level, it is the individual impairment = encountered by >>> > every packet when the network in operation. But we don't want to = have to >>> > track every packet - 1:1 scale maps are pretty useless. So we need = to >>> > abstract that in order to create a model that has value. >>> > >>> > Next abstraction: an improper random variable. This unifies loss = and delay >>> > into a single stochastic object. >>> > Next abstraction: received transport, which is a CDF where we are >>> > interested in the properties of the "tail". >>> > >>> > Next abstraction, that joins network performance and application = QoE (as >>> > relates to performance): relate the CDF to the application through = a >>> > Quality Transport Agreement. This "stochastic contract" is both = necessary >>> > and sufficient to deliver the application outcome. >>> > >>> > Next concretisation towards QoE: offered load of demand, as a CDF. >>> > Next concretisation towards QoE: breach hazard metric, which = abstracts the >>> > application performance. Indicates the likelihood of the QTA = contract being >>> > broken, and how badly. >>> > Final concretisation: the individual application performance = encountered by >>> > every user. Again, a 1:1 map that isn't very helpful. >>> > >>> > So as you can see, it's about as far away from a single point = average >>> > metric as you can possibly get. A far richer model is required in = order to >>> > achieve robust performance engineering. >>> > >>> > It is "computed" using multi-point measurements to capture the >>> > distribution. The G/S/V charts you see are based on processing = that data to >>> > account for various issues, including clock skew. >>> > >>> > I hope that helps. We need to document more of this in public, = which is an >>> > ongoing process. >>> > >>> > Martin >>> > >>> > On 25 July 2014 16:58, Sebastian Moeller wrote: >>> > >>> >> Hi Martin, >>> >> >>> >> thanks for the pointers, >>> >> >>> >> >>> >> On Jul 25, 2014, at 16:25 , Martin Geddes >>> wrote: >>> >> >>> >>> You may find the following useful background reading on the = state of >>> the >>> >> art in network measurement, and a primer on =CE=94Q (which is the >>> property we >>> >> wish to measure). >>> >>> >>> >>> First, start with this presentation: Network performance >>> optimisation >>> >> using high-fidelity measures >>> >>> Then read this one to decompose =CE=94Q into G, S and V: >>> Fundamentals of >>> >> network performance engineering >>> >>> Then read this one to get a bit more sense on what =CE=94Q is >>> about: >>> >> Introduction to =CE=94Q and Network Performance Science = (extracts) >>> >>> >>> >>> Then read these essays: >>> >>> >>> >>> Foundation of Network Science >>> >>> How to do network performance chemistry >>> >>> How to X-ray a telecoms network >>> >>> There is no quality in averages: IPX case study >>> >> >>> >> All of this makes intuitively sense, but it is a bit light on >>> how >>> >> deltaQ is to be computed ;). >>> >> As far as I understand it also has not much bearing on my home >>> >> network; the only one under my control. Now, following the buffer = bloat >>> >> discussion for some years, I have internalized the idea that = bandwidth >>> >> alone does not suffice to describe the quality of my network = connection. >>> I >>> >> think that the latency increase under load (for unrelated flows) = is the >>> >> best of all the bad single number measures of network = dynamics/quality. >>> I >>> >> should be related to what I understood deltaQ to depend on (as = packet >>> loss >>> >> for non real time flows will cause an increase in latency). I = think >>> that >>> >> continuous measurements make a to n of sense for ISPs, >>> backbone-operators, >>> >> mobile carriers =E2=80=A6 but at home, basically, I operate as my = own >>> network >>> >> quality monitor ;) (that is I try to pin point and debug = (transient) >>> >> anomalies). >>> >> >>> >>> >>> >>> Martin >>> >>> >>> >>> For fresh thinking about telecoms sign up for my free newsletter = or >>> >> visit the Geddes Think Tank. >>> >>> LinkedIn Twitter Mobile: +44 7957 499219 Skype: mgeddes >>> >>> Martin Geddes Consulting Ltd, Incorporated in Scotland, number >>> SC275827 >>> >> VAT Number: 859 5634 72 Registered office: 17-19 East London = Street, >>> >> Edinburgh, EH7 4BN >>> >>> >>> >>> >>> >>> >>> >>> On 25 July 2014 15:17, Sebastian Moeller >>> wrote: >>> >>> Hi Neil, >>> >>> >>> >>> >>> >>> On Jul 25, 2014, at 14:24 , Neil Davies >>> wrote: >>> >>> >>> >>>> Rich >>> >>>> >>> >>>> I have a deep worry over this style of single point measurement = - >>> and >>> >> hence speed - as an appropriate measure. >>> >>> >>> >>> But how do you propose to measure the (bottleneck) link >>> capacity >>> >> then? It turns out for current CPE and CMTS/DSLAM equipment one >>> typically >>> >> can not relay on good QoE out of the box, since typically these = devices >>> do >>> >> not use their (largish) buffers wisely. Instead the current = remedy is to >>> >> take back control over the bottleneck link by shaping the = actually sent >>> >> traffic to stay below the hardware link capacity thereby avoiding >>> feeling >>> >> the consequences of the over-buffering. But to do this is is = quite >>> helpful >>> >> to get an educated guess what the bottleneck links capacity = actually is. >>> >> And for that purpose a speediest seems useful. >>> >>> >>> >>> >>> >>>> We know, and have evidence, that throughput/utilisation is not = a >>> good >>> >> proxy for the network delivering suitable quality of experience. = We work >>> >> with organisation (Telco=E2=80=99s, large system integrators etc) = where we >>> spend a >>> >> lot of time having to =E2=80=9Cundo=E2=80=9D the consequences of >>> =E2=80=9Cmaximising speed=E2=80=9D. Just >>> >> like there is more to life than work, there is more to QoE than = speed. >>> >>>> >>> >>>> For more specific comments see inline >>> >>>> >>> >>>> On 25 Jul 2014, at 13:09, Rich Brown >>> wrote: >>> >>>> >>> >>>>> Neil, >>> >>>>> >>> >>>>> Thanks for the note and the observations. My thoughts: >>> >>>>> >>> >>>>> 1) I note that speedof.me does seem to overstate the speed >>> results. >>> >> At my home, it reports 5.98mbps down, and 638kbps up, while >>> >> betterspeedtest.sh shows 5.49/0.61 mbps. (speedtest.net gives = numbers >>> >> similar to the betterspeedtest.net script.) >>> >>>>> >>> >>>>> 2) I think we're in agreement about the peak upload rate that >>> you >>> >> point out is too high. Their measurement code runs in the = browser. It >>> seems >>> >> likely that the browser pumps out a few big packets before = getting flow >>> >> control information, thus giving the impression that they can = send at a >>> >> higher rate. This comports with the obvious decay that ramps = toward the >>> >> long-term rate. >>> >>>> >>> >>>> I think that its simpler than that, it is measuring the rate at >>> which >>> >> it can push packets out the interface - its real time rate is = precisely >>> >> that - it can not be the rate being reported by the far end, it = can >>> never >>> >> exceed the limiting link. The long term average (if it is like = other >>> speed >>> >> testers we=E2=80=99ve had to look into) is being measured at the = TCP/IP SDU >>> level >>> >> by measuring the difference in time between the first and last >>> timestamps >>> >> of data stream and dividing that into the total data sent. Their >>> >> =E2=80=9Cover-estimate=E2=80=9D is because there are packets = buffered in the >>> CPE that have >>> >> left the machine but not arrived at the far end. >>> >>> >>> >>> Testing from an openwrt router located at a >>> >> high-symmetric-bandwidth location shows that speedof.me does not = scale >>> >> higher than ~ 130 Mbps server to client and ~15Mbps client to = server (on >>> >> the same connection I can get 130Mbps S2C and ~80Mbps C2S, so the >>> asymmetry >>> >> in the speedof.me results is not caused by my local environment). >>> >>> @Rich and Dave, this probably means that for the upper end >>> of >>> >> fiber and cable and VDSL connections speed of.me is not going to = be a >>> >> reliable speed measure=E2=80=A6 Side note www.sppedtest.net shows = ~100Mbps >>> S2C >>> >> and ~100Mbps C2S, so might be better suited to high-upload = links... >>> >>> >>> >>>> >>> >>>>> >>> >>>>> 3) But that long-term speed should be at or below the >>> theoretical >>> >> long-term rate, not above it. >>> >>>> >>> >>>> Agreed, but in this case knowing the sync rate already defines >>> that >>> >> maximum. >>> >>> >>> >>> I fully agree, but for ADSL the sync rate also contains a lot >>> of >>> >> encapsulation, so the maximum achievable TCP rate is at best ~90% = of >>> link >>> >> rate. Note for cerowrt=E2=80=99s SQM system the link rate is = exactly the >>> right >>> >> number to start out with at that system can take the = encapsulation into >>> >> account. But even then it is somewhat unintuitive to deduce the = expected >>> >> good-put from the link rate. >>> >>> >>> >>>> >>> >>>>> >>> >>>>> Two experiments for you to try: >>> >>>>> >>> >>>>> a) What does betterspeedtest.sh show? (It's in the latest >>> CeroWrt, in >>> >> /usr/lib/CeroWrtScripts, or get it from github: >>> >> https://github.com/richb-hanover/CeroWrtScripts ) >>> >>>>> >>> >>>>> b) What does www.speedtest.net show? >>> >>>>> >>> >>>>> I will add your question (about the inaccuracy) to the note >>> that I >>> >> want to send out to speedof.me this weekend. I will also ask that = they >>> >> include min/max latency measurements to their test, and an option = to >>> send >>> >> for > 10 seconds to minimize any effect of PowerBoost=E2=80=A6 >>> >>> >>> >>> I think they do already, at least for the download >>> bandwidth; >>> >> they start with 128Kb and keep doubling the file size until a = file takes >>> >> longer than 8 seconds to transfer, they only claim to report the = numbers >>> >> from that last transferred file, so worst case with a stable link = and a >>> >> bandwidth > 16kbps ;), it has taken at least 12 seconds (4 plus = 8) of >>> >> measuring before the end of the plot, so the bandwidth of at = least the >>> last >>> >> half of the download plot should be representative even assuming = power >>> >> boost. Caveat, I assume that power boost will not be reset by the >>> transient >>> >> lack of data transfer between the differently sized files (but = since it >>> >> should involve the same IPs and port# why should power boost = reset >>> itself?). >>> >>> >>> >>> Best Regards >>> >>> Sebastian >>> >>> >>> >>> >>> >>> >>> >>>>> >>> >>>>> Best regards, >>> >>>>> >>> >>>>> Rich >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> On Jul 25, 2014, at 5:10 AM, Neil Davies >>> >>> >> wrote: >>> >>>>> >>> >>>>>> Rich >>> >>>>>> >>> >>>>>> You may want to check how accurate they are to start. >>> >>>>>> >>> >>>>>> I just ran a =E2=80=9Cspeed test=E2=80=9D on my line (which I >>> have complete control >>> >> and visibility over the various network elements) and it reports = an >>> average >>> >> =E2=80=9Cspeed=E2=80=9D (in the up direction) that is in excess = of the >>> capacity of the >>> >> line, it reports the maximum rate at nearly twice the best = possible rate >>> of >>> >> the ADSL connection. >>> >>>>>> >>> >>>>>> Doesn=E2=80=99t matter how pretty it is, if its not >>> accurate it is of no >>> >> use. This is rather ironic as the web site claims it is the >>> =E2=80=9Csmartest and >>> >> most accurate=E2=80=9D! >>> >>>>>> >>> >>>>>> Neil >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> PS pretty clear to me what mistake they=E2=80=99ve made in >>> the measurement >>> >> process - its to do with incorrect inference and hence missing = the >>> >> buffering effects. >>> >>>>>> >>> >>>>>> On 20 Jul 2014, at 14:19, Rich Brown >>> >>> >> wrote: >>> >>>>>> >>> >>>>>>> Doc Searls ( >>> >> >>> = http://blogs.law.harvard.edu/doc/2014/07/20/the-cliff-peronal-clouds-need-= to-climb/) >>> >> mentioned in passing that he uses a new speed test website. I = checked it >>> >> out, and it was very cool=E2=80=A6 >>> >>>>>>> >>> >>>>>>> www.speedof.me is an all-HTML5 website that seems to >>> make accurate >>> >> measurements of the up and download speeds of your internet = connection. >>> >> It=E2=80=99s also very attractive, and the real-time plots of the = speed >>> show >>> >> interesting info. (screen shot at: = http://richb-hanover.com/speedof-me/) >>> >>>>>>> >>> >>>>>>> Now if we could get them to a) allow longer/bigger >>> tests to >>> >> circumvent PowerBoost, and b) include a latency measurement so = people >>> could >>> >> point out their bufferbloated equipment. >>> >>>>>>> >>> >>>>>>> I'm going to send them a note. Anything else I should >>> add? >>> >>>>>>> >>> >>>>>>> Rich >>> >>>>>>> _______________________________________________ >>> >>>>>>> Bloat mailing list >>> >>>>>>> Bloat@lists.bufferbloat.net >>> >>>>>>> https://lists.bufferbloat.net/listinfo/bloat >>> >>>>>> >>> >>>>> >>> >>>> >>> >>>> _______________________________________________ >>> >>>> Bloat mailing list >>> >>>> Bloat@lists.bufferbloat.net >>> >>>> https://lists.bufferbloat.net/listinfo/bloat >>> >>> >>> >>> _______________________________________________ >>> >>> Bloat mailing list >>> >>> Bloat@lists.bufferbloat.net >>> >>> https://lists.bufferbloat.net/listinfo/bloat >>> >>> >>> >> >>> >> >>> >_______________________________________________ >>> Cerowrt-devel mailing list >>> Cerowrt-devel@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/cerowrt-devel >>> _______________________________________________ >>> Cerowrt-devel mailing list >>> Cerowrt-devel@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/cerowrt-devel > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat