From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-x22e.google.com (mail-ig0-x22e.google.com [IPv6:2607:f8b0:4001:c05::22e]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id CC10321F1F0 for ; Tue, 28 Apr 2015 19:49:42 -0700 (PDT) Received: by igbyr2 with SMTP id yr2so108117990igb.0 for ; Tue, 28 Apr 2015 19:49:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=o3vMTX9EmyPa1JF2fMxhBmny5sV6ZEBSn2DflfKNKRY=; b=tGPjL2WqPKUxaqz7vd4lDl6Jpvlmx5RBEvLArlitbiQ072MkqeNwsGJ1KvvTtiOn9i RRC24J6SbIA29yvPZgNdMs67yc7jR8Y53D+vvSOmPi1tthyC7t9OQFEy8d18f7FagTUc XEE+G4KuWmwIZaxk6GJ6eBapOvKVaE3ZNigpwpdOX4DrUiExdlFA3XxCtS/7LhT0xrWT WIn6QxT6xK4or7cl2E3ykLwz9oKrjfGbifNdLUGYOSsLsuk2zZG/zce09Z9a8DDdnHTS ayUPMppkyzKG+2nfErlQJBO2M6shJQIRco4jIuWKDmc/CUVicf8fPZB94no/p21p8tst vjcA== MIME-Version: 1.0 X-Received: by 10.50.64.244 with SMTP id r20mr25054909igs.48.1430275781679; Tue, 28 Apr 2015 19:49:41 -0700 (PDT) Sender: justinbeech@gmail.com Received: by 10.50.107.42 with HTTP; Tue, 28 Apr 2015 19:49:41 -0700 (PDT) In-Reply-To: References: Date: Wed, 29 Apr 2015 12:49:41 +1000 X-Google-Sender-Auth: NMqkRtQTTEdPV1FEmrSoNtQCayc Message-ID: From: jb To: Jonathan Morton , bloat Content-Type: multipart/alternative; boundary=047d7bea3d22e5e1210514d407fe Subject: Re: [Bloat] extremely good dslreports result for bufferbloat on free.fr X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Apr 2015 02:50:11 -0000 --047d7bea3d22e5e1210514d407fe Content-Type: text/plain; charset=UTF-8 Ok thanks I think I will stay away from the quagmire of rating ISPs on buffer bloat. And first try to boil any bloat measurement down to an easy to understand letter grade between A+ and F, in a way that you guys think stands inspection of individual results. On Wed, Apr 29, 2015 at 11:39 AM, Jonathan Morton wrote: > These are pretty good questions, actually. But as pointed out, when you > want to start ranking, it's important to distinguish the performance of the > ISP itself from equipment under the subscriber's control, which itself > might be configured to hide faults in the ISP. > > Statistics is a hard subject. If you feel at all confused by what I > describe below, it might be a good idea to sit down with a real expert. The > basic calculations are not difficult; it's knowing WHAT to calculate and > what it means once you've done it. > > Notably, the upload queue usually depends much more on the CPE than on > anything the ISP controls directly. I don't think there are that many ISPs > left which absolutely insist on using their own modem, but most will supply > a preferred, preconfigured model on request, and many users will accept > that, not knowing better. Unfortunately, I can't think of an easy, robust > way to detect what CPE is in use our how it has been configured, so it's > hard to control for it statistically. > > In general, downstream queuing is much more under the ISP's control. To > account for the (growing?) subset of users who apply ingress shaping, you > could look at the upper percentiles of latency, since usually ingress > shaping will improve matters. Caveat: it's also possible for ingress > shaping to make things worse, either through accidental misconfiguration or > even maliciously, so don't just take the peak value. > > Initially I suggest you present a histogram of the results that fall into > particular grades. That'll help you get a feel for the statistics, and it's > easy to pick out a modal value by eye. Beware of trying to calculate a > modal value simplistically, since the histogram might not show a simple > mode if results tend to straddle two adjacent grades. (This is why the mode > is the least used of the three basic averages; it's hard to calculate it > such that it's reliably useful, even though it's often intuitively useful.) > > I also suggest you add a basic question to the user: are you using a > router with QoS features turned on (yes/no/dunno), with dunno as the > default. That'll give you a way to point out that most of the lower latency > results are probably a result of this. You could draw stacked or adjacent > histograms with this information colour coded. > > As for the single numbers to plug into the formula, this also requires > some care. > > For the idle/baseline, I suggest using only the samples from before the > bandwidth tests begin, rather than also including those from between and > afterwards. Then you should use the harmonic mean on these, to get a value > biased on the low end. > > For each of the load sets, you want a value biased high. Taking the 90th > percentile might be a reasonable approach here; it'll discard outliers and > brief transients that a good AQM acting alone (ie. without FQ) might leave, > but should still expose the sorts of obvious problems that we're trying to > tackle. > > I do think that the idle latency and the total loaded latency can usefully > be reported as frequencies. The total loaded latency can be taken to be the > idle latency plus the induced download latency plus the induced upload > latency, as an approximation of the latency when both directions are loaded > at once. > > As for ranking ISPs overall... this is hard to do in a way that's > perceived to be fair. My recommendation on this front would be to allow > your users to select criteria with weights. Some might consider download > and/or upload speed to be important as well as latency, while others might > see a choice between overall latency (important for games) and jitter > (important for VoIP). This makes it harder to claim that you're personally > presenting a controversial opinion on relative merits. > > - Jonathan Morton > --047d7bea3d22e5e1210514d407fe Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Ok thanks

I think I will stay away from= the quagmire of rating ISPs on buffer bloat.

And first = try to boil any bloat measurement down to an easy to understand letter grad= e
between A+ and F, in a way that you guys think stands inspectio= n of individual
results.

On Wed, Apr 29, 2015 at 11:39 AM, Jonath= an Morton <chromatix99@gmail.com> wrote:

These are pretty good questions, actually= . But as pointed out, when you want to start ranking, it's important to= distinguish the performance of the ISP itself from equipment under the sub= scriber's control, which itself might be configured to hide faults in t= he ISP.

Statistics is a hard subject. If you feel at all confused by= what I describe below, it might be a good idea to sit down with a real exp= ert. The basic calculations are not difficult; it's knowing WHAT to cal= culate and what it means once you've done it.

Notably, the upload queue usually depends much more on the C= PE than on anything the ISP controls directly. I don't think there are = that many ISPs left which absolutely insist on using their own modem, but m= ost will supply a preferred, preconfigured model on request, and many users= will accept that, not knowing better. Unfortunately, I can't think of = an easy, robust way to detect what CPE is in use our how it has been config= ured, so it's hard to control for it statistically.

In general, downstream queuing is much more under the ISP= 9;s control. To account for the (growing?) subset of users who apply ingres= s shaping, you could look at the upper percentiles of latency, since usuall= y ingress shaping will improve matters. Caveat: it's also possible for = ingress shaping to make things worse, either through accidental misconfigur= ation or even maliciously, so don't just take the peak value.

Initially I suggest you present a histogram of the results t= hat fall into particular grades. That'll help you get a feel for the st= atistics, and it's easy to pick out a modal value by eye. Beware of try= ing to calculate a modal value simplistically, since the histogram might no= t show a simple mode if results tend to straddle two adjacent grades. (This= is why the mode is the least used of the three basic averages; it's ha= rd to calculate it such that it's reliably useful, even though it's= often intuitively useful.)

I also suggest you add a basic question to the user: are you= using a router with QoS features turned on (yes/no/dunno), with dunno as t= he default. That'll give you a way to point out that most of the lower = latency results are probably a result of this. You could draw stacked or ad= jacent histograms with this information colour coded.

As for the single numbers to plug into the formula, this als= o requires some care.

For the idle/baseline, I suggest using only the samples from= before the bandwidth tests begin, rather than also including those from be= tween and afterwards. Then you should use the harmonic mean on these, to ge= t a value biased on the low end.

For each of the load sets, you want a value biased high. Tak= ing the 90th percentile might be a reasonable approach here; it'll disc= ard outliers and brief transients that a good AQM acting alone (ie. without= FQ) might leave, but should still expose the sorts of obvious problems tha= t we're trying to tackle.

I do think that the idle latency and the total loaded latenc= y can usefully be reported as frequencies. The total loaded latency can be = taken to be the idle latency plus the induced download latency plus the ind= uced upload latency, as an approximation of the latency when both direction= s are loaded at once.

As for ranking ISPs overall... this is hard to do in a way t= hat's perceived to be fair. My recommendation on this front would be to= allow your users to select criteria with weights. Some might consider down= load and/or upload speed to be important as well as latency, while others m= ight see a choice between overall latency (important for games) and jitter = (important for VoIP). This makes it harder to claim that you're persona= lly presenting a controversial opinion on relative merits.

- Jonathan Morton


--047d7bea3d22e5e1210514d407fe--