Ok thanks

I think I will stay away from the quagmire of rating ISPs on buffer bloat.

And first try to boil any bloat measurement down to an easy to understand
letter grade
between A+ and F, in a way that you guys think stands inspection of
individual
results.

On Wed, Apr 29, 2015 at 11:39 AM, Jonathan Morton <chromatix99@gmail.com>
wrote:

> These are pretty good questions, actually. But as pointed out, when you
> want to start ranking, it's important to distinguish the performance of the
> ISP itself from equipment under the subscriber's control, which itself
> might be configured to hide faults in the ISP.
>
> Statistics is a hard subject. If you feel at all confused by what I
> describe below, it might be a good idea to sit down with a real expert. The
> basic calculations are not difficult; it's knowing WHAT to calculate and
> what it means once you've done it.
>
> Notably, the upload queue usually depends much more on the CPE than on
> anything the ISP controls directly. I don't think there are that many ISPs
> left which absolutely insist on using their own modem, but most will supply
> a preferred, preconfigured model on request, and many users will accept
> that, not knowing better. Unfortunately, I can't think of an easy, robust
> way to detect what CPE is in use our how it has been configured, so it's
> hard to control for it statistically.
>
> In general, downstream queuing is much more under the ISP's control. To
> account for the (growing?) subset of users who apply ingress shaping, you
> could look at the upper percentiles of latency, since usually ingress
> shaping will improve matters. Caveat: it's also possible for ingress
> shaping to make things worse, either through accidental misconfiguration or
> even maliciously, so don't just take the peak value.
>
> Initially I suggest you present a histogram of the results that fall into
> particular grades. That'll help you get a feel for the statistics, and it's
> easy to pick out a modal value by eye. Beware of trying to calculate a
> modal value simplistically, since the histogram might not show a simple
> mode if results tend to straddle two adjacent grades. (This is why the mode
> is the least used of the three basic averages; it's hard to calculate it
> such that it's reliably useful, even though it's often intuitively useful.)
>
> I also suggest you add a basic question to the user: are you using a
> router with QoS features turned on (yes/no/dunno), with dunno as the
> default. That'll give you a way to point out that most of the lower latency
> results are probably a result of this. You could draw stacked or adjacent
> histograms with this information colour coded.
>
> As for the single numbers to plug into the formula, this also requires
> some care.
>
> For the idle/baseline, I suggest using only the samples from before the
> bandwidth tests begin, rather than also including those from between and
> afterwards. Then you should use the harmonic mean on these, to get a value
> biased on the low end.
>
> For each of the load sets, you want a value biased high. Taking the 90th
> percentile might be a reasonable approach here; it'll discard outliers and
> brief transients that a good AQM acting alone (ie. without FQ) might leave,
> but should still expose the sorts of obvious problems that we're trying to
> tackle.
>
> I do think that the idle latency and the total loaded latency can usefully
> be reported as frequencies. The total loaded latency can be taken to be the
> idle latency plus the induced download latency plus the induced upload
> latency, as an approximation of the latency when both directions are loaded
> at once.
>
> As for ranking ISPs overall... this is hard to do in a way that's
> perceived to be fair. My recommendation on this front would be to allow
> your users to select criteria with weights. Some might consider download
> and/or upload speed to be important as well as latency, while others might
> see a choice between overall latency (important for games) and jitter
> (important for VoIP). This makes it harder to claim that you're personally
> presenting a controversial opinion on relative merits.
>
> - Jonathan Morton
>