Re: [Bloat] extremely good dslreports result for bufferbloat on free.fr

[Jonathan Morton <chromatix99@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3718 bytes --]

These are pretty good questions, actually. But as pointed out, when you
want to start ranking, it's important to distinguish the performance of the
ISP itself from equipment under the subscriber's control, which itself
might be configured to hide faults in the ISP.

Statistics is a hard subject. If you feel at all confused by what I
describe below, it might be a good idea to sit down with a real expert. The
basic calculations are not difficult; it's knowing WHAT to calculate and
what it means once you've done it.

Notably, the upload queue usually depends much more on the CPE than on
anything the ISP controls directly. I don't think there are that many ISPs
left which absolutely insist on using their own modem, but most will supply
a preferred, preconfigured model on request, and many users will accept
that, not knowing better. Unfortunately, I can't think of an easy, robust
way to detect what CPE is in use our how it has been configured, so it's
hard to control for it statistically.

In general, downstream queuing is much more under the ISP's control. To
account for the (growing?) subset of users who apply ingress shaping, you
could look at the upper percentiles of latency, since usually ingress
shaping will improve matters. Caveat: it's also possible for ingress
shaping to make things worse, either through accidental misconfiguration or
even maliciously, so don't just take the peak value.

Initially I suggest you present a histogram of the results that fall into
particular grades. That'll help you get a feel for the statistics, and it's
easy to pick out a modal value by eye. Beware of trying to calculate a
modal value simplistically, since the histogram might not show a simple
mode if results tend to straddle two adjacent grades. (This is why the mode
is the least used of the three basic averages; it's hard to calculate it
such that it's reliably useful, even though it's often intuitively useful.)

I also suggest you add a basic question to the user: are you using a router
with QoS features turned on (yes/no/dunno), with dunno as the default.
That'll give you a way to point out that most of the lower latency results
are probably a result of this. You could draw stacked or adjacent
histograms with this information colour coded.

As for the single numbers to plug into the formula, this also requires some
care.

For the idle/baseline, I suggest using only the samples from before the
bandwidth tests begin, rather than also including those from between and
afterwards. Then you should use the harmonic mean on these, to get a value
biased on the low end.

For each of the load sets, you want a value biased high. Taking the 90th
percentile might be a reasonable approach here; it'll discard outliers and
brief transients that a good AQM acting alone (ie. without FQ) might leave,
but should still expose the sorts of obvious problems that we're trying to
tackle.

I do think that the idle latency and the total loaded latency can usefully
be reported as frequencies. The total loaded latency can be taken to be the
idle latency plus the induced download latency plus the induced upload
latency, as an approximation of the latency when both directions are loaded
at once.

As for ranking ISPs overall... this is hard to do in a way that's perceived
to be fair. My recommendation on this front would be to allow your users to
select criteria with weights. Some might consider download and/or upload
speed to be important as well as latency, while others might see a choice
between overall latency (important for games) and jitter (important for
VoIP). This makes it harder to claim that you're personally presenting a
controversial opinion on relative merits.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 4009 bytes --]