From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vn0-x234.google.com (mail-vn0-x234.google.com [IPv6:2607:f8b0:400c:c0f::234]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id F076C21F1F0 for ; Tue, 28 Apr 2015 18:39:27 -0700 (PDT) Received: by vnbg62 with SMTP id g62so1804060vnb.6 for ; Tue, 28 Apr 2015 18:39:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=R9PiJ237RwaCgmgCFoB1vg45guFvNRDp58Pi9Yjjd9s=; b=wNP5cDWECe9nOOT0jg/flS7cmchqHm1H7L9NbxOTO/m63WhpmS8zQTwVTVVJeoC+tN eoUw1TTg6RUx3fSQuYpVN80/+ZI634lR1SEetm4QH4uKqeoQRH+yktLkP6vNvh+x9vLZ nUjwFn7BIqwDg5oUwyFqb/4GOY+srfeP5jjSLWOTZZ6PAkMIZyTJHNpdtrlmUnHiJZNx I0W3prDa2pAo4e1494L843iB4eosdonG+UbIFXT4SxOw4Daui6Vn8B5SvsoG4rV4pryU U6FEF6UKwYEHFKv4ZXBA1TvTVwv/qv4hxJy6dSfpEQerJ/zU3ghUXyWKk+/h7B9a1BxY 6aWA== MIME-Version: 1.0 X-Received: by 10.52.126.235 with SMTP id nb11mr6072368vdb.58.1430271566413; Tue, 28 Apr 2015 18:39:26 -0700 (PDT) Received: by 10.52.12.167 with HTTP; Tue, 28 Apr 2015 18:39:26 -0700 (PDT) Received: by 10.52.12.167 with HTTP; Tue, 28 Apr 2015 18:39:26 -0700 (PDT) In-Reply-To: References: Date: Wed, 29 Apr 2015 04:39:26 +0300 Message-ID: From: Jonathan Morton To: David Lang Content-Type: multipart/alternative; boundary=bcaec520ec87a6051e0514d30c4e Cc: bloat Subject: Re: [Bloat] extremely good dslreports result for bufferbloat on free.fr X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Apr 2015 01:39:56 -0000 --bcaec520ec87a6051e0514d30c4e Content-Type: text/plain; charset=UTF-8 These are pretty good questions, actually. But as pointed out, when you want to start ranking, it's important to distinguish the performance of the ISP itself from equipment under the subscriber's control, which itself might be configured to hide faults in the ISP. Statistics is a hard subject. If you feel at all confused by what I describe below, it might be a good idea to sit down with a real expert. The basic calculations are not difficult; it's knowing WHAT to calculate and what it means once you've done it. Notably, the upload queue usually depends much more on the CPE than on anything the ISP controls directly. I don't think there are that many ISPs left which absolutely insist on using their own modem, but most will supply a preferred, preconfigured model on request, and many users will accept that, not knowing better. Unfortunately, I can't think of an easy, robust way to detect what CPE is in use our how it has been configured, so it's hard to control for it statistically. In general, downstream queuing is much more under the ISP's control. To account for the (growing?) subset of users who apply ingress shaping, you could look at the upper percentiles of latency, since usually ingress shaping will improve matters. Caveat: it's also possible for ingress shaping to make things worse, either through accidental misconfiguration or even maliciously, so don't just take the peak value. Initially I suggest you present a histogram of the results that fall into particular grades. That'll help you get a feel for the statistics, and it's easy to pick out a modal value by eye. Beware of trying to calculate a modal value simplistically, since the histogram might not show a simple mode if results tend to straddle two adjacent grades. (This is why the mode is the least used of the three basic averages; it's hard to calculate it such that it's reliably useful, even though it's often intuitively useful.) I also suggest you add a basic question to the user: are you using a router with QoS features turned on (yes/no/dunno), with dunno as the default. That'll give you a way to point out that most of the lower latency results are probably a result of this. You could draw stacked or adjacent histograms with this information colour coded. As for the single numbers to plug into the formula, this also requires some care. For the idle/baseline, I suggest using only the samples from before the bandwidth tests begin, rather than also including those from between and afterwards. Then you should use the harmonic mean on these, to get a value biased on the low end. For each of the load sets, you want a value biased high. Taking the 90th percentile might be a reasonable approach here; it'll discard outliers and brief transients that a good AQM acting alone (ie. without FQ) might leave, but should still expose the sorts of obvious problems that we're trying to tackle. I do think that the idle latency and the total loaded latency can usefully be reported as frequencies. The total loaded latency can be taken to be the idle latency plus the induced download latency plus the induced upload latency, as an approximation of the latency when both directions are loaded at once. As for ranking ISPs overall... this is hard to do in a way that's perceived to be fair. My recommendation on this front would be to allow your users to select criteria with weights. Some might consider download and/or upload speed to be important as well as latency, while others might see a choice between overall latency (important for games) and jitter (important for VoIP). This makes it harder to claim that you're personally presenting a controversial opinion on relative merits. - Jonathan Morton --bcaec520ec87a6051e0514d30c4e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

These are pretty good questions, actually. But as pointed ou= t, when you want to start ranking, it's important to distinguish the pe= rformance of the ISP itself from equipment under the subscriber's contr= ol, which itself might be configured to hide faults in the ISP.

Statistics is a hard subject. If you feel at all confused by= what I describe below, it might be a good idea to sit down with a real exp= ert. The basic calculations are not difficult; it's knowing WHAT to cal= culate and what it means once you've done it.

Notably, the upload queue usually depends much more on the C= PE than on anything the ISP controls directly. I don't think there are = that many ISPs left which absolutely insist on using their own modem, but m= ost will supply a preferred, preconfigured model on request, and many users= will accept that, not knowing better. Unfortunately, I can't think of = an easy, robust way to detect what CPE is in use our how it has been config= ured, so it's hard to control for it statistically.

In general, downstream queuing is much more under the ISP= 9;s control. To account for the (growing?) subset of users who apply ingres= s shaping, you could look at the upper percentiles of latency, since usuall= y ingress shaping will improve matters. Caveat: it's also possible for = ingress shaping to make things worse, either through accidental misconfigur= ation or even maliciously, so don't just take the peak value.

Initially I suggest you present a histogram of the results t= hat fall into particular grades. That'll help you get a feel for the st= atistics, and it's easy to pick out a modal value by eye. Beware of try= ing to calculate a modal value simplistically, since the histogram might no= t show a simple mode if results tend to straddle two adjacent grades. (This= is why the mode is the least used of the three basic averages; it's ha= rd to calculate it such that it's reliably useful, even though it's= often intuitively useful.)

I also suggest you add a basic question to the user: are you= using a router with QoS features turned on (yes/no/dunno), with dunno as t= he default. That'll give you a way to point out that most of the lower = latency results are probably a result of this. You could draw stacked or ad= jacent histograms with this information colour coded.

As for the single numbers to plug into the formula, this als= o requires some care.

For the idle/baseline, I suggest using only the samples from= before the bandwidth tests begin, rather than also including those from be= tween and afterwards. Then you should use the harmonic mean on these, to ge= t a value biased on the low end.

For each of the load sets, you want a value biased high. Tak= ing the 90th percentile might be a reasonable approach here; it'll disc= ard outliers and brief transients that a good AQM acting alone (ie. without= FQ) might leave, but should still expose the sorts of obvious problems tha= t we're trying to tackle.

I do think that the idle latency and the total loaded latenc= y can usefully be reported as frequencies. The total loaded latency can be = taken to be the idle latency plus the induced download latency plus the ind= uced upload latency, as an approximation of the latency when both direction= s are loaded at once.

As for ranking ISPs overall... this is hard to do in a way t= hat's perceived to be fair. My recommendation on this front would be to= allow your users to select criteria with weights. Some might consider down= load and/or upload speed to be important as well as latency, while others m= ight see a choice between overall latency (important for games) and jitter = (important for VoIP). This makes it harder to claim that you're persona= lly presenting a controversial opinion on relative merits.

- Jonathan Morton

--bcaec520ec87a6051e0514d30c4e--