From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ob0-x22c.google.com (mail-ob0-x22c.google.com [IPv6:2607:f8b0:4003:c01::22c]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 7859821F19A for ; Tue, 28 Apr 2015 19:01:04 -0700 (PDT) Received: by obbeb7 with SMTP id eb7so10196878obb.3 for ; Tue, 28 Apr 2015 19:01:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=HVdftnHiECnRlE08iVPnLGXHCRhC8WbmSafEbj/Ehww=; b=ml2gORLWFa+BBoxXFNIORH6R1WZNsjGh2yhE1SoaScs13jV6pMLaFK4cBRFI6KbOPq Kox5AcvkEZtQGvxhZDbqwsrjY8teyRe84LCL1JRvN+SNu0TaB9uRNqnKmFtEiHCJTVTO mQQnpTt5PLdWOQPvFTpzO/FJgsxWZcxxbuRDFTP1KWhemb8ZpMycaI/B44YFTN+q0qpw jTvr5wot83Nwcck0RbAc3XdvlfAYc42ecQBSgzLve5oQljTCwuACwiSp6WhgCQ0/8+d3 rEr+ZzepYr2x8AYEpXVyGjtgT2Y+PIbyJ9dn+bIL/Ej0p3N5JmJtst0gAZfG5sf/+ca1 851Q== MIME-Version: 1.0 X-Received: by 10.202.216.87 with SMTP id p84mr3521763oig.133.1430272864024; Tue, 28 Apr 2015 19:01:04 -0700 (PDT) Received: by 10.202.71.139 with HTTP; Tue, 28 Apr 2015 19:01:03 -0700 (PDT) In-Reply-To: References: Date: Tue, 28 Apr 2015 19:01:03 -0700 Message-ID: From: Dave Taht To: Jonathan Morton Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: bloat Subject: Re: [Bloat] extremely good dslreports result for bufferbloat on free.fr X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Apr 2015 02:01:33 -0000 my short term answer is to ask you merely collect lots more data and slice it up different ways, trying to isolate actual browser bugs from actual data (and NOT discarding what appears to be outliers, because the outliers are interesting - I am perpetually thinking of the discovery of the cosmic background radiation when it comes to looking at bufferbloat stuff), then work the raw data with one or more of the folk here (under NDA, I would assume) to get aggregate stats that make sense and are publishable. Some things that may help: A) You can be generally assured that anything that has ECT(3) asserted on the header is coming from a aqm-enabled router. There are also ack markings for that, not in the header. (are you also getting captures?) B) Nearly anything from free.fr will be either SFQed or FQ_Codeled C) Hopefully your users will distinguish wifi from non wifi connections but that seems dicey. D) We can add tests as similar to yours as possible as yours evolve, to the netperf-wrapper suite so as to get comparisons from our tools, without having a browser/web server/vm to deal with. I would like (after seeing it finally, on OSX) for the bufferbloat dial to be front and center instead of the static ping radar image. On Tue, Apr 28, 2015 at 6:39 PM, Jonathan Morton wr= ote: > These are pretty good questions, actually. But as pointed out, when you w= ant > to start ranking, it's important to distinguish the performance of the IS= P > itself from equipment under the subscriber's control, which itself might = be > configured to hide faults in the ISP. > > Statistics is a hard subject. If you feel at all confused by what I descr= ibe > below, it might be a good idea to sit down with a real expert. The basic > calculations are not difficult; it's knowing WHAT to calculate and what i= t > means once you've done it. > > Notably, the upload queue usually depends much more on the CPE than on > anything the ISP controls directly. I don't think there are that many ISP= s > left which absolutely insist on using their own modem, but most will supp= ly > a preferred, preconfigured model on request, and many users will accept > that, not knowing better. Unfortunately, I can't think of an easy, robust > way to detect what CPE is in use our how it has been configured, so it's > hard to control for it statistically. > > In general, downstream queuing is much more under the ISP's control. To > account for the (growing?) subset of users who apply ingress shaping, you > could look at the upper percentiles of latency, since usually ingress > shaping will improve matters. Caveat: it's also possible for ingress shap= ing > to make things worse, either through accidental misconfiguration or even > maliciously, so don't just take the peak value. > > Initially I suggest you present a histogram of the results that fall into > particular grades. That'll help you get a feel for the statistics, and it= 's > easy to pick out a modal value by eye. Beware of trying to calculate a mo= dal > value simplistically, since the histogram might not show a simple mode if > results tend to straddle two adjacent grades. (This is why the mode is th= e > least used of the three basic averages; it's hard to calculate it such th= at > it's reliably useful, even though it's often intuitively useful.) > > I also suggest you add a basic question to the user: are you using a rout= er > with QoS features turned on (yes/no/dunno), with dunno as the default. > That'll give you a way to point out that most of the lower latency result= s > are probably a result of this. You could draw stacked or adjacent histogr= ams > with this information colour coded. > > As for the single numbers to plug into the formula, this also requires so= me > care. > > For the idle/baseline, I suggest using only the samples from before the > bandwidth tests begin, rather than also including those from between and > afterwards. Then you should use the harmonic mean on these, to get a valu= e > biased on the low end. > > For each of the load sets, you want a value biased high. Taking the 90th > percentile might be a reasonable approach here; it'll discard outliers an= d > brief transients that a good AQM acting alone (ie. without FQ) might leav= e, > but should still expose the sorts of obvious problems that we're trying t= o > tackle. > > I do think that the idle latency and the total loaded latency can usefull= y > be reported as frequencies. The total loaded latency can be taken to be t= he > idle latency plus the induced download latency plus the induced upload > latency, as an approximation of the latency when both directions are load= ed > at once. > > As for ranking ISPs overall... this is hard to do in a way that's perceiv= ed > to be fair. My recommendation on this front would be to allow your users = to > select criteria with weights. Some might consider download and/or upload > speed to be important as well as latency, while others might see a choice > between overall latency (important for games) and jitter (important for > VoIP). This makes it harder to claim that you're personally presenting a > controversial opinion on relative merits. > > - Jonathan Morton > > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > --=20 Dave T=C3=A4ht Open Networking needs **Open Source Hardware** https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67