From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw0-x230.google.com (mail-yw0-x230.google.com [IPv6:2607:f8b0:4002:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 386FF3B260 for ; Thu, 5 May 2016 17:45:22 -0400 (EDT) Received: by mail-yw0-x230.google.com with SMTP id o66so166195058ywc.3 for ; Thu, 05 May 2016 14:45:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=Tt11GU2iXEr0iiBYo8zU9y33NZCIcS38wCF5OJawliM=; b=stwcvaGeCfm9suEYTrs4QluUuifMItt5of0mhGdv/RGCa6Vau15VlUlhMIdW5sMYk2 RYTXbNqM2AYaRMC1iEn7Nc2aRtfBwYqytlYwAFrUWL4F0LqTrN/slS9PPqTBsNmSv8wx EN+tgE+6rnXLZCKB+6Wdk/q8LQf0qhV8N+dlCvDwIOX27XpC6o2qjD4QpKJTdKtja4HJ rUv4Yo5YtZFcZoa3Kopj30eaS8qovCcu1ZhRpiud76m81wLiwgOJF+pYO9zEb0vrXqWy c+0dsjrwjUxQOK8HwQS6iQhOmgYf/XcsnoSreeIH8aXEKM88e0uRg2qVkXqk/0J1c1us KwVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=Tt11GU2iXEr0iiBYo8zU9y33NZCIcS38wCF5OJawliM=; b=Q/q9VhcA0+J/ABhvYev9v1LJZf1GQWxI67aLTtsSJDmAjuC+sgSeHM6rGDUvgPi6KI dEHXDdDFhedSPuMb7gNIakBoipSr2TnC8b+WEgqXEuqeP8Tjxxl4Xg3S8VTgGvxLB8BT 8K0A4lWnlpN9Xj/8jqZN5+IeHaChgRLuvQjnhDA2okZY2QelHysaZPVqJzo5YOp3YOoE VymH2aYQOcizRvRNv/ZD2XM3zRa3LnXZI8BLGHQRcdFivrU2fBrheEVtm/JmYP2RjLac azGqlJsCUuBfluLiQZwnaxtpvnPDKR9QtpTw1MYBAu94g/1y1E7TH1cwgGop06G0YpqT 93pw== X-Gm-Message-State: AOPr4FUUDgwUxUrdxbm394AmcKhznUfRPiT3pBsbniVIsPDSWuVkass+MNYHqQlqNngRTEXMRkasD+1vPq8x9g== MIME-Version: 1.0 X-Received: by 10.159.40.99 with SMTP id c90mr10747205uac.85.1462484721707; Thu, 05 May 2016 14:45:21 -0700 (PDT) Received: by 10.103.44.70 with HTTP; Thu, 5 May 2016 14:45:21 -0700 (PDT) In-Reply-To: <87mvo49oqj.fsf@toke.dk> References: <87mvo49oqj.fsf@toke.dk> Date: Thu, 5 May 2016 14:45:21 -0700 Message-ID: From: Aaron Wood To: =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= Cc: Dave Taht , make-wifi-fast@lists.bufferbloat.net Content-Type: multipart/alternative; boundary=94eb2c04861853b4fe05321f4278 Subject: Re: [Make-wifi-fast] hacking on the candelatech and qca ath10k firmware X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2016 21:45:22 -0000 --94eb2c04861853b4fe05321f4278 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Thu, May 5, 2016 at 2:30 PM, Toke H=C3=B8iland-J=C3=B8rgensen wrote: > Dave Taht writes: > > > On Thu, May 5, 2016 at 10:05 AM, Aaron Wood wrote: > >> I think you might be mis-reading the box-plots as error-bars (since > their > >> quartile plots). I'll need to crunch the numbers, but I'm pretty sure > that > >> the fq results are going to show a higher median throughput (and lower > >> median latency), with a fair bit of significance. I'll see if I can > figure > >> out how to calculate the SD of the mean (and other quartiles) from the > flent > >> output (I have scripts that can do this for iperf3's json output). > > > > Thanks in advance!!! > > > > I hate box plots honestly. They often lie. I'd rather look at a > > detailed time series first, and the box plot *only* after I verified > > that that was sane. And I'm not good at reading box plots right! > > Also note that a box plot of a single test will show you "error bars" > which are really computed from the samples of the single flow; so they > are not independent samples, and so care should be taken when > interpreting them. > Not "error-bars", which imply the Standard Error (of the mean), but a box-and-whisker which shows the 5-number summary (quartiles), which is quite different (SEM should be much narrower than the inter-quartile range)= . For instance, in the data that Dave has here, I computed: run mean bw Std Err Mean CT_10_1 95.3 1.06 CT_10_1 91.0 1.14 CT_10_1 102.0 1.21 CT_10_1_fq 109.0 0.78 OTOH, these runs are picking enough "other" noise that it's clear that the computed SEM isn't true (given that three runs of the same setup each have means that are way too far apart). And further, using tools built around gaussian distributions on something that most definitely is not doesn't help make it any clearer. -Aaron --94eb2c04861853b4fe05321f4278 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

= On Thu, May 5, 2016 at 2:30 PM, Toke H=C3=B8iland-J=C3=B8rgensen <toke@toke.dk= > wrote:
Dave Taht <<= a href=3D"mailto:dave.taht@gmail.com">dave.taht@gmail.com> writes:
> On Thu, May 5, 2016 at 10:05 AM, Aaron Wood <woody77@gmail.com> wrote:
>> I think you might be mis-reading the box-plots as error-bars (sinc= e their
>> quartile plots).=C2=A0 I'll need to crunch the numbers, but I&= #39;m pretty sure that
>> the fq results are going to show a higher median throughput (and l= ower
>> median latency), with a fair bit of significance.=C2=A0 I'll s= ee if I can figure
>> out how to calculate the SD of the mean (and other quartiles) from= the flent
>> output (I have scripts that can do this for iperf3's json outp= ut).
>
> Thanks in advance!!!
>
> I hate box plots honestly. They often lie. I'd rather look at a > detailed time series first, and the box plot *only* after I verified > that that was sane. And I'm not good at reading box plots right!
Also note that a box plot of a single test will show you "error= bars"
which are really computed from the samples of the single flow; so they
are not independent samples, and so care should be taken when
interpreting them.

Not "error-bars= ", which imply the Standard Error (of the mean), but a box-and-whisker= which shows the 5-number summary (quartiles), which is quite different (SE= M should be much narrower than the inter-quartile range).

For instance, in the data that Dave has here, I computed:

run =C2=A0 =C2=A0 =C2=A0 =C2=A0 mean bw =C2=A0 =C2=A0Std Er= r Mean
CT_10_1 =C2=A0 95.3 =C2=A0 1.06
CT_10_1 =C2=A0 9= 1.0 =C2=A0 1.14
CT_10_1 =C2=A0 102.0 =C2=A0 1.21
CT_10_= 1_fq =C2=A0 109.0 =C2=A0 0.78

OTOH, these runs are= picking enough "other" noise that it's clear that the comput= ed SEM isn't true (given that three runs of the same setup each have me= ans that are way too far apart).

And further, usin= g tools built around gaussian distributions on something that most definite= ly is not doesn't help make it any clearer.

-A= aron
--94eb2c04861853b4fe05321f4278--