Re: [Make-wifi-fast] hacking on the candelatech and qca ath10k firmware

[Aaron Wood <woody77@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1942 bytes --]

On Thu, May 5, 2016 at 2:30 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Dave Taht <dave.taht@gmail.com> writes:
>
> > On Thu, May 5, 2016 at 10:05 AM, Aaron Wood <woody77@gmail.com> wrote:
> >> I think you might be mis-reading the box-plots as error-bars (since
> their
> >> quartile plots).  I'll need to crunch the numbers, but I'm pretty sure
> that
> >> the fq results are going to show a higher median throughput (and lower
> >> median latency), with a fair bit of significance.  I'll see if I can
> figure
> >> out how to calculate the SD of the mean (and other quartiles) from the
> flent
> >> output (I have scripts that can do this for iperf3's json output).
> >
> > Thanks in advance!!!
> >
> > I hate box plots honestly. They often lie. I'd rather look at a
> > detailed time series first, and the box plot *only* after I verified
> > that that was sane. And I'm not good at reading box plots right!
>
> Also note that a box plot of a single test will show you "error bars"
> which are really computed from the samples of the single flow; so they
> are not independent samples, and so care should be taken when
> interpreting them.
>

Not "error-bars", which imply the Standard Error (of the mean), but a
box-and-whisker which shows the 5-number summary (quartiles), which is
quite different (SEM should be much narrower than the inter-quartile range).

For instance, in the data that Dave has here, I computed:

run         mean bw    Std Err Mean
CT_10_1   95.3   1.06
CT_10_1   91.0   1.14
CT_10_1   102.0   1.21
CT_10_1_fq   109.0   0.78

OTOH, these runs are picking enough "other" noise that it's clear that the
computed SEM isn't true (given that three runs of the same setup each have
means that are way too far apart).

And further, using tools built around gaussian distributions on something
that most definitely is not doesn't help make it any clearer.

-Aaron

[-- Attachment #2: Type: text/html, Size: 2727 bytes --]