Re: [Make-wifi-fast] hacking on the candelatech and qca ath10k firmware

[Dave Taht <dave.taht@gmail.com>]

On Thu, May 5, 2016 at 10:05 AM, Aaron Wood <woody77@gmail.com> wrote:
> I think you might be mis-reading the box-plots as error-bars (since their
> quartile plots).  I'll need to crunch the numbers, but I'm pretty sure that
> the fq results are going to show a higher median throughput (and lower
> median latency), with a fair bit of significance.  I'll see if I can figure
> out how to calculate the SD of the mean (and other quartiles) from the flent
> output (I have scripts that can do this for iperf3's json output).

Thanks in advance!!!

I hate box plots honestly. They often lie. I'd rather look at a
detailed time series first, and the box plot *only* after I verified
that that was sane. And I'm not good at reading box plots right!

Tthat said, what I meant by error bars was that I mentally disregard
any eyeball comparison variance of ~10% as a possible artifact of the
usually single or dual test, and rely on doing extensive, repeated
and/or long term tests to get that down to significance. Eventually.
After all the bugs are out. Toke uses 30 tests in a row to get
somewhere, which takes weeks, so I fly by the seat of my pants in this
way for as long as I can.

I've had so many cases where I'd look at a box plot and not understand
what was going on. The one that sticks in my memory best (never got
around to writing it up though) were the ones where we were dealing
with the unaligned access bugs in tcp on cerowrt. We'd see overall
throughput drop by like 20% for ipv6 vs ipv4 in the box plots. We'd
see periodic total losses in throughput on the detailed time based
ones.

Here's another case where box plots lie, showing the impact of
"something" every 2 minutes:

see second plot on:

http://blog.cerowrt.org/post/cs5_lockout/

I'd rather look at a
https://en.wikipedia.org/wiki/Seven-number_summary in terms of box
plot. A howto or lecture on how to better interpret various flent
tests would be nice to do up, I don't think it's clear to many people
how the width of the sawtooths on most of the flent tests relative to
the direct latency measurement still show the effectiveness of the
underlying AQM even with fq in place, because the relate better in a
single queue aqm - I've seen so many of the aqm alone vs fq+aqm plots
- and also the backlog plots which I more rarely collect and publish -
that I just filter them into a mental something that works.

People using box plots exclusively to analyze tcp throughput are on
drugs. I have consciously focused on doing plots rather than reporting
single number results like "Got 110Mbits throughput! Ship it", 'cause
that doesn't show the sawtooth.

This message brought to you by the "Society to Save the Sawtooth"


>
> -Aaron
>
> On Thu, May 5, 2016 at 12:09 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> see: http://blog.cerowrt.org/post/ath10_ath9k_1/
>>
>> the regular qca firmware survived the rrul better, and seemed to do
>> wmm better. (CS6 for example, was fine) Aside from that it was slower
>> and more jittery than the candelatech firmware. some pics there. Am
>> too tired to write it up right now.
>>
>> https://github.com/dtaht/blog-cerowrt/tree/master/content/flent/qca-10.2
>>
>> I guess I gotta go boot into baseline kernels now and pray I haven't
>> been deluding myself at these speeds. For all I know everything is
>> actually better with those than all these patches.
>>
>> night
>>
>> --
>> Dave Täht
>> Let's go make home routers and wifi faster! With better software!
>> http://blog.cerowrt.org
>> _______________________________________________
>> Make-wifi-fast mailing list
>> Make-wifi-fast@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/make-wifi-fast
>
>



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org