[Make-wifi-fast] hacking on the candelatech and qca ath10k firmware

[Make-wifi-fast] hacking on the candelatech and qca ath10k firmware

[Dave Taht]

see: http://blog.cerowrt.org/post/ath10_ath9k_1/

the regular qca firmware survived the rrul better, and seemed to do
wmm better. (CS6 for example, was fine) Aside from that it was slower
and more jittery than the candelatech firmware. some pics there. Am
too tired to write it up right now.

https://github.com/dtaht/blog-cerowrt/tree/master/content/flent/qca-10.2

I guess I gotta go boot into baseline kernels now and pray I haven't
been deluding myself at these speeds. For all I know everything is
actually better with those than all these patches.

night

-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Make-wifi-fast] hacking on the candelatech and qca ath10k firmware

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1434 bytes --]

I think you might be mis-reading the box-plots as error-bars (since their
quartile plots).  I'll need to crunch the numbers, but I'm pretty sure that
the fq results are going to show a higher median throughput (and lower
median latency), with a fair bit of significance.  I'll see if I can figure
out how to calculate the SD of the mean (and other quartiles) from the
flent output (I have scripts that can do this for iperf3's json output).

-Aaron

On Thu, May 5, 2016 at 12:09 AM, Dave Taht <dave.taht@gmail.com> wrote:

> see: http://blog.cerowrt.org/post/ath10_ath9k_1/
>
> the regular qca firmware survived the rrul better, and seemed to do
> wmm better. (CS6 for example, was fine) Aside from that it was slower
> and more jittery than the candelatech firmware. some pics there. Am
> too tired to write it up right now.
>
> https://github.com/dtaht/blog-cerowrt/tree/master/content/flent/qca-10.2
>
> I guess I gotta go boot into baseline kernels now and pray I haven't
> been deluding myself at these speeds. For all I know everything is
> actually better with those than all these patches.
>
> night
>
> --
> Dave Täht
> Let's go make home routers and wifi faster! With better software!
> http://blog.cerowrt.org
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
>

[-- Attachment #2: Type: text/html, Size: 2313 bytes --]

Re: [Make-wifi-fast] hacking on the candelatech and qca ath10k firmware

[Dave Taht]

On Thu, May 5, 2016 at 10:05 AM, Aaron Wood <woody77@gmail.com> wrote:
> I think you might be mis-reading the box-plots as error-bars (since their
> quartile plots).  I'll need to crunch the numbers, but I'm pretty sure that
> the fq results are going to show a higher median throughput (and lower
> median latency), with a fair bit of significance.  I'll see if I can figure
> out how to calculate the SD of the mean (and other quartiles) from the flent
> output (I have scripts that can do this for iperf3's json output).

Thanks in advance!!!

I hate box plots honestly. They often lie. I'd rather look at a
detailed time series first, and the box plot *only* after I verified
that that was sane. And I'm not good at reading box plots right!

Tthat said, what I meant by error bars was that I mentally disregard
any eyeball comparison variance of ~10% as a possible artifact of the
usually single or dual test, and rely on doing extensive, repeated
and/or long term tests to get that down to significance. Eventually.
After all the bugs are out. Toke uses 30 tests in a row to get
somewhere, which takes weeks, so I fly by the seat of my pants in this
way for as long as I can.

I've had so many cases where I'd look at a box plot and not understand
what was going on. The one that sticks in my memory best (never got
around to writing it up though) were the ones where we were dealing
with the unaligned access bugs in tcp on cerowrt. We'd see overall
throughput drop by like 20% for ipv6 vs ipv4 in the box plots. We'd
see periodic total losses in throughput on the detailed time based
ones.

Here's another case where box plots lie, showing the impact of
"something" every 2 minutes:

see second plot on:

http://blog.cerowrt.org/post/cs5_lockout/

I'd rather look at a
https://en.wikipedia.org/wiki/Seven-number_summary in terms of box
plot. A howto or lecture on how to better interpret various flent
tests would be nice to do up, I don't think it's clear to many people
how the width of the sawtooths on most of the flent tests relative to
the direct latency measurement still show the effectiveness of the
underlying AQM even with fq in place, because the relate better in a
single queue aqm - I've seen so many of the aqm alone vs fq+aqm plots
- and also the backlog plots which I more rarely collect and publish -
that I just filter them into a mental something that works.

People using box plots exclusively to analyze tcp throughput are on
drugs. I have consciously focused on doing plots rather than reporting
single number results like "Got 110Mbits throughput! Ship it", 'cause
that doesn't show the sawtooth.

This message brought to you by the "Society to Save the Sawtooth"


>
> -Aaron
>
> On Thu, May 5, 2016 at 12:09 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> see: http://blog.cerowrt.org/post/ath10_ath9k_1/
>>
>> the regular qca firmware survived the rrul better, and seemed to do
>> wmm better. (CS6 for example, was fine) Aside from that it was slower
>> and more jittery than the candelatech firmware. some pics there. Am
>> too tired to write it up right now.
>>
>> https://github.com/dtaht/blog-cerowrt/tree/master/content/flent/qca-10.2
>>
>> I guess I gotta go boot into baseline kernels now and pray I haven't
>> been deluding myself at these speeds. For all I know everything is
>> actually better with those than all these patches.
>>
>> night
>>
>> --
>> Dave Täht
>> Let's go make home routers and wifi faster! With better software!
>> http://blog.cerowrt.org
>> _______________________________________________
>> Make-wifi-fast mailing list
>> Make-wifi-fast@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/make-wifi-fast
>
>



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Make-wifi-fast] hacking on the candelatech and qca ath10k firmware

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> On Thu, May 5, 2016 at 10:05 AM, Aaron Wood <woody77@gmail.com> wrote:
>> I think you might be mis-reading the box-plots as error-bars (since their
>> quartile plots).  I'll need to crunch the numbers, but I'm pretty sure that
>> the fq results are going to show a higher median throughput (and lower
>> median latency), with a fair bit of significance.  I'll see if I can figure
>> out how to calculate the SD of the mean (and other quartiles) from the flent
>> output (I have scripts that can do this for iperf3's json output).
>
> Thanks in advance!!!
>
> I hate box plots honestly. They often lie. I'd rather look at a
> detailed time series first, and the box plot *only* after I verified
> that that was sane. And I'm not good at reading box plots right!

Also note that a box plot of a single test will show you "error bars"
which are really computed from the samples of the single flow; so they
are not independent samples, and so care should be taken when
interpreting them.

-Toke

Re: [Make-wifi-fast] hacking on the candelatech and qca ath10k firmware

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1942 bytes --]

On Thu, May 5, 2016 at 2:30 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Dave Taht <dave.taht@gmail.com> writes:
>
> > On Thu, May 5, 2016 at 10:05 AM, Aaron Wood <woody77@gmail.com> wrote:
> >> I think you might be mis-reading the box-plots as error-bars (since
> their
> >> quartile plots).  I'll need to crunch the numbers, but I'm pretty sure
> that
> >> the fq results are going to show a higher median throughput (and lower
> >> median latency), with a fair bit of significance.  I'll see if I can
> figure
> >> out how to calculate the SD of the mean (and other quartiles) from the
> flent
> >> output (I have scripts that can do this for iperf3's json output).
> >
> > Thanks in advance!!!
> >
> > I hate box plots honestly. They often lie. I'd rather look at a
> > detailed time series first, and the box plot *only* after I verified
> > that that was sane. And I'm not good at reading box plots right!
>
> Also note that a box plot of a single test will show you "error bars"
> which are really computed from the samples of the single flow; so they
> are not independent samples, and so care should be taken when
> interpreting them.
>

Not "error-bars", which imply the Standard Error (of the mean), but a
box-and-whisker which shows the 5-number summary (quartiles), which is
quite different (SEM should be much narrower than the inter-quartile range).

For instance, in the data that Dave has here, I computed:

run         mean bw    Std Err Mean
CT_10_1   95.3   1.06
CT_10_1   91.0   1.14
CT_10_1   102.0   1.21
CT_10_1_fq   109.0   0.78

OTOH, these runs are picking enough "other" noise that it's clear that the
computed SEM isn't true (given that three runs of the same setup each have
means that are way too far apart).

And further, using tools built around gaussian distributions on something
that most definitely is not doesn't help make it any clearer.

-Aaron

[-- Attachment #2: Type: text/html, Size: 2727 bytes --]

Re: [Make-wifi-fast] hacking on the candelatech and qca ath10k firmware

[Toke Høiland-Jørgensen]

On 5 May 2016 23:45:21 CEST, Aaron Wood <woody77@gmail.com> wrote:

>And further, using tools built around gaussian distributions on
>something
>that most definitely is not doesn't help make it any clearer.

Exactly! That was what I was trying to say ;)

-Toke

Re: [Make-wifi-fast] hacking on the candelatech and qca ath10k firmware

[Dave Taht]

On Thu, May 5, 2016 at 2:59 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>
> On 5 May 2016 23:45:21 CEST, Aaron Wood <woody77@gmail.com> wrote:
>
>>And further, using tools built around gaussian distributions on
>>something
>>that most definitely is not doesn't help make it any clearer.
>
> Exactly! That was what I was trying to say ;)

Heh.

I think something like:

"using tools built around gaussian distributions on datasets that most
definitely are not gaussian won't make the results any clearer."

is a great tag line for (someone else's) blog entry or paper.




-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org