[Make-wifi-fast] crypto-fq bug postmortem is up

[Make-wifi-fast] crypto-fq bug postmortem is up

[Dave Taht]

at:

http://blog.cerowrt.org/post/crypto_fq_bug/

I am very interested to try to remember more of the suggestions, and
blind alleys, and why we thought they were possibilities... and draw
more conclusions as to what we did right, and wrong.

Please comment, and send links.

If I have more time, I'll make it shorter. Or funnier. And add more
pictures, captures, and links. Finding this bug *eventually* got to be
fun.

-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[Loganaden Velvindron]

On Sat, Sep 17, 2016 at 1:42 AM, Dave Taht <dave.taht@gmail.com> wrote:
> at:
>
> http://blog.cerowrt.org/post/crypto_fq_bug/
>
> I am very interested to try to remember more of the suggestions, and
> blind alleys, and why we thought they were possibilities... and draw
> more conclusions as to what we did right, and wrong.
>

amazing.

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[Loganaden Velvindron]

On Sat, Sep 17, 2016 at 1:42 AM, Dave Taht <dave.taht@gmail.com> wrote:
> at:
>
> http://blog.cerowrt.org/post/crypto_fq_bug/
>
> I am very interested to try to remember more of the suggestions, and
> blind alleys, and why we thought they were possibilities... and draw
> more conclusions as to what we did right, and wrong.
>
> Please comment, and send links.
>
> If I have more time, I'll make it shorter. Or funnier. And add more
> pictures, captures, and links. Finding this bug *eventually* got to be
> fun.
>

My home network:

FTTH modem (bridge) --- tp-link archer c7
(pppoe+sqm+simple_pppoe(fq_codel)+ap) ---- (lan cable) --- tp-link
archer c7 v2 with the experimental firmware(ap):
Quick and dirty test:

on non-patched openwrt wifi(wpa-psk):
https://www.dslreports.com/speedtest/5028614

on tp-link archer c7 v2 with toke's firmware(wpa-psk):
https://www.dslreports.com/speedtest/5028349

My brother said that "It feels like ethernet both in terms of
throughput and latency". My brother is hammering the the tp-link
archer c7 box with bittorrent, whatsapp, viber, and normal http.

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[Loganaden Velvindron]

On Sat, Sep 17, 2016 at 9:12 PM, Loganaden Velvindron
<loganaden@gmail.com> wrote:
> On Sat, Sep 17, 2016 at 1:42 AM, Dave Taht <dave.taht@gmail.com> wrote:
>> at:
>>
>> http://blog.cerowrt.org/post/crypto_fq_bug/
>>
>> I am very interested to try to remember more of the suggestions, and
>> blind alleys, and why we thought they were possibilities... and draw
>> more conclusions as to what we did right, and wrong.
>>
>> Please comment, and send links.
>>
>> If I have more time, I'll make it shorter. Or funnier. And add more
>> pictures, captures, and links. Finding this bug *eventually* got to be
>> fun.
>>
>
> My home network:
>
> FTTH modem (bridge) --- tp-link archer c7
> (pppoe+sqm+simple_pppoe(fq_codel)+ap) ---- (lan cable) --- tp-link
> archer c7 v2 with the experimental firmware(ap):
> Quick and dirty test:
>
> on non-patched openwrt wifi(wpa-psk):
> https://www.dslreports.com/speedtest/5028614
>
> on tp-link archer c7 v2 with toke's firmware(wpa-psk):
> https://www.dslreports.com/speedtest/5028349
>
> My brother said that "It feels like ethernet both in terms of
> throughput and latency". My brother is hammering the the tp-link
> archer c7 box with bittorrent, whatsapp, viber, and normal http.

Testing on my android smartphone (android 6.0.1):

Unpatched ap:
https://www.dslreports.com/speedtest/5028965

Patched ap:
https://www.dslreports.com/speedtest/5028940

On smartphones, it looks like you can get better throughput with
toke's firmware.

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[Dave Taht]

On Sat, Sep 17, 2016 at 10:26 AM, Loganaden Velvindron
<loganaden@gmail.com> wrote:
> On Sat, Sep 17, 2016 at 9:12 PM, Loganaden Velvindron
> <loganaden@gmail.com> wrote:
>> On Sat, Sep 17, 2016 at 1:42 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>> at:
>>>
>>> http://blog.cerowrt.org/post/crypto_fq_bug/
>>>
>>> I am very interested to try to remember more of the suggestions, and
>>> blind alleys, and why we thought they were possibilities... and draw
>>> more conclusions as to what we did right, and wrong.
>>>
>>> Please comment, and send links.
>>>
>>> If I have more time, I'll make it shorter. Or funnier. And add more
>>> pictures, captures, and links. Finding this bug *eventually* got to be
>>> fun.
>>>
>>
>> My home network:
>>
>> FTTH modem (bridge) --- tp-link archer c7
>> (pppoe+sqm+simple_pppoe(fq_codel)+ap) ---- (lan cable) --- tp-link
>> archer c7 v2 with the experimental firmware(ap):
>> Quick and dirty test:
>>
>> on non-patched openwrt wifi(wpa-psk):
>> https://www.dslreports.com/speedtest/5028614
>>
>> on tp-link archer c7 v2 with toke's firmware(wpa-psk):
>> https://www.dslreports.com/speedtest/5028349
>>
>> My brother said that "It feels like ethernet both in terms of
>> throughput and latency". My brother is hammering the the tp-link
>> archer c7 box with bittorrent, whatsapp, viber, and normal http.
>
> Testing on my android smartphone (android 6.0.1):
>
> Unpatched ap:
> https://www.dslreports.com/speedtest/5028965
>
> Patched ap:
> https://w7ww.dslreports.com/speedtest/5028940
>7
> On smartphones, it looks like you can get better throughput with
> toke's firmware.

I would repeat that test a couple times to make sure. It would not
surprise me that we also won on the dslreports tests for wifi for
normal clients - dslreports does use multiple flows after all and the
fq is going to help a lot here  - but seeing bandwidth improve from 7
to 12mbits as a result of all this for that test, under these
circumstances, strikes me as a bit much.

-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[Toke Høiland-Jørgensen]

Loganaden Velvindron <loganaden@gmail.com> writes:

> On Sat, Sep 17, 2016 at 9:12 PM, Loganaden Velvindron
> <loganaden@gmail.com> wrote:
>> On Sat, Sep 17, 2016 at 1:42 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>> at:
>>>
>>> http://blog.cerowrt.org/post/crypto_fq_bug/
>>>
>>> I am very interested to try to remember more of the suggestions, and
>>> blind alleys, and why we thought they were possibilities... and draw
>>> more conclusions as to what we did right, and wrong.
>>>
>>> Please comment, and send links.
>>>
>>> If I have more time, I'll make it shorter. Or funnier. And add more
>>> pictures, captures, and links. Finding this bug *eventually* got to be
>>> fun.
>>>
>>
>> My home network:
>>
>> FTTH modem (bridge) --- tp-link archer c7
>> (pppoe+sqm+simple_pppoe(fq_codel)+ap) ---- (lan cable) --- tp-link
>> archer c7 v2 with the experimental firmware(ap):
>> Quick and dirty test:
>>
>> on non-patched openwrt wifi(wpa-psk):
>> https://www.dslreports.com/speedtest/5028614
>>
>> on tp-link archer c7 v2 with toke's firmware(wpa-psk):
>> https://www.dslreports.com/speedtest/5028349
>>
>> My brother said that "It feels like ethernet both in terms of
>> throughput and latency". My brother is hammering the the tp-link
>> archer c7 box with bittorrent, whatsapp, viber, and normal http.
>
> Testing on my android smartphone (android 6.0.1):
>
> Unpatched ap:
> https://www.dslreports.com/speedtest/5028965
>
> Patched ap:
> https://www.dslreports.com/speedtest/5028940
>
> On smartphones, it looks like you can get better throughput with
> toke's firmware.

Is this with other stations active? And how is the WiFi configured
(HT20? HT40? 2.4Ghz or 5Ghz?). It's odd if the WiFi is the bottleneck at
those speeds...

-Toke

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[Loganaden Velvindron]

On Sat, Sep 17, 2016 at 9:35 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> Loganaden Velvindron <loganaden@gmail.com> writes:
>
>> On Sat, Sep 17, 2016 at 9:12 PM, Loganaden Velvindron
>> <loganaden@gmail.com> wrote:
>>> On Sat, Sep 17, 2016 at 1:42 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>>> at:
>>>>
>>>> http://blog.cerowrt.org/post/crypto_fq_bug/
>>>>
>>>> I am very interested to try to remember more of the suggestions, and
>>>> blind alleys, and why we thought they were possibilities... and draw
>>>> more conclusions as to what we did right, and wrong.
>>>>
>>>> Please comment, and send links.
>>>>
>>>> If I have more time, I'll make it shorter. Or funnier. And add more
>>>> pictures, captures, and links. Finding this bug *eventually* got to be
>>>> fun.
>>>>
>>>
>>> My home network:
>>>
>>> FTTH modem (bridge) --- tp-link archer c7
>>> (pppoe+sqm+simple_pppoe(fq_codel)+ap) ---- (lan cable) --- tp-link
>>> archer c7 v2 with the experimental firmware(ap):
>>> Quick and dirty test:
>>>
>>> on non-patched openwrt wifi(wpa-psk):
>>> https://www.dslreports.com/speedtest/5028614
>>>
>>> on tp-link archer c7 v2 with toke's firmware(wpa-psk):
>>> https://www.dslreports.com/speedtest/5028349
>>>
>>> My brother said that "It feels like ethernet both in terms of
>>> throughput and latency". My brother is hammering the the tp-link
>>> archer c7 box with bittorrent, whatsapp, viber, and normal http.
>>
>> Testing on my android smartphone (android 6.0.1):
>>
>> Unpatched ap:
>> https://www.dslreports.com/speedtest/5028965
>>
>> Patched ap:
>> https://www.dslreports.com/speedtest/5028940
>>
>> On smartphones, it looks like you can get better throughput with
>> toke's firmware.
>
> Is this with other stations active? And how is the WiFi configured
> (HT20? HT40? 2.4Ghz or 5Ghz?). It's odd if the WiFi is the bottleneck at
> those speeds...
>
2.4Ghz.

By stations, you mean other people connecting to that Access Point ?
Yes, right now, my brother is using bittorrent, whatsapp, viber,
facebook , and other apps.

dmesg shows that the selected algorithm is minstrel_ht.

At least 2 other people are going to switch to the new Access Point.


> -Toke

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[Loganaden Velvindron]

[-- Attachment #1: Type: text/plain, Size: 48 bytes --]

Here are the rrul_all_scaled & ping_cdf graphs:

[-- Attachment #2: cdf_no_patches.png --]
[-- Type: image/png, Size: 71478 bytes --]

[-- Attachment #3: cdf_with_patches.png --]
[-- Type: image/png, Size: 68212 bytes --]

[-- Attachment #4: rrul_no_patches.png --]
[-- Type: image/png, Size: 211272 bytes --]

[-- Attachment #5: rrul_with_patches.png --]
[-- Type: image/png, Size: 212623 bytes --]

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[Dave Taht]

There does not seem to be a statistically significant difference
there. I would encourage you to use rrul_be instead, as we have not
tackled the hardware queue problem in wifi as yet.

It's also unclear what you are testing, the wifi or the link itself.
The uplink and downlink on ethernet are?

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[Loganaden Velvindron]

On Sun, Sep 18, 2016 at 10:19 AM, Dave Taht <dave.taht@gmail.com> wrote:
> There does not seem to be a statistically significant difference
> there. I would encourage you to use rrul_be instead, as we have not
> tackled the hardware queue problem in wifi as yet.
>
> It's also unclear what you are testing, the wifi or the link itself.
> The uplink and downlink on ethernet are?

Yes, the uplink and downlink are on ethernet.

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[Loganaden Velvindron]

On the tp-link archer c7 v2, and toke's firmware, I hit a non reproducible bug:

After running flent a couple of times, the wifi stops working. It's no
longer reachable.

I was using whatsapp a few minutes ago, when the same thing happened.

I can narrow it down to the wifi (bgn), because when I disable the
wifi interface via luci, and re-enable it, wifi works properly.
Unfortunately, there are no relevant messages in the kernel or system
log. I'm using wpa-psk (ccmp) for encryption.

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[Sebastian Moeller]

[-- Attachment #1: Type: text/html, Size: 2609 bytes --]

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[Loganaden Velvindron]

On Mon, Sep 19, 2016 at 12:50 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Loganaden,
>
> this sounds somewhat familiar to me (see
> https://bugs.lede-project.org/index.php?do=details&task_id=176 ). At least
> the symptoms are similar and so is the temporary remedy (calling wifi on the
> CLI). Are you using LEDE firmwares as well? In my case it seems bad radio
> conditions help expose the issue somewhat easier... My current idea is that
> r1482 (commit    372d0fea29e60b02154fd7176ba32e7742f6640e, ' ath9k: add a
> bunch of powersave handling fixes') somehow introduced a new or exposed an
> old and hidden bug for the atheros radios, r1483 (commit
> a894a535ff7e6c37bd853e951663130482bc0ff2, 'mac80211: add fixes for dealing
> with unexpected BlockAck frames') might either have introduced an easier way
> to trigger the offensive conditions, but that is conjecture...
>

I'm using this firmware:

https://kau.toke.dk/lede/airtime-fairness-builds/ar71xx/generic/lede-r1526%2B2-ar71xx-generic-archer-c7-v2-squashfs-factory.bin

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[moeller0]

Hi Loganaden, hi Toke,

> On Sep 19, 2016, at 05:56 , Loganaden Velvindron <loganaden@gmail.com> wrote:
> 
> On Mon, Sep 19, 2016 at 12:50 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi Loganaden,
>> 
>> this sounds somewhat familiar to me (see
>> https://bugs.lede-project.org/index.php?do=details&task_id=176 ). At least
>> the symptoms are similar and so is the temporary remedy (calling wifi on the
>> CLI). Are you using LEDE firmwares as well? In my case it seems bad radio
>> conditions help expose the issue somewhat easier... My current idea is that
>> r1482 (commit    372d0fea29e60b02154fd7176ba32e7742f6640e, ' ath9k: add a
>> bunch of powersave handling fixes') somehow introduced a new or exposed an
>> old and hidden bug for the atheros radios, r1483 (commit
>> a894a535ff7e6c37bd853e951663130482bc0ff2, 'mac80211: add fixes for dealing
>> with unexpected BlockAck frames') might either have introduced an easier way
>> to trigger the offensive conditions, but that is conjecture...
>> 
> 
> I'm using this firmware:
> 
> https://kau.toke.dk/lede/airtime-fairness-builds/ar71xx/generic/lede-r1526%2B2-ar71xx-generic-archer-c7-v2-squashfs-factory.bin


Since this is based on LEDE r1526 it should contain the suspect commits 372d0fea29e60b02154fd7176ba32e7742f6640e and a894a535ff7e6c37bd853e951663130482bc0ff2, so this might be the exact same issue. I seem to be able to force this issue with moving from area of good reception to an area of bad reception (yeah for living in an old house with massive brich walls), maybe that cild work for you as well? I am not sure whether the issue is regarded by the LEDE developers as very critical yet (and rightfully so IMHO, there are only a few reports), so any added report might turn this more into the development spot light…

Best Regards
	Sebastian

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[Toke Høiland-Jørgensen]

moeller0 <moeller0@gmx.de> writes:

> Hi Loganaden, hi Toke,
>
>> On Sep 19, 2016, at 05:56 , Loganaden Velvindron <loganaden@gmail.com> wrote:
>> 
>> On Mon, Sep 19, 2016 at 12:50 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>> 
>>> Hi Loganaden,
>>> 
>>> this sounds somewhat familiar to me (see
>>> https://bugs.lede-project.org/index.php?do=details&task_id=176 ). At least
>>> the symptoms are similar and so is the temporary remedy (calling wifi on the
>>> CLI). Are you using LEDE firmwares as well? In my case it seems bad radio
>>> conditions help expose the issue somewhat easier... My current idea is that
>>> r1482 (commit    372d0fea29e60b02154fd7176ba32e7742f6640e, ' ath9k: add a
>>> bunch of powersave handling fixes') somehow introduced a new or exposed an
>>> old and hidden bug for the atheros radios, r1483 (commit
>>> a894a535ff7e6c37bd853e951663130482bc0ff2, 'mac80211: add fixes for dealing
>>> with unexpected BlockAck frames') might either have introduced an easier way
>>> to trigger the offensive conditions, but that is conjecture...
>>> 
>> 
>> I'm using this firmware:
>> 
>> https://kau.toke.dk/lede/airtime-fairness-builds/ar71xx/generic/lede-r1526%2B2-ar71xx-generic-archer-c7-v2-squashfs-factory.bin
>
> Since this is based on LEDE r1526 it should contain the suspect
> commits 372d0fea29e60b02154fd7176ba32e7742f6640e and
> a894a535ff7e6c37bd853e951663130482bc0ff2, so this might be the exact
> same issue. I seem to be able to force this issue with moving from
> area of good reception to an area of bad reception (yeah for living in
> an old house with massive brich walls), maybe that cild work for you
> as well? I am not sure whether the issue is regarded by the LEDE
> developers as very critical yet (and rightfully so IMHO, there are
> only a few reports), so any added report might turn this more into the
> development spot light…

The patches you are talking about went in as part of debugging the
intermediate queueing implementation. Opening an issue on the LEDE bug
tracker for this might be a good idea...

-Toke

Re: [Make-wifi-fast] crypto-fq bug postmortem is up

[moeller0]

Hi Toke,

> On Sep 19, 2016, at 13:27 , Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> moeller0 <moeller0@gmx.de> writes:
> 
>> Hi Loganaden, hi Toke,
>> 
>>> On Sep 19, 2016, at 05:56 , Loganaden Velvindron <loganaden@gmail.com> wrote:
>>> 
>>> On Mon, Sep 19, 2016 at 12:50 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>>> 
>>>> Hi Loganaden,
>>>> 
>>>> this sounds somewhat familiar to me (see
>>>> https://bugs.lede-project.org/index.php?do=details&task_id=176 ). At least
>>>> the symptoms are similar and so is the temporary remedy (calling wifi on the
>>>> CLI). Are you using LEDE firmwares as well? In my case it seems bad radio
>>>> conditions help expose the issue somewhat easier... My current idea is that
>>>> r1482 (commit    372d0fea29e60b02154fd7176ba32e7742f6640e, ' ath9k: add a
>>>> bunch of powersave handling fixes') somehow introduced a new or exposed an
>>>> old and hidden bug for the atheros radios, r1483 (commit
>>>> a894a535ff7e6c37bd853e951663130482bc0ff2, 'mac80211: add fixes for dealing
>>>> with unexpected BlockAck frames') might either have introduced an easier way
>>>> to trigger the offensive conditions, but that is conjecture...
>>>> 
>>> 
>>> I'm using this firmware:
>>> 
>>> https://kau.toke.dk/lede/airtime-fairness-builds/ar71xx/generic/lede-r1526%2B2-ar71xx-generic-archer-c7-v2-squashfs-factory.bin
>> 
>> Since this is based on LEDE r1526 it should contain the suspect
>> commits 372d0fea29e60b02154fd7176ba32e7742f6640e and
>> a894a535ff7e6c37bd853e951663130482bc0ff2, so this might be the exact
>> same issue. I seem to be able to force this issue with moving from
>> area of good reception to an area of bad reception (yeah for living in
>> an old house with massive brich walls), maybe that cild work for you
>> as well? I am not sure whether the issue is regarded by the LEDE
>> developers as very critical yet (and rightfully so IMHO, there are
>> only a few reports), so any added report might turn this more into the
>> development spot light…
> 
> The patches you are talking about went in as part of debugging the
> intermediate queueing implementation. Opening an issue on the LEDE bug
> tracker for this might be a good idea…

	I believe/hope I already did: https://bugs.lede-project.org/index.php?do=details&task_id=176 There are other reports of similar failures in the tracker, but I am not fully convinced they all have the exact same root cause. I would not be amazed if the commits I indicated above do not introduce real bugs, but rather expose existing issues by making the conditions to trigger the radio loss more likely. 

Best Regards
	Sebastian


> 
> -Toke