* [Make-wifi-fast] Instrumented ATH9K for Crashes?
@ 2017-02-17 3:18 Frank Horowitz
2017-02-17 3:50 ` Dave Taht
0 siblings, 1 reply; 10+ messages in thread
From: Frank Horowitz @ 2017-02-17 3:18 UTC (permalink / raw)
To: make-wifi-fast, toke
[-- Attachment #1: Type: text/plain, Size: 2339 bytes --]
Hi All,
TL/DR: I’ve been seeing reliable crashes from ATH9K drivers in net-next kernels for weeks, but have been unable to capture a crash log.
In an attempt at having a reliably/regularly updatable router running the ATF and BBR codes, I’ve assembled an Atom based Zotac mini-itx board with two different ATH9K based radios. I’ve installed Ubuntu 16.10, with a kernel compiled from Dave Miller’s net-next tree (currently running 4.10-rc7). The radios are set up using 2 different hostapd.conf files (one for the 2.4GHz radio, and one for the 5GHz radio). The motherboard has an RTL8169 ethernet onboard, and I’ve got a 4 port Intel ethernet card also in the mix. The RTL8169 is my WAN port, fed by a DSL modem (running LEDE), and all but one of the other network ports are part of a LAN bridge — the last port is ultimately meant to feed a DMZ, but there’s nothing on it at the moment.
When the radios are not connected to the bridge, everything has run stably for days. When the radios are connected to the bridge, but have no clients, the result has run stably for about 24 hours before I stopped the test.
When a radio is connected to the bridge and has a client, the system reliably crashes within an hour or two.
I’ve tried to get netconsole logs from another linux box on my bridged LAN. but thats a Heisenbug because I can’t get the ATH9K’s to play well with netconsole over the bridge. I think this is due to the lack of polling in the ATH9K driver, but would be delighted to find out that it’s something configurable for those radios. Bottom line, I’ve had no luck in snagging a log from the crashes via netconsole. I’ve also tried looking at the systemd logs, but nothing made it to the log database before the crash.
I could reconfigure my network such that the unbridged DMZ is feeding my external linux box.
Before I try that, I thought I’d ask Toke and the list for advice about any configs for the ATH9K driver that might help with A) capturing a crash log, and/or B) debugging the drivers.
Hopefully, by the time this bites someone else in 4.11 kernels, we’ll have been able to squish this bug. (Just to be explicit, I’m volunteering to be a testbed. Don’t tell my wife! ;-) )
TIA for any hints on how best to proceed.
Frank Horowitz
frank@horow.net
[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 841 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Make-wifi-fast] Instrumented ATH9K for Crashes?
2017-02-17 3:18 [Make-wifi-fast] Instrumented ATH9K for Crashes? Frank Horowitz
@ 2017-02-17 3:50 ` Dave Taht
2017-02-17 9:56 ` Toke Høiland-Jørgensen
0 siblings, 1 reply; 10+ messages in thread
From: Dave Taht @ 2017-02-17 3:50 UTC (permalink / raw)
To: Frank Horowitz; +Cc: make-wifi-fast, Toke Høiland-Jørgensen
It is my hope felix squished it last week.
https://patchwork.kernel.org/patch/9568369/
On Thu, Feb 16, 2017 at 7:18 PM, Frank Horowitz <frank@horow.net> wrote:
> Hi All,
>
> TL/DR: I’ve been seeing reliable crashes from ATH9K drivers in net-next kernels for weeks, but have been unable to capture a crash log.
>
> In an attempt at having a reliably/regularly updatable router running the ATF and BBR codes, I’ve assembled an Atom based Zotac mini-itx board with two different ATH9K based radios. I’ve installed Ubuntu 16.10, with a kernel compiled from Dave Miller’s net-next tree (currently running 4.10-rc7). The radios are set up using 2 different hostapd.conf files (one for the 2.4GHz radio, and one for the 5GHz radio). The motherboard has an RTL8169 ethernet onboard, and I’ve got a 4 port Intel ethernet card also in the mix. The RTL8169 is my WAN port, fed by a DSL modem (running LEDE), and all but one of the other network ports are part of a LAN bridge — the last port is ultimately meant to feed a DMZ, but there’s nothing on it at the moment.
>
> When the radios are not connected to the bridge, everything has run stably for days. When the radios are connected to the bridge, but have no clients, the result has run stably for about 24 hours before I stopped the test.
>
> When a radio is connected to the bridge and has a client, the system reliably crashes within an hour or two.
>
> I’ve tried to get netconsole logs from another linux box on my bridged LAN. but thats a Heisenbug because I can’t get the ATH9K’s to play well with netconsole over the bridge. I think this is due to the lack of polling in the ATH9K driver, but would be delighted to find out that it’s something configurable for those radios. Bottom line, I’ve had no luck in snagging a log from the crashes via netconsole. I’ve also tried looking at the systemd logs, but nothing made it to the log database before the crash.
>
> I could reconfigure my network such that the unbridged DMZ is feeding my external linux box.
>
> Before I try that, I thought I’d ask Toke and the list for advice about any configs for the ATH9K driver that might help with A) capturing a crash log, and/or B) debugging the drivers.
>
> Hopefully, by the time this bites someone else in 4.11 kernels, we’ll have been able to squish this bug. (Just to be explicit, I’m volunteering to be a testbed. Don’t tell my wife! ;-) )
>
> TIA for any hints on how best to proceed.
>
> Frank Horowitz
> frank@horow.net
>
>
>
>
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
--
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Make-wifi-fast] Instrumented ATH9K for Crashes?
2017-02-17 3:50 ` Dave Taht
@ 2017-02-17 9:56 ` Toke Høiland-Jørgensen
2017-02-17 15:47 ` Frank Horowitz
0 siblings, 1 reply; 10+ messages in thread
From: Toke Høiland-Jørgensen @ 2017-02-17 9:56 UTC (permalink / raw)
To: Dave Taht; +Cc: Frank Horowitz, make-wifi-fast
Dave Taht <dave.taht@gmail.com> writes:
> It is my hope felix squished it last week.
>
> https://patchwork.kernel.org/patch/9568369/
That is a fix to the airtime fairness patch, though, which is not in
4.10. But if the kernel is from net-next, it's probably not actually
4.10; Frank, did you get that from git describe? Because then it's
probably not 4.10-rc7, but 4.10-rc7+something, which is different (yeah,
git describe is annoying that way).
As far as instrumenting a box for capturing logs, I ended up using an
old-fashioned serial cable to another machine in my test setup...
-Toke
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Make-wifi-fast] Instrumented ATH9K for Crashes?
2017-02-17 9:56 ` Toke Høiland-Jørgensen
@ 2017-02-17 15:47 ` Frank Horowitz
2017-02-17 16:22 ` Toke Høiland-Jørgensen
0 siblings, 1 reply; 10+ messages in thread
From: Frank Horowitz @ 2017-02-17 15:47 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Dave Taht, make-wifi-fast
[-- Attachment #1: Type: text/plain, Size: 903 bytes --]
Git describe shows v4.10-rc7-1866-g135f8e5 and I’m definitely on the net-next tree. I’m going to try the patch Dave just pointed to to see if it applies cleanly.
Frank
> On Feb 17, 2017, at 4:56 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Dave Taht <dave.taht@gmail.com> writes:
>
>> It is my hope felix squished it last week.
>>
>> https://patchwork.kernel.org/patch/9568369/
>
> That is a fix to the airtime fairness patch, though, which is not in
> 4.10. But if the kernel is from net-next, it's probably not actually
> 4.10; Frank, did you get that from git describe? Because then it's
> probably not 4.10-rc7, but 4.10-rc7+something, which is different (yeah,
> git describe is annoying that way).
>
> As far as instrumenting a box for capturing logs, I ended up using an
> old-fashioned serial cable to another machine in my test setup...
>
> -Toke
[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 841 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Make-wifi-fast] Instrumented ATH9K for Crashes?
2017-02-17 15:47 ` Frank Horowitz
@ 2017-02-17 16:22 ` Toke Høiland-Jørgensen
2017-02-17 16:24 ` Frank Horowitz
0 siblings, 1 reply; 10+ messages in thread
From: Toke Høiland-Jørgensen @ 2017-02-17 16:22 UTC (permalink / raw)
To: Frank Horowitz; +Cc: Dave Taht, make-wifi-fast
Frank Horowitz <frank@horow.net> writes:
> Git describe shows v4.10-rc7-1866-g135f8e5 and I’m definitely on the
> net-next tree. I’m going to try the patch Dave just pointed to to see
> if it applies cleanly.
Ah, right, yeah, that is 1866 commits *after* 4.10-rc7.
I think the patch was recently merged into net-next, actually. At least
it was in a pull request...
-Toke
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Make-wifi-fast] Instrumented ATH9K for Crashes?
2017-02-17 16:22 ` Toke Høiland-Jørgensen
@ 2017-02-17 16:24 ` Frank Horowitz
2017-02-19 13:58 ` Dave Taht
0 siblings, 1 reply; 10+ messages in thread
From: Frank Horowitz @ 2017-02-17 16:24 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Dave Taht, make-wifi-fast
[-- Attachment #1: Type: text/plain, Size: 667 bytes --]
Well, Felix’s patch applied cleanly at least. I’m waiting for the compile to finish, and will install and test shortly.
> On Feb 17, 2017, at 11:22 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Frank Horowitz <frank@horow.net> writes:
>
>> Git describe shows v4.10-rc7-1866-g135f8e5 and I’m definitely on the
>> net-next tree. I’m going to try the patch Dave just pointed to to see
>> if it applies cleanly.
>
> Ah, right, yeah, that is 1866 commits *after* 4.10-rc7.
>
> I think the patch was recently merged into net-next, actually. At least
> it was in a pull request...
>
> -Toke
Frank Horowitz
frank@horow.net
[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 841 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Make-wifi-fast] Instrumented ATH9K for Crashes?
2017-02-17 16:24 ` Frank Horowitz
@ 2017-02-19 13:58 ` Dave Taht
2017-02-19 14:27 ` Frank Horowitz
0 siblings, 1 reply; 10+ messages in thread
From: Dave Taht @ 2017-02-19 13:58 UTC (permalink / raw)
To: Frank Horowitz; +Cc: Toke Høiland-Jørgensen, make-wifi-fast
On Fri, Feb 17, 2017 at 8:24 AM, Frank Horowitz <frank@horow.net> wrote:
> Well, Felix’s patch applied cleanly at least. I’m waiting for the compile to finish, and will install and test shortly.
Really long compile...
>
>> On Feb 17, 2017, at 11:22 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Frank Horowitz <frank@horow.net> writes:
>>
>>> Git describe shows v4.10-rc7-1866-g135f8e5 and I’m definitely on the
>>> net-next tree. I’m going to try the patch Dave just pointed to to see
>>> if it applies cleanly.
>>
>> Ah, right, yeah, that is 1866 commits *after* 4.10-rc7.
>>
>> I think the patch was recently merged into net-next, actually. At least
>> it was in a pull request...
>>
>> -Toke
>
> Frank Horowitz
> frank@horow.net
>
>
>
--
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Make-wifi-fast] Instrumented ATH9K for Crashes?
2017-02-19 13:58 ` Dave Taht
@ 2017-02-19 14:27 ` Frank Horowitz
2017-02-19 14:45 ` Dave Taht
0 siblings, 1 reply; 10+ messages in thread
From: Frank Horowitz @ 2017-02-19 14:27 UTC (permalink / raw)
To: Dave Taht; +Cc: Toke Høiland-Jørgensen, make-wifi-fast
[-- Attachment #1: Type: text/plain, Size: 1252 bytes --]
Yeah. Life intervened.
I’ve now had the radios attached to the bridge and with clients for a solid 24 hours without crashes. So far, so good.
Fingers crossed…
Frank
> On Feb 19, 2017, at 8:58 AM, Dave Taht <dave.taht@gmail.com> wrote:
>
> On Fri, Feb 17, 2017 at 8:24 AM, Frank Horowitz <frank@horow.net> wrote:
>> Well, Felix’s patch applied cleanly at least. I’m waiting for the compile to finish, and will install and test shortly.
>
> Really long compile...
>
>>
>>> On Feb 17, 2017, at 11:22 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>
>>> Frank Horowitz <frank@horow.net> writes:
>>>
>>>> Git describe shows v4.10-rc7-1866-g135f8e5 and I’m definitely on the
>>>> net-next tree. I’m going to try the patch Dave just pointed to to see
>>>> if it applies cleanly.
>>>
>>> Ah, right, yeah, that is 1866 commits *after* 4.10-rc7.
>>>
>>> I think the patch was recently merged into net-next, actually. At least
>>> it was in a pull request...
>>>
>>> -Toke
>>
>> Frank Horowitz
>> frank@horow.net
>>
>>
>>
>
>
>
> --
> Dave Täht
> Let's go make home routers and wifi faster! With better software!
> http://blog.cerowrt.org
Frank Horowitz
frank@horow.net
[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 841 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Make-wifi-fast] Instrumented ATH9K for Crashes?
2017-02-19 14:27 ` Frank Horowitz
@ 2017-02-19 14:45 ` Dave Taht
2017-02-19 14:48 ` Frank Horowitz
0 siblings, 1 reply; 10+ messages in thread
From: Dave Taht @ 2017-02-19 14:45 UTC (permalink / raw)
To: Frank Horowitz; +Cc: Toke Høiland-Jørgensen, make-wifi-fast
On Sun, Feb 19, 2017 at 6:27 AM, Frank Horowitz <frank@horow.net> wrote:
> Yeah. Life intervened.
>
> I’ve now had the radios attached to the bridge and with clients for a solid 24 hours without crashes. So far, so good.
My apologies to your wife^Hi and family for the bug.
>
> Fingers crossed…
>
> Frank
>
>> On Feb 19, 2017, at 8:58 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> On Fri, Feb 17, 2017 at 8:24 AM, Frank Horowitz <frank@horow.net> wrote:
>>> Well, Felix’s patch applied cleanly at least. I’m waiting for the compile to finish, and will install and test shortly.
>>
>> Really long compile...
>>
>>>
>>>> On Feb 17, 2017, at 11:22 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>>
>>>> Frank Horowitz <frank@horow.net> writes:
>>>>
>>>>> Git describe shows v4.10-rc7-1866-g135f8e5 and I’m definitely on the
>>>>> net-next tree. I’m going to try the patch Dave just pointed to to see
>>>>> if it applies cleanly.
>>>>
>>>> Ah, right, yeah, that is 1866 commits *after* 4.10-rc7.
>>>>
>>>> I think the patch was recently merged into net-next, actually. At least
>>>> it was in a pull request...
>>>>
>>>> -Toke
>>>
>>> Frank Horowitz
>>> frank@horow.net
>>>
>>>
>>>
>>
>>
>>
>> --
>> Dave Täht
>> Let's go make home routers and wifi faster! With better software!
>> http://blog.cerowrt.org
>
> Frank Horowitz
> frank@horow.net
>
>
>
--
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Make-wifi-fast] Instrumented ATH9K for Crashes?
2017-02-19 14:45 ` Dave Taht
@ 2017-02-19 14:48 ` Frank Horowitz
0 siblings, 0 replies; 10+ messages in thread
From: Frank Horowitz @ 2017-02-19 14:48 UTC (permalink / raw)
To: Dave Taht; +Cc: Toke Høiland-Jørgensen, make-wifi-fast
[-- Attachment #1: Type: text/plain, Size: 1969 bytes --]
Bugs happen.
Kudos to all (and especially Felix!) for finding and (apparently) squashing this nasty!
If I haven’t said it before, this entire community just ROCKS. Testing is what I can do to contribute.
Frank
> On Feb 19, 2017, at 9:45 AM, Dave Taht <dave.taht@gmail.com> wrote:
>
> On Sun, Feb 19, 2017 at 6:27 AM, Frank Horowitz <frank@horow.net> wrote:
>> Yeah. Life intervened.
>>
>> I’ve now had the radios attached to the bridge and with clients for a solid 24 hours without crashes. So far, so good.
>
> My apologies to your wife^Hi and family for the bug.
>>
>> Fingers crossed…
>>
>> Frank
>>
>>> On Feb 19, 2017, at 8:58 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>> On Fri, Feb 17, 2017 at 8:24 AM, Frank Horowitz <frank@horow.net> wrote:
>>>> Well, Felix’s patch applied cleanly at least. I’m waiting for the compile to finish, and will install and test shortly.
>>>
>>> Really long compile...
>>>
>>>>
>>>>> On Feb 17, 2017, at 11:22 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>>>
>>>>> Frank Horowitz <frank@horow.net> writes:
>>>>>
>>>>>> Git describe shows v4.10-rc7-1866-g135f8e5 and I’m definitely on the
>>>>>> net-next tree. I’m going to try the patch Dave just pointed to to see
>>>>>> if it applies cleanly.
>>>>>
>>>>> Ah, right, yeah, that is 1866 commits *after* 4.10-rc7.
>>>>>
>>>>> I think the patch was recently merged into net-next, actually. At least
>>>>> it was in a pull request...
>>>>>
>>>>> -Toke
>>>>
>>>> Frank Horowitz
>>>> frank@horow.net
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Dave Täht
>>> Let's go make home routers and wifi faster! With better software!
>>> http://blog.cerowrt.org
>>
>> Frank Horowitz
>> frank@horow.net
>>
>>
>>
>
>
>
> --
> Dave Täht
> Let's go make home routers and wifi faster! With better software!
> http://blog.cerowrt.org
Frank Horowitz
frank@horow.net
[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 841 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2017-02-19 14:48 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-02-17 3:18 [Make-wifi-fast] Instrumented ATH9K for Crashes? Frank Horowitz
2017-02-17 3:50 ` Dave Taht
2017-02-17 9:56 ` Toke Høiland-Jørgensen
2017-02-17 15:47 ` Frank Horowitz
2017-02-17 16:22 ` Toke Høiland-Jørgensen
2017-02-17 16:24 ` Frank Horowitz
2017-02-19 13:58 ` Dave Taht
2017-02-19 14:27 ` Frank Horowitz
2017-02-19 14:45 ` Dave Taht
2017-02-19 14:48 ` Frank Horowitz
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox