<div dir="ltr">Hi everybody,<br><br>I've been using cerowrt as a secondary wifi network (just a single AP for now) for a few weeks now. Recently, my wndr3800 got stuck in a bad state and eventually rebooted. I've had this happen a few times now and am looking for ways to debug the issue. I'm new to cerowrt and openwrt so any advice is appreciated.<div>
<div><br></div><div>Since I use it as a secondary network, this is no way critical. I'm not looking for free tech support but I couldn't find anything on the wiki about troubleshooting. I'd love to start a page and write some shell scripts to diagnose and report issues. I know that a cerowrt router is meant to be a research project rather a consumer device but these things seem helpful regardless.</div>
<div><br></div><div>Please let me know if you'd prefer I not email the list with these issues or if you'd rather I used trac or a different forum.</div><div><br></div><div><br></div><div>--<br></div><div><br>In this state, I can connect to the cerowrt base station via wifi but am unable to route packets to the internet. I can connect to :81 and see the login page but logging in results in a lua error at `/cgi-bin/luci`</div>
<div><br><br> /usr/lib/lua/luci/dispatcher.lua:448: Failed to execute function dispatcher target for entry '/'.<br>
The called action terminated with an exception:<br> /usr/lib/lua/luci/sauth.lua:87: Session data invalid!<br> stack traceback:<br> [C]: in function 'assert'<br> /usr/lib/lua/luci/dispatcher.lua:448: in function 'dispatch'<br>
/usr/lib/lua/luci/dispatcher.lua:195: in function </usr/lib/lua/luci/dispatcher.lua:194><br><br>I can ssh into the device and cat various log files until the router hangs and reboots. here's a few relevant lines from my terminal history before the device rebooted (I'm assuming a watchdog kicked in and rebooted it).</div>
<div><br>root@buffy2-1:~# ping <a href="http://google.com" target="_blank">google.com</a><br>ping: bad address '<a href="http://google.com" target="_blank">google.com</a>'<br>root@buffy2-1:~# free<br>
total used free shared buffers<br>Mem: 126336 110332 16004 0 5616<br>-/+ buffers: 104716 21620<br>Swap: 0 0 0<br>
root@buffy2-1:~# uptime<br> 02:08:54 up 2 days, 1:26, load average: 0.10, 0.21, 0.17<br>root@buffy2-1:~# dmesg<br>[ 0.000000] Linux version 3.10.24 (cero2@snapon) (gcc version 4.6.4 (OpenWrt/Linaro GCC 4.6-2013.05 r38226) ) #1 Tue Dec 24 <br>
10:50:15 PST 2013<br>[skipping some lines]<br><br>[ 13.156250] Error: Driver 'gpio-keys-polled' is already registered, aborting...<br>[ 19.414062] IPv6: ADDRCONF(NETDEV_UP): ge00: link is not ready<br>[ 19.421875] ar71xx: pll_reg 0xb8050010: 0x11110000<br>
[ 19.429687] se00: link up (1000Mbps/Full duplex)<br>[ 22.140625] IPv6: ADDRCONF(NETDEV_UP): sw00: link is not ready<br>[ 23.351562] IPv6: ADDRCONF(NETDEV_CHANGE): sw00: link becomes ready<br>[ 23.757812] ar71xx: pll_reg 0xb8050014: 0x11110000<br>
[ 23.757812] ge00: link up (1000Mbps/Full duplex)<br>[ 23.773437] IPv6: ADDRCONF(NETDEV_CHANGE): ge00: link becomes ready<br><br>root@buffy2-1:~# ifconfig<br>
ge00 Link encap:Ethernet HWaddr 2C:B0:5D:A0:C5:B1 <br> inet addr:192.168.1.138 Bcast:192.168.1.255 Mask:255.255.255.0<br> inet6 addr: fe80::2eb0:5dff:fea0:c5b1/64 Scope:Link<br> inet6 addr: 2602:30a:2cdb:330:2eb0:5dff:fea0:c5b1/64 Scope:Global<br>
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br> RX packets:1469670 errors:0 dropped:8 overruns:0 frame:0<br> TX packets:547733 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:1000 <br>
RX bytes:229243410 (218.6 MiB) TX bytes:57304808 (54.6 MiB)<br> Interrupt:5 <br><br>lo Link encap:Local Loopback <br> inet addr:127.0.0.1 Mask:255.0.0.0<br> inet6 addr: ::1/128 Scope:Host<br>
UP LOOPBACK RUNNING MTU:65536 Metric:1<br> RX packets:23689 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:23689 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:0 <br>
RX bytes:2612713 (2.4 MiB) TX bytes:2612713 (2.4 MiB)<br><br>pimreg Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 <br> UP RUNNING NOARP MTU:1472 Metric:1<br> RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br>
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:0 <br> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)<br><br>se00 Link encap:Ethernet HWaddr 2E:B0:5D:A0:C5:B0 <br>
inet addr:172.30.42.1 Bcast:172.30.42.31 Mask:255.255.255.224<br> inet6 addr: 2602:30a:2cdb:330:2eb0:5dff:fea0:c5b1/64 Scope:Global<br> inet6 addr: fe80::2cb0:5dff:fea0:c5b0/64 Scope:Link<br>
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br> RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:191740 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:1000 <br>
RX bytes:0 (0.0 B) TX bytes:42184988 (40.2 MiB)<br> Interrupt:4 <br><br>sw00 Link encap:Ethernet HWaddr 2C:B0:5D:A0:C5:B0 <br> inet addr:172.30.42.65 Bcast:172.30.42.95 Mask:255.255.255.224<br>
inet6 addr: 2602:30a:2cdb:330:2eb0:5dff:fea0:c5b1/64 Scope:Global<br> inet6 addr: fe80::2eb0:5dff:fea0:c5b0/64 Scope:Link<br> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br> RX packets:70239 errors:0 dropped:0 overruns:0 frame:0<br>
TX packets:286967 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:1000 <br> RX bytes:15590189 (14.8 MiB) TX bytes:127357293 (121.4 MiB)<br><br>root@buffy2-1:~# less /var/log/babeld.log<br>
Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>
send: Cannot assign requested address<br>send: Cannot assign requested address<br>send: Cannot assign requested address<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>
Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>netlink_read: recvmsg(): No buffer space available<br>Couldn't determine channel of interface sw00: Invalid argument.<br>
Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>
Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>
Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>
Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>
Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>
Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>Couldn't determine channel of interface sw00: Invalid argument.<br>
</quote><br><br>and then the system hang. Note: the first time I ran 'less<br>/var/log/babeld.log', it was Killed. I would assume that's an OOM<br>killer?</div><div>
<br>I'm running cerowrt 3.10.24-8 on a wndr3800. I have a spare wndr3800</div><div>that I'm getting ready to flash with the same version and I can see if I get</div><div>the same issue. What's the best way to clone a configuration?<br>
<br>---<br><br>Here are some log lines from a separate instance of the same problem, I</div><div>recovered these log lines from the web server (I was able to login to</div><div>the web interface for another hour until I got the same lua exception</div>
<div>as in the beginning of this email)</div><div><br></div><div>This time I was completely unable to connect to wifi from my iPhone</div><div>and you can see where hostapd disassociates the iPhone soon after</div><div>it connects.</div>
<div><br></div><div>the kernel log:<br><br>[158995.550781] icmp6_send: no reply to icmp error<br>
<br>[160930.527343] ath: phy0: Failed to stop TX DMA, queues=0x004!<br><br>[160930.542968] ath: phy0: DMA failed to stop in 10 ms AR_CR=0x00000024 AR_DIAG_SW=0x42000020 DMADBG_7=0x000084c0<br><br>[160930.554687] ath: phy0: Could not stop RX, we could be confusing the DMA engine when we start RX up<br>
<br>and in the system log:<br><br>Fri Jan 24 15:51:15 2014 daemon.err minissdpd[4933]: recvmsg(s, &hdr, 0): No buffer space available<br><br>Fri Jan 24 15:51:15 2014 daemon.err avahi-daemon[1273]: netlink.c: recvmsg() failed: No buffer space available<br>
<br>Fri Jan 24 15:52:18 2014 daemon.err minissdpd[4933]: recvmsg(s, &hdr, 0): No buffer space available<br><br><br>root@buffy2-1:/tmp/log# free<br><br> total used free shared buffers<br>
<br>Mem: 126336 110128 16208 0 5616<br><br>-/+ buffers: 104512 21824<br><br>Swap: 0 0 0<br><br><br>Fri Jan 24 17:14:53 2014 daemon.err minissdpd[4933]: recvmsg(s, &hdr, 0): No buffer space available<br>
<br>Fri Jan 24 17:15:53 2014 daemon.err minissdpd[4933]: recvmsg(s, &hdr, 0): No buffer space available<br>
<br>Fri Jan 24 17:16:18 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> hostapd: sw00: STA 90:72:40:e9:9c:b6 IEEE 802.11: authenticated<br><br>Fri Jan 24 17:16:18 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> hostapd: sw00: STA 90:72:40:e9:9c:b6 IEEE 802.11: associated (aid 1)<br>
<br>Fri Jan 24 17:16:18 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> hostapd: sw00: STA 90:72:40:e9:9c:b6 RADIUS: starting accounting session 52DC70A7-00000010<br><br>Fri Jan 24 17:16:18 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> hostapd: sw00: STA 90:72:40:e9:9c:b6 WPA: pairwise key handshake completed (RSN)<br>
<br>Fri Jan 24 17:16:18 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> dnsmasq-dhcp[4999]: DHCPREQUEST(sw00) 192.168.1.114 90:72:40:e9:9c:b6 <br><br>Fri Jan 24 17:16:18 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> dnsmasq-dhcp[4999]: DHCPNAK(sw00) 192.168.1.114 90:72:40:e9:9c:b6 wrong network<br>
<br>Fri Jan 24 17:16:21 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> dnsmasq-dhcp[4999]: DHCPDISCOVER(sw00) 90:72:40:e9:9c:b6 <br><br>Fri Jan 24 17:16:21 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> dnsmasq-dhcp[4999]: DHCPOFFER(sw00) 172.30.42.82 90:72:40:e9:9c:b6 <br>
<br>Fri Jan 24 17:16:21 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> dnsmasq-dhcp[4999]: DHCPDISCOVER(sw00) 90:72:40:e9:9c:b6 <br><br>Fri Jan 24 17:16:21 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> dnsmasq-dhcp[4999]: DHCPOFFER(sw00) 172.30.42.82 90:72:40:e9:9c:b6 <br>
<br>Fri Jan 24 17:16:22 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> dnsmasq-dhcp[4999]: DHCPREQUEST(sw00) 172.30.42.82 90:72:40:e9:9c:b6 <br><br>Fri Jan 24 17:16:22 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> dnsmasq-dhcp[4999]: DHCPACK(sw00) 172.30.42.82 90:72:40:e9:9c:b6 stevejs-iPhone<br>
<br>Fri Jan 24 17:16:35 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> hostapd: sw00: STA 90:72:40:e9:9c:b6 IEEE 802.11: disassociated<br><br>Fri Jan 24 17:16:36 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> hostapd: sw00: STA 90:72:40:e9:9c:b6 IEEE 802.11: deauthenticated due to inactivity (timer DEAUTH/REMOVE)<br>
<br>Fri Jan 24 17:16:57 2014 daemon.err minissdpd[4933]: recvmsg(s, &hdr, 0): No buffer space available<br><br>Fri Jan 24 17:17:59 2014 daemon.err minissdpd[4933]: recvmsg(s, &hdr, 0): No buffer space available<br>
<br>Fri Jan 24 17:19:04 2014 daemon.err minissdpd[4933]: recvmsg(s, &hdr, 0): No buffer space available<br><br>Fri Jan 24 17:19:04 2014 daemon.err avahi-daemon[1273]: netlink.c: recvmsg() failed: No buffer space available<br>
<br>Fri Jan 24 17:19:45 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> hostapd: sw00: STA 90:72:40:e9:9c:b6 IEEE 802.11: authenticated<br><br>Fri Jan 24 17:19:45 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> hostapd: sw00: STA 90:72:40:e9:9c:b6 IEEE 802.11: associated (aid 1)<br>
<br>Fri Jan 24 17:19:45 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> hostapd: sw00: STA 90:72:40:e9:9c:b6 RADIUS: starting accounting session 52DC70A7-00000011<br><br>Fri Jan 24 17:19:45 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> hostapd: sw00: STA 90:72:40:e9:9c:b6 WPA: pairwise key handshake completed (RSN)<br>
<br>Fri Jan 24 17:19:45 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> dnsmasq-dhcp[4999]: DHCPDISCOVER(sw00) 90:72:40:e9:9c:b6 <br><br>Fri Jan 24 17:19:45 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> dnsmasq-dhcp[4999]: DHCPOFFER(sw00) 172.30.42.82 90:72:40:e9:9c:b6 <br>
<br>Fri Jan 24 17:19:47 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> dnsmasq-dhcp[4999]: DHCPREQUEST(sw00) 172.30.42.82 90:72:40:e9:9c:b6 <br><br>Fri Jan 24 17:19:47 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> dnsmasq-dhcp[4999]: DHCPACK(sw00) 172.30.42.82 90:72:40:e9:9c:b6 stevejs-iPhone<br>
<br>Fri Jan 24 17:20:01 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> hostapd: sw00: STA 90:72:40:e9:9c:b6 IEEE 802.11: disassociated<br><br>Fri Jan 24 17:20:02 2014 <a href="http://daemon.info" target="_blank">daemon.info</a> hostapd: sw00: STA 90:72:40:e9:9c:b6 IEEE 802.11: deauthenticated due to inactivity (timer DEAUTH/REMOVE)<br>
<br>Fri Jan 24 17:20:09 2014 daemon.err minissdpd[4933]: recvmsg(s, &hdr, 0): No buffer space available<br><br>Fri Jan 24 17:21:10 2014 daemon.err minissdpd[4933]: recvmsg(s, &hdr, 0): No buffer space available<br>
<br>and then it hung and rebooted.</div><div><br></div><div>I bought this particular wndr3800 used so it's possible I have a bad piece of hardware. Are there any hardware testing scripts or anything like memtest86 for these devices? Are there entries in /sys that I should be looking at? I've noticed about 9k unaligned_instructions on boot but that never grows.</div>
<div><br>Thanks!<br>Steve Jenson<br>@stevej<br><br><br><br>
</div></div></div>