From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ie0-f172.google.com (mail-ie0-f172.google.com [209.85.223.172]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 0348321F09E for ; Mon, 27 Jan 2014 13:10:59 -0800 (PST) Received: by mail-ie0-f172.google.com with SMTP id e14so6625726iej.31 for ; Mon, 27 Jan 2014 13:10:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=YsQWiE3lfHd8YE3aw+pF6QjylEYt4Xc6GD6rchm/XDk=; b=PhfRlQDsNSjIvFU/K3rIZ/m5bGEaxyRurQcSWgKy6csjii4IGBuNPAI3nfowjtqgtP M+LzeicTLzoOFBZgoKzSX8z+4TM9noeqf4megZc/VRiAhKMzl4SYNru2R53AGR6Tik7G B1vGnL3vSdmkepveB/dB4iLogJSbgXiU+fsn7upjIQDpih3YLZTq1uaSn3Vo27F5oG4E IB3byheOFzWIGLm+Eoh0487gQHGq59gyffFZzrBJaTvFzczdEXlX6vEREigyogh9Q1qY jP1rFjDFj+6E9Fl4rmYyw41WQW65j7biQpAukFSuwGGlRDMTPMlR9A8uHNplr1vIEt0q xv4A== X-Gm-Message-State: ALoCoQk5p2Syq2HZrGKA83v1oxFC3P/tN9rijQXM+O9zUMKmwwD3Pdj6ev6LjUXiDF0VocXRDIzr X-Received: by 10.51.15.130 with SMTP id fo2mr19464050igd.28.1390857059212; Mon, 27 Jan 2014 13:10:59 -0800 (PST) MIME-Version: 1.0 Received: by 10.50.60.33 with HTTP; Mon, 27 Jan 2014 13:10:39 -0800 (PST) X-Originating-IP: [162.205.176.51] In-Reply-To: References: From: Steve Jenson Date: Mon, 27 Jan 2014 13:10:39 -0800 Message-ID: To: Dave Taht Content-Type: multipart/alternative; boundary=001a1134b50ef2b6ad04f0fa24e8 Cc: "cerowrt-devel@lists.bufferbloat.net" Subject: Re: [Cerowrt-devel] cerowrt issues (3.10.24-8) X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jan 2014 21:11:00 -0000 --001a1134b50ef2b6ad04f0fa24e8 Content-Type: text/plain; charset=ISO-8859-1 Looking more, the buffer errors are showing up in syslog well before tmpfs fills up. Is the memtester openwrt package available for cerowrt? I don't see it under `Available packages`. Thanks, Steve On Mon, Jan 27, 2014 at 1:06 PM, Steve Jenson wrote: > On Fri, Jan 24, 2014 at 3:23 PM, Dave Taht wrote: > >> On Fri, Jan 24, 2014 at 6:08 PM, Steve Jenson >> wrote: >> > Hi everybody, >> > >> > I've been using cerowrt as a secondary wifi network (just a single AP >> for >> > now) for a few weeks now. Recently, my wndr3800 got stuck in a bad >> state and >> > eventually rebooted. I've had this happen a few times now and am >> looking for >> > ways to debug the issue. I'm new to cerowrt and openwrt so any advice is >> > appreciated. >> > >> > Since I use it as a secondary network, this is no way critical. >> >> Yea! I appreciate caution before putting alpha software on your gw. >> >> > I'm not >> > looking for free tech support but I couldn't find anything on the wiki >> about >> > troubleshooting. I'd love to start a page and write some shell scripts >> to >> > diagnose and report issues. I know that a cerowrt router is meant to be >> a >> > research project rather a consumer device but these things seem helpful >> > regardless. >> >> Sure, let me know your wiki account. I have been lax about granting >> access of late as the signup process is overrun by spammers. > > > My username is stevej on the wiki. Thanks! > > > >> > Please let me know if you'd prefer I not email the list with these >> issues or >> > if you'd rather I used trac or a different forum. >> > >> >> The list is where most stuff happens. Also in the irc channel. >> >> If it gets to where it needs to be tracked we have a bugtracker at >> >> http://www.bufferbloat.net/projects/cerowrt/issues >> >> The first question I have is: Are you on comcast? Cerowrt >> had a dhcpv6-pd implementation that "just worked" from feburary through >> december. Regrettably they changed the RA announcement interval >> to a really low number around then... and this triggers a firewall reload >> every minute on everything prior to the release I point to below. >> >> If there is a memory leak somewhere that would have triggered it. > > > I am on AT&T ADSL2+ with a Motorola NVG510 modem. > > > >> > In this state, I can connect to the cerowrt base station via wifi but am >> > unable to route packets to the internet. I can connect to :81 and see >> the >> > login page but logging in results in a lua error at `/cgi-bin/luci` >> > >> > >> > /usr/lib/lua/luci/dispatcher.lua:448: Failed to execute function >> > dispatcher target for entry '/'. >> > The called action terminated with an exception: >> > /usr/lib/lua/luci/sauth.lua:87: Session data invalid! >> > stack traceback: >> > [C]: in function 'assert' >> > /usr/lib/lua/luci/dispatcher.lua:448: in function 'dispatch' >> > /usr/lib/lua/luci/dispatcher.lua:195: in function >> > >> > >> > I can ssh into the device and cat various log files until the router >> hangs >> > and reboots. here's a few relevant lines from my terminal history >> before the >> > device rebooted (I'm assuming a watchdog kicked in and rebooted it). >> > >> > root@buffy2-1:~# ping google.com >> > ping: bad address 'google.com' >> > root@buffy2-1:~# free >> > total used free shared buffers >> > Mem: 126336 110332 16004 0 5616 >> > -/+ buffers: 104716 21620 >> > Swap: 0 0 0 >> > root@buffy2-1:~# uptime >> > 02:08:54 up 2 days, 1:26, load average: 0.10, 0.21, 0.17 >> > root@buffy2-1:~# dmesg >> > [ 0.000000] Linux version 3.10.24 (cero2@snapon) (gcc version 4.6.4 >> > (OpenWrt/Linaro GCC 4.6-2013.05 r38226) ) #1 Tue Dec 24 >> > 10:50:15 PST 2013 >> > [skipping some lines] >> > >> > [ 13.156250] Error: Driver 'gpio-keys-polled' is already registered, >> > aborting... >> > [ 19.414062] IPv6: ADDRCONF(NETDEV_UP): ge00: link is not ready >> > [ 19.421875] ar71xx: pll_reg 0xb8050010: 0x11110000 >> > [ 19.429687] se00: link up (1000Mbps/Full duplex) >> > [ 22.140625] IPv6: ADDRCONF(NETDEV_UP): sw00: link is not ready >> > [ 23.351562] IPv6: ADDRCONF(NETDEV_CHANGE): sw00: link becomes ready >> > [ 23.757812] ar71xx: pll_reg 0xb8050014: 0x11110000 >> > [ 23.757812] ge00: link up (1000Mbps/Full duplex) >> > [ 23.773437] IPv6: ADDRCONF(NETDEV_CHANGE): ge00: link becomes ready >> > >> > root@buffy2-1:~# ifconfig >> > ge00 Link encap:Ethernet HWaddr 2C:B0:5D:A0:C5:B1 >> > inet addr:192.168.1.138 Bcast:192.168.1.255 >> Mask:255.255.255.0 >> > inet6 addr: fe80::2eb0:5dff:fea0:c5b1/64 Scope:Link >> > inet6 addr: 2602:30a:2cdb:330:2eb0:5dff:fea0:c5b1/64 >> Scope:Global >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> > RX packets:1469670 errors:0 dropped:8 overruns:0 frame:0 >> > TX packets:547733 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:1000 >> > RX bytes:229243410 (218.6 MiB) TX bytes:57304808 (54.6 MiB) >> > Interrupt:5 >> > >> > lo Link encap:Local Loopback >> > inet addr:127.0.0.1 Mask:255.0.0.0 >> > inet6 addr: ::1/128 Scope:Host >> > UP LOOPBACK RUNNING MTU:65536 Metric:1 >> > RX packets:23689 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:23689 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:0 >> > RX bytes:2612713 (2.4 MiB) TX bytes:2612713 (2.4 MiB) >> > >> > pimreg Link encap:UNSPEC HWaddr >> > 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 >> > UP RUNNING NOARP MTU:1472 Metric:1 >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:0 >> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) >> > >> > se00 Link encap:Ethernet HWaddr 2E:B0:5D:A0:C5:B0 >> > inet addr:172.30.42.1 Bcast:172.30.42.31 >> Mask:255.255.255.224 >> > inet6 addr: 2602:30a:2cdb:330:2eb0:5dff:fea0:c5b1/64 >> Scope:Global >> >> How are you assigning your ipv6 addresses? >> > > It's been a while since I messed with this but I think IPv6 is assigned > thanks to 6relayd? My modem has IPv6 enabled but no DHCPv6 options that I > can find. Here's how cerowrt is configured. > > root@buffy2-1:/overlay/etc/config# cat 6relayd > config server 'default' > option fallback_relay 'rd dhcpv6 ndp' > list network 'ge00' > list network 'ge01' > list network 'gw00' > list network 'gw01' > list network 'gw10' > list network 'gw11' > list network 'se00' > list network 'sw00' > list network 'sw10' > option rd 'relay' > option dhcpv6 'relay' > option ndp 'relay' > option master 'ge00' > > >> > inet6 addr: fe80::2cb0:5dff:fea0:c5b0/64 Scope:Link >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:191740 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:1000 >> > RX bytes:0 (0.0 B) TX bytes:42184988 (40.2 MiB) >> > Interrupt:4 >> > >> > sw00 Link encap:Ethernet HWaddr 2C:B0:5D:A0:C5:B0 >> > inet addr:172.30.42.65 Bcast:172.30.42.95 >> Mask:255.255.255.224 >> > inet6 addr: 2602:30a:2cdb:330:2eb0:5dff:fea0:c5b1/64 >> Scope:Global >> > inet6 addr: fe80::2eb0:5dff:fea0:c5b0/64 Scope:Link >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> > RX packets:70239 errors:0 dropped:0 overruns:0 frame:0 >> > TX packets:286967 errors:0 dropped:0 overruns:0 carrier:0 >> > collisions:0 txqueuelen:1000 >> > RX bytes:15590189 (14.8 MiB) TX bytes:127357293 (121.4 MiB) >> > >> > root@buffy2-1:~# less /var/log/babeld.log >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > send: Cannot assign requested address >> > send: Cannot assign requested address >> > send: Cannot assign requested address >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > netlink_read: recvmsg(): No buffer space available >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> > Couldn't determine channel of interface sw00: Invalid argument. >> >> This is a problem in babel detecting the channel on a "normal" >> rather than a mesh interface. It's bugged me a long while, but >> haven't got around to finding what triggers it. Might "fix" it by >> acquiring the channel at babel start time from /etc/config/wireless. >> >> It messes up the diversity routing calculation, grump. >> >> There is a possibility a logfile got really big, but this one >> generally doesn't, but I should turn off logging in some >> future release... > > > I believe I've tracked down part of what's going on. It looks like my > tmpfs is filling up 100% and then the device enters a bad state: > > After 24 hours, with tmpfs at 50%, babeld.log is the largest file by far > in tmpfs and the only file that appears to be growing (based on `du`). It > takes about 48 hours from reboot to fill up tmpfs on my device. > > # sort babeld.log | uniq -c |sort -rn |head > > 503236 Couldn't determine channel of interface sw00: Invalid argument. > > 1376 netlink_read: recvmsg(): No buffer space available > > 3 send: Cannot assign requested address > > # wc -l babeld.log > > 504617 babeld.log > I sped up system failure by using `dd` to fill up tmpfs and the system > became immediately unusable. > > This also explains the luci session store errors as sessions are stored in > tmpfs. > > The other buffer issues may or may not be related to this. > > Best, > Steve > --001a1134b50ef2b6ad04f0fa24e8 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Looking more, the buffer errors are showing up in syslog w= ell before tmpfs fills up. Is the memtester openwrt package available for c= erowrt? I don't see it under `Available packages`.

Thanks,
Steve


On Mon, Jan 27, 2014 at 1:06 PM, Steve Jenson <st= evej@fruitless.org> wrote:
=
On Fri, Jan 24, 2014 at 3:23 P= M, Dave Taht <dave.taht@gmail.com> wrote:
On Fri, Jan 24, 2014 at 6:08 PM, Steve Jenson <stevej@fruitless.org= > wrote:
> Hi everybody,
>
> I've been using cerowrt as a secondary wifi network (just a single= AP for
> now) for a few weeks now. Recently, my wndr3800 got stuck in a bad sta= te and
> eventually rebooted. I've had this happen a few times now and am l= ooking for
> ways to debug the issue. I'm new to cerowrt and openwrt so any adv= ice is
> appreciated.
>
> Since I use it as a secondary network, this is no way critical.

Yea! I appreciate caution before putting alpha software on your gw.

> I'm not
> looking for free tech support but I couldn't find anything on the = wiki about
> troubleshooting. I'd love to start a page and write some shell scr= ipts to
> diagnose and report issues. I know that a cerowrt router is meant to b= e a
> research project rather a consumer device but these things seem helpfu= l
> regardless.

Sure, let me know your wiki account. I have been lax about granting access of late as the signup process is overrun by spammers.

My username is stevej on the wiki. Thanks!=A0
=

=A0
> Please let me know if you'd prefer I not email the list with these= issues or
> if you'd rather I used trac or a different forum.
>

The list is where most stuff happens. Also in the irc channel.

If it gets to where it needs to be tracked we have a bugtracker at

http://www.bufferbloat.net/projects/cerowrt/issues

The first question I have is: Are you on comcast? Cerowrt
had a dhcpv6-pd implementation that "just worked" from feburary t= hrough
december. Regrettably they changed the RA announcement interval
to a really low number around then... and this triggers a firewall reload every minute on everything prior to the release I point to below.

If there is a memory leak somewhere that would have triggered it.

I am on AT&T ADSL2+ with a Motorola NVG510= modem.=A0

=A0
> In this state, I can connect to the cerowrt base station via= wifi but am
> unable to route packets to the internet. I can connect to :81 and see = the
> login page but logging in results in a lua error at `/cgi-bin/luci` >
>
> =A0 /usr/lib/lua/luci/dispatcher.lua:448: Failed to execute function > dispatcher target for entry '/'.
> =A0 The called action terminated with an exception:
> =A0 /usr/lib/lua/luci/sauth.lua:87: Session data invalid!
> =A0 stack traceback:
> =A0 [C]: in function 'assert'
> =A0 /usr/lib/lua/luci/dispatcher.lua:448: in function 'dispatch= 9;
> =A0 /usr/lib/lua/luci/dispatcher.lua:195: in function
> </usr/lib/lua/luci/dispatcher.lua:194>
>
> I can ssh into the device and cat various log files until the router h= angs
> and reboots. here's a few relevant lines from my terminal history = before the
> device rebooted (I'm assuming a watchdog kicked in and rebooted it= ).
>
> root@buffy2-1:~# ping = google.com
> ping: bad address 'google.com'
> root@buffy2-1:~# free
> =A0 =A0 =A0 =A0 =A0 =A0 =A0total =A0 =A0 =A0 =A0 used =A0 =A0 =A0 =A0 = free =A0 =A0 =A0 shared =A0 =A0 =A0buffers
> Mem: =A0 =A0 =A0 =A0126336 =A0 =A0 =A0 110332 =A0 =A0 =A0 =A016004 =A0= =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 5616
> -/+ buffers: =A0 =A0 =A0 =A0 =A0 =A0 104716 =A0 =A0 =A0 =A021620
> Swap: =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 = =A0 =A0 =A00
> root@buffy2-1:~# uptime
> =A002:08:54 up 2 days, =A01:26, =A0load average: 0.10, 0.21, 0.17
> root@buffy2-1:~# dmesg
> [ =A0 =A00.000000] Linux version 3.10.24 (cero2@snapon) (gcc version 4= .6.4
> (OpenWrt/Linaro GCC 4.6-2013.05 r38226) ) #1 Tue Dec 24
> 10:50:15 PST 2013
> [skipping some lines]
>
> [ =A0 13.156250] Error: Driver 'gpio-keys-polled' is already r= egistered,
> aborting...
> [ =A0 19.414062] IPv6: ADDRCONF(NETDEV_UP): ge00: link is not ready > [ =A0 19.421875] ar71xx: pll_reg 0xb8050010: 0x11110000
> [ =A0 19.429687] se00: link up (1000Mbps/Full duplex)
> [ =A0 22.140625] IPv6: ADDRCONF(NETDEV_UP): sw00: link is not ready > [ =A0 23.351562] IPv6: ADDRCONF(NETDEV_CHANGE): sw00: link becomes rea= dy
> [ =A0 23.757812] ar71xx: pll_reg 0xb8050014: 0x11110000
> [ =A0 23.757812] ge00: link up (1000Mbps/Full duplex)
> [ =A0 23.773437] IPv6: ADDRCONF(NETDEV_CHANGE): ge00: link becomes rea= dy
>
> root@buffy2-1:~# ifconfig
> ge00 =A0 =A0 =A0Link encap:Ethernet =A0HWaddr 2C:B0:5D:A0:C5:B1
> =A0 =A0 =A0 =A0 =A0 inet addr:192.168.1.138 =A0Bcast:192.168.1.255 =A0= Mask:255.255.255.0
> =A0 =A0 =A0 =A0 =A0 inet6 addr: fe80::2eb0:5dff:fea0:c5b1/64 Scope:Lin= k
> =A0 =A0 =A0 =A0 =A0 inet6 addr: 2602:30a:2cdb:330:2eb0:5dff:fea0:c5b1/= 64 Scope:Global
> =A0 =A0 =A0 =A0 =A0 UP BROADCAST RUNNING MULTICAST =A0MTU:1500 =A0Metr= ic:1
> =A0 =A0 =A0 =A0 =A0 RX packets:1469670 errors:0 dropped:8 overruns:0 f= rame:0
> =A0 =A0 =A0 =A0 =A0 TX packets:547733 errors:0 dropped:0 overruns:0 ca= rrier:0
> =A0 =A0 =A0 =A0 =A0 collisions:0 txqueuelen:1000
> =A0 =A0 =A0 =A0 =A0 RX bytes:229243410 (218.6 MiB) =A0TX bytes:5730480= 8 (54.6 MiB)
> =A0 =A0 =A0 =A0 =A0 Interrupt:5
>
> lo =A0 =A0 =A0 =A0Link encap:Local Loopback
> =A0 =A0 =A0 =A0 =A0 inet addr:127.0.0.1 =A0Mask:255.0.0.0
> =A0 =A0 =A0 =A0 =A0 inet6 addr: ::1/128 Scope:Host
> =A0 =A0 =A0 =A0 =A0 UP LOOPBACK RUNNING =A0MTU:65536 =A0Metric:1
> =A0 =A0 =A0 =A0 =A0 RX packets:23689 errors:0 dropped:0 overruns:0 fra= me:0
> =A0 =A0 =A0 =A0 =A0 TX packets:23689 errors:0 dropped:0 overruns:0 car= rier:0
> =A0 =A0 =A0 =A0 =A0 collisions:0 txqueuelen:0
> =A0 =A0 =A0 =A0 =A0 RX bytes:2612713 (2.4 MiB) =A0TX bytes:2612713 (2.= 4 MiB)
>
> pimreg =A0 =A0Link encap:UNSPEC =A0HWaddr
> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
> =A0 =A0 =A0 =A0 =A0 UP RUNNING NOARP =A0MTU:1472 =A0Metric:1
> =A0 =A0 =A0 =A0 =A0 RX packets:0 errors:0 dropped:0 overruns:0 frame:0=
> =A0 =A0 =A0 =A0 =A0 TX packets:0 errors:0 dropped:0 overruns:0 carrier= :0
> =A0 =A0 =A0 =A0 =A0 collisions:0 txqueuelen:0
> =A0 =A0 =A0 =A0 =A0 RX bytes:0 (0.0 B) =A0TX bytes:0 (0.0 B)
>
> se00 =A0 =A0 =A0Link encap:Ethernet =A0HWaddr 2E:B0:5D:A0:C5:B0
> =A0 =A0 =A0 =A0 =A0 inet addr:172.30.42.1 =A0Bcast:172.30.42.31 =A0Mas= k:255.255.255.224
> =A0 =A0 =A0 =A0 =A0 inet6 addr: 2602:30a:2cdb:330:2eb0:5dff:fea0:c5b1/= 64 Scope:Global

How are you assigning your ipv6 addresses?

It's been a while since I messed with this = but I think IPv6 is assigned thanks to 6relayd? My modem has IPv6 enabled b= ut no DHCPv6 options that I can find. Here's how cerowrt is configured.=

root@buffy2-1:/overlay/etc/config# cat 6relayd
conf= ig server 'default'
=A0 option fallback_relay 'rd dhcpv6 ndp= '
=A0 list network 'ge00'
=A0 list network 'ge01'=
=A0 list network 'gw00'
=A0 list network 'gw01'
=A0 l= ist network 'gw10'
=A0 list network 'gw11'
=A0 list n= etwork 'se00'
=A0 list network 'sw00'
=A0 list networ= k 'sw10'
=A0 option rd 'relay'
=A0 option dhcpv6 'relay'
=A0 o= ption ndp 'relay'
=A0 option master 'ge00'


> =A0 =A0 =A0 =A0 =A0 inet6 addr: fe80::2cb0:5dff:fea0:c5b0/64 Scope:Lin= k
> =A0 =A0 =A0 =A0 =A0 UP BROADCAST RUNNING MULTICAST =A0MTU:1500 =A0Metr= ic:1
> =A0 =A0 =A0 =A0 =A0 RX packets:0 errors:0 dropped:0 overruns:0 frame:0=
> =A0 =A0 =A0 =A0 =A0 TX packets:191740 errors:0 dropped:0 overruns:0 ca= rrier:0
> =A0 =A0 =A0 =A0 =A0 collisions:0 txqueuelen:1000
> =A0 =A0 =A0 =A0 =A0 RX bytes:0 (0.0 B) =A0TX bytes:42184988 (40.2 MiB)=
> =A0 =A0 =A0 =A0 =A0 Interrupt:4
>
> sw00 =A0 =A0 =A0Link encap:Ethernet =A0HWaddr 2C:B0:5D:A0:C5:B0
> =A0 =A0 =A0 =A0 =A0 inet addr:172.30.42.65 =A0Bcast:172.30.42.95 =A0Ma= sk:255.255.255.224
> =A0 =A0 =A0 =A0 =A0 inet6 addr: 2602:30a:2cdb:330:2eb0:5dff:fea0:c5b1/= 64 Scope:Global
> =A0 =A0 =A0 =A0 =A0 inet6 addr: fe80::2eb0:5dff:fea0:c5b0/64 Scope:Lin= k
> =A0 =A0 =A0 =A0 =A0 UP BROADCAST RUNNING MULTICAST =A0MTU:1500 =A0Metr= ic:1
> =A0 =A0 =A0 =A0 =A0 RX packets:70239 errors:0 dropped:0 overruns:0 fra= me:0
> =A0 =A0 =A0 =A0 =A0 TX packets:286967 errors:0 dropped:0 overruns:0 ca= rrier:0
> =A0 =A0 =A0 =A0 =A0 collisions:0 txqueuelen:1000
> =A0 =A0 =A0 =A0 =A0 RX bytes:15590189 (14.8 MiB) =A0TX bytes:127357293= (121.4 MiB)
>
> root@buffy2-1:~# less /var/log/babeld.log
> Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > send: Cannot assign requested address
> send: Cannot assign requested address
> send: Cannot assign requested address
> Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > netlink_read: recvmsg(): No buffer space available
> Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument. > Couldn't determine channel of interface sw00: Invalid argument.
This is a problem in babel detecting the channel on a "nor= mal"
rather than a mesh interface. It's bugged me a long while, but
haven't got around to finding what triggers it. Might "fix" i= t by
acquiring the channel at babel start time from /etc/config/wireless.

It messes up the diversity routing calculation, grump.

There is a possibility a logfile got really big, but this one
generally doesn't, but I should turn off logging in some
future release...

I believe I&#= 39;ve tracked down part of what's going on. It looks like my tmpfs is f= illing up 100% and then the device enters a bad state:

After 24 hours, with tmpfs at 50%, babeld.log is the largest file by f= ar in tmpfs and the only file that appears to be growing (based on `du`). I= t takes about 48 hours from reboot to fill up tmpfs on my device.

#=A0 sort babeld.log | uniq -c |sort -rn |head

=A0503236 Couldn't determine channel of interface sw00: Invalid argu= ment.

=A0=A0 1376 netlink_read: recvmsg(): No buffer space available

=A0 =A0 =A0 3 send: Cannot assign requested address

<= p># wc -l babeld.log

504617 babeld.log

I sped up = system failure by using `dd` to fill up tmpfs and the system became immedia= tely unusable.

This also explains the luci session store errors as ses= sions are stored in tmpfs.

The other buffer issues= may or may not be related to this.

Best,
Steve

--001a1134b50ef2b6ad04f0fa24e8--