Re: [Cerowrt-devel] 3.3.6-2

[Sebastian Moeller <moeller0@gmx.de>]

Hi Robert,


On May 25, 2012, at 4:11 AM, Robert Bradley wrote:

> On 25 May 2012 07:41, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi Robert,
>> 
>> since I see the same log file on my router as Jim, I just want to report
>> my observations below.
>> 
>> On May 24, 2012, at 11:58 AM, Robert Bradley wrote:
> <snip>
> (re. guest interfaces on wireless)
>>> Are these disabled on your routers at the moment?  I suppose in the
>>> worst case you could try setting an explicit channel for both of the
>>> non-mesh guest interfaces and see if the logs clear up (or somehow pass "-L
>>> /dev/null" to babeld).
>> 
>>        After setting the 2.4GHz channel to 1 instead of auto
>> /tmp/babeld.log still grows with the same entries. And on a WNDR3700v2 there
>> are 30840 KB of tmpfs on /tmp so the babeld.log size of 256KB should not by
>> itself cause the router to crash. That said, while testing this hypothesis
>> by filling most of /tmp (dd if=/dev/zero of=/tmp/delete_me bs=1024
>> count=30000, so that around 340KB stayed free) the router reliably went
>> first into OOM and the rebooted itself. Might it be that the size of the
>> /tmp filesystem is too large if actually used? If I naively add the VSZs of
>> most processes I end up at around 90% of available memory, so worst case
>> there actually only seems to be room for a much smaller /tmp than 30MB. .
>> Maybe restricting /tmp to 6000 KB might make this problem go away (or
>> hooking up a swap device). Does this reasoning sound sane? Once I figure out
>> how to reduce the size of /tmp I will test this.
>> 
> 
> Using "mount -o remount -o size=6000k /tmp" should apparently work for
> that.  The reasoning sounds good to me, too.  

	I will go and test that.

> That said, unless we can
> find an obvious reason for /tmp overfilling, I'm not sure we should do
> that, since it will cause problems upgrading.  

	But if I create a file of 30000 1KB blocks in /tmp (so that around 400 KB stay available), the router goes into OOM, so I do not think that upgrading would work well if it really needs so much memory? I have a hunch that the openwork base under cerowrt does not assume something as big and demanding as the 11MB bind9 named process running :)

> There's also the issue
> that in bug #379, only wireless traffic caused problems.  I think that
> even if excessive logs are the problem, the real issue must be
> somewhere within the wireless driver, but I could well be wrong…

	Oh I agree the /tmp issue is a tangent, but it does not seem healthy that the router spirals into reboot once /tmp fills up (BTW if I remove my 30000KB file from /tmp while the first OOM is in process the router recovers) My hunch is that the falmost fully instantiated tmpfs takes to o much memory from the system for it to handle its usual business.
	On top of that are the wireless issues, say what about a kernel memory leak caused by ath wireless that grows and grows until the problematic /tmp size is in the single digit MBs that starts the spiral to reboot?

> 
> I'm thinking that maybe flooding wireless->wired with UDP traffic for
> 5-10 minutes is the right approach, and then vice-versa (restarting
> the router inbetween?).  If there are problems like infinite retries
> or packet memory leaks, that might show them up quickly.

	That sounds like the right way to process, except I am no expert at setting netsurf up so that might take a while until I get around to actually test that hypothesis. (Do you by any chance know a publicly available net server process running in the internets to which I could point a local netperf, and do you have any recommendations how to create the UDP flood with netperf ?)

Best
	Sebastian

> 
> --
> Robert Bradley