[Cerowrt-users] Setting up bridging and debugging problems with LAN ports with WNDR3800

[Cerowrt-users] Setting up bridging and debugging problems with LAN ports with WNDR3800

[Marc MERLIN]

Ok, it's a bit long, sorry, I spent too many hours today trying to fix some
issues in cerowrt and get bridging working.

This is cerowrt 3.3.8-26.

Before I get to bridging, openwrt could get my ethernet LAN ports working if I
recall correcly, but it seems that cerowrt can't (the WAN port is ok, and so
is wireless, but none of my LAN ports seem to be able to send IP traffic
even though I see STP and other traffic from them).

The first issue is while I had wireless working, wired just wasn't.
I never got an IP on wired ports, and for that matter when I forced the IP
on my laptop, I couldn't ping the interface

Since I can't get a DHCP answer on LAN, I forced an IP:
gandalfthegreat:~# ifconfig eth0 172.30.42.4 netmask 255.255.255.224
gandalfthegreat:~# ping 172.30.42.1
PING 172.30.42.1 (172.30.42.1) 56(84) bytes of data.
From 172.30.42.4 icmp_seq=1 Destination Host Unreachable
11:21:05.175560 ARP, Request who-has 172.30.42.1 tell 172.30.42.4, length 28
11:21:05.763975 IP 172.30.42.1 > 224.0.0.1: igmp query v2
11:21:06.171874 ARP, Request who-has 172.30.42.1 tell 172.30.42.4, length 28
11:21:06.365442 IP 172.30.42.1 > 224.0.0.251: igmp v2 report 224.0.0.251
11:21:07.171869 ARP, Request who-has 172.30.42.1 tell 172.30.42.4, length 28

This shows that my laptop sees traffic from 172.30.42.1 but won't get an ARP
reply.

Then from my router, I tried to ping my laptop and got this:
12:51:29.954271 ARP, Request who-has 172.30.42.4 tell 172.30.42.1, length 46
12:51:29.954294 ARP, Reply 172.30.42.4 is-at f0:de:f1:f6:03:80, length 28
12:51:30.130065 IP 172.30.42.1 > 224.0.0.13: PIMv2, Bootstrap, length 36

This shows my router is sending an arp, my laptop gets it, replies, but
the reply isn't making it to the router.

So it seems that inbound traffic on the router's LAN ports isn't working.
Is it a bug/misconfiguration with the switch chip (rtl8366s) that breaks only IP
and somehow seems to ignore/discard arp queries too?
I looked in firewalling and didn't see anything obviously wrong.

> Question #1:
What am I doing wrong or how do I debug further?


Then, ignoring the LAN ports for now, I got bridging to work like this:
brctl addbr br-lan
brctl addif br-lan  se00
brctl addif br-lan  sw00
brctl addif br-lan  sw10
brctl setpathcost br-lan se00 5
brctl setpathcost br-lan sw00 15
brctl setpathcost br-lan sw10 10
ifconfig se00 0.0.0.0
ifconfig sw00 0.0.0.0
ifconfig sw10 0.0.0.0
ifconfig br-lan 192.168.206.249
brctl stp br-lan on
brctl show
bridge name	bridge id		STP enabled	interfaces
br-lan		8000.2cb05da0c8a4	yes		se00
							sw00
							sw10

This is working and I can switch between the 2 wireless interfaces without
dropping a connection, but
> Question #2:
How do I get this saved properly in the cerowrt config so that it's not overwritten
by the UI when I touch anything at all, and of course lost at boot.
(if I had a 2nd router with openwrt it'd be easier since I could see how
they save their bridging config)


> Question #3:
I had to hand edit /var/etc/dnsmasq.conf to get DHCP working with the new
data after boot (and restart the daemon). 
Should I save stuff in /etc/config/wireless somehow? but if so, I'm not sure
what syntax to use for the bridge config.

> Question #4:
how do I get debugging/logs from dnsmasq? Is it done through syslog?

> Question #5:
Why can't I get the :81 web interface to respond on its outside IP (kind of
useful when I'm mucking on the internal one).
/etc/lighttpd/lighttpd.conf says:
## bind to port (default: 80)
server.port = 81

## bind to localhost (default: all interfaces)
#server.bind = "localhost"

> Question #6:
Why is the admin interface on :81 not using https?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/

Re: [Cerowrt-users] Setting up bridging and debugging problems with LAN ports with WNDR3800

[Marc MERLIN]

On Sat, Nov 17, 2012 at 03:44:37PM -0800, Marc MERLIN wrote:
> Ok, it's a bit long, sorry, I spent too many hours today trying to fix some
> issues in cerowrt and get bridging working.
> 
> This is cerowrt 3.3.8-26.
> 
> Before I get to bridging, openwrt could get my ethernet LAN ports working if I
> recall correcly, but it seems that cerowrt can't (the WAN port is ok, and so
> is wireless, but none of my LAN ports seem to be able to send IP traffic
> even though I see STP and other traffic from them).
> 
> The first issue is while I had wireless working, wired just wasn't.
> I never got an IP on wired ports, and for that matter when I forced the IP
> on my laptop, I couldn't ping the interface
 
Ok, after a full reset it worked again and I figured out what the problem
is.
If you enable VLAN functionality in network/switch, the LAN ports stop
working even if they are marked as 'untagged' which is the default.
When I used tcpdump, I did look for whether there was a tagging problem,
but the packets didn't seem tagged.
But it gets worse, even after turning vlan off in the GUI
config switch          
        option enable_vlan4k '1'
stays and prevents the LAN ports from working.
Actually, also
        option enable_vlan '0'
needs to be restored and without that line, your LAN ports just will not
work.

That's a pretty bad GUI trap, 4H of my time down the drain :(

> > Question #1:
> What am I doing wrong or how do I debug further?
 
There wasn't much to find, this required wiping everything, starting over, 
taking config diffs, finding the option that broke everything, and further
finding that unchecking the GUI option didn't clean the config file enough
to recover.
 
> > Question #4:
> how do I get debugging/logs from dnsmasq? Is it done through syslog?
 
logread or logread -f

> > Question #5:
> Why can't I get the :81 web interface to respond on its outside IP (kind of
> useful when I'm mucking on the internal one).
> /etc/lighttpd/lighttpd.conf says:
> ## bind to port (default: 80)
> server.port = 81

81 is firewalled off on the ge00 interface.

> > Question #6:
> Why is the admin interface on :81 not using https?

Seems that openwrt didn't seem to hink it was a good idea. 

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: [Cerowrt-users] Setting up bridging and debugging problems with LAN ports with WNDR3800

[Dave Taht]

On Mon, Nov 19, 2012 at 6:55 PM, Marc MERLIN <marc@merlins.org> wrote:
> On Sat, Nov 17, 2012 at 03:44:37PM -0800, Marc MERLIN wrote:
>> Ok, it's a bit long, sorry, I spent too many hours today trying to fix some
>> issues in cerowrt and get bridging working.
>>
>> This is cerowrt 3.3.8-26.
>>
>> Before I get to bridging, openwrt could get my ethernet LAN ports working if I
>> recall correcly, but it seems that cerowrt can't (the WAN port is ok, and so
>> is wireless, but none of my LAN ports seem to be able to send IP traffic
>> even though I see STP and other traffic from them).
>>
>> The first issue is while I had wireless working, wired just wasn't.
>> I never got an IP on wired ports, and for that matter when I forced the IP
>> on my laptop, I couldn't ping the interface
>
> Ok, after a full reset it worked again and I figured out what the problem
> is.
> If you enable VLAN functionality in network/switch, the LAN ports stop
> working even if they are marked as 'untagged' which is the default.
> When I used tcpdump, I did look for whether there was a tagging problem,
> but the packets didn't seem tagged.
> But it gets worse, even after turning vlan off in the GUI
> config switch
>         option enable_vlan4k '1'
> stays and prevents the LAN ports from working.
> Actually, also
>         option enable_vlan '0'
> needs to be restored and without that line, your LAN ports just will not
> work.

I can't remember what the option is, but it helps to turn on jumbo
frames. This cuts the buffering in the switch down to something less
unreasonable.

> That's a pretty bad GUI trap, 4H of my time down the drain :(

I'm glad that you got somewhere and documented it publicly.

>> > Question #1:
>> What am I doing wrong or how do I debug further?
>
> There wasn't much to find, this required wiping everything, starting over,
> taking config diffs, finding the option that broke everything, and further
> finding that unchecking the GUI option didn't clean the config file enough
> to recover.

that was a deep hole dug!

>
>> > Question #4:
>> how do I get debugging/logs from dnsmasq? Is it done through syslog?
>
> logread or logread -f
>
>> > Question #5:
>> Why can't I get the :81 web interface to respond on its outside IP (kind of
>> useful when I'm mucking on the internal one).
>> /etc/lighttpd/lighttpd.conf says:
>> ## bind to port (default: 80)
>> server.port = 81
>
> 81 is firewalled off on the ge00 interface.

I might argue it be firewalled off from wifi entirely, too. The reason
why it isn't is that I fairly often have to reflash a large number of
routers, remotely, which tends to blow up something or another, and
it's helpful to be able to get back into the wifi side.

(this is horribly understaffed research project. Security is a concern
(see the xinet.d stuff) but first up is making the network behave
better.)

>> > Question #6:
>> Why is the admin interface on :81 not using https?
>
> Seems that openwrt didn't seem to hink it was a good idea.

The problem here is threefold:

1) The amount of available randomness (entropy) in the linux kernel,
on an embedded device, is nearly nil. This has all sorts of bad
side-effects.

We've had multiple cases where routers generated the same ssh key, for example.

This is quite scary to me, as 10s of millions of commercial devices
with lousy entropy and yet supposedly secure "wpa2" are shipping in
the field.

While cerowrt contains a few patches intended to increase entropy,
they were far from satisfactory. Even with the merge of the new random
number code from 3.6, multiple device drivers need to be enhanced
before it's truly useful, and I'll remain unhappy until I find a
device with hardware RNG, or am convinced enough entropy is wedged
into the pool on boot to be random enough.

2) For doing https, there are two solutions. A) create a self signed
SSL key on first boot. This has the problems of 1, above, and
additionally causes your browser to throw an error saying it's a self
signed key, even though, normally, self signed keys are more secure
than anything else, as the various breaks in the chain of trust via
conventional authorities shows.

B) ship with a per router key that is the same on all firmware. This
is not horrible, in conjunction with C). Pretty horrible without.

C) somehow acquire a ssl key on first boot. Unfortunately you are not
connected to anything on first boot, you need a key. And automating
key creation and signing is something of a problem, when you don't
control a certificate authority and have secure paths in the first
place.

Sigh.

https IS supported in the web server and the only problem with
enabling it are the problems noted above.

I felt that it was best to not "lie" about the level of security by
using https, and try to firewall off the core interfaces, more than
pursue either of these alternatives. That said, I'd like to make https
available by default, and if I just had that hardware rng or trust in
the new 3.6 based randomness code, I'd do so in a heartbeat with a
self signed key.


>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/
> _______________________________________________
> Cerowrt-users mailing list
> Cerowrt-users@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-users



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-users] Setting up bridging and debugging problems with LAN ports with WNDR3800

[Marc MERLIN]

First, I got bridging to work this morning over wired and wireless. I'll post
my diffs when I have verified everything is still ok.
Should I post on the list, or some web page of yours or mine?

Second, I think I only had to change network, wireless, dhcp, and firewall.
Are there other magic files where you have your congestion code applying
settings to the interfaces that are now wrong after I renamed them?

I want to make sure I don't drop some of your work by posting an incomplete
patch.

On Mon, Nov 19, 2012 at 07:15:36PM +0100, Dave Taht wrote:
> > That's a pretty bad GUI trap, 4H of my time down the drain :(
> 
> I'm glad that you got somewhere and documented it publicly.

I'm not sure how well this list is indexed. Is your wiki something others
like me can contribute to?
(and/or can the dangerous option be removed from the GUI for your next
version?)

> > 81 is firewalled off on the ge00 interface.
> 
> I might argue it be firewalled off from wifi entirely, too. The reason

Yes, although if it had been, I'd have rendered my router unusable after the
bug above since lan wifi was my only way in.

> why it isn't is that I fairly often have to reflash a large number of
> routers, remotely, which tends to blow up something or another, and
> it's helpful to be able to get back into the wifi side.

Yes :)

> > Seems that openwrt didn't seem to hink it was a good idea.
> 
> The problem here is threefold:
> 
> 1) The amount of available randomness (entropy) in the linux kernel,
> on an embedded device, is nearly nil. This has all sorts of bad
> side-effects.
 
I didn't realize there was no good entropy on those devices. I was hoping
there were floating IO pins that you could do analogread on, or other such
stuff. Shows how much I know.

> We've had multiple cases where routers generated the same ssh key, for example.

Doh, that's obviously very bad.

> This is quite scary to me, as 10s of millions of commercial devices
> with lousy entropy and yet supposedly secure "wpa2" are shipping in
> the field.

Yes :(
That said, sounds like the kernel needs to fix that, not openwrt by
disabling https, arguably making security even worse.
A good compromize would be for https to work by default and have a message
at the top that says "you likely have a bad http cert, please make a good
one linux with 'type foo bar', paste the output in "/path/to/filebaz",
and reboot.

> While cerowrt contains a few patches intended to increase entropy,
> they were far from satisfactory. Even with the merge of the new random
> number code from 3.6, multiple device drivers need to be enhanced
> before it's truly useful, and I'll remain unhappy until I find a
> device with hardware RNG, or am convinced enough entropy is wedged
> into the pool on boot to be random enough.

Fair enough.

> 2) For doing https, there are two solutions. A) create a self signed
> SSL key on first boot. This has the problems of 1, above, and
> additionally causes your browser to throw an error saying it's a self
> signed key, even though, normally, self signed keys are more secure
> than anything else, as the various breaks in the chain of trust via
> conventional authorities shows.

Yep.

> C) somehow acquire a ssl key on first boot. Unfortunately you are not
> connected to anything on first boot, you need a key. And automating
> key creation and signing is something of a problem, when you don't
> control a certificate authority and have secure paths in the first
> place.
> 
> Sigh.

Indeed.
 
> https IS supported in the web server and the only problem with
> enabling it are the problems noted above.

Understood.

> I felt that it was best to not "lie" about the level of security by
> using https, and try to firewall off the core interfaces, more than

If I ssh in though, it's no better, so I'm not sure it's a win.
Probably putting an http banner on the https site with the problem and how
to fix it would be a good compromize.

Thanks for the answers,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: [Cerowrt-users] Setting up bridging and debugging problems with LAN ports with WNDR3800

[Marc MERLIN]

On Mon, Nov 19, 2012 at 01:24:36PM -0800, Marc MERLIN wrote:
> First, I got bridging to work this morning over wired and wireless. I'll post
> my diffs when I have verified everything is still ok.
> Should I post on the list, or some web page of yours or mine?
> 
> Second, I think I only had to change network, wireless, dhcp, and firewall.
> Are there other magic files where you have your congestion code applying
> settings to the interfaces that are now wrong after I renamed them?
 
I'm still happy to post something. I'll do it on my blog if it's the best
place.
Before posting info though, I want to make sure I'm not giving people bad
advise. Is what I wrote above enough not to disable to latency tweaks, or is
there more I should do?

For the stuff below, if it makes sense to you, I can try to appeal to the
openwrt folks directly since they are your upstream.
In the end, providing ssh but not https just doesn't make sense, and the
http interface is just as bad as asking me to telnet into the router to
configure it :)

Happy black friday :)

Marc

> > This is quite scary to me, as 10s of millions of commercial devices
> > with lousy entropy and yet supposedly secure "wpa2" are shipping in
> > the field.
> 
> Yes :(
> That said, sounds like the kernel needs to fix that, not openwrt by
> disabling https, arguably making security even worse.
> A good compromize would be for https to work by default and have a message
> at the top that says "you likely have a bad http cert, please make a good
> one linux with 'type foo bar', paste the output in "/path/to/filebaz",
> and reboot.
>
> > While cerowrt contains a few patches intended to increase entropy,
> > they were far from satisfactory. Even with the merge of the new random
> > number code from 3.6, multiple device drivers need to be enhanced
> > before it's truly useful, and I'll remain unhappy until I find a
> > device with hardware RNG, or am convinced enough entropy is wedged
> > into the pool on boot to be random enough.
> 
> Fair enough.
> 
> > 2) For doing https, there are two solutions. A) create a self signed
> > SSL key on first boot. This has the problems of 1, above, and
> > additionally causes your browser to throw an error saying it's a self
> > signed key, even though, normally, self signed keys are more secure
> > than anything else, as the various breaks in the chain of trust via
> > conventional authorities shows.
> 
> Yep.
> 
> > C) somehow acquire a ssl key on first boot. Unfortunately you are not
> > connected to anything on first boot, you need a key. And automating
> > key creation and signing is something of a problem, when you don't
> > control a certificate authority and have secure paths in the first
> > place.
> > 
> > Sigh.
> 
> Indeed.
>  
> > https IS supported in the web server and the only problem with
> > enabling it are the problems noted above.
> 
> Understood.
> 
> > I felt that it was best to not "lie" about the level of security by
> > using https, and try to firewall off the core interfaces, more than
> 
> If I ssh in though, it's no better, so I'm not sure it's a win.
> Probably putting an http banner on the https site with the problem and how
> to fix it would be a good compromize.
> 
> Thanks for the answers,
> Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/