[Cerowrt-devel] shaggy dog story on 3.8.8-4 experiences

[Cerowrt-devel] shaggy dog story on 3.8.8-4 experiences

[Michael Richardson]

On my todo list was moving from 3.3 installed last summer to 3.8.8-4,
via the forklift method of configuring a new device and installing it.

Something was up on my network, and I discovered that I could not ssh
into my 3800, and so on Sunday afternoon when I figured that I wasn't
going to interrupt anything, I rebooted it from the web interface, which
was still working.
Sadly, it did come back, but it wasn't in the configuration I expected,
and I could not access it.  I'm way too used to having a serial console
on everything...
I was about to flash it, when I thought... good time for 3.8.8-4, so 
I put that one in place, and uploaded my config back from November,
and... that unit promptly went away.  I've since concluded after
examining that configuration, that this wasn't the last and final
configuration, and that likely the previous unit ("55" I call it) had
never properly saved it's config.

A few hours later (BBQ-Pizza and bed time stories were in the way), I
got things back to normal....  I've now extracted the config backup into
a git tree and actually read it to confirm that it's sane.

My observations:

1) I find it really confusing to have a default route on each interface
   details.  It's clear to me now that I want to only set the default
   route on the interface which is my uplink, but it seems like maybe
   I should set the same thing on the other interfaces, but that leads
   to bad things.

2) I don't think that I got RAs turned on properly on at least one of
   my interfaces.  I did do it today with unit 55 on the bench, so I'm
   unclear what I did wrong.

3) mDNS does not announce the actual hostname that I gave to the device
   only "cerowrt".

4) I guess the "prefix routed to this device for other interfaces" is
   the beginnings of 6204/homenet support.  I'm unclear if it makes
   sense for it to be settable on multiple interfaces, at least not 
   I think it belongs in the advanced settings pane.
   
5) I did get prefixes assigned to interfaces... alas, it duplicated
   prefixes which were already assigned to other interfaces!!!
   I did setup the hints, but perhaps not for all interfaces.
   At least one prefix is behind another router, so this router can not
   see that prefix is already in use.  Suggestion: start at highest
   number available and work downwards.
   These assignments should probably go somewhere near the DHCP device
   assignment list.

6) is there anything between complete flash and boot?
   If I hold down "factory reset" until it's yellow, what does that 
   mean?  I'd like a "reset to LAN-1 has no-VLANs, just DHCP+v6-ULA-RA"
   and configuration is ignored, but is still present.  Is that
   possible?

7) I tried the AQM page, and I think it worked.  

8) I can't make the firewall do what I want, given that I have routed
   public IPv4, so I wind up just writing iptables command in the
   firewall.user file... 
   That's also where I removed the /64 routes for the prefixes that were
   assigned to the wrong interfaces...

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [ 
	



	

Re: [Cerowrt-devel] shaggy dog story on 3.8.8-4 experiences

[Dave Taht]

On Tue, Apr 30, 2013 at 12:29 PM, Michael Richardson <mcr@sandelman.ca> wrote:
>
> On my todo list was moving from 3.3 installed last summer to 3.8.8-4,
> via the forklift method of configuring a new device and installing it.
>
> Something was up on my network, and I discovered that I could not ssh
> into my 3800, and so on Sunday afternoon when I figured that I wasn't
> going to interrupt anything, I rebooted it from the web interface, which
> was still working.
> Sadly, it did come back, but it wasn't in the configuration I expected,
> and I could not access it.  I'm way too used to having a serial console
> on everything...
> I was about to flash it, when I thought... good time for 3.8.8-4, so
> I put that one in place, and uploaded my config back from November,
> and...

yea, well, I make no warrantees about configs being backward
compatible. Sorry, tons of stuff has changed.

> that unit promptly went away.  I've since concluded after
> examining that configuration, that this wasn't the last and final
> configuration, and that likely the previous unit ("55" I call it) had
> never properly saved it's config.
>
> A few hours later (BBQ-Pizza and bed time stories were in the way), I
> got things back to normal....  I've now extracted the config backup into
> a git tree and actually read it to confirm that it's sane.
>
> My observations:
>
> 1) I find it really confusing to have a default route on each interface
>    details.  It's clear to me now that I want to only set the default
>    route on the interface which is my uplink, but it seems like maybe
>    I should set the same thing on the other interfaces, but that leads
>    to bad things.

The default route is a bad leftover from the days nobody could decide
on a routing protocol. Which is still the days.

In the case of the default gw on the router, it comes from the
dhcp/dhcp-pd client  getting the default gw from it's upstream.

In the case of the default gw on the clients, it gets it from it's
dhcp server, (dnsmasq) and is specific to the interface ip the
interface is on. Since this is a /27, you get things like .1 or .65 or
.97.

Me I stopped dealing with this sort of nonsense years ago and just run
a routing daemon on everything and ignore default routes except what I
get from that....

> 2) I don't think that I got RAs turned on properly on at least one of
>    my interfaces.  I did do it today with unit 55 on the bench, so I'm
>    unclear what I did wrong.

The new RA support is not fully integrated with the gui yet. dnsmasq
2.66 just landed in openwrt head a few days ago, so expect uci and gui
changes there soon.

So how cero does it is in /etc/dnsmasq.conf and there are a variety of
ways to do it.
>
> 3) mDNS does not announce the actual hostname that I gave to the device
>    only "cerowrt".

that is presently set in /etc/avahi/something.

The long term plan has become to follow the mdnsext work and fold it
into dnsmasq. Probably. configured with something via a ubus
interface, which certainly includes getting the hostname right. It
looks like simon finally got funded to do the dnssec work so I don't
expect mdsnext to even start thinking about happening before the next
ietf meeting unless someone else gets hot on it.

>
> 4) I guess the "prefix routed to this device for other interfaces" is
>    the beginnings of 6204/homenet support.  I'm unclear if it makes
>    sense for it to be settable on multiple interfaces, at least not
>    I think it belongs in the advanced settings pane.

Not sure what you mean. I don't really get the 6relayd stuff.

> 5) I did get prefixes assigned to interfaces... alas, it duplicated
>    prefixes which were already assigned to other interfaces!!!
>    I did setup the hints, but perhaps not for all interfaces.

I'm not sure what you are talking about here. ipassign 64 will take
the 64 hint and assign something to it.

>    At least one prefix is behind another router, so this router can not
>    see that prefix is already in use.  Suggestion: start at highest
>    number available and work downwards.

Two interior prefix allocation methods have been described by the
homenet and hipnet rfcs. openwrt follows neither at present.


>    These assignments should probably go somewhere near the DHCP device
>    assignment list.
>
> 6) is there anything between complete flash and boot?
>    If I hold down "factory reset" until it's yellow, what does that
>    mean?  I'd like a "reset to LAN-1 has no-VLANs, just DHCP+v6-ULA-RA"
>    and configuration is ignored, but is still present.  Is that
>    possible?

No.

> 7) I tried the AQM page, and I think it worked.

:)

tc -s qdisc show dev ge00

will show htb and fq_codel in that case.
>
> 8) I can't make the firewall do what I want, given that I have routed
>    public IPv4, so I wind up just writing iptables command in the
>    firewall.user file...

yea, that's become such an uncommon case....

>    That's also where I removed the /64 routes for the prefixes that were
>    assigned to the wrong interfaces...

I think this is a little overthought, but ok....
> --
> ]               Never tell me the odds!                 | ipv6 mesh networks [
> ]   Michael Richardson, Sandelman Software Works        | network architect  [
> ]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [
>
>
>
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] shaggy dog story on 3.8.8-4 experiences

[Robert Bradley]

[-- Attachment #1: Type: text/plain, Size: 925 bytes --]

On 30 April 2013 21:45, Dave Taht <dave.taht@gmail.com> wrote:

> On Tue, Apr 30, 2013 at 12:29 PM, Michael Richardson <mcr@sandelman.ca>
> wrote:
> > 5) I did get prefixes assigned to interfaces... alas, it duplicated
> >    prefixes which were already assigned to other interfaces!!!
> >    I did setup the hints, but perhaps not for all interfaces.
>
> I'm not sure what you are talking about here. ipassign 64 will take
> the 64 hint and assign something to it.
>

This is referring to something I have run into before, where the
auto-assignment code ignores any manually assigned /64 prefixes.  If you
hypothetically assign fd00:0:1::/64 to se00 and fd00:0:2::/64 to sw00, the
auto-assignment has a tendency to assign fd00:0:1::/64 to sw00 along with
your manually-assigned prefix.  I don't know how well auto-assignment plays
with AHCP, so I have tended to set things up manually to get around this.
-- 
Robert Bradley

[-- Attachment #2: Type: text/html, Size: 1375 bytes --]

Re: [Cerowrt-devel] shaggy dog story on 3.8.8-4 experiences

[Michael Richardson]

>>>>> "Dave" == Dave Taht <dave.taht@gmail.com> writes:
    >> Something was up on my network, and I discovered that I could not ssh
    >> into my 3800, and so on Sunday afternoon when I figured that I wasn't
    >> going to interrupt anything, I rebooted it from the web interface, which
    >> was still working.
    >> Sadly, it did come back, but it wasn't in the configuration I expected,
    >> and I could not access it.  I'm way too used to having a serial console
    >> on everything...
    >> I was about to flash it, when I thought... good time for 3.8.8-4, so
    >> I put that one in place, and uploaded my config back from November,
    >> and...

    Dave> yea, well, I make no warrantees about configs being backward
    Dave> compatible. Sorry, tons of stuff has changed.

no, I think that the my saved config was in fact borked.

    >> 1) I find it really confusing to have a default route on each interface
    >> details.  It's clear to me now that I want to only set the default
    >> route on the interface which is my uplink, but it seems like maybe
    >> I should set the same thing on the other interfaces, but that leads
    >> to bad things.

    Dave> The default route is a bad leftover from the days nobody could decide
    Dave> on a routing protocol. Which is still the days.

Yes, most people will config via DHCP.  I'm suggesting that it's
confusing to have a default route box in basic config on interfaces
which are not the uplink. ...  I'm just saying that removing it would
make sense.

    >> 3) mDNS does not announce the actual hostname that I gave to the device
    >> only "cerowrt".

    Dave> that is presently set in /etc/avahi/something.

    Dave> The long term plan has become to follow the mdnsext work and fold it
    Dave> into dnsmasq. Probably. configured with something via a ubus

okay, so dnsmasq is not doing mDNS announcements at this point, we have
an avahi running?   I just built openwrt, I need to pull your git tree
now that I'm sure that I have a working build environment.

    >> 4) I guess the "prefix routed to this device for other interfaces" is
    >> the beginnings of 6204/homenet support.  I'm unclear if it makes
    >> sense for it to be settable on multiple interfaces, at least not
    >> I think it belongs in the advanced settings pane.

    Dave> Not sure what you mean. I don't really get the 6relayd stuff.

There are three new boxes on *each* interface definition:
a) "subnet which is routed here for allocation"
b) prefix length to assign
c) hint as to prefix to assign.

Since my ISP doesn't run DHCPv6 (PD), I enter my prefix on ge00.

(I noticed that the IPv6 DHCP client is on a virtual interface "ge01",
which I think it really confusing... at least, it should be named
ge00_v6 or something which makes it clear it's not a VLAN either)

Entering (a) is good because it produces a:
	 ip -6 route add blackhole prefix/len

which means that any subnet that isn't allocated anywhere does not
result in a routing loop.  It also causes the 6relayd to assign prefixes
to each interface. 

I will see if I can hack on 6relayd, because I really like this.

    >> At least one prefix is behind another router, so this router can not
    >> see that prefix is already in use.  Suggestion: start at highest
    >> number available and work downwards.

    Dave> Two interior prefix allocation methods have been described by the
    Dave> homenet and hipnet rfcs. openwrt follows neither at present.

sure, that's not the point.  The point is that the set of "prefixes in
use" is not limited to just those on local interfaces, but also ones
that might have a static route elsewhere.   Yes, I agree that routing
protocols are important and useful... that's why I have 4x 3800 now for
play and testing :-)

The incidence of conflict would be less if the auto-assigned numbers
started from highest number.  At least for me, it would.

    >> 6) is there anything between complete flash and boot?
    >> If I hold down "factory reset" until it's yellow, what does that
    >> mean?  I'd like a "reset to LAN-1 has no-VLANs, just DHCP+v6-ULA-RA"
    >> and configuration is ignored, but is still present.  Is that
    >> possible?

    Dave> No.

Is such a thing possible?  Maybe using one of the other buttons on the
front?  I don't know how the 

    >> 8) I can't make the firewall do what I want, given that I have routed
    >> public IPv4, so I wind up just writing iptables command in the
    >> firewall.user file...

    Dave> yea, that's become such an uncommon case....

I understand that :-)
I asked my ISP if I could give them some v4 space back, but they never
answered me.  I suspect that they don't want any back until ARIN is
truly empty..

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [ 
	

Re: [Cerowrt-devel] shaggy dog story on 3.8.8-4 experiences

[Steven Barth]

Hi,
I was on holiday until yesterday so my reply is a bit late.


On 01.05.2013 14:58, Michael Richardson wrote:
>
> Entering (a) is good because it produces a:
> 	 ip -6 route add blackhole prefix/len
>
> which means that any subnet that isn't allocated anywhere does not
> result in a routing loop.  It also causes the 6relayd to assign prefixes
> to each interface.
>
> I will see if I can hack on 6relayd, because I really like this.
This is actually done by netifd, not 6relayd. 6relayd is the 
RA/DHCPv6-server used in OpenWrt trunk (CeroWrt uses dnsmasq here I 
think) and provides most of the homenet functionality (RFC 6204 should 
be pretty much covered by OpenWrt with very few exceptions - I think 
only one)


>
>      >> At least one prefix is behind another router, so this router can not
>      >> see that prefix is already in use.  Suggestion: start at highest
>      >> number available and work downwards.
>
>      Dave> Two interior prefix allocation methods have been described by the
>      Dave> homenet and hipnet rfcs. openwrt follows neither at present.
>
> sure, that's not the point.  The point is that the set of "prefixes in
> use" is not limited to just those on local interfaces, but also ones
> that might have a static route elsewhere.   Yes, I agree that routing
> protocols are important and useful... that's why I have 4x 3800 now for
> play and testing :-)
>
> The incidence of conflict would be less if the auto-assigned numbers
> started from highest number.  At least for me, it would.

That's imo not a good idea because it doesn't really solve the issue.

I could modify the assignment logic in netifd to scan through all 
ip6addrs configured before distributing any prefixes and exclude ip6addr 
assigned prefix parts from being reassigned again.


Regarding the other cases: It is possible to set ip6assign to something 
<64 so that a bigger prefix is assigned to a downstream-interface. With 
6relayd as DHCPv6-server this automatically makes anything but the first 
/64 available on the interface to downstream routers via DHCPv6-PD (the 
first /64 is announced using RAs on the interface).


At the moment you could simply use ip6hint with every ip6assign 
consequently on every interface and keep some prefix-parts from being 
used by the distribution logic.


So this is all fine with static / 6in4 however the problem here is that 
we cannot guarantee anything when it comes to DHCPv6-PD:
* We do not know beforehand how big the prefix from the ISP will be and 
if the reservation would fit.
* We do not know beforehand if the ISP excludes a certain part of the 
prefix using the RFC 6603 method which would clash with the reservation.


In cases of size issues or clashes with excluded prefixes the 
distribution logic might ignore the ip6assign / ip6hint values to get at 
least something working anyway. So with anything non-static like 
DHCPv6-PD it is in general not possible to define a reservation and just 
assume it will work.


The technically better solution would be a mechanism where programs 
could request a prefix via some RPC-mechanism from netifd. However this 
would need synchronization and callback-mechanism to inform the routing 
daemon when a prefix is assigned / deassigned etc.

Additionally this logic would need to be builtin into each individual 
routing daemon or at least we need to have some kind of hotplug-logic 
that triggers a reconfiguration and restart of such a daemon every time 
a prefix changes so that it doesn't use outdated information. This 
sounds all very painful to me.

I'm a bit lost on this issue and don't really know how to proceed right 
now so any input on this matter is very much appreciated.


Cheers,

Steven

Re: [Cerowrt-devel] shaggy dog story on 3.8.8-4 experiences

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 3620 bytes --]


>>>>> "Steven" == Steven Barth <cyrus@openwrt.org> writes:
    Steven> I was on holiday until yesterday so my reply is a bit late.

Thank you for the reply!

    >> which means that any subnet that isn't allocated anywhere does not
    >> result in a routing loop.  It also causes the 6relayd to assign prefixes
    >> to each interface.
    >> 
    >> I will see if I can hack on 6relayd, because I really like this.

    Steven> This is actually done by netifd, not 6relayd. 6relayd is the
    Steven> RA/DHCPv6-server 
    Steven> used in OpenWrt trunk (CeroWrt uses dnsmasq here I think) and provides most
    Steven> of the homenet functionality (RFC 6204 should be pretty much covered by
    Steven> OpenWrt with very few exceptions - I think only one)

okay, I see. I would have thought that since the prefix came down via
DHCPv6, that this was being done there too.  I will read through netifd.

    >> >> At least one prefix is behind another router, so this router can not
    >> >> see that prefix is already in use.  Suggestion: start at highest
    >> >> number available and work downwards.

    Dave> Two interior prefix allocation methods have been described by the
    Dave> homenet and hipnet rfcs. openwrt follows neither at present.

:-)

    >> sure, that's not the point.  The point is that the set of "prefixes in
    >> use" is not limited to just those on local interfaces, but also ones
    >> that might have a static route elsewhere.   Yes, I agree that routing
    >> protocols are important and useful... that's why I have 4x 3800 now for
    >> play and testing :-)
    >> 
    >> The incidence of conflict would be less if the auto-assigned numbers
    >> started from highest number.  At least for me, it would.

    Steven> That's imo not a good idea because it doesn't really solve the issue.

I understand... I wasn't suggesting it would solve the problem, just
make it less of a toe stumbing event.

    Steven> I could modify the assignment logic in netifd to scan through all ip6addrs
    Steven> configured before distributing any prefixes and exclude ip6addr assigned
    Steven> prefix parts from being reassigned again.

That's really what we need.

    Steven> Regarding the other cases: It is possible to set ip6assign to something <64
    Steven> so that a bigger prefix is assigned to a downstream-interface. With 6relayd
    Steven> as DHCPv6-server this automatically makes anything but the first /64
    Steven> available on the interface to downstream routers via
    Steven> DHCPv6-PD (the first /64 
    Steven> is announced using RAs on the interface).

wow, that's really cool.  I will actually use this then so that my main
router can then play ISP for my test router...

    Steven> The technically better solution would be a mechanism where programs could
    Steven> request a prefix via some RPC-mechanism from netifd. However
    Steven> this would need 
    Steven> synchronization and callback-mechanism to inform the routing daemon when a
    Steven> prefix is assigned / deassigned etc.

dare I suggest... dbus?
I know that network manager already sends out dbus messages about new
prefixes configured, and things like pidgin listen and reconnect when
they see a new prefix.

"everyone is doing it"  (even the Automotive people)

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [ 
	

[-- Attachment #2: Type: application/pgp-signature, Size: 307 bytes --]