[Cerowrt-devel] mdns reflector issues on ipv6/babel routing through nat.

[Cerowrt-devel] mdns reflector issues on ipv6/babel routing through nat.

[Dave Taht]

Once you get to a few routers, a few deep, (3 in series in this case,
9 overall), the avahi mdns proxy starts to malfunction over ipv6, and
I ended up with a rather nasty broadcast storm.

So I had to disable the ipv6 multicast of mdns in order to get my
network back in this (excessively) complex network.

use-ipv6=no in the /etc/avahi/avahi-daemon file

Seems to work fine, two deep. Curiously, I did not observe a similar
storm for ipv4...

Now this is across like 5 different versions of cerowrt, but it would
not surprise me that this is a  generic problem with avahi on ipv6,
and/or a symptom of the brain-damaged-ness of mdns in the first place.

use-ipv6=no

I note that when you connect cero boxes together in a babel mesh
configuration, site-local multicast is not a problem, because it
doesn't work in the first place (by design). This can be construed as
an advantage (no broadcast storm), or disadvantage (mdns and
site-local multicast doesn't work across meshed links)

Incidentally, I don't know if anyone would purposely inflict a network
this complex on themselves:

http://pastebin.com/LzeeiCXg

but it does illustrate that a complex, automagically routed, fault
tolerant ipv4 and ipv6 network IS feasible, so long as all internal
addresses are unique.

The biggest problem I run into is that 'failover-capable,
fault-tolerant routing' introduces major headaches with firewall
rules.

Another thing the above paste illustrates that you can mix and match
ipv4 nat with ipv6 fully meshed routing.

The box I took that trace off has babel enabled on all interfaces, and
has the following rule at the top of it's babeld.conf file

out if ge00 ip 0.0.0.0/0 deny

(as do multiple other boxes in the lab on the external network)

this prohibits announcing ipv4 routes across the natted ge00
interface, but allows ipv6. In the caseof that paste, this particular
router has NO internal wired connections at all, it just meshes
internally for ipv4, and because ge00 is a higher quality (ethernet)
interface, babel chooses it for the default for ipv6 for most routes.



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Cerowrt-devel] mdns reflector issues on ipv6/babel routing through nat.

[Dave Taht]

On Sun, Mar 18, 2012 at 2:22 PM, Dave Taht <dave.taht@gmail.com> wrote:
> Once you get to a few routers, a few deep, (3 in series in this case,
> 9 overall), the avahi mdns proxy starts to malfunction over ipv6, and
> I ended up with a rather nasty broadcast storm.
>
> So I had to disable the ipv6 multicast of mdns in order to get my
> network back in this (excessively) complex network.
>
> use-ipv6=no in the /etc/avahi/avahi-daemon file
>
> Seems to work fine, two deep. Curiously, I did not observe a similar
> storm for ipv4...
>
> Now this is across like 5 different versions of cerowrt, but it would
> not surprise me that this is a  generic problem with avahi on ipv6,
> and/or a symptom of the brain-damaged-ness of mdns in the first place.
>
> use-ipv6=no
>
> I note that when you connect cero boxes together in a babel mesh
> configuration, site-local multicast is not a problem, because it
> doesn't work in the first place (by design). This can be construed as
> an advantage (no broadcast storm), or disadvantage (mdns and
> site-local multicast doesn't work across meshed links)

Actually  I was wrong. I'm STILL observing a broadcast storm, AND it
is taking place across the meshed links too.... aggggh.....

> Incidentally, I don't know if anyone would purposely inflict a network
> this complex on themselves:
>
> http://pastebin.com/LzeeiCXg
>
> but it does illustrate that a complex, automagically routed, fault
> tolerant ipv4 and ipv6 network IS feasible, so long as all internal
> addresses are unique.
>
> The biggest problem I run into is that 'failover-capable,
> fault-tolerant routing' introduces major headaches with firewall
> rules.
>
> Another thing the above paste illustrates that you can mix and match
> ipv4 nat with ipv6 fully meshed routing.
>
> The box I took that trace off has babel enabled on all interfaces, and
> has the following rule at the top of it's babeld.conf file
>
> out if ge00 ip 0.0.0.0/0 deny
>
> (as do multiple other boxes in the lab on the external network)
>
> this prohibits announcing ipv4 routes across the natted ge00
> interface, but allows ipv6. In the caseof that paste, this particular
> router has NO internal wired connections at all, it just meshes
> internally for ipv4, and because ge00 is a higher quality (ethernet)
> interface, babel chooses it for the default for ipv6 for most routes.
>
>
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://www.bufferbloat.net



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Cerowrt-devel] mdns reflector issues on ipv6/babel routing through nat.

[Jim Gettys]

Oh, cool.  Broadcast storms are soooo much fun...
                - Jim


On 03/18/2012 05:24 PM, Dave Taht wrote:
> On Sun, Mar 18, 2012 at 2:22 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> Once you get to a few routers, a few deep, (3 in series in this case,
>> 9 overall), the avahi mdns proxy starts to malfunction over ipv6, and
>> I ended up with a rather nasty broadcast storm.
>>
>> So I had to disable the ipv6 multicast of mdns in order to get my
>> network back in this (excessively) complex network.
>>
>> use-ipv6=no in the /etc/avahi/avahi-daemon file
>>
>> Seems to work fine, two deep. Curiously, I did not observe a similar
>> storm for ipv4...
>>
>> Now this is across like 5 different versions of cerowrt, but it would
>> not surprise me that this is a  generic problem with avahi on ipv6,
>> and/or a symptom of the brain-damaged-ness of mdns in the first place.
>>
>> use-ipv6=no
>>
>> I note that when you connect cero boxes together in a babel mesh
>> configuration, site-local multicast is not a problem, because it
>> doesn't work in the first place (by design). This can be construed as
>> an advantage (no broadcast storm), or disadvantage (mdns and
>> site-local multicast doesn't work across meshed links)
> Actually  I was wrong. I'm STILL observing a broadcast storm, AND it
> is taking place across the meshed links too.... aggggh.....
>
>> Incidentally, I don't know if anyone would purposely inflict a network
>> this complex on themselves:
>>
>> http://pastebin.com/LzeeiCXg
>>
>> but it does illustrate that a complex, automagically routed, fault
>> tolerant ipv4 and ipv6 network IS feasible, so long as all internal
>> addresses are unique.
>>
>> The biggest problem I run into is that 'failover-capable,
>> fault-tolerant routing' introduces major headaches with firewall
>> rules.
>>
>> Another thing the above paste illustrates that you can mix and match
>> ipv4 nat with ipv6 fully meshed routing.
>>
>> The box I took that trace off has babel enabled on all interfaces, and
>> has the following rule at the top of it's babeld.conf file
>>
>> out if ge00 ip 0.0.0.0/0 deny
>>
>> (as do multiple other boxes in the lab on the external network)
>>
>> this prohibits announcing ipv4 routes across the natted ge00
>> interface, but allows ipv6. In the caseof that paste, this particular
>> router has NO internal wired connections at all, it just meshes
>> internally for ipv4, and because ge00 is a higher quality (ethernet)
>> interface, babel chooses it for the default for ipv6 for most routes.
>>
>>
>>
>> --
>> Dave Täht
>> SKYPE: davetaht
>> US Tel: 1-239-829-5608
>> http://www.bufferbloat.net
>
>

Re: [Cerowrt-devel] mdns reflector issues on ipv6/babel routing through nat.

[Dave Taht]

On Sun, Mar 18, 2012 at 2:27 PM, Jim Gettys <jg@freedesktop.org> wrote:
> Oh, cool.  Broadcast storms are soooo much fun...

I ended up having to disable avahi entirely.

the NEAT thing about this, was that although my network was basically
saturated, it still more or less worked and I hardly noticed I had a
problem, due to having sfq everywhere....

>                - Jim
>
>
> On 03/18/2012 05:24 PM, Dave Taht wrote:
>> On Sun, Mar 18, 2012 at 2:22 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>> Once you get to a few routers, a few deep, (3 in series in this case,
>>> 9 overall), the avahi mdns proxy starts to malfunction over ipv6, and
>>> I ended up with a rather nasty broadcast storm.
>>>
>>> So I had to disable the ipv6 multicast of mdns in order to get my
>>> network back in this (excessively) complex network.
>>>
>>> use-ipv6=no in the /etc/avahi/avahi-daemon file
>>>
>>> Seems to work fine, two deep. Curiously, I did not observe a similar
>>> storm for ipv4...
>>>
>>> Now this is across like 5 different versions of cerowrt, but it would
>>> not surprise me that this is a  generic problem with avahi on ipv6,
>>> and/or a symptom of the brain-damaged-ness of mdns in the first place.
>>>
>>> use-ipv6=no
>>>
>>> I note that when you connect cero boxes together in a babel mesh
>>> configuration, site-local multicast is not a problem, because it
>>> doesn't work in the first place (by design). This can be construed as
>>> an advantage (no broadcast storm), or disadvantage (mdns and
>>> site-local multicast doesn't work across meshed links)
>> Actually  I was wrong. I'm STILL observing a broadcast storm, AND it
>> is taking place across the meshed links too.... aggggh.....
>>
>>> Incidentally, I don't know if anyone would purposely inflict a network
>>> this complex on themselves:
>>>
>>> http://pastebin.com/LzeeiCXg
>>>
>>> but it does illustrate that a complex, automagically routed, fault
>>> tolerant ipv4 and ipv6 network IS feasible, so long as all internal
>>> addresses are unique.
>>>
>>> The biggest problem I run into is that 'failover-capable,
>>> fault-tolerant routing' introduces major headaches with firewall
>>> rules.
>>>
>>> Another thing the above paste illustrates that you can mix and match
>>> ipv4 nat with ipv6 fully meshed routing.
>>>
>>> The box I took that trace off has babel enabled on all interfaces, and
>>> has the following rule at the top of it's babeld.conf file
>>>
>>> out if ge00 ip 0.0.0.0/0 deny
>>>
>>> (as do multiple other boxes in the lab on the external network)
>>>
>>> this prohibits announcing ipv4 routes across the natted ge00
>>> interface, but allows ipv6. In the caseof that paste, this particular
>>> router has NO internal wired connections at all, it just meshes
>>> internally for ipv4, and because ge00 is a higher quality (ethernet)
>>> interface, babel chooses it for the default for ipv6 for most routes.
>>>
>>>
>>>
>>> --
>>> Dave Täht
>>> SKYPE: davetaht
>>> US Tel: 1-239-829-5608
>>> http://www.bufferbloat.net
>>
>>
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Cerowrt-devel] mdns reflector issues on ipv6/babel routing through nat.

[Dave Taht]

On Sun, Mar 18, 2012 at 2:30 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Sun, Mar 18, 2012 at 2:27 PM, Jim Gettys <jg@freedesktop.org> wrote:
>> Oh, cool.  Broadcast storms are soooo much fun...
>
> I ended up having to disable avahi entirely.

(as I found ipv4 multicast going wild, too, on a few links)

I note that I have had pimd installed for ages, but nothing to try it
on seems to work.

uftp does not want to route across multiple interfaces, either.

now, after todays diversion, some might think that's a good thing, but
routed multicast USED to sort of work a decade ago....

I'd had to rip out mrd6 earlier due to excessive cpu usage, so I have
no idea if that works.

About the only 'current' multicast capable daemon that looks well
maintained is Xorp, but it's huge.
Quagga has some pimd support but it's basically a glorified igmp implementation.

multicast research seems to have died in the last decade.

I sure would like to beat up on multicast in the large, so I can get a
better grip on how it's malfunctioning in the small....

>
> the NEAT thing about this, was that although my network was basically
> saturated, it still more or less worked and I hardly noticed I had a
> problem, due to having sfq everywhere....
>
>>                - Jim
>>
>>
>> On 03/18/2012 05:24 PM, Dave Taht wrote:
>>> On Sun, Mar 18, 2012 at 2:22 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>>> Once you get to a few routers, a few deep, (3 in series in this case,
>>>> 9 overall), the avahi mdns proxy starts to malfunction over ipv6, and
>>>> I ended up with a rather nasty broadcast storm.
>>>>
>>>> So I had to disable the ipv6 multicast of mdns in order to get my
>>>> network back in this (excessively) complex network.
>>>>
>>>> use-ipv6=no in the /etc/avahi/avahi-daemon file
>>>>
>>>> Seems to work fine, two deep. Curiously, I did not observe a similar
>>>> storm for ipv4...
>>>>
>>>> Now this is across like 5 different versions of cerowrt, but it would
>>>> not surprise me that this is a  generic problem with avahi on ipv6,
>>>> and/or a symptom of the brain-damaged-ness of mdns in the first place.
>>>>
>>>> use-ipv6=no
>>>>
>>>> I note that when you connect cero boxes together in a babel mesh
>>>> configuration, site-local multicast is not a problem, because it
>>>> doesn't work in the first place (by design). This can be construed as
>>>> an advantage (no broadcast storm), or disadvantage (mdns and
>>>> site-local multicast doesn't work across meshed links)
>>> Actually  I was wrong. I'm STILL observing a broadcast storm, AND it
>>> is taking place across the meshed links too.... aggggh.....
>>>
>>>> Incidentally, I don't know if anyone would purposely inflict a network
>>>> this complex on themselves:
>>>>
>>>> http://pastebin.com/LzeeiCXg
>>>>
>>>> but it does illustrate that a complex, automagically routed, fault
>>>> tolerant ipv4 and ipv6 network IS feasible, so long as all internal
>>>> addresses are unique.
>>>>
>>>> The biggest problem I run into is that 'failover-capable,
>>>> fault-tolerant routing' introduces major headaches with firewall
>>>> rules.
>>>>
>>>> Another thing the above paste illustrates that you can mix and match
>>>> ipv4 nat with ipv6 fully meshed routing.
>>>>
>>>> The box I took that trace off has babel enabled on all interfaces, and
>>>> has the following rule at the top of it's babeld.conf file
>>>>
>>>> out if ge00 ip 0.0.0.0/0 deny
>>>>
>>>> (as do multiple other boxes in the lab on the external network)
>>>>
>>>> this prohibits announcing ipv4 routes across the natted ge00
>>>> interface, but allows ipv6. In the caseof that paste, this particular
>>>> router has NO internal wired connections at all, it just meshes
>>>> internally for ipv4, and because ge00 is a higher quality (ethernet)
>>>> interface, babel chooses it for the default for ipv6 for most routes.
>>>>
>>>>
>>>>
>>>> --
>>>> Dave Täht
>>>> SKYPE: davetaht
>>>> US Tel: 1-239-829-5608
>>>> http://www.bufferbloat.net
>>>
>>>
>>
>
>
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://www.bufferbloat.net



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Cerowrt-devel] mdns reflector issues on ipv6/babel routing through nat.

[Robert Bradley]

On 18 March 2012 21:22, Dave Taht <dave.taht@gmail.com> wrote:
>
> Once you get to a few routers, a few deep, (3 in series in this case,
> 9 overall), the avahi mdns proxy starts to malfunction over ipv6, and
> I ended up with a rather nasty broadcast storm.

<snip>

> Now this is across like 5 different versions of cerowrt, but it would
> not surprise me that this is a  generic problem with avahi on ipv6,
> and/or a symptom of the brain-damaged-ness of mdns in the first place.
>

I thought that broadcast storms were a known issue with Avahi's mdns
reflector.  The man pages for avahi-daemon.conf
(http://linux.die.net/man/5/avahi-daemon.conf) have this to say:

"Make sure to not run multiple reflectors between the same networks,
this might cause them to play Ping Pong with mDNS packets."

I think the official answer for mesh networks (and subnetted networks
in general) is to switch to wide-area Bonjour - in other words,
dynamically publishing records to a real DNS server.  Avahi had
problems with publishing records in the past, but as far as I can tell
those have been resolved.

One problem with using Wide-Area Bonjour is configuration.  As far as
I can tell, you have to specify a specific (non-.local) domain for
wide-area browsing, and there is no good automated solution for home
users.  For Avahi, this means editing /etc/avahi/avahi-daemon.conf and
setting the "domain-name" and "browse-domains" options.  Setting this
up on OS X and Windows is difficult for most home users, and involves
either installing the SDK or editing the Registry or configuration
files by hand.  A set of DHCP options (similar to those for WINS and
NBT on Windows) for disabling mdns and setting up wide-area Bonjour
would be useful...

Worse than that, though, some sites (e.g.
http://www.afp548.com/article.php?story=20090225001154457) suggest
that Apple's Bonjour daemon will only register global IP addresses
with the DNS server, and not private addresses.  This makes the whole
thing worse than useless for CeroWRT!

Solutions to this?

- Give up on cross-subnet publishing entirely.
- Write a variant of Avahi's mDNS reflector that listens to local mDNS
traffic and reissues queries over unicast DNS.  Any local record
announcements it sees would need to be registered with the unicast DNS
server too.
- Get the DHCP server to add initial SRV/TXT records to the DNS
server, so you're not dependent on OS X doing the right thing.
mDNSResponder can then update these records later if it feels like it.
 The disadvantage is that you're either reporting services that may
not exist, or are not reporting some services.
- Advertise global IPv6 addresses throughout the mesh?  This might
work, although Windows tends to object to mDNS over IPv6...
--
Robert Bradley

Re: [Cerowrt-devel] mdns reflector issues on ipv6/babel routing through nat.

[Robert Bradley]

On 19/03/12 19:41, Robert Bradley wrote:
> Avahi had
> problems with publishing records in the past, but as far as I can tell
> those have been resolved.
>

I take that back - patches were apparently written for this but never 
made public, as far as I can see.

-- 
Robert Bradley

Re: [Cerowrt-devel] mdns reflector issues on ipv6/babel routing through nat.

[Dave Taht]

On Mon, Mar 19, 2012 at 3:07 PM, Robert Bradley
<robert.bradley1@gmail.com> wrote:
> On 19/03/12 19:41, Robert Bradley wrote:
>>
>> Avahi had
>> problems with publishing records in the past, but as far as I can tell
>> those have been resolved.
>>
>
> I take that back - patches were apparently written for this but never made
> public, as far as I can see.

links? emails? This has got to get fixed in the long run... nagging
the developers to publish and polish is generally useful.
>
>
> --
> Robert Bradley
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Cerowrt-devel] mdns reflector issues on ipv6/babel routing through nat.

[Robert Bradley]

[-- Attachment #1: Type: text/plain, Size: 1478 bytes --]

On 19/03/12 22:11, Dave Taht wrote:
> On Mon, Mar 19, 2012 at 3:07 PM, Robert Bradley
> <robert.bradley1@gmail.com>  wrote:
>> On 19/03/12 19:41, Robert Bradley wrote:
>>> Avahi had
>>> problems with publishing records in the past, but as far as I can tell
>>> those have been resolved.
>>>
>> I take that back - patches were apparently written for this but never made
>> public, as far as I can see.
> links? emails? This has got to get fixed in the long run... nagging
> the developers to publish and polish is generally useful.
>

The email mentioning this is at 
http://lists.freedesktop.org/archives/avahi/2008-December/001523.html, 
by Federico Lucifredi (//flucifredi@acm.org).  That was in December 
2008.  There was an email in July 2011 asking about this 
(http://lists.freedesktop.org/archives/avahi/2011-June/002037.html) 
which did not get anywhere.  I have emailed Federico tonight about his 
patches to Avahi, so maybe(!) these can finally be published and integrated.

I should mention at this point (although it's not ideal) that various 
workarounds exist.  For example, there is a Ruby D-Bus daemon to do 
wide-area updates at <https://github.com/Roguelazer/wamupd/>, and 
Federico Lucifredi has a Perl script at 
<http://primates.ximian.com/~flucifredi/mybook/dns-update.html> for 
one-off registration.  If one of these (or a similar script) were 
installed by default in Linux distributions, that might be enough for 
desktop machines.

-- 
Robert Bradley

[-- Attachment #2: Type: text/html, Size: 2720 bytes --]

Re: [Cerowrt-devel] mdns reflector issues on ipv6/babel routing through nat.

[Robert Bradley]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

On 20/03/12 00:54, Robert Bradley wrote:
>  The email mentioning this is at
>  http://lists.freedesktop.org/archives/avahi/2008-December/001523.html, by
>  Federico Lucifredi (//flucifredi@acm.org). That was in December 2008. There
>  was an email in July 2011 asking about this
>  (http://lists.freedesktop.org/archives/avahi/2011-June/002037.html) which did
>  not get anywhere. I have emailed Federico tonight about his patches to Avahi,
>  so maybe(!) these can finally be published and integrated.
>

Belated update: Having heard from Federico, he sounds willing to get the patches cleaned up.  Also, if anyone here has access to Harvard's Grossman library, the patches are apparently included in Federico's thesis:

http://hollis.harvard.edu/?itemid=|library/m/aleph|011831208


-- 
Robert Bradley


[-- Attachment #2: Type: text/html, Size: 1477 bytes --]