Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[Dave Taht]

On Mon, Jul 2, 2012 at 7:52 AM, Denis Ovsienko <infrastation@yandex.ru> wrote:
> 01.07.2012, 03:53, "Dave Taht" <dave.taht@gmail.com>:
>> On Sat, Jun 30, 2012 at 7:44 PM, Juliusz Chroboczek <jch@pps.jussieu.fr> wrote:
>>
>>>>>>    babel rxcost 100
>>>  Implemented, committed, tested, pushed, and documented.
>>
>> Packaged, compiled, pushed to ceropackages-3-3 on github, and built in
>> the (as yet untested) development-only build of cerowrt, here:
>>
>> http://snapon.lab.bufferbloat.net/~cero1/3.3/3.3.8-8/
>
> There are 7 boxes in my temporary possession: one WNDR3700v2 and six WNDR3800. For the next couple of weeks these won't be used for anything but CeroWrt build tests (no idea what happens after that). I have just flashed all of them with 3.3.8-8 and passed through a custom config script, which besides other things currently configures Babel exchanges to be authenticated. The only major outstanding issue seems to be the default route one (which doesn't reproduce in today's topology, but I know it exists).

How goeth ipv6?

>
> The project may get more options, if we drive the prototype towards a finished deliverable.

I am very enthusiastic about babel's new authenticated mesh routing!

It is also my opinion that without a decent drop and packet mixing
strategy that mesh networks will perform badly under load. I'm hoping
that fq_codel does well, although it seems very likely that an
aggregation aware and fq_codel-like strategy needs to move into the
mac80211 layer, which is perhaps years worth of work.

What would a deliverable look like? What would interest people enough
to get some good data, papers written, progress made, more
users/developers and cash in the door?

...

I presently have 4 Nanostation M5, 3 picostation HP, and 5 wndr3700s.
I've found a good site to work with 2.4ghz and 5.x ghz radios (a 110
acre campground that has given me permission to play here) and test
fq_codel in various forms of cerowrt in, for as long as I like.

One of the things we've really struggled with was in today's saturated
2.4ghz environment there is no way to benchmark both radios. We've
been reduced to suggesting we 'flee to the mountains' in order to get
good results. OK, I just did that.

My own primary purpose *right now* is to be repeating a swath of tests
we ran on predecessors to fq_codel, (sfq, sfqred) to get a baseline on
what happens with the current OS and then to be able to compare them
against fq_codel. BEFORE linux 3.5 goes stable in august.

Some of this can happen in bloatlab #1 and on the x86 hw, but as for
wireless, that needs something of a greenfield environment to do
right.

This was the first post-BQL result I'd got back in november, 2011 -
the first one that showed that there was hope we could, indeed, fix
bufferbloat, on ethernet

http://www.teklibre.com/~d/bloat/pfifo_fast_vs_sfq_qfq_log.png

We'd finally got (under ethernet only) enough control of the stack to
reduce latency under load on linux by nearly 2 orders of magnitude.
Now, QFQ was heavyweight, so a string of mods to sfq went into linux
3.2 and 3.3, ultimately culminating in sfqred, which was very nice,
although
we had to rip out a feature (head of queue) as unstable that had made
it very competitive with QFQ...

http://www.teklibre.com/~d/bloat/hoqvssfqred.ps (april 15, 2012)

(the pfifo_fast results are so bad as to be NOT worth showing)

but the red component of sfqred proved hard to tune, although it did
manage to hold latency under wireless load to a relatively sane
60-90ms "band".

http://www.teklibre.com/~d/bloat/ping_log.ps

...

and then


...

codel

http://www.bufferbloat.net/projects/codel/wiki

arrived from van jacobson and kathie nichols. A short time later eric
dumazet wrote fq_codel... which basically ripped out the red component
of sfqred AND found a middle path towards having new stream entry be
less under-optimized than in sfq...

and all the results are looking amazing...

but I simply haven't had time/energy/money to repeat these three tests
to see how much/if we won or not.

Anyway, I'm planning on repeating these tests on current
cerowrt/openwrt software and a variety of hardware.  And I/someone
need to make the test and statistics collection more robust, and
writing that code over the next few months in the yurt here seems to
be a relaxing prospect.

(I will be in Paris July 15-29 however not enough spare cash to make
ietf in vancouver. :( )

Also, as in the current cerowrt codebase are also patches to be able
to dynamically reduce queuing in the ath9k drivers, my secondary
purpose is to gather some data on wireless's behavior w and w/o
fq_codel and sane amounts of buffering at that layer. I really don't
expect to win enough for it to be useful but that's why we experiment.

In any case, I find babel very useful in letting me setup (and keep
up, and then crash) large numbers of nodes, and as it's a layer 3
protocol it's a lot easier to spread it across ethernet and wifi and
also observe what's going vs a vs other mesh protocols.

1) I could start publishing the images/source for a babel-mesh-enabled
nanostation m5 and picostation hp which are cerowrt derived but
seriously cut down in size.

2) Trying to have a project with a definable short and long term goal
seems a good idea. I outlined my own main ones above.

3) Given all the hoopla about crowdfunding, I was thinking of setting
up an idea like this as a kickstarter project.

The amount of time/money I've spent on the bufferbloat projects long
ago exceeded my own resources. Yet: I just turned down a lucrative day
job because I'm uncomfortable with fq_codel being undertested - and
what I'd like to be doing far exceeds my own resources.

Ideas towards a project definition/deliverable that would be useful,
interesting, & sustainable
would be useful.

> --
>     Denis Ovsienko
>
> _______________________________________________
> Babel-users mailing list
> Babel-users@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users



-- 
Dave Täht
http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-6 is out
with fq_codel!"

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[L. Aaron Kaplan]

[-- Attachment #1: Type: text/plain, Size: 2156 bytes --]


On Jul 2, 2012, at 5:36 PM, Dave Taht wrote:

> How goeth ipv6?
> 
>> 
>> The project may get more options, if we drive the prototype towards a finished deliverable.
> 
> I am very enthusiastic about babel's new authenticated mesh routing!
> 
> It is also my opinion that without a decent drop and packet mixing
> strategy that mesh networks will perform badly under load. I'm hoping
> that fq_codel does well, although it seems very likely that an
> aggregation aware and fq_codel-like strategy needs to move into the
> mac80211 layer, which is perhaps years worth of work.
Or have better cross-layer communication - see below....
> 
> What would a deliverable look like? What would interest people enough
> to get some good data, papers written, progress made, more
> users/developers and cash in the door?


I would like to point you towards a nice feature that Henning is currently implementing: DLEP - a (mostly routing protocol independent) framework for communicating settings and metrics between a radio and a routing daemon.
See https://datatracker.ietf.org/doc/draft-ietf-manet-dlep/

Abstract


   When routing devices rely on modems to effect communications over 
   wireless links, they need timely and accurate knowledge of the 
   characteristics of the link (speed, state, etc.) in order to make 
   forwarding decisions. In mobile or other environments where these 
   characteristics change frequently, manual configurations or the  
   inference of state through routing or transport protocols does not 
   allow the router to make the best decisions. A bidirectional, event-
   driven communication channel between the router and the modem is 
   necessary.



More info and code samples on the olsr.org git repo. I'd like to also encourage Babel coders to take a look at it and collectively improve this.
It will be - as Dave already mentioned - anyway a hard problem to have this working for many radios in a standardized way, so it would really be helpful if different mesh developers for once could work *together* on some general feature which improves everybody's metrics.


Aaron.


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[Dave Taht]

On Mon, Jul 2, 2012 at 12:16 PM, L. Aaron Kaplan <aaron@lo-res.org> wrote:
>
> On Jul 2, 2012, at 5:36 PM, Dave Taht wrote:
>
>> How goeth ipv6?
>>
>>>
>>> The project may get more options, if we drive the prototype towards a finished deliverable.
>>
>> I am very enthusiastic about babel's new authenticated mesh routing!
>>
>> It is also my opinion that without a decent drop and packet mixing
>> strategy that mesh networks will perform badly under load. I'm hoping
>> that fq_codel does well, although it seems very likely that an
>> aggregation aware and fq_codel-like strategy needs to move into the
>> mac80211 layer, which is perhaps years worth of work.
> Or have better cross-layer communication - see below....
>>
>> What would a deliverable look like? What would interest people enough
>> to get some good data, papers written, progress made, more
>> users/developers and cash in the door?
>
>
> I would like to point you towards a nice feature that Henning is currently implementing: DLEP - a (mostly routing protocol independent) framework for communicating settings and metrics between a radio and a routing daemon.
> See https://datatracker.ietf.org/doc/draft-ietf-manet-dlep/
>
> Abstract
>
>
>    When routing devices rely on modems to effect communications over
>    wireless links, they need timely and accurate knowledge of the
>    characteristics of the link (speed, state, etc.) in order to make
>    forwarding decisions. In mobile or other environments where these
>    characteristics change frequently, manual configurations or the
>    inference of state through routing or transport protocols does not
>    allow the router to make the best decisions. A bidirectional, event-
>    driven communication channel between the router and the modem is
>    necessary.
>
>
>
> More info and code samples on the olsr.org git repo. I'd like to also encourage Babel coders to take a look at it and collectively improve this.

I just did. I like it.

I note that minstrel also supplies currently unreachable-via-api
information as to rates and retries that would be useful to be
exposed, but has a higher complexity than the above spec.
(note I only just read it, soo...). Andrew mcgregor has a
too-big-for-the-MPU paper on how minstrel works he's been distributing
privately.

Also I was fiddling with supplying GPS data and sensor data (over
babel, but it's not the best choice) both for more optimal wifi access
and for a prototype of an earthquake early warning system... and I'd
argue that GPS (and GPS time) data is generally useful and gps's are
now everywhere, a good way to distribute it could be added to the
above.

> It will be - as Dave already mentioned - anyway a hard problem to have this working for many radios in a standardized way, so it would really be helpful if different mesh developers for once could work *together* on some general feature which improves everybody's metrics.

YES! Please, can we mesh people find a way to work together for a change?

my own work on bufferbloat to a large extent is driven by a (5 years
worth of) quest to solve the "dense mesh" problem that OLPCs had. The
fq_codel work at least theoretically (but currently at the wrong
layer) is a means to improve that for all wired/wireless networks and
any given mesh protocol in particular would benefit. I have been
tracking batman-adv as well, but not olsr to any extent. Where does
"working code" lie?

I note that fq_codel throws an interesting congestion-related stat
away right now, which I'd like to find a way to include in a route
metric.

I also note that I am intensely dissatisfied with babel's current
metric, which is basically "ping" over multicast, but I'm happy as
babel's metric is pluggable and can be changed.

>
> Aaron.
>



-- 
Dave Täht
http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-6 is out
with fq_codel!"

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[L. Aaron Kaplan]

[-- Attachment #1: Type: text/plain, Size: 4906 bytes --]


On Jul 2, 2012, at 6:44 PM, Dave Taht wrote:

> On Mon, Jul 2, 2012 at 12:16 PM, L. Aaron Kaplan <aaron@lo-res.org> wrote:
>> 
>> On Jul 2, 2012, at 5:36 PM, Dave Taht wrote:
>> 
>>> How goeth ipv6?
>>> 
>>>> 
>>>> The project may get more options, if we drive the prototype towards a finished deliverable.
>>> 
>>> I am very enthusiastic about babel's new authenticated mesh routing!
>>> 
>>> It is also my opinion that without a decent drop and packet mixing
>>> strategy that mesh networks will perform badly under load. I'm hoping
>>> that fq_codel does well, although it seems very likely that an
>>> aggregation aware and fq_codel-like strategy needs to move into the
>>> mac80211 layer, which is perhaps years worth of work.
>> Or have better cross-layer communication - see below....
>>> 
>>> What would a deliverable look like? What would interest people enough
>>> to get some good data, papers written, progress made, more
>>> users/developers and cash in the door?
>> 
>> 
>> I would like to point you towards a nice feature that Henning is currently implementing: DLEP - a (mostly routing protocol independent) framework for communicating settings and metrics between a radio and a routing daemon.
>> See https://datatracker.ietf.org/doc/draft-ietf-manet-dlep/
>> 
>> Abstract
>> 
>> 
>>   When routing devices rely on modems to effect communications over
>>   wireless links, they need timely and accurate knowledge of the
>>   characteristics of the link (speed, state, etc.) in order to make
>>   forwarding decisions. In mobile or other environments where these
>>   characteristics change frequently, manual configurations or the
>>   inference of state through routing or transport protocols does not
>>   allow the router to make the best decisions. A bidirectional, event-
>>   driven communication channel between the router and the modem is
>>   necessary.
>> 
>> 
>> 
>> More info and code samples on the olsr.org git repo. I'd like to also encourage Babel coders to take a look at it and collectively improve this.
> 
> I just did. I like it.
> 

Nice :)

> I note that minstrel also supplies currently unreachable-via-api
> information as to rates and retries that would be useful to be
> exposed, but has a higher complexity than the above spec.
> (note I only just read it, soo...). Andrew mcgregor has a
> too-big-for-the-MPU paper on how minstrel works he's been distributing
> privately.
> 
> Also I was fiddling with supplying GPS data and sensor data (over
> babel, but it's not the best choice) both for more optimal wifi access
> and for a prototype of an earthquake early warning system... and I'd
> argue that GPS (and GPS time) data is generally useful and gps's are
> now everywhere, a good way to distribute it could be added to the
> above.
> 
>> It will be - as Dave already mentioned - anyway a hard problem to have this working for many radios in a standardized way, so it would really be helpful if different mesh developers for once could work *together* on some general feature which improves everybody's metrics.
> 
> YES! Please, can we mesh people find a way to work together for a change?

Absolutely!

> 
> my own work on bufferbloat to a large extent is driven by a (5 years
> worth of) quest to solve the "dense mesh" problem that OLPCs had. The

Oh!... I remember being at One Kendall Sq. and discussing this with Michailis LOL...
I know exactly what you mean :) 30 XOs in close proximity was a recipe for disaster :)

> fq_codel work at least theoretically (but currently at the wrong
> layer) is a means to improve that for all wired/wireless networks and
> any given mesh protocol in particular would benefit. I have been
> tracking batman-adv as well, but not olsr to any extent. Where does
> "working code" lie?

olsr.org/git/

What you are looking for in particular is:
http://www.olsr.org/git/?p=framework.git;a=summary
and DLEP:
http://www.olsr.org/git/?p=dlep_app.git;a=summary

> 
> I note that fq_codel throws an interesting congestion-related stat
> away right now, which I'd like to find a way to include in a route
> metric.
> 
> I also note that I am intensely dissatisfied with babel's current
> metric, which is basically "ping" over multicast, but I'm happy as

;-) the dirty little secret is : all of the current mesh protos have unsatisfactory metrics :)

> babel's metric is pluggable and can be changed.

Same for OLSR. Good design choice.


Okay, Dave . I recommend that you get a grip on the DELP draft and get in touch with Henning.
I shifted more towards the "making things happen" side of mesh R&D than to the implementation side. So you need to talk with Henning in this respect. I would love to see a common framework - well tested and hardened by multiple developers which is usable for all of us.

Aaron.



[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

[Cerowrt-devel] DLEP [was: switching cerowrt to quagga-babeld issues]

[Juliusz Chroboczek]

> I would like to point you towards a nice feature that Henning is
> currently implementing: DLEP - a (mostly routing protocol independent)
> framework for communicating settings and metrics between a radio and
> a routing daemon.

I reviewed an early draft of that.

As I understand DLEP, it is a standardised way to implement the "thin
access point" model that Cisco and friends are promoting -- a complex
(and expensive) controller associated with a number of dumb APs.  This
is a good way to enable mobility between APs with unmodified stations,
it's a good way to make network administrators happy (centralised
administration), and also a good way to make Cisco happy (since the
controller can be sold for much, much more than an AP).

Standardising the protocol that the controller uses to communicate with
the APs is a worthy goal, but pretty much orthogonal to mesh networking
research.

-- Juliusz

Re: [Cerowrt-devel] [Babel-users] DLEP [was: switching cerowrt to quagga-babeld issues]

[Henning Rogge]

On Mon, Jul 2, 2012 at 7:13 PM, Juliusz Chroboczek <jch@pps.jussieu.fr> wrote:
>> I would like to point you towards a nice feature that Henning is
>> currently implementing: DLEP - a (mostly routing protocol independent)
>> framework for communicating settings and metrics between a radio and
>> a routing daemon.
>
> I reviewed an early draft of that.

The current draft is not really in a good shape, still a lot of
"proprietary" stuff and complexity that has the get out.

> As I understand DLEP, it is a standardised way to implement the "thin
> access point" model that Cisco and friends are promoting -- a complex
> (and expensive) controller associated with a number of dumb APs.  This
> is a good way to enable mobility between APs with unmodified stations,
> it's a good way to make network administrators happy (centralised
> administration), and also a good way to make Cisco happy (since the
> controller can be sold for much, much more than an AP).

It also will allow us with an easy way to access link layer metrics,
even from radios we can only reach over ethernet.

> Standardising the protocol that the controller uses to communicate with
> the APs is a worthy goal, but pretty much orthogonal to mesh networking
> research.

Yes, it is... it might become a good raw data source for the metric
calculation of any kind of mesh routing protocol.

Henning Rogge

-- 
Steven Hawkings about cosmic inflation: "An increase of billions of
billions of percent in a tiny fraction of a second. Of course, that
was before the present government."

Re: [Cerowrt-devel] DLEP [was: switching cerowrt to quagga-babeld issues]

[L. Aaron Kaplan]

[-- Attachment #1: Type: text/plain, Size: 1158 bytes --]

Juliusz, remember that there are two approaches right now concernig DLEP:  CISCO and the rest of the world ;-)
nuff said ;)))


On Jul 2, 2012, at 7:13 PM, Juliusz Chroboczek wrote:

>> I would like to point you towards a nice feature that Henning is
>> currently implementing: DLEP - a (mostly routing protocol independent)
>> framework for communicating settings and metrics between a radio and
>> a routing daemon.
> 
> I reviewed an early draft of that.
> 
> As I understand DLEP, it is a standardised way to implement the "thin
> access point" model that Cisco and friends are promoting -- a complex
> (and expensive) controller associated with a number of dumb APs.  This
> is a good way to enable mobility between APs with unmodified stations,
> it's a good way to make network administrators happy (centralised
> administration), and also a good way to make Cisco happy (since the
> controller can be sold for much, much more than an AP).
> 
> Standardising the protocol that the controller uses to communicate with
> the APs is a worthy goal, but pretty much orthogonal to mesh networking
> research.
> 
> -- Juliusz
> 


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[Denis Ovsienko]

> How goeth ipv6?

IPv6 default route is lost somewhere on the way from kernel to zebra process (and not promoted by babeld respectively). I'll try looking at it tomorrow and also pulling together some thoughts on the rest of your message.

-- 
    Denis Ovsienko

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[Denis Ovsienko]

>>  How goeth ipv6?
>
> IPv6 default route is lost somewhere on the way from kernel to zebra process (and not promoted by babeld respectively). I'll try looking at it tomorrow and also pulling together some thoughts on the rest of your message.
>

This seems to be caused by a difference in kernel behaviour between current CeroWrt and, for example, my notebook (3.3.7-1.fc16.x86_64). Plugged into the same network one after another, they end up with different default routes in kernel:

(cero)
default via fe80::230:48ff:fed4:63e4 dev ge00  proto kernel  metric 1024  expires 1797sec

(notebook)
default via fe80::230:48ff:fed4:63e4 dev em1  proto static  metric 1 
default via fe80::230:48ff:fed4:63e4 dev em1  proto kernel  metric 1024  expires 1786sec

Note the kernel/static route protocol. The established practice of zebra process used to be ignoring any RTPROT_KERNEL netlink messages. It is not clear if RTPROT_BOOT/RTPROT_RA should be used for a RA-originated route, but RTPROT_STATIC does the job in the latter case anyway and the default route reaches zebra. In the former case the default route is effectively isolated from zebra.

Does anybody know where this difference comes from?

-- 
    Denis Ovsienko

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[Denis Ovsienko]

> Does anybody know where this difference comes from?

The difference comes from NetworkManager. Its efforts in reproducing high-metric RTPROT_KERNEL routes with low-metric RTPROT_STATIC ones are effectively hiding the kernel issue outside of CeroWrt runtime. Would it be better to add a watchdog shell script, which does the same, or patch the kernel?

-- 
    Denis Ovsienko

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[Gabriel Kerneis]

On Tue, Jul 03, 2012 at 04:35:10PM +0400, Denis Ovsienko wrote:
> > Does anybody know where this difference comes from?
> 
> The difference comes from NetworkManager. […] Would it be better to add a
> watchdog shell script, which does the same, or patch the kernel?

It would be better to remove NetworkManager ;-)

-- Gabriel

P.S. I do understand that it wouldn't solve the issue, just kidding.

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[Dave Taht]

On Tue, Jul 3, 2012 at 8:35 AM, Denis Ovsienko <infrastation@yandex.ru> wrote:
>> Does anybody know where this difference comes from?
>
> The difference comes from NetworkManager. Its efforts in reproducing high-metric RTPROT_KERNEL routes with low-metric RTPROT_STATIC ones are effectively hiding the kernel issue outside of CeroWrt runtime. Would it be better to add a watchdog shell script, which does the same, or patch the kernel?

I would *much rather* patch the kernel than have a watchdog. However I
don't quite understand
the redistribution issue vs a vs ipv6 here. If I have a "redistribute
kernel" on for ipv4, it does propagate the default route.

(I note that I dislike network manager too as it tries too hard to
work around bugs in the base OS and my own view of the world is far
more "meshy")

I'll gladly try pushing a patch up to the mainline if that's what is needed.

>
> --
>     Denis Ovsienko
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht
http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-6 is out
with fq_codel!"

[Cerowrt-devel] switching cerowrt to quagga-babeld issues

[Denis Ovsienko]

03.07.2012, 17:18, "Dave Taht" <dave.taht@gmail.com>:

>  On Tue, Jul 3, 2012 at 8:35 AM, Denis Ovsienko <infrastation@yandex.ru> wrote:
>>>   Does anybody know where this difference comes from?
>>   The difference comes from NetworkManager. Its efforts in reproducing high-metric RTPROT_KERNEL routes with low-metric RTPROT_STATIC ones are effectively hiding the kernel issue outside of CeroWrt runtime. Would it be better to add a watchdog shell script, which does the same, or patch the kernel?
>  I would *much rather* patch the kernel than have a watchdog. However I
>  don't quite understand
>  the redistribution issue vs a vs ipv6 here. If I have a "redistribute
>  kernel" on for ipv4, it does propagate the default route.

The matter is, IPv4 default route comes flagged as either "static" or "boot" (both cases are displayed without "proto" column by /sbin/ip route). This is properly picked up. IPv6 default route comes flagged as "kernel"..

>  (I note that I dislike network manager too as it tries too hard to
>  work around bugs in the base OS and my own view of the world is far
>  more "meshy")
>
>  I'll gladly try pushing a patch up to the mainline if that's what is needed.

I've got no required expertise to make such change safe for all, but starting with a Cero-only patch seems possible.

-- 
    Denis Ovsienko

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[Denis Ovsienko]

[-- Attachment #1: Type: text/plain, Size: 849 bytes --]

> I've got no required expertise to make such change safe for all, but starting with a Cero-only patch seems possible.

Hello, all.

Let me make an update on the problem and the solution. NetworkManager does not belong to the problem space of the current IPv6 default route issue in CeroWrt just because it is not packaged in CeroWrt. NetworkManager sample data is provided to explain the machinery of the kernel/zebra route delivery issue in different environments. So far I would prefer to get this issue resolved in CeroWrt and focus on the next show stopper.

There is a kernel patch attached, which should fix the issue with IPv6 default route delivery in CeroWrt (it does fix it on my PC). Offering it for generic kernels is a different story, which shouldn't start until, say, 1-2 weeks of safe test-driving.

Cheers.

-- 
    Denis Ovsienko

[-- Attachment #2: 0001-fix-RTPROT_RA-markup-of-some-RA-routes-in-netlink.patch --]
[-- Type: text/plain, Size: 2241 bytes --]

From 1d969903c6221980360f76abb5e063300e5cf3bb Mon Sep 17 00:00:00 2001
From: Denis Ovsienko <infrastation@yandex.ru>
Date: Fri, 6 Jul 2012 18:08:18 +0400
Subject: [PATCH] fix RTPROT_RA markup of some RA routes in netlink

There are three types of IPv6 routes originated by kernel ND RA code:

* Default routes standing for RA packets with non-zero router lifetime.
* Connected prefix routes standing for a Prefix Information (3) RA TLV.
* Any prefix routes standing for a Route Information (24) RA TLV.

All three types are internally stored using RTPROT_KERNEL or RTPROT_BOOT
protocol code and RTF_ADDRCONF route flag (this is the only purpose for
this flag). The flag isn't directly available in netlink socket space.
Given the need to tell route origin in userspace, for routes with
nexthops in the first turn, rt6_fill_node() tries to distinguish default
router case sending the netlink route structure with RTPROT_RA (this is
respectively the only use case for this protocol code), but to no
success due to a test condition taken wrong. All three types are
delivered with RTPROT_KERNEL.

This change is modelled after the original mailing list posting by Jeff
Haran. It fixes the test condition for the default router case and
extends it for the Route Information case.
---
 net/ipv6/route.c |   12 +++++++++---
 1 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 999a982..2f070d6 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -2441,9 +2441,15 @@ static int rt6_fill_node(struct net *net,
 	if (rt->rt6i_flags & RTF_DYNAMIC)
 		rtm->rtm_protocol = RTPROT_REDIRECT;
 	else if (rt->rt6i_flags & RTF_ADDRCONF)
-		rtm->rtm_protocol = RTPROT_KERNEL;
-	else if (rt->rt6i_flags & RTF_DEFAULT)
-		rtm->rtm_protocol = RTPROT_RA;
+	{
+		/* any ND RA route, most probably originated by kernel */
+		if (rt->rt6i_flags & RTF_DEFAULT) /* default router */
+			rtm->rtm_protocol = RTPROT_RA;
+		else if (rt->rt6i_flags & RTF_ROUTEINFO) /* any route w/nexthop */
+			rtm->rtm_protocol = RTPROT_RA;
+		else /* RTF_PREFIX_RT, interface connected prefix route */
+			rtm->rtm_protocol = RTPROT_KERNEL;
+	}
 
 	if (rt->rt6i_flags & RTF_CACHE)
 		rtm->rtm_flags |= RTM_F_CLONED;
-- 
1.7.7.6

[Cerowrt-devel] IPv6 RA and RTPROT_whatever [was: switching cerowrt to quagga-babeld issues]

[Juliusz Chroboczek]

> There is a kernel patch attached, which should fix the issue with IPv6
> default route delivery in CeroWrt (it does fix it on my PC).

I'm a little bit confused.

RAs are supposed to be ignored by routers -- the RFCs are very clear on
that.  Linux obeys the RFCs, it will disable accept_ra when forwarding
is set on an interface.

Now there's a race condition -- if the RA was accepted before the
routing daemon set the forwarding knob, then the default route will
remain.  However, it will not be renewed -- when the RA expires, you'll
lose your default route.

The workaround to this race condition is to have the routing daemon
ignore RA-originated routes.  This is traditionally done by making such
routes RTPROT_BOOT and ignoring such routes in the routing daemon (and
installing routing-protocol-originated routes with a lower metric).

So am I'm confused, or is marking RA-originated routes as anything else
than RTPROT_BOOT exactly the wrong thing to do?

-- Juliusz

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[Jim Gettys]

On 07/03/2012 09:18 AM, Dave Taht wrote:
> On Tue, Jul 3, 2012 at 8:35 AM, Denis Ovsienko <infrastation@yandex.ru> wrote:
>>> Does anybody know where this difference comes from?
>> The difference comes from NetworkManager. Its efforts in reproducing high-metric RTPROT_KERNEL routes with low-metric RTPROT_STATIC ones are effectively hiding the kernel issue outside of CeroWrt runtime. Would it be better to add a watchdog shell script, which does the same, or patch the kernel?
> I would *much rather* patch the kernel than have a watchdog. However I
> don't quite understand
> the redistribution issue vs a vs ipv6 here. If I have a "redistribute
> kernel" on for ipv4, it does propagate the default route.
>
> (I note that I dislike network manager too as it tries too hard to
> work around bugs in the base OS and my own view of the world is far
> more "meshy")
>
> I'll gladly try pushing a patch up to the mainline if that's what is needed.
>

Hey, guys: if NM has a problem, let Dan Williams know, rather than just
removing NM and flaming about NM.  Dan's a nice guy, and works hard to
make NM work well, and it's a nearly impossible task.

And as far as him trying to work around problems, I remember him
discovering that most 802.11 drivers not setting the time when they saw
announcements properly, and then going and patching as many drivers as
he could lay his hands on...  (having lost half his hair figuring out
why people were associating with the wrong AP after resume)....  So he
has given at the blood bank fixing the kernel.
                        - Jim

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[David Lamparter]

On Tue, Jul 03, 2012 at 09:18:43AM -0400, Dave Taht wrote:
> On Tue, Jul 3, 2012 at 8:35 AM, Denis Ovsienko <infrastation@yandex.ru> wrote:
> >> Does anybody know where this difference comes from?
> >
> > The difference comes from NetworkManager. Its efforts in reproducing
> > high-metric RTPROT_KERNEL routes with low-metric RTPROT_STATIC ones
> > are effectively hiding the kernel issue outside of CeroWrt runtime.
> > Would it be better to add a watchdog shell script, which does the
> > same, or patch the kernel?
> 
> I would *much rather* patch the kernel than have a watchdog. However I
> don't quite understand
> the redistribution issue vs a vs ipv6 here. If I have a "redistribute
> kernel" on for ipv4, it does propagate the default route.

I'm not sure I understood your problem here, but if it boils down to
"zebra doesn't redistribute an IPv6 RA default route", then that's by
design and shouldn't be touched.

IPv6 RA is a router to host protocol.  Routers should never accept
information from it, it is neither secure nor able to convey enough
details to prevent loops or dead-end routes.

This is also why enabling IPv6 forwarding disables reception of route
advertisements in-kernel.

If I understand correctly, your use-case is a mesh router that acts as a
host on a "parent" network.  If so, this case should be handled by a
separate daemon that receives and processes IPv6 RAs, hopefully applies
some filtering.  Also, this absolutely cannot be default behaviour.

If I misunderstood the issues, please ignore my mail.

Cheers,


-David


P.S.: Also, NetworkManager and Quagga should never run on the same host.
NetworkManager does Host processing, Quagga does Router processing, and
those two are mutually exclusive.

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 4183 bytes --]


Here's the architectural issue at the root of this issue (I think):
 
"P.S.: Also, NetworkManager and Quagga should never run on the same host.
NetworkManager does Host processing, Quagga does Router processing, and
those two are mutually exclusive."
 
It should be the case that a host and a router can exist and run in the same physical box, this is part of the IP architecture.   But Linux's network stack may make this hard to achieve (maybe impossible without careful isolation of all the parts).  If so, it is a Linux stack issue that arises from mingling host and router datagrams in a way that does not fully separate them.
 
So, it could be viewed as a bug in Linux (including NetworkManager and Quagga) that that separation is not achievable(achieved).   NetworkManager spans the layers and manages Ethernet mostly, but sometimes views it through the IP lens, making some assumptions that are not always valid architectural invariants.  NetworkManager assumes that there is only one primary host IP interface, for example, as I recall, even though IPv6 especially supports many IPv6 addresses in the same box for Hosts and for routers.
 
So the interfaces and IP addresses for the router must be managed separately from those in the host, even if on the same box.
 
The easiest way to do a very clean separation in Linux when sharing the physical NICs as link-endpoints among router and host functions is perhaps using virtualization of the NICs with VLANs and/or the TAP interface providing a virtual link between the router and host on the same box.  (I haven't studied this in detail, but it seems straightforward with no gotchas).  This would separate the host interfaces from the router ones unambiguously.
 
Then let NetworkManager see only the host-controlled interface, but let the other interfaces be out of NetworkManager's control.
 
 
 
 
-----Original Message-----
From: "David Lamparter" <equinox@diac24.net>
Sent: Friday, July 6, 2012 12:59pm
To: "Dave Taht" <dave.taht@gmail.com>
Cc: "babel-users" <babel-users@lists.alioth.debian.org>, "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>
Subject: Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues



On Tue, Jul 03, 2012 at 09:18:43AM -0400, Dave Taht wrote:
> On Tue, Jul 3, 2012 at 8:35 AM, Denis Ovsienko <infrastation@yandex.ru> wrote:
> >> Does anybody know where this difference comes from?
> >
> > The difference comes from NetworkManager. Its efforts in reproducing
> > high-metric RTPROT_KERNEL routes with low-metric RTPROT_STATIC ones
> > are effectively hiding the kernel issue outside of CeroWrt runtime.
> > Would it be better to add a watchdog shell script, which does the
> > same, or patch the kernel?
> 
> I would *much rather* patch the kernel than have a watchdog. However I
> don't quite understand
> the redistribution issue vs a vs ipv6 here. If I have a "redistribute
> kernel" on for ipv4, it does propagate the default route.

I'm not sure I understood your problem here, but if it boils down to
"zebra doesn't redistribute an IPv6 RA default route", then that's by
design and shouldn't be touched.

IPv6 RA is a router to host protocol.  Routers should never accept
information from it, it is neither secure nor able to convey enough
details to prevent loops or dead-end routes.

This is also why enabling IPv6 forwarding disables reception of route
advertisements in-kernel.

If I understand correctly, your use-case is a mesh router that acts as a
host on a "parent" network.  If so, this case should be handled by a
separate daemon that receives and processes IPv6 RAs, hopefully applies
some filtering.  Also, this absolutely cannot be default behaviour.

If I misunderstood the issues, please ignore my mail.

Cheers,


-David


P.S.: Also, NetworkManager and Quagga should never run on the same host.
NetworkManager does Host processing, Quagga does Router processing, and
those two are mutually exclusive.
_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

[-- Attachment #2: Type: text/html, Size: 5319 bytes --]

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[Robert Bradley]

On 03/07/12 13:35, Denis Ovsienko wrote:
>> Does anybody know where this difference comes from?
> The difference comes from NetworkManager. Its efforts in reproducing high-metric RTPROT_KERNEL routes with low-metric RTPROT_STATIC ones are effectively hiding the kernel issue outside of CeroWrt runtime. Would it be better to add a watchdog shell script, which does the same, or patch the kernel?
>
Isn't the main issue that IPv4 cannot autoconfigure routes?  For IPv4, 
you *need* to set default routes either manually or via DHCP. In IPv6 
this all tends to be done via RA announcements and autoconfiguration.

In fact, I think the NetworkManager static route only comes from 
DHCPv6.  I have a router here (WNR2000v2) that only handles 6to4 traffic 
and dishes out addresses via DHCPv6.  To get proper conectivity, I also 
have a separate gogoc tunnel that is advertised only via RA packets.  
NetworkManager adds a static default route, but only for the 6to4 
router.  The gogc machine only gets a kernel-derived route.

I cannot see that a kernel patch that reports kernel-derived routes as 
static would be accepted upstream.  I think it is more likely that 
quagga will need patching to obtain the autoconfigured routes from the 
kernel somehow.

-- 
Robert Bradley

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[Robert Bradley]

On 03/07/12 16:28, Robert Bradley wrote:
> In fact, I think the NetworkManager static route only comes from 
> DHCPv6.  I have a router here (WNR2000v2) that only handles 6to4 
> traffic and dishes out addresses via DHCPv6.  To get proper 
> conectivity, I also have a separate gogoc tunnel that is advertised 
> only via RA packets.  NetworkManager adds a static default route, but 
> only for the 6to4 router.  The gogc machine only gets a kernel-derived 
> route.

It looks like I spoke slightly too soon.  If I disable the router's IPv6 
support, NetworkManager reassigns the static route using the advertised 
gogoc tunnel.
-- 
Robert Bradley

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[Denis Ovsienko]

>>  The project may get more options, if we drive the prototype towards a finished deliverable.
>
> I am very enthusiastic about babel's new authenticated mesh routing!
>
> It is also my opinion that without a decent drop and packet mixing
> strategy that mesh networks will perform badly under load. I'm hoping
> that fq_codel does well, although it seems very likely that an
> aggregation aware and fq_codel-like strategy needs to move into the
> mac80211 layer, which is perhaps years worth of work.
>
> What would a deliverable look like? What would interest people enough
> to get some good data, papers written, progress made, more
> users/developers and cash in the door?

Hello.

Considering CeroWrt as a free software project, it may be the right time to measure each of the following:

1. Amount of manpower/cash required to keep the project afloat and developing.
2. Added value, which exists due to its unique properties.
3. Population of developers willing to invest their manpower/cash.
4. Population of users willing to use the outcomes as long as it helps them stay focused on their own needs.

Accounting and distributing CeroWrt daily duties will make some space for a day job, at least part-time, which is very important. Focusing the 2nd item to help users understand the point of switching a Netgear box to CeroWrt will help the community grow naturally. The deliverable could come like this:

"Here is a small automatic test, which will measure the jitter, delay and RTT of your connection through a Netgear box to a server far on the Internet. Record these numbers and repeat the test after flashing the 3800 with this stable CeroWrt release. Let us know the difference and consider these unique features, which are available in CeroWrt only:
* something good
* something else
...
Have fun!
"

>
> ...
>
> I presently have 4 Nanostation M5, 3 picostation HP, and 5 wndr3700s.
> I've found a good site to work with 2.4ghz and 5.x ghz radios (a 110
> acre campground that has given me permission to play here) and test
> fq_codel in various forms of cerowrt in, for as long as I like.
>
> One of the things we've really struggled with was in today's saturated
> 2.4ghz environment there is no way to benchmark both radios. We've
> been reduced to suggesting we 'flee to the mountains' in order to get
> good results. OK, I just did that.

Picking some antenna types other than omnidirectional may be another solution, but it depends.

-- 
    Denis Ovsienko

Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 4034 bytes --]


Regarding radio benchmarking, note that the effect of the surface of the dirt/vegetation below line of sight seriously affects 2.4 and 5 GHz.  It's been measured and reported (forget which journal) by folks from Berkeley who just did some empirical studies.  E.g. 6 inch vs 11 inch grass 5 feet below the antenna makes a big difference, etc.
 
On the other hand, you know that in the real world, lab benchmarks of radios mean nothing at all, especially in a protocol based on contention where the energy in the beginning of the packet is crucial, independent of the decodability of the bits, but the decodability of the bits affects the backoff, etc.
 
I would suggest that tests that matter will be carried out in the densely populated worlds of cities, towns, ...  If a mesh cannot survive in that environment, it's going to be of very, very limited usefulness, other than to provide Ph.D. dissertations in "optimal" routing in *imaginary* conditions.
 
Forget "optimal".   Stable, scalable, resilient, simple, and good enough is far more important, practically.
 
-----Original Message-----
From: "Denis Ovsienko" <infrastation@yandex.ru>
Sent: Wednesday, July 4, 2012 7:34am
To: "cerowrt-devel" <cerowrt-devel@lists.bufferbloat.net>
Subject: Re: [Cerowrt-devel] [Babel-users] switching cerowrt to quagga-babeld issues



>>  The project may get more options, if we drive the prototype towards a finished deliverable.
>
> I am very enthusiastic about babel's new authenticated mesh routing!
>
> It is also my opinion that without a decent drop and packet mixing
> strategy that mesh networks will perform badly under load. I'm hoping
> that fq_codel does well, although it seems very likely that an
> aggregation aware and fq_codel-like strategy needs to move into the
> mac80211 layer, which is perhaps years worth of work.
>
> What would a deliverable look like? What would interest people enough
> to get some good data, papers written, progress made, more
> users/developers and cash in the door?

Hello.

Considering CeroWrt as a free software project, it may be the right time to measure each of the following:

1. Amount of manpower/cash required to keep the project afloat and developing.
2. Added value, which exists due to its unique properties.
3. Population of developers willing to invest their manpower/cash.
4. Population of users willing to use the outcomes as long as it helps them stay focused on their own needs.

Accounting and distributing CeroWrt daily duties will make some space for a day job, at least part-time, which is very important. Focusing the 2nd item to help users understand the point of switching a Netgear box to CeroWrt will help the community grow naturally. The deliverable could come like this:

"Here is a small automatic test, which will measure the jitter, delay and RTT of your connection through a Netgear box to a server far on the Internet. Record these numbers and repeat the test after flashing the 3800 with this stable CeroWrt release. Let us know the difference and consider these unique features, which are available in CeroWrt only:
* something good
* something else
...
Have fun!
"

>
> ...
>
> I presently have 4 Nanostation M5, 3 picostation HP, and 5 wndr3700s.
> I've found a good site to work with 2.4ghz and 5.x ghz radios (a 110
> acre campground that has given me permission to play here) and test
> fq_codel in various forms of cerowrt in, for as long as I like.
>
> One of the things we've really struggled with was in today's saturated
> 2.4ghz environment there is no way to benchmark both radios. We've
> been reduced to suggesting we 'flee to the mountains' in order to get
> good results. OK, I just did that.

Picking some antenna types other than omnidirectional may be another solution, but it depends.

-- 
 Denis Ovsienko
_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

[-- Attachment #2: Type: text/html, Size: 4828 bytes --]