[Cerowrt-devel] atomic route updates?

[Cerowrt-devel] atomic route updates?

[Dave Taht]

iproute2's ip/iproute2.c code has a function called iproute_modify,
which to a blurring eye appears to be capable of doing an atomic route
update via it's netlink interface.

"        if (matches(*argv, "change") == 0 || strcmp(*argv, "chg") == 0)
                return iproute_modify(RTM_NEWROUTE, NLM_F_REPLACE,
                                      argc-1, argv+1);
        if (matches(*argv, "replace") == 0)
                return iproute_modify(RTM_NEWROUTE, NLM_F_CREATE|NLM_F_REPLACE,
                                      argc-1, argv+1);"

The babel  native daemon and the quagga-re code, however, does a
delete/add, which results in packets dropping on the floor when a
route changes.

Everyone that's tried to make this code do atomic updates has failed.

Juliusz thinks this section of the codebase is cursed, and I
personally, gave up, because dave miller had spent most of the last 3+
years eliminating the linux kernel route cache, (which was finally
eliminated a few kernel versions back (3.6?)) and I figured all
attempts at doing anything fancy with routing during that phase was
going to break in odd ways until the new cache-free-linux-kernel
routing code stabilized.

OK, so, like, it's kernel 3.8 time now, and ...  I find myself too
scarred by previous attempts to give it a go myself, but I know that
out there are intrepid explorers out there, just dying to delve into
the gnarly details of netlink programming to keep a few more streams
going full throttle in the face of a routing change! Yes? Anyone? So
take a look at iproute2 and the relevant netlink code here:

git://github.com/Quagga-RE/quagga-RE.git
git://github.com/jech/babeld.git

I note that I have a few other ideas for netlink-related changes to
cero. Two of the big ones is that I'd like to be able to have a
userspace daemon get back more details as to when *fq_codel drops a
packet (e.g, send a multicast to a listener of the dropped the packet,
and why), and get some sort of ongoing bandwidth estimate when it
starts dropping, etc.

I also have some hope for multi-prefix multi-homed routing too...

but I find netlink really intimidating. It has an interface that only
a bit-banger would love.

-- 
Dave Täht

PS: I have a build of 3.8.8 at the moment that uses the new PROCD
replacement for init that I'm losing a battle on, too. Also toke has
got cero mostly independently buildable now.

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] atomic route updates?

[Maciej Soltysiak]

[-- Attachment #1: Type: text/plain, Size: 3135 bytes --]

I think I remember to fix ssdp you brought some of route cache and igmp
back. Do we know if cero routed setup without route cache can work with
ssdp? I wouldn't like to force bringing back route cache just for ssdp.

I'm OK to test this scenario.

Maciej
On 20 Apr 2013 18:01, "Dave Taht" <dave.taht@gmail.com> wrote:

> iproute2's ip/iproute2.c code has a function called iproute_modify,
> which to a blurring eye appears to be capable of doing an atomic route
> update via it's netlink interface.
>
> "        if (matches(*argv, "change") == 0 || strcmp(*argv, "chg") == 0)
>                 return iproute_modify(RTM_NEWROUTE, NLM_F_REPLACE,
>                                       argc-1, argv+1);
>         if (matches(*argv, "replace") == 0)
>                 return iproute_modify(RTM_NEWROUTE,
> NLM_F_CREATE|NLM_F_REPLACE,
>                                       argc-1, argv+1);"
>
> The babel  native daemon and the quagga-re code, however, does a
> delete/add, which results in packets dropping on the floor when a
> route changes.
>
> Everyone that's tried to make this code do atomic updates has failed.
>
> Juliusz thinks this section of the codebase is cursed, and I
> personally, gave up, because dave miller had spent most of the last 3+
> years eliminating the linux kernel route cache, (which was finally
> eliminated a few kernel versions back (3.6?)) and I figured all
> attempts at doing anything fancy with routing during that phase was
> going to break in odd ways until the new cache-free-linux-kernel
> routing code stabilized.
>
> OK, so, like, it's kernel 3.8 time now, and ...  I find myself too
> scarred by previous attempts to give it a go myself, but I know that
> out there are intrepid explorers out there, just dying to delve into
> the gnarly details of netlink programming to keep a few more streams
> going full throttle in the face of a routing change! Yes? Anyone? So
> take a look at iproute2 and the relevant netlink code here:
>
> git://github.com/Quagga-RE/quagga-RE.git
> git://github.com/jech/babeld.git
>
> I note that I have a few other ideas for netlink-related changes to
> cero. Two of the big ones is that I'd like to be able to have a
> userspace daemon get back more details as to when *fq_codel drops a
> packet (e.g, send a multicast to a listener of the dropped the packet,
> and why), and get some sort of ongoing bandwidth estimate when it
> starts dropping, etc.
>
> I also have some hope for multi-prefix multi-homed routing too...
>
> but I find netlink really intimidating. It has an interface that only
> a bit-banger would love.
>
> --
> Dave Täht
>
> PS: I have a build of 3.8.8 at the moment that uses the new PROCD
> replacement for init that I'm losing a battle on, too. Also toke has
> got cero mostly independently buildable now.
>
> Fixing bufferbloat with cerowrt:
> http://www.teklibre.com/cerowrt/subscribe.html
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 4018 bytes --]

Re: [Cerowrt-devel] atomic route updates?

[Dave Taht]

On Tue, Apr 23, 2013 at 11:14 PM, Maciej Soltysiak <maciej@soltysiak.com> wrote:
> I think I remember to fix ssdp you brought some of route cache and igmp
> back.

No. What I did observe was that for the past 2+ years, routing
behavior kept changing in subtle ways, and I am hoping it's all
perfect now.

the igmp bug was due to openwrt being too agressive on removing
useless cruft from /proc, thus breaking a few userspace tools. Been
fixed a while now.

> Do we know if cero routed setup without route cache can work with
> ssdp? I wouldn't like to force bringing back route cache just for ssdp.

Routes are routes. The route cache was an inefficient means of
theoretically speeding up route lookup and has been eliminated, that's
all. Route lookups still happen, just not through an over-abstracted
cache mechanism.

> I'm OK to test this scenario.

goferit.

>
> Maciej
>
> On 20 Apr 2013 18:01, "Dave Taht" <dave.taht@gmail.com> wrote:
>>
>> iproute2's ip/iproute2.c code has a function called iproute_modify,
>> which to a blurring eye appears to be capable of doing an atomic route
>> update via it's netlink interface.
>>
>> "        if (matches(*argv, "change") == 0 || strcmp(*argv, "chg") == 0)
>>                 return iproute_modify(RTM_NEWROUTE, NLM_F_REPLACE,
>>                                       argc-1, argv+1);
>>         if (matches(*argv, "replace") == 0)
>>                 return iproute_modify(RTM_NEWROUTE,
>> NLM_F_CREATE|NLM_F_REPLACE,
>>                                       argc-1, argv+1);"
>>
>> The babel  native daemon and the quagga-re code, however, does a
>> delete/add, which results in packets dropping on the floor when a
>> route changes.
>>
>> Everyone that's tried to make this code do atomic updates has failed.
>>
>> Juliusz thinks this section of the codebase is cursed, and I
>> personally, gave up, because dave miller had spent most of the last 3+
>> years eliminating the linux kernel route cache, (which was finally
>> eliminated a few kernel versions back (3.6?)) and I figured all
>> attempts at doing anything fancy with routing during that phase was
>> going to break in odd ways until the new cache-free-linux-kernel
>> routing code stabilized.
>>
>> OK, so, like, it's kernel 3.8 time now, and ...  I find myself too
>> scarred by previous attempts to give it a go myself, but I know that
>> out there are intrepid explorers out there, just dying to delve into
>> the gnarly details of netlink programming to keep a few more streams
>> going full throttle in the face of a routing change! Yes? Anyone? So
>> take a look at iproute2 and the relevant netlink code here:
>>
>> git://github.com/Quagga-RE/quagga-RE.git
>> git://github.com/jech/babeld.git
>>
>> I note that I have a few other ideas for netlink-related changes to
>> cero. Two of the big ones is that I'd like to be able to have a
>> userspace daemon get back more details as to when *fq_codel drops a
>> packet (e.g, send a multicast to a listener of the dropped the packet,
>> and why), and get some sort of ongoing bandwidth estimate when it
>> starts dropping, etc.
>>
>> I also have some hope for multi-prefix multi-homed routing too...
>>
>> but I find netlink really intimidating. It has an interface that only
>> a bit-banger would love.
>>
>> --
>> Dave Täht
>>
>> PS: I have a build of 3.8.8 at the moment that uses the new PROCD
>> replacement for init that I'm losing a battle on, too. Also toke has
>> got cero mostly independently buildable now.
>>
>> Fixing bufferbloat with cerowrt:
>> http://www.teklibre.com/cerowrt/subscribe.html
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] atomic route updates?

[Robert Bradley]

On 20/04/13 10:56, Dave Taht wrote:
> iproute2's ip/iproute2.c code has a function called iproute_modify,
> which to a blurring eye appears to be capable of doing an atomic route
> update via it's netlink interface.
>
> "        if (matches(*argv, "change") == 0 || strcmp(*argv, "chg") == 0)
>                  return iproute_modify(RTM_NEWROUTE, NLM_F_REPLACE,
>                                        argc-1, argv+1);
>          if (matches(*argv, "replace") == 0)
>                  return iproute_modify(RTM_NEWROUTE, NLM_F_CREATE|NLM_F_REPLACE,
>                                        argc-1, argv+1);"
>
> The babel  native daemon and the quagga-re code, however, does a
> delete/add, which results in packets dropping on the floor when a
> route changes.
>

I think quagga-RE would take a lot of work, as you'd need to teach it 
about modifying routes.  The standalone babeld looks promising though, 
and I have a basic patch up on GitHub 
(git://github.com/rb12345/babeld.git).  One issue is that it seems 
difficult to actually flush the IPv6 route cache, at least on Ubuntu, as 
writes to /proc/sys/net/ipv6/route/flush appear to be ignored.  The 
iproute2 approach is to query all cached routes and misuse its internal 
print_route function to remove each route when flushing the IPv6 cache.

At the moment, the code compiles fine, and I am in the process of 
testing it here.

-- 
Robert Bradley

Re: [Cerowrt-devel] atomic route updates?

[Robert Bradley]

On 24/04/13 14:27, Robert Bradley wrote:
> At the moment, the code compiles fine, and I am in the process of 
> testing it here.
>

For the moment, it seems to work on the Ubuntu 3.5 kernel fine, but 
still generates stuck unreachable routes as mentioned in the babeld 
1.3.4 changelog.  It may still be worth trying this fork on 3.8+ kernels 
though to see if anything changes.

-- 
Robert Bradley

Re: [Cerowrt-devel] atomic route updates?

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

There is a possibly related bug being discussed on the babel-users list
with an easy fix.

Not sure what kernel versions it happens on.
On Apr 24, 2013 11:39 AM, "Robert Bradley" <robert.bradley1@gmail.com>
wrote:

> On 20/04/13 10:56, Dave Taht wrote:
>
>> iproute2's ip/iproute2.c code has a function called iproute_modify,
>> which to a blurring eye appears to be capable of doing an atomic route
>> update via it's netlink interface.
>>
>> "        if (matches(*argv, "change") == 0 || strcmp(*argv, "chg") == 0)
>>                  return iproute_modify(RTM_NEWROUTE, NLM_F_REPLACE,
>>                                        argc-1, argv+1);
>>          if (matches(*argv, "replace") == 0)
>>                  return iproute_modify(RTM_NEWROUTE,
>> NLM_F_CREATE|NLM_F_REPLACE,
>>                                        argc-1, argv+1);"
>>
>> The babel  native daemon and the quagga-re code, however, does a
>> delete/add, which results in packets dropping on the floor when a
>> route changes.
>>
>>
> I think quagga-RE would take a lot of work, as you'd need to teach it
> about modifying routes.  The standalone babeld looks promising though, and
> I have a basic patch up on GitHub (git://github.com/rb12345/**babeld.git<http://github.com/rb12345/babeld.git>).
>  One issue is that it seems difficult to actually flush the IPv6 route
> cache, at least on Ubuntu, as writes to /proc/sys/net/ipv6/route/flush
> appear to be ignored.  The iproute2 approach is to query all cached routes
> and misuse its internal print_route function to remove each route when
> flushing the IPv6 cache.
>
> At the moment, the code compiles fine, and I am in the process of testing
> it here.
>
> --
> Robert Bradley
>
> ______________________________**_________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.**bufferbloat.net<Cerowrt-devel@lists.bufferbloat.net>
> https://lists.bufferbloat.net/**listinfo/cerowrt-devel<https://lists.bufferbloat.net/listinfo/cerowrt-devel>
>

[-- Attachment #2: Type: text/html, Size: 2589 bytes --]

Re: [Cerowrt-devel] atomic route updates?

[Robert Bradley]

On 24/04/13 16:46, Dave Taht wrote:
> There is a possibly related bug being discussed on the babel-users list  with an
> easy fix.
>
> Not sure what kernel versions it happens on.
>

I was seeing that bug on the quantal 3.5 kernel as well.  So far, 
testing with the raring 3.8 kernel seems to fix both the unreachable 
route bug and the random /128 route bugs.  My patched babeld also 
appears to work no worse than the distro-provided 1.3.1 build.  That 
said, with both versions I see that babeld switches to wireless routes 
and stays there if eth0 is disconnected and later restored.

It may be time for me to try running the babeld HEAD code without the 
patches and compare the behaviour of that to the patched version next...

-- 
Robert Bradley

Re: [Cerowrt-devel] atomic route updates?

[Dave Taht]

On Thu, Apr 25, 2013 at 8:05 AM, Robert Bradley
<robert.bradley1@gmail.com> wrote:
> On 24/04/13 16:46, Dave Taht wrote:
>>
>> There is a possibly related bug being discussed on the babel-users list
>> with an
>> easy fix.
>>
>> Not sure what kernel versions it happens on.
>>
>
> I was seeing that bug on the quantal 3.5 kernel as well.  So far, testing
> with the raring 3.8 kernel seems to fix both the unreachable route bug and
> the random /128 route bugs.  My patched babeld also appears to work no worse
> than the distro-provided 1.3.1 build.  That said, with both versions I see
> that babeld switches to wireless routes and stays there if eth0 is
> disconnected and later restored.

Still stuck on this? I'm planning a new build this week with your
atomic route stuff in it. babel 1.3.6 and 1.40 have been released.

>
> It may be time for me to try running the babeld HEAD code without the
> patches and compare the behaviour of that to the patched version next...
>
>
> --
> Robert Bradley
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] atomic route updates?

[Daniel Ezell]

[-- Attachment #1: Type: text/plain, Size: 1739 bytes --]

If minidlna will compile, and you have the time, could you include it in
the next release?
On May 5, 2013 6:28 AM, "Dave Taht" <dave.taht@gmail.com> wrote:

> On Thu, Apr 25, 2013 at 8:05 AM, Robert Bradley
> <robert.bradley1@gmail.com> wrote:
> > On 24/04/13 16:46, Dave Taht wrote:
> >>
> >> There is a possibly related bug being discussed on the babel-users list
> >> with an
> >> easy fix.
> >>
> >> Not sure what kernel versions it happens on.
> >>
> >
> > I was seeing that bug on the quantal 3.5 kernel as well.  So far, testing
> > with the raring 3.8 kernel seems to fix both the unreachable route bug
> and
> > the random /128 route bugs.  My patched babeld also appears to work no
> worse
> > than the distro-provided 1.3.1 build.  That said, with both versions I
> see
> > that babeld switches to wireless routes and stays there if eth0 is
> > disconnected and later restored.
>
> Still stuck on this? I'm planning a new build this week with your
> atomic route stuff in it. babel 1.3.6 and 1.40 have been released.
>
> >
> > It may be time for me to try running the babeld HEAD code without the
> > patches and compare the behaviour of that to the patched version next...
> >
> >
> > --
> > Robert Bradley
> >
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-devel@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
>
> --
> Dave Täht
>
> Fixing bufferbloat with cerowrt:
> http://www.teklibre.com/cerowrt/subscribe.html
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 2576 bytes --]

Re: [Cerowrt-devel] atomic route updates?

[Dave Taht]

When I try to add minidlna to the build package database (e.g.
./scripts/feeds add minidlna ), nothing happens. So something's either
broke in my build or the minidlna package.


On Sun, May 5, 2013 at 6:58 AM, Daniel Ezell <dezell@stonescry.com> wrote:
> If minidlna will compile, and you have the time, could you include it in the
> next release?
>
> On May 5, 2013 6:28 AM, "Dave Taht" <dave.taht@gmail.com> wrote:
>>
>> On Thu, Apr 25, 2013 at 8:05 AM, Robert Bradley
>> <robert.bradley1@gmail.com> wrote:
>> > On 24/04/13 16:46, Dave Taht wrote:
>> >>
>> >> There is a possibly related bug being discussed on the babel-users list
>> >> with an
>> >> easy fix.
>> >>
>> >> Not sure what kernel versions it happens on.
>> >>
>> >
>> > I was seeing that bug on the quantal 3.5 kernel as well.  So far,
>> > testing
>> > with the raring 3.8 kernel seems to fix both the unreachable route bug
>> > and
>> > the random /128 route bugs.  My patched babeld also appears to work no
>> > worse
>> > than the distro-provided 1.3.1 build.  That said, with both versions I
>> > see
>> > that babeld switches to wireless routes and stays there if eth0 is
>> > disconnected and later restored.
>>
>> Still stuck on this? I'm planning a new build this week with your
>> atomic route stuff in it. babel 1.3.6 and 1.40 have been released.
>>
>> >
>> > It may be time for me to try running the babeld HEAD code without the
>> > patches and compare the behaviour of that to the patched version next...
>> >
>> >
>> > --
>> > Robert Bradley
>> >
>> > _______________________________________________
>> > Cerowrt-devel mailing list
>> > Cerowrt-devel@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>>
>>
>> --
>> Dave Täht
>>
>> Fixing bufferbloat with cerowrt:
>> http://www.teklibre.com/cerowrt/subscribe.html
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] atomic route updates?

[Robert Bradley]

On 05/05/13 13:02, Dave Taht wrote:
> On Thu, Apr 25, 2013 at 8:05 AM, Robert Bradley
> <robert.bradley1@gmail.com> wrote:
>> I was seeing that bug on the quantal 3.5 kernel as well.  So far, testing
>> with the raring 3.8 kernel seems to fix both the unreachable route bug and
>> the random /128 route bugs.  My patched babeld also appears to work no worse
>> than the distro-provided 1.3.1 build.  That said, with both versions I see
>> that babeld switches to wireless routes and stays there if eth0 is
>> disconnected and later restored.
> Still stuck on this? I'm planning a new build this week with your
> atomic route stuff in it. babel 1.3.6 and 1.40 have been released.
>

The current code is behaving identically to the upstream master branch 
to me, so I think this is fine for the new build.

-- 
Robert Bradley