[Cerowrt-devel] Fwd: Dave's wishlist [was: Source-specific routing merged]

[Cerowrt-devel] Fwd: Dave's wishlist [was: Source-specific routing merged]

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 5908 bytes --]

---------- Forwarded message ----------
From: Dave Taht <dave.taht@gmail.com>
Date: Tue, Mar 17, 2015 at 7:41 AM
Subject: Re: Dave's wishlist [was: Source-specific routing merged]
To: Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr>
Cc: "babel-users@lists.alioth.debian.org" <
babel-users@lists.alioth.debian.org>, Gabriel Kerneis <gabriel@kerneis.info>,
Steven Barth <cyrus@openwrt.org>, Henning Rogge <hrogge@gmail.com>, Paul
McKenney <paulmck@linux.vnet.ibm.com>, Felix Fietkau <nbd@nbd.name>


My quest is always for an extra "9" of reliability. Anyplace where you can
make something more robust (even if it is out at the .9999999999) level, I
tend to like to do in order to have the highest MTBF possible in
combination with all the other moving parts on the spacecraft (spaceship
earth).

One of the reasons why I love paul mckenney so much is that he deeply cares
about stuff that happens only one in a billion times.

From this blog post of his: http://paulmck.livejournal.com/37782.html

"I quickly learned that the bug is difficult to reproduce, requiring
something like 100 hours of focused rcutorture testing. Bisection based on
100-hour tests would have consumed the remainder of 2014 and a significant
fraction of 2015, so something better was required. In fact, something *way*
better was required because there was only a very small number of failures,
which meant that the expected test time to reproduce the bug might well
have been 200 hours or even 300 hours instead of my best guess of 100
hours."

so, thus, I get picky on system daemons.

On Tue, Mar 17, 2015 at 4:45 AM, Juliusz Chroboczek <
jch@pps.univ-paris-diderot.fr> wrote:

> > 1) Did the issue with procd ever get resolved? (sighup I think it was)
>
> Gabriel, Steven?  Can procd be configured not to send SIGHUP, or shall
> I add an option to babeld to ignore it?  (Currently babeld terminates on
> SIGHUP, and I like it that way, since it prevents a babeld from sticking
> around after you log off.)
>

this basically logjammed on this issue. Either procd needed to be modified
to be able to
send an arbitrary signal, or babel changed to take sighup as a reload.

I'd done the simple patch to babel.

But I understand your use case also (stop routing on hup via remote
access), and have never poked into procd. Perhaps it could be changed to
take a var for the actual signal number it uses per daemon, but I will
argue that has overhead the openwrt devs would be loathe to take. But: I
will look. Babel can't be the only thing that needs a different signal to
reload...

a third way out is just to patch babel for openwrt...


> > 2) got the new vars into openwrt, or shall I do?
>

rtt branch had 2 new vars as best as I recall, and I envisioned the babels
package being retired, which as best as I recall has extra vars like src-eq.

Secondly the command lines would get complex on me, and I figured just
re-writing the conf file was saner than command line args. Get a
sigWHATEVER reload (or mmap) the conf file, checksum it against the
previous version, do nothing if it didn't change.

Thirdly, having an openwrt specific uci and/or ubus parser that could be
compiled in would be more reliable than a script, simpler and faster. I can
try to find funding for doing that... (in like 1.7's timeframe!!) I looked
over the libubus and libuci interfaces and staggered away confused.

Gabriel?
>
> > ecn
>
> No, since I don't understand why you think that setting ECN on Babel
> packets makes sense.  (It might make sense to set ECN on some Babel
> packets -- the ones that are marked as "urgent" -- but I'm interested in
> hearing your reasoning.)
>

fq_codel is the default on openwrt. ECN is enabled on that. Basic ECN
marking is 2 characters of new code. (tracking it harder, but that's
boilerplate code now). hnetd is presently very dumb about coalescing /60s
out of /64s...  I'd like to be trying much faster update schedules on
ethernet as per some of the discussion on homenet.

But, let me take this subject to another thread than this.


>
> > atomic route updates
>
> Ausgeschlossen.


Nothing is impossible.


> Last time I tried, I got a number of complaints that it
> broke operational networks.
>

As the new FIB table patches have landed in linux 4.0 and later, it has
done some odd things with RCU that I am not sure would be a good thing with
the present delete+add routes system everything like quagga+ babel seems to
use.

I'd written about it here while discussing the amazing new FIB patches (7x
reductions in lookup time or more), but was not aware that henning had
actually got atomic route updates that worked.

http://lists.openwall.net/netdev/2015/03/11/136

So, perhaps autodetection of some sort here, also, would be of help.

And figuring out why it used to break.

And ooh! atomic route changes! no packet loss at all! Look at that extra 9!


> It's also less important than it used to be, since the hysteresis
> algorithm in 1.5.0 dramatically reduced the number of route switches --
> current versions of Babel should not be loosing a measurable number of
> packets due to non-atomic switches.
>

How many 9s do you want?


>
> > IPV6_SUBTREES autodetection
>
> That should definitely be a runtime option.  Matthieu?
>
> Not so sure about autodetection, Steven, Henning, do you have any ideas?
>

My thought here was to try to insert and then retrieve a route that would
fail if IPV6_SUBTREES was not defined in the kernel.


> -- Juliusz
>



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

[-- Attachment #2: Type: text/html, Size: 9184 bytes --]

Re: [Cerowrt-devel] Fwd: Dave's wishlist [was: Source-specific routing merged]

[David Lang]

On Tue, 17 Mar 2015, Dave Taht wrote:

> My quest is always for an extra "9" of reliability. Anyplace where you can
> make something more robust (even if it is out at the .9999999999) level, I
> tend to like to do in order to have the highest MTBF possible in
> combination with all the other moving parts on the spacecraft (spaceship
> earth).

There are different ways to add reliability

one is to try and make sure nothing ever fails

the second is to have a way of recovering when things go wrong.


Bufferbloat came about because people got trapped into the first mode of 
thinking (packets should never get lost), when the right answer ended up being 
to realize that we have a recovery method and use it.

Sometimes trying to make sure nothing ever fails adds a lot of complexity to the 
code to handle all the corner cases, and the overall reliability will improve by 
instead simplify normal flow, even if it add a small number of failures, if that 
means that you can have a common set of recovery code that's well excercised and 
tested.

As you are talking about loosing packets with route changes, watch out that you 
don't fall into this trap.

David Lang

Re: [Cerowrt-devel] Fwd: Dave's wishlist [was: Source-specific routing merged]

[dpreed]

I agree wholeheartedly with your point, David.

One other clarifying point (I'm not trying to be pedantic, here, but it may sound that way):

Reliability is not the same as Availability.  The two are quite different.

 Bufferbloat is pretty much an "availability" issue, not a reliability issue.  In other words, packets are not getting lost.  The system is just preventing desired use.

Availability issues can be due to actual failures of components, but there are lots of availability issues that are caused (as you suggest) by attempts to focus narrowly on "loss of data" or "component failures".

When you build a system, there is a temptation to apply what is called the Fallacy of Composition (look it up on Wikipedia for precise definition).  The key thing in the Fallacy of Composition is that when a system of components has a property as a whole, then every component of the system must by definition have that property.

(The end-to-end argument is a specific rule that is based on a recognition of the Fallacy of Composition in one case.)

We all know that there is never a single moment when any moderately large part of the Internet does not contain failed components.  Yet the Internet has *very* high availability - 24x7x365, and we don't need to know very much about what parts are failing.  That's by design, of course. And it is a design that does not derive its properties from a trivial notion of "proof of correctness", or even "bug freeness"

The relevance of a "failure" or even a "design flaw" to system availability is a matter of a much bigger perspective of what the system does, and what its users perceive as to whether they can get work done.




On Tuesday, March 17, 2015 3:30pm, "David Lang" <david@lang.hm> said:

> On Tue, 17 Mar 2015, Dave Taht wrote:
> 
>> My quest is always for an extra "9" of reliability. Anyplace where you can
>> make something more robust (even if it is out at the .9999999999) level, I
>> tend to like to do in order to have the highest MTBF possible in
>> combination with all the other moving parts on the spacecraft (spaceship
>> earth).
> 
> There are different ways to add reliability
> 
> one is to try and make sure nothing ever fails
> 
> the second is to have a way of recovering when things go wrong.
> 
> 
> Bufferbloat came about because people got trapped into the first mode of
> thinking (packets should never get lost), when the right answer ended up being
> to realize that we have a recovery method and use it.
> 
> Sometimes trying to make sure nothing ever fails adds a lot of complexity to the
> code to handle all the corner cases, and the overall reliability will improve by
> instead simplify normal flow, even if it add a small number of failures, if that
> means that you can have a common set of recovery code that's well excercised and
> tested.
> 
> As you are talking about loosing packets with route changes, watch out that you
> don't fall into this trap.
> 
> David Lang
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>