[Cerowrt-devel] An interesting application of tunneling and ipv6 mesh networking

[Dave Taht]

---------- Forwarded message ----------
From: <jp at nexedi.com>
Date: Wed, Jan 23, 2013 at 6:30 AM
Subject: [Babel-users] A happy babel user: re6st
To: babel-users at lists.alioth.debian.org


Hi,

Very often, people complain on mailing lists. Today, I would like to say
thank you.

Last summer, we have implemented a wired mesh network system based on babel
which can provide stable IPv6 to all nodes of a decentralized cloud
operation system. It works great.

Thank you babel.

If you are in a hurry, here is the code:
http://git.erp5.org/gitweb/re6stnet.git
What you can do with that code: provide reliable IPv6 to the world

If you think re6st is useful, please feel free to add it to the list of
babel links.

Details bellow.

Regards,

JP Smets.
Nexedi CEO
+33 629 02 44 25
---

1- The problem to solve

We implemented a couple of years ago (http://bit.ly/SWVQlx) a Cloud system
called SlapOS (http://www.slapos.org) which relies on servers located in
people's home and now also in offices, data centers or even your
smartphone, tablet or TV. SlapOS is now used by some large corporations.
One of its main applications is to create a disaster recovery cloud which
can resist any force majeur event (ex. war, terrorism, political
instability, software bug) which does affect traditional clouds from time
to time (http://iwgcr.org). It is also much cheaper and environmental
friendly.

SlapOS relies on IPv6 in order to interconnect all nodes. Each node is
allocated usually 100 global IPv6 addresses or more.

This is where our problem started: all IPv6 providers we tried were unable
to provide reliable connectivity. We tried providers in France, Germany,
Japan, Norway. For example, in France among 200 IPv6 adresses provided by a
Freebox (Free), 3 becomes unreachable from time to time, during a couple of
minutes or hours. OVH routers sometimes no longer route packers to Free,
but only for IPv6, during a couple of hours. Telia routers somtimes "eat" a
few bytes during the initialization of a session.

Overall, the use of native IPv6 of ISPs lead to a service availability of
99% or worse. We we searching for a solution.

We also had had the experience that from time to time, IPv4 transit between
ISPs can be cut for a while - a couple of hours -although less often. Our
ideal solution should also solve that.

2- The solution: re6st + babel

Step 1: create a wired mesh

We coded a litlle deaemon called re6st which is able to find 10 IPv4
neighbours randomly and create a tunnel to each neighbour. re6st can be
placed behind a NAT. It is able to capture public IPv4 address of your
router through UPnP. After some time, all nodes which run re6st form a
global mesh.

Step 2: start babel

Once tunnels are created, babel is used for routing. Babel then finds the
best route to interconnect all re6st nodes.

3- Results

After a couple of month of using re6st + babel we can say that it works
quite well. SlapOS no longer experiences the connectivity problems of
native IPv6. We can safely host websites with SlapOS over re6st+babel.

4- Next steps

A report will be published.

5- Remaining problems to solve

The problems which remain to be solved are the following:

a- How can we prevent one babel participant to act against other
participants by providing wrong information to other participants ? Imagine
for example that a bad organization joins re6st + babel network and starts
capturing all routes in order to analyze traffic or even block it.

b- How can we create a hierarchical addressing system ? The idea here is to
group participants dynamically and assign them a "big" IPv6 address range.
Each participant connects to another participant through another
participant by first connecting randomly to one participant in a dynamic
group and next connect to other participant in the same group. With this
grouping approach, there is no need to create a hierachical network with a
bakbone. It also solves the problem of scalability.

c- How can we implement more policies (ex. latency) ?

d- How could we implement accounting and billing in a way or another ?
(open question, but quite important for example to solve the problem of
FTTH participants with upload limited to 3GB / day as in Japan)

6- Credits

Most of the coding of re6st was done by Julien Muchembled (Nexedi),  Ulysse
Beaugnon (ENS) and Guillaume Bury (ENS).

7- Alternatives

We could have used other routing protocols (ex. OLSR). But we felt that
Babel pluggable policy system was a key design difference which could be
used to later customize it to different needs of Cloud applications (ex.
low latency). We would also feel ashamed to use a protocol which babel's
creator proved that it was flawed.

We could have used tinc. But tinc creates a fully connected mesh. There is
also a difference between what it claims to do and what it actually does.
Last, mixing tunneling and routing is a bad idea as we were suggested by
Juliusz C.

We could have used gre instead of OpenVPN for tunnels. But that does work
behind an IPv4 NAT. Yet, nothing prevents use from later using gre.
_______________________________________________
Babel-users mailing list
Babel-users at lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users



-- 
Dave Täht

Fixing bufferbloat with cerowrt:
http://www.teklibre.com/cerowrt/subscribe.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/cerowrt-devel/attachments/20130123/cf4b7cbd/attachment-0002.html>