From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp191.iad.emailsrvr.com (smtp191.iad.emailsrvr.com [207.97.245.191]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 8A80421F0A2 for ; Wed, 23 Jan 2013 10:34:13 -0800 (PST) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp49.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 49210190549; Wed, 23 Jan 2013 13:34:12 -0500 (EST) X-Virus-Scanned: OK Received: from legacy19.wa-web.iad1a (legacy19.wa-web.iad1a.rsapps.net [192.168.2.205]) by smtp49.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 2FA66190357; Wed, 23 Jan 2013 13:34:12 -0500 (EST) Received: from reed.com (localhost.localdomain [127.0.0.1]) by legacy19.wa-web.iad1a (Postfix) with ESMTP id 1DB852D5806B; Wed, 23 Jan 2013 13:34:12 -0500 (EST) Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com) with HTTP; Wed, 23 Jan 2013 13:34:12 -0500 (EST) Date: Wed, 23 Jan 2013 13:34:12 -0500 (EST) From: dpreed@reed.com To: "Dave Taht" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20130123133412000000_72016" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: Message-ID: <1358966052.119718192@apps.rackspace.com> X-Mailer: webmail7.0 Cc: cerowrt-devel@lists.bufferbloat.net Subject: Re: [Cerowrt-devel] An interesting application of tunneling and ipv6 mesh networking X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jan 2013 18:34:13 -0000 ------=_20130123133412000000_72016 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0ACool! Let's make sure the OpenVPN links on it don't have bufferbloat i= n them - VPNs often do, and perhaps need a layer of fq_codel on top of each= link (also underneath, of course, where the routers can be fixed).=0A =0AM= ore interestingly - is this an openly growing network? Or is it intended t= o be an island?=0A =0AFor those stuck in IPv4 land (which includes folks wh= o live inside corporate Intranets, sadly, maybe this is a good step.=0A =0A= -----Original Message-----=0AFrom: "Dave Taht" =0ASent= : Wednesday, January 23, 2013 12:11pm=0ATo: cerowrt-devel@lists.bufferbloat= .net=0ASubject: [Cerowrt-devel] An interesting application of tunneling and= ipv6 mesh networking=0A=0A=0A=0A=0A=0A---------- Forwarded message -------= ---=0AFrom: <[mailto:jp@nexedi.com] jp@nexedi.com>=0ADate: Wed, Jan 23, 2= 013 at 6:30 AM=0A Subject: [Babel-users] A happy babel user: re6st=0ATo: [m= ailto:babel-users@lists.alioth.debian.org] babel-users@lists.alioth.debian.= org=0A=0A=0AHi,=0A=0A Very often, people complain on mailing lists. Today, = I would like to say thank you.=0A=0A Last summer, we have implemented a wir= ed mesh network system based on babel which can provide stable IPv6 to all = nodes of a decentralized cloud operation system. It works great.=0A=0A Than= k you babel.=0A=0A If you are in a hurry, here is the code: [http://git.erp= 5.org/gitweb/re6stnet.git] http://git.erp5.org/gitweb/re6stnet.git=0A What = you can do with that code: provide reliable IPv6 to the world=0A=0A If you = think re6st is useful, please feel free to add it to the list of babel link= s.=0A=0A Details bellow.=0A=0A Regards,=0A=0A JP Smets.=0A Nexedi CEO=0A[te= l:%2B33%20629%2002%2044%2025] +33 629 02 44 25=0A ---=0A=0A 1- The problem = to solve=0A=0A We implemented a couple of years ago ([http://bit.ly/SWVQlx]= http://bit.ly/SWVQlx) a Cloud system called SlapOS ([http://www.slapos.org= ] http://www.slapos.org) which relies on servers located in people's home a= nd now also in offices, data centers or even your smartphone, tablet or TV.= SlapOS is now used by some large corporations. One of its main application= s is to create a disaster recovery cloud which can resist any force majeur = event (ex. war, terrorism, political instability, software bug) which does = affect traditional clouds from time to time ([http://iwgcr.org] http://iwgc= r.org). It is also much cheaper and environmental friendly.=0A=0A SlapOS re= lies on IPv6 in order to interconnect all nodes. Each node is allocated usu= ally 100 global IPv6 addresses or more.=0A=0A This is where our problem sta= rted: all IPv6 providers we tried were unable to provide reliable connectiv= ity. We tried providers in France, Germany, Japan, Norway. For example, in = France among 200 IPv6 adresses provided by a Freebox (Free), 3 becomes unre= achable from time to time, during a couple of minutes or hours. OVH routers= sometimes no longer route packers to Free, but only for IPv6, during a cou= ple of hours. Telia routers somtimes "eat" a few bytes during the initializ= ation of a session.=0A=0A Overall, the use of native IPv6 of ISPs lead to a= service availability of 99% or worse. We we searching for a solution.=0A= =0A We also had had the experience that from time to time, IPv4 transit bet= ween ISPs can be cut for a while - a couple of hours -although less often. = Our ideal solution should also solve that.=0A=0A 2- The solution: re6st + b= abel=0A=0A Step 1: create a wired mesh=0A=0A We coded a litlle deaemon call= ed re6st which is able to find 10 IPv4 neighbours randomly and create a tun= nel to each neighbour. re6st can be placed behind a NAT. It is able to capt= ure public IPv4 address of your router through UPnP. After some time, all n= odes which run re6st form a global mesh.=0A=0A Step 2: start babel=0A=0A On= ce tunnels are created, babel is used for routing. Babel then finds the bes= t route to interconnect all re6st nodes.=0A=0A 3- Results=0A=0A After a cou= ple of month of using re6st + babel we can say that it works quite well. Sl= apOS no longer experiences the connectivity problems of native IPv6. We can= safely host websites with SlapOS over re6st+babel.=0A=0A 4- Next steps=0A= =0A A report will be published.=0A=0A 5- Remaining problems to solve=0A=0A = The problems which remain to be solved are the following:=0A=0A a- How can = we prevent one babel participant to act against other participants by provi= ding wrong information to other participants ? Imagine for example that a b= ad organization joins re6st + babel network and starts capturing all routes= in order to analyze traffic or even block it.=0A=0A b- How can we create a= hierarchical addressing system ? The idea here is to group participants dy= namically and assign them a "big" IPv6 address range. Each participant conn= ects to another participant through another participant by first connecting= randomly to one participant in a dynamic group and next connect to other p= articipant in the same group. With this grouping approach, there is no need= to create a hierachical network with a bakbone. It also solves the problem= of scalability.=0A=0A c- How can we implement more policies (ex. latency) = ?=0A=0A d- How could we implement accounting and billing in a way or anothe= r ? (open question, but quite important for example to solve the problem of= FTTH participants with upload limited to 3GB / day as in Japan)=0A=0A 6- C= redits=0A=0A Most of the coding of re6st was done by Julien Muchembled (Nex= edi), Ulysse Beaugnon (ENS) and Guillaume Bury (ENS).=0A=0A 7- Alternative= s=0A=0A We could have used other routing protocols (ex. OLSR). But we felt = that Babel pluggable policy system was a key design difference which could = be used to later customize it to different needs of Cloud applications (ex.= low latency). We would also feel ashamed to use a protocol which babel's c= reator proved that it was flawed.=0A=0A We could have used tinc. But tinc c= reates a fully connected mesh. There is also a difference between what it c= laims to do and what it actually does. Last, mixing tunneling and routing i= s a bad idea as we were suggested by Juliusz C.=0A=0A We could have used gr= e instead of OpenVPN for tunnels. But that does work behind an IPv4 NAT. Ye= t, nothing prevents use from later using gre.=0A___________________________= ____________________=0A Babel-users mailing list=0A[mailto:Babel-users@list= s.alioth.debian.org] Babel-users@lists.alioth.debian.org=0A[http://lists.al= ioth.debian.org/cgi-bin/mailman/listinfo/babel-users] http://lists.alioth.d= ebian.org/cgi-bin/mailman/listinfo/babel-users=0A=0A=0A-- =0ADave T=C3=A4ht= =0A=0A Fixing bufferbloat with cerowrt: [http://www.teklibre.com/cerowrt/su= bscribe.html] http://www.teklibre.com/cerowrt/subscribe.html ------=_20130123133412000000_72016 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

= Cool!   Let's make sure the OpenVPN links on it don't have buffer= bloat in them - VPNs often do, and perhaps need a layer of fq_codel on top = of each link (also underneath, of course, where the routers can be fixed).<= /p>=0A

 

=0A

More interestingly - is this an openly growing network?  Or = is it intended to be an island?

=0A

&nbs= p;

=0A

For those stuck in IPv4 land (whi= ch includes folks who live inside corporate Intranets, sadly, maybe this is= a good step.

=0A

 

=0A

-----Original Message-----
From: "Dave Taht" = <dave.taht@gmail.com>
Sent: Wednesday, January 23, 2013 12:11pm<= br />To: cerowrt-devel@lists.bufferbloat.net
Subject: [Cerowrt-devel] = An interesting application of tunneling and ipv6 mesh networking

=0A


=0A
---------- Forwarded message ----------
From: <jp@nexedi.com>
Date: Wed, Jan 23, 2013 at 6:30= AM
Subject: [Babel-users] A happy babel user: re6st
To: babel-users@lists.alioth.de= bian.org


Hi,

Very often, people complain o= n mailing lists. Today, I would like to say thank you.

Last sum= mer, we have implemented a wired mesh network system based on babel which c= an provide stable IPv6 to all nodes of a decentralized cloud operation syst= em. It works great.

Thank you babel.

If you are in = a hurry, here is the code: http://git.erp5.org/gitweb/re6stnet.git
Wha= t you can do with that code: provide reliable IPv6 to the world

= If you think re6st is useful, please feel free to add it to the list of ba= bel links.

Details bellow.

Regards,

JP = Smets.
Nexedi CEO
+33= 629 02 44 25
---

1- The problem to solve

= We implemented a couple of years ago (http://bit.ly/SWVQlx) a Cloud system called SlapOS (http://www.slapos.org) = which relies on servers located in people's home and now also in offices, d= ata centers or even your smartphone, tablet or TV. SlapOS is now used by so= me large corporations. One of its main applications is to create a disaster= recovery cloud which can resist any force majeur event (ex. war, terrorism= , political instability, software bug) which does affect traditional clouds= from time to time (http://i= wgcr.org). It is also much cheaper and environmental friendly.
SlapOS relies on IPv6 in order to interconnect all nodes. Each node is = allocated usually 100 global IPv6 addresses or more.

This is wh= ere our problem started: all IPv6 providers we tried were unable to provide= reliable connectivity. We tried providers in France, Germany, Japan, Norwa= y. For example, in France among 200 IPv6 adresses provided by a Freebox (Fr= ee), 3 becomes unreachable from time to time, during a couple of minutes or= hours. OVH routers sometimes no longer route packers to Free, but only for= IPv6, during a couple of hours. Telia routers somtimes "eat" a few bytes d= uring the initialization of a session.

Overall, the use of nati= ve IPv6 of ISPs lead to a service availability of 99% or worse. We we searc= hing for a solution.

We also had had the experience that from t= ime to time, IPv4 transit between ISPs can be cut for a while - a couple of= hours -although less often. Our ideal solution should also solve that.

2- The solution: re6st + babel

Step 1: create a wired = mesh

We coded a litlle deaemon called re6st which is able to fi= nd 10 IPv4 neighbours randomly and create a tunnel to each neighbour. re6st= can be placed behind a NAT. It is able to capture public IPv4 address of y= our router through UPnP. After some time, all nodes which run re6st form a = global mesh.

Step 2: start babel

Once tunnels are c= reated, babel is used for routing. Babel then finds the best route to inter= connect all re6st nodes.

3- Results

After a couple = of month of using re6st + babel we can say that it works quite well. SlapOS= no longer experiences the connectivity problems of native IPv6. We can saf= ely host websites with SlapOS over re6st+babel.

4- Next steps
A report will be published.

5- Remaining problems to= solve

The problems which remain to be solved are the following= :

a- How can we prevent one babel participant to act against ot= her participants by providing wrong information to other participants ? Ima= gine for example that a bad organization joins re6st + babel network and st= arts capturing all routes in order to analyze traffic or even block it.

b- How can we create a hierarchical addressing system ? The idea h= ere is to group participants dynamically and assign them a "big" IPv6 addre= ss range. Each participant connects to another participant through another = participant by first connecting randomly to one participant in a dynamic gr= oup and next connect to other participant in the same group. With this grou= ping approach, there is no need to create a hierachical network with a bakb= one. It also solves the problem of scalability.

c- How can we i= mplement more policies (ex. latency) ?

d- How could we implemen= t accounting and billing in a way or another ? (open question, but quite im= portant for example to solve the problem of FTTH participants with upload l= imited to 3GB / day as in Japan)

6- Credits

Most of= the coding of re6st was done by Julien Muchembled (Nexedi),  Ulysse B= eaugnon (ENS) and Guillaume Bury (ENS).

7- Alternatives
We could have used other routing protocols (ex. OLSR). But we felt tha= t Babel pluggable policy system was a key design difference which could be = used to later customize it to different needs of Cloud applications (ex. lo= w latency). We would also feel ashamed to use a protocol which babel's crea= tor proved that it was flawed.

We could have used tinc. But tin= c creates a fully connected mesh. There is also a difference between what i= t claims to do and what it actually does. Last, mixing tunneling and routin= g is a bad idea as we were suggested by Juliusz C.

We could hav= e used gre instead of OpenVPN for tunnels. But that does work behind an IPv= 4 NAT. Yet, nothing prevents use from later using gre.
_______________= ________________________________
Babel-users mailing list
Babel-users@lists.alioth.d= ebian.org
http://lists.alioth.debian.org/cg= i-bin/mailman/listinfo/babel-users
=0A


--
D= ave T=C3=A4ht

Fixing bufferbloat with cerowrt: http://www.tek= libre.com/cerowrt/subscribe.html
 ------=_20130123133412000000_72016--