From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ie0-f170.google.com (mail-ie0-f170.google.com [209.85.223.170]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 8CF1421F0AE for ; Wed, 23 Jan 2013 11:23:59 -0800 (PST) Received: by mail-ie0-f170.google.com with SMTP id k10so14351974iea.15 for ; Wed, 23 Jan 2013 11:23:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=YT8Bg2vX7IGSG6RVmyn2ObAtXfGdw/+Kf1yTofEOpu8=; b=O+MBNGduoTBj5BD2CLSeeFp/zwz9aXez0cLfMEyBjGcLDWOmz78mCJLwoVBUPCkF+H P+4nQb/19d5KRfs1MMdB6yierDwg5qiuRCJUfjvD/syg+CzCmW3nC8EFDGCJtTrHKnYW W9aqVm9BTyY60YmjaL7LygDB87xciFFY1L5typXvH+8984YfQYstKUJ/YyNT9kmwd7yr 4HjTohtXdE8PXSaEEo2PblC83yQQtdJax+c6NfkApGRNmpnzjsL/km9Jg5i0eCYHfxTW JeHXSAQ1bxIiFFducZi79CBUIAMO9gUYmmuPkQMYrF7QtrUOUzw76dMRHS296JXeXMPn VUsg== MIME-Version: 1.0 X-Received: by 10.42.247.8 with SMTP id ma8mr1870519icb.1.1358969038587; Wed, 23 Jan 2013 11:23:58 -0800 (PST) Received: by 10.64.135.39 with HTTP; Wed, 23 Jan 2013 11:23:58 -0800 (PST) In-Reply-To: <1358966052.119718192@apps.rackspace.com> References: <1358966052.119718192@apps.rackspace.com> Date: Wed, 23 Jan 2013 14:23:58 -0500 Message-ID: From: Dave Taht To: dpreed@reed.com, jp@nexedi.com Content-Type: multipart/alternative; boundary=90e6ba1efca0ce48e404d3f9a2c4 Cc: cerowrt-devel@lists.bufferbloat.net Subject: Re: [Cerowrt-devel] An interesting application of tunneling and ipv6 mesh networking X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jan 2013 19:23:59 -0000 --90e6ba1efca0ce48e404d3f9a2c4 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Wed, Jan 23, 2013 at 1:34 PM, wrote: > Cool! Let's make sure the OpenVPN links on it don't have bufferbloat in > them - VPNs often do, and perhaps need a layer of fq_codel on top of each > link (also underneath, of course, where the routers can be fixed). > > I have seen extremely interesting behavior with openvpn. vtun will drop packets like crazy under load. > > > More interestingly - is this an openly growing network? Or is it intende= d > to be an island? > > > The author of the original post, cc'd. > For those stuck in IPv4 land (which includes folks who live inside > corporate Intranets, sadly, maybe this is a good step. > > > I note that I've been told that osx for the "backtomymac" feature uses a private ipv6 backbone. It's probably one of the largest in the world. > -----Original Message----- > From: "Dave Taht" > Sent: Wednesday, January 23, 2013 12:11pm > To: cerowrt-devel@lists.bufferbloat.net > Subject: [Cerowrt-devel] An interesting application of tunneling and ipv6 > mesh networking > > > > ---------- Forwarded message ---------- > From: > Date: Wed, Jan 23, 2013 at 6:30 AM > Subject: [Babel-users] A happy babel user: re6st > To: babel-users@lists.alioth.debian.org > > > Hi, > > Very often, people complain on mailing lists. Today, I would like to say > thank you. > > Last summer, we have implemented a wired mesh network system based on > babel which can provide stable IPv6 to all nodes of a decentralized cloud > operation system. It works great. > > Thank you babel. > > If you are in a hurry, here is the code: > http://git.erp5.org/gitweb/re6stnet.git > What you can do with that code: provide reliable IPv6 to the world > > If you think re6st is useful, please feel free to add it to the list of > babel links. > > Details bellow. > > Regards, > > JP Smets. > Nexedi CEO > +33 629 02 44 25 > --- > > 1- The problem to solve > > We implemented a couple of years ago (http://bit.ly/SWVQlx) a Cloud > system called SlapOS (http://www.slapos.org) which relies on servers > located in people's home and now also in offices, data centers or even yo= ur > smartphone, tablet or TV. SlapOS is now used by some large corporations. > One of its main applications is to create a disaster recovery cloud which > can resist any force majeur event (ex. war, terrorism, political > instability, software bug) which does affect traditional clouds from time > to time (http://iwgcr.org). It is also much cheaper and environmental > friendly. > > SlapOS relies on IPv6 in order to interconnect all nodes. Each node is > allocated usually 100 global IPv6 addresses or more. > > This is where our problem started: all IPv6 providers we tried were unabl= e > to provide reliable connectivity. We tried providers in France, Germany, > Japan, Norway. For example, in France among 200 IPv6 adresses provided by= a > Freebox (Free), 3 becomes unreachable from time to time, during a couple = of > minutes or hours. OVH routers sometimes no longer route packers to Free, > but only for IPv6, during a couple of hours. Telia routers somtimes "eat"= a > few bytes during the initialization of a session. > > Overall, the use of native IPv6 of ISPs lead to a service availability of > 99% or worse. We we searching for a solution. > > We also had had the experience that from time to time, IPv4 transit > between ISPs can be cut for a while - a couple of hours -although less > often. Our ideal solution should also solve that. > > 2- The solution: re6st + babel > > Step 1: create a wired mesh > > We coded a litlle deaemon called re6st which is able to find 10 IPv4 > neighbours randomly and create a tunnel to each neighbour. re6st can be > placed behind a NAT. It is able to capture public IPv4 address of your > router through UPnP. After some time, all nodes which run re6st form a > global mesh. > > Step 2: start babel > > Once tunnels are created, babel is used for routing. Babel then finds the > best route to interconnect all re6st nodes. > > 3- Results > > After a couple of month of using re6st + babel we can say that it works > quite well. SlapOS no longer experiences the connectivity problems of > native IPv6. We can safely host websites with SlapOS over re6st+babel. > > 4- Next steps > > A report will be published. > > 5- Remaining problems to solve > > The problems which remain to be solved are the following: > > a- How can we prevent one babel participant to act against other > participants by providing wrong information to other participants ? Imagi= ne > for example that a bad organization joins re6st + babel network and start= s > capturing all routes in order to analyze traffic or even block it. > > b- How can we create a hierarchical addressing system ? The idea here is > to group participants dynamically and assign them a "big" IPv6 address > range. Each participant connects to another participant through another > participant by first connecting randomly to one participant in a dynamic > group and next connect to other participant in the same group. With this > grouping approach, there is no need to create a hierachical network with = a > bakbone. It also solves the problem of scalability. > > c- How can we implement more policies (ex. latency) ? > > d- How could we implement accounting and billing in a way or another ? > (open question, but quite important for example to solve the problem of > FTTH participants with upload limited to 3GB / day as in Japan) > > 6- Credits > > Most of the coding of re6st was done by Julien Muchembled (Nexedi), > Ulysse Beaugnon (ENS) and Guillaume Bury (ENS). > > 7- Alternatives > > We could have used other routing protocols (ex. OLSR). But we felt that > Babel pluggable policy system was a key design difference which could be > used to later customize it to different needs of Cloud applications (ex. > low latency). We would also feel ashamed to use a protocol which babel's > creator proved that it was flawed. > > We could have used tinc. But tinc creates a fully connected mesh. There i= s > also a difference between what it claims to do and what it actually does. > Last, mixing tunneling and routing is a bad idea as we were suggested by > Juliusz C. > > We could have used gre instead of OpenVPN for tunnels. But that does work > behind an IPv4 NAT. Yet, nothing prevents use from later using gre. > _______________________________________________ > Babel-users mailing list > Babel-users@lists.alioth.debian.org > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users > > > > -- > Dave T=E4ht > > Fixing bufferbloat with cerowrt: > http://www.teklibre.com/cerowrt/subscribe.html > --=20 Dave T=E4ht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html --90e6ba1efca0ce48e404d3f9a2c4 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

On Wed, Jan 23, 2013 at 1:34 PM, <dpreed= @reed.com> wrote:

Cool!=A0=A0 = Let's make sure the OpenVPN links on it don't have bufferbloat in t= hem - VPNs often do, and perhaps need a layer of fq_codel on top of each li= nk (also underneath, of course, where the routers can be fixed).


I have see= n extremely interesting behavior with openvpn. vtun will drop packets like = crazy under load.
=A0

=A0

More interestingly - is this an openly grow= ing network?=A0 Or is it intended to be an island?

=A0


The aut= hor of the original post, cc'd.
=A0

For those stuck in IPv4 land (which include= s folks who live inside corporate Intranets, sadly, maybe this is a good st= ep.

=A0


I note that I've been told that osx for the "backtomymac"= ; feature uses a private ipv6 backbone. It's probably one of the larges= t in the world.
=A0
=

-----Original Message-----
From: "D= ave Taht" <dave.taht@gmail.com>
Sent: Wednesday, January 23, 2013 12:11pm To: cerowrt-devel@lists.bufferbloat.net
Subject: [Cerowrt-devel] An in= teresting application of tunneling and ipv6 mesh networking



---------- Forwarded message ----------
From:= <jp@nexedi.com>
Date: We= d, Jan 23, 2013 at 6:30 AM
Subject: [Babel-users] A happy babel user: re6st
To: babel-users@lists.al= ioth.debian.org


Hi,

Very often, people complain on m= ailing lists. Today, I would like to say thank you.

Last summer, we have implemented a wired mesh network system based on = babel which can provide stable IPv6 to all nodes of a decentralized cloud o= peration system. It works great.

Thank you babel.

If you ar= e in a hurry, here is the code: http://git.erp5.org/gitweb/re6stnet.git
What you can do with that code: provide reliable IPv6 to the world

= If you think re6st is useful, please feel free to add it to the list of ba= bel links.

Details bellow.

Regards,

JP Smets.
Nexedi CEO
+33 629 02 44 25
---

1- The problem to solve

We im= plemented a couple of years ago (http://bit.ly/SWVQlx) a Cloud system called SlapOS (http://www.slapos.org) which = relies on servers located in people's home and now also in offices, dat= a centers or even your smartphone, tablet or TV. SlapOS is now used by some= large corporations. One of its main applications is to create a disaster r= ecovery cloud which can resist any force majeur event (ex. war, terrorism, = political instability, software bug) which does affect traditional clouds f= rom time to time (http://iwg= cr.org). It is also much cheaper and environmental friendly.

SlapOS relies on IPv6 in order to interconnect all nodes. Each node is= allocated usually 100 global IPv6 addresses or more.

This is where= our problem started: all IPv6 providers we tried were unable to provide re= liable connectivity. We tried providers in France, Germany, Japan, Norway. = For example, in France among 200 IPv6 adresses provided by a Freebox (Free)= , 3 becomes unreachable from time to time, during a couple of minutes or ho= urs. OVH routers sometimes no longer route packers to Free, but only for IP= v6, during a couple of hours. Telia routers somtimes "eat" a few = bytes during the initialization of a session.

Overall, the use of native IPv6 of ISPs lead to a service availability= of 99% or worse. We we searching for a solution.

We also had had t= he experience that from time to time, IPv4 transit between ISPs can be cut = for a while - a couple of hours -although less often. Our ideal solution sh= ould also solve that.

2- The solution: re6st + babel

Step 1: create a wired mesh
=
We coded a litlle deaemon called re6st which is able to find 10 IPv4 n= eighbours randomly and create a tunnel to each neighbour. re6st can be plac= ed behind a NAT. It is able to capture public IPv4 address of your router t= hrough UPnP. After some time, all nodes which run re6st form a global mesh.=

Step 2: start babel

Once tunnels are created, babel is used fo= r routing. Babel then finds the best route to interconnect all re6st nodes.=

3- Results

After a couple of month of using re6st + babel = we can say that it works quite well. SlapOS no longer experiences the conne= ctivity problems of native IPv6. We can safely host websites with SlapOS ov= er re6st+babel.

4- Next steps

A report will be published.

5- Remaining= problems to solve

The problems which remain to be solved are the f= ollowing:

a- How can we prevent one babel participant to act agains= t other participants by providing wrong information to other participants ?= Imagine for example that a bad organization joins re6st + babel network an= d starts capturing all routes in order to analyze traffic or even block it.=

b- How can we create a hierarchical addressing system ? The idea here = is to group participants dynamically and assign them a "big" IPv6= address range. Each participant connects to another participant through an= other participant by first connecting randomly to one participant in a dyna= mic group and next connect to other participant in the same group. With thi= s grouping approach, there is no need to create a hierachical network with = a bakbone. It also solves the problem of scalability.

c- How can we implement more policies (ex. latency) ?

d- How c= ould we implement accounting and billing in a way or another ? (open questi= on, but quite important for example to solve the problem of FTTH participan= ts with upload limited to 3GB / day as in Japan)

6- Credits

Most of the coding of re6st was done by Julien Much= embled (Nexedi), =A0Ulysse Beaugnon (ENS) and Guillaume Bury (ENS).

= 7- Alternatives

We could have used other routing protocols (ex. OL= SR). But we felt that Babel pluggable policy system was a key design differ= ence which could be used to later customize it to different needs of Cloud = applications (ex. low latency). We would also feel ashamed to use a protoco= l which babel's creator proved that it was flawed.

We could have used tinc. But tinc creates a fully connected mesh. Ther= e is also a difference between what it claims to do and what it actually do= es. Last, mixing tunneling and routing is a bad idea as we were suggested b= y Juliusz C.

We could have used gre instead of OpenVPN for tunnels. But that does w= ork behind an IPv4 NAT. Yet, nothing prevents use from later using gre.
= _______________________________________________
Babel-users mailing lis= t
Ba= bel-users@lists.alioth.debian.org
http://list= s.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users



--
Dave T=E4ht

Fixing bufferbloat with cerowrt: h= ttp://www.teklibre.com/cerowrt/subscribe.html
<= /blockquote>


--
Dave T=E4ht

Fixing bufferbloa= t with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html=20 --90e6ba1efca0ce48e404d3f9a2c4--