From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp181.iad.emailsrvr.com (smtp181.iad.emailsrvr.com [207.97.245.181]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 2F884201251 for ; Mon, 30 Apr 2012 12:15:46 -0700 (PDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp58.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 8CC91308227; Mon, 30 Apr 2012 15:15:44 -0400 (EDT) X-Virus-Scanned: OK Received: from legacy7.wa-web.iad1a (legacy7.wa-web.iad1a.rsapps.net [192.168.2.216]) by smtp58.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 6D24F30821F; Mon, 30 Apr 2012 15:15:44 -0400 (EDT) Received: from reed.com (localhost [127.0.0.1]) by legacy7.wa-web.iad1a (Postfix) with ESMTP id 40F733200B0; Mon, 30 Apr 2012 15:15:44 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com) with HTTP; Mon, 30 Apr 2012 15:15:44 -0400 (EDT) Date: Mon, 30 Apr 2012 15:15:44 -0400 (EDT) From: dpreed@reed.com To: "Dave Taht" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20120430151544000000_72142" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: <1335798890.08871546@apps.rackspace.com> Message-ID: <1335813344.264819197@apps.rackspace.com> X-Mailer: webmail7.0 Cc: cerowrt-devel@lists.bufferbloat.net Subject: Re: [Cerowrt-devel] =?utf-8?q?Pointers_on_build_setup=3F?= X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Apr 2012 19:15:46 -0000 ------=_20120430151544000000_72142 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0AI'll see about setting a login up for you - it's a bit complicated becau= se the cluster is not exposed at the moment, so I need to secure it and exp= ose it. I don't want to distract from feature completion... so it may be = a few days (given that I'm busy during the week mostly with my day job at S= AP, which involves systems engineering experiments with *really big* comput= ing systems - my personal research toy there is 400 Westmere-EX cores with = 5 TB of RAM, and 30 TB of PCIe Flash, and that's on the small end.... my te= am runs several other, larger systems of the same sort for "production" res= earch :-) It runs Linux, of course. )=0A =0AMy focus is related to dense = mesh stuff, because scalable urban/mobile usage is the big wall for radio i= nternetworking after IPv6 and controlled latency (bufferbloat). But actual= ly I'm not trying to solve the whole "decentralized mesh" problem - what I'= m trying to do is do propagation monitoring and modeling in real home envir= onments on all 2.4 and 5 GHz channels (and maybe create a tool for that). = It's my opinion that we need to understand a lot more about real-world phys= ics *as seen by real radios*, and without that, we can't really design prot= ocols for adaptive, cooperative use of these license-by-rule bands.=0A =0AT= he physics problem is one that may never be addressable by academic researc= hers - they don't have the ability to reach into homes, apartments, busines= ses, etc. to do the experiments. But an open, modern home router that anyo= ne can buy for at most a couple hundred bucks can provide that platform. I= t's my "Internet 3 experiment". I firmly believe you need to do systems en= gineering experiments before designing answers. I can't use smartphones or= commercial home routers to do this.=0A =0AOf course, once I've created a w= ay to measure real-time propagation, the next step will be to create cooper= ation/adaptatation protocols (not just routing, but the complement of routi= ng - where not to transmit given the physical environment, kind of "real-ti= me radio engineering" to maximize utilization among the systems).=0A =0ADav= id=0A =0A =0A-----Original Message-----=0AFrom: "Dave Taht" =0ASent: Monday, April 30, 2012 11:50am=0ATo: dpreed@reed.com=0ACc: "= Outback Dingo" , cerowrt-devel@lists.bufferbloat.ne= t=0ASubject: Re: Pointers on build setup?=0A=0A=0A=0AOn Mon, Apr 30, 2012 a= t 8:14 AM, wrote:=0A> Regarding builds - can someone who= is building cerowrt from source=0A> (including kernel) point me at the rec= ipe for building it completely from=0A> scratch?=0A=0AWe have most of a bui= ld script that has worked at various points in time,=0Aat the moment, I dou= bt it's correct. Ketan posted some mods recently...=0A=0Aas things stand to= day, it's kind of fiddly, and requires multiple attempts=0Aat getting the .= config right, there's a few tool dependencies, etc.=0A=0AAs soon as I hit f= eature complete - and had hardware that could turn=0Aaround builds fast eno= ugh - I'd planned to fix that up.=0A=0AIn the interim...=0A=0AI can get a c= orrect build setup for you in 10 minutes, if you give me a login.=0ACertain= ly having a script that 'just works', and is a one liner, has=0Abeen a goal= , once things get more stable. Which I would hope be late=0Athis week. Hope= fully.=0A=0AI note I have not checked in all the 'fixes' to the ipv6 alignm= ent=0Aissue (bugs 360, 371, 379) in the past week, as alternatives are bein= g=0Adiscussed on the openwrt-devel list.=0A=0A> Besides running 3.3-2 on a = 3800, I've begun my project to validate that=0A> Cerowrt can run on my ubnt= RouterStation Pro box, which is preliminary to=0A> doing some experimentat= ion on dense scalability that interests me. I=0A=0AIf you are talking abo= ut what I call the 'dense mesh' problem, the predecessor=0Aproject to cerow= rt had it's original genesis in trying to solve the=0Ameltdown issues olpc = had with 802.11s.=0A=0A"Diversity routing" seemed to be an answer.=0A=0Awww= .pps.jussieu.fr/~jch/software/babel/wbmv4.pdf=0A=0AAnd the routerstation pr= o (3 radios! ooh!) is a perfect platform to be=0Atrying that on.=0AJuliusz = was looking into doing experiments on that hardware...=0A=0ASadly it shares= the unaligned ipv6 problem that we've been struggling to solve,=0Aand the = max thruput is less than what those 3 radios could achieve.=0A=0A> verified= that it runs the latest stable Backfire release of OpenWRT this=0A> weeken= d, and it does so quite well.=0A>=0A>=0A>=0A> My own private home build ser= ver is (essentially) a cluster of two AMD8160=0A> 8-core 32 GB servers conn= ected with 40 Gb/s Infiniband and a 6-way RAID5=0A> storage array on one of= them that gives quite nice read performance. I'm=0A> not worried about ho= w long it takes to do a build...=0A=0AI will note that SSDs really rock, an= d seek times of ~0 do wonders for=0Athe link step.=0A=0AThat said, I was pr= oud of my new hardware, until I read that. :/=0A=0A>=0A>=0A>=0A> Anyway - t= he recipe doesn't have to be perfected for me to use it - and I=0A> suspect= it's similar to OpenWRT's build, so maybe pointing me to the best=0A> help= for that and what the differences are would be great.=0A=0AFrom a basic pe= rspective, you have additional feeds in feeds.conf,=0Aa specialized config,= a special filesystem that does a few different=0Athings - notably changing= the device naming so you can have multiple=0Arouted interfaces and simplif= ied firewall rules -=0A=0ASo you start by checking out cerofiles. The build= _cero.sh script there=0Ais obsolete but can supply clue as to the repos req= uired and ketan had=0Asome patches to that.=0A=0AWhat I do nowadays is mild= ly different than the script, in that after=0Athe repos are pulled down, fe= eds.conf is configured, scripts/env new=0Awhatever has been run and cerofil= es imported into it, I tend to hammer=0Aat it with=0A=0A./scripts/feeds ins= tall -p cero `cat env/override.list`=0A./scripts/feeds install `cat env/pac= kages.list`=0A./scripts/feeds install -p cero `cat env/override.list`=0A=0A= cp env/config-* .config=0Amake defconfig=0Agrep =3Dm .config | sort -u > t1= .txt=0Agrep =3Dm env/config* | sort -u > t2.txt=0A=0Adiff t1.txt t2.txt=0A= =0Aadd the packages via feeds install (I'm always adding new ones)=0A=0Athe= script ALMOST does all that.=0A=0A=0A(or I can can get it going for you in= 10 minutes)=0A=0A>=0A>=0A>=0A> I promise I will write the "port" experienc= e up and post it on the Wiki, in=0A> case others want to port cerowrt to ot= her "OpenWRT platforms" of sufficient=0A> capabilities.=0A>=0A>=0A>=0A> Dav= id=0A>=0A>=0A>=0A> -----Original Message-----=0A> From: "Outback Dingo" =0A> Sent: Sunday, April 29, 2012 9:42pm=0A> To: "Dave= Taht" =0A> Cc: cerowrt-devel@lists.bufferbloat.net=0A= > Subject: Re: [Cerowrt-devel] speeding up builds=0A>=0A> On Sun, Apr 29, 2= 012 at 8:15 PM, Dave Taht wrote:=0A>> I finally acqui= red a machine with 32GB of ram, an intel 3930k (6=0A>> cores), and an SSD.= =0A>>=0A>> I put the build_dir, /tmp and /var/tmp on ramdisks, and...=0A>>= =0A>> This cut a complete cerowrt build (including toolchain) down from >= =0A>> 3.5 hrs down to under 45 minutes.=0A>>=0A>> Without the toolchain reb= uild, but after a make clean (to rebuild the=0A>> packages and kernel), it'= s about 28 minutes.=0A>>=0A>> I can see that it is possible to parallelize = things more to maybe chop=0A>> another 30% of of things...=0A>> ...but I'm = glad to have 3 hrs of my life back, per build.=0A>>=0A>> I wanted to figure= out to what extent modern hardware would enhance=0A>> the existing buildbo= t system.=0A>> Now I know...=0A>=0A> odd my laptop will do a full build wit= h tool chain in about an=0A> hour..... its only a core i3 with 6gb and an s= sd=0A>=0A>>=0A>>=0A>> --=0A>> Dave T=C3=A4ht=0A>> SKYPE: davetaht=0A>> US T= el: 1-239-829-5608=0A>> http://www.bufferbloat.net=0A>> ___________________= ____________________________=0A>> Cerowrt-devel mailing list=0A>> Cerowrt-d= evel@lists.bufferbloat.net=0A>> https://lists.bufferbloat.net/listinfo/cero= wrt-devel=0A> _______________________________________________=0A> Cerowrt-d= evel mailing list=0A> Cerowrt-devel@lists.bufferbloat.net=0A> https://lists= .bufferbloat.net/listinfo/cerowrt-devel=0A=0A=0A=0A-- =0ADave T=C3=A4ht=0AS= KYPE: davetaht=0AUS Tel: 1-239-829-5608=0Ahttp://www.bufferbloat.net ------=_20120430151544000000_72142 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

I'll see a= bout setting a login up for you - it's a bit complicated because the cluste= r is not exposed at the moment, so I need to secure it and expose it. = I don't want to distract from feature completion...  so it may be a f= ew days (given that I'm busy during the week mostly with my day job at SAP,= which involves systems engineering experiments with *really big* computing= systems - my personal research toy there is 400 Westmere-EX cores with 5 T= B of RAM, and 30 TB of PCIe Flash, and that's on the small end.... my team = runs several other, larger systems of the same sort for "production" resear= ch  :-)  It runs Linux, of course. )

=0A

 

=0A

My focus is relate= d to dense mesh stuff, because scalable urban/mobile usage is the big wall = for radio internetworking after IPv6 and controlled latency (bufferbloat).&= nbsp; But actually I'm not trying to solve the whole "decentralized mesh" p= roblem - what I'm trying to do is do propagation monitoring and modeling in= real home environments on all 2.4 and 5 GHz channels (and maybe create a t= ool for that).  It's my opinion that we need to understand a lot more = about real-world physics *as seen by real radios*, and without that, we can= 't really design protocols for adaptive, cooperative use of these license-b= y-rule bands.

=0A

 

=0A

The physics problem is one that may never be addre= ssable by academic researchers - they don't have the ability to reach into = homes, apartments, businesses, etc. to do the experiments.  But an ope= n, modern home router that anyone can buy for at most a couple hundred buck= s can provide that platform.  It's my "Internet 3 experiment".  I= firmly believe you need to do systems engineering experiments before desig= ning answers.  I can't use smartphones or commercial home routers to d= o this.

=0A

 

=0A

Of course, once I've created a way to measure real-time p= ropagation, the next step will be to create cooperation/adaptatation protoc= ols (not just routing, but the complement of routing - where not to transmi= t given the physical environment, kind of "real-time radio engineering" to = maximize utilization among the systems).

=0A

 

=0A

David

=0A

 

=0A

 =0A

-----Original Message-----
From:= "Dave Taht" <dave.taht@gmail.com>
Sent: Monday, April 30, 2012 = 11:50am
To: dpreed@reed.com
Cc: "Outback Dingo" <outbackdingo@= gmail.com>, cerowrt-devel@lists.bufferbloat.net
Subject: Re: Pointe= rs on build setup?

=0A
=0AOn Mon, Apr 30, 2012 at 8:14 AM, <dpreed= @reed.com> wrote:
> Regarding builds - can someone who is buildi= ng cerowrt from source
> (including kernel) point me at the recipe = for building it completely from
> scratch?

We have most = of a build script that has worked at various points in time,
at the mo= ment, I doubt it's correct. Ketan posted some mods recently...

a= s things stand today, it's kind of fiddly, and requires multiple attemptsat getting the .config right, there's a few tool dependencies, etc.

As soon as I hit feature complete - and had hardware that could tur= n
around builds fast enough - I'd planned to fix that up.

I= n the interim...

I can get a correct build setup for you in 10 m= inutes, if you give me a login.
Certainly having a script that 'just w= orks', and is a one liner, has
been a goal, once things get more stabl= e. Which I would hope be late
this week. Hopefully.

I note = I have not checked in all the 'fixes' to the ipv6 alignment
issue (bug= s 360, 371, 379) in the past week, as alternatives are being
discussed= on the openwrt-devel list.

> Besides running 3.3-2 on a 3800= , I've begun my project to validate that
> Cerowrt can run on my ub= nt RouterStation Pro box, which is preliminary to
> doing some expe= rimentation on dense scalability that interests me.   I

If you are talking about what I call the 'dense mesh' problem, the predec= essor
project to cerowrt had it's original genesis in trying to solve = the
meltdown issues olpc had with 802.11s.

"Diversity routi= ng" seemed to be an answer.

www.pps.jussieu.fr/~jch/software/bab= el/wbmv4.pdf

And the routerstation pro (3 radios! ooh!) is a per= fect platform to be
trying that on.
Juliusz was looking into doin= g experiments on that hardware...

Sadly it shares the unaligned = ipv6 problem that we've been struggling to solve,
and the max thruput = is less than what those 3 radios could achieve.

> verified th= at it runs the latest stable Backfire release of OpenWRT this
> wee= kend, and it does so quite well.
>
>
>
> My= own private home build server is (essentially) a cluster of two AMD8160> 8-core 32 GB servers connected with 40 Gb/s Infiniband and a 6-way = RAID5
> storage array on one of them that gives quite nice read per= formance.  I'm
> not worried about how long it takes to do a b= uild...

I will note that SSDs really rock, and seek times of ~0 = do wonders for
the link step.

That said, I was proud of my = new hardware, until I read that. :/

>
>
>
> Anyway - the recipe doesn't have to be perfected for me to use it - = and I
> suspect it's similar to OpenWRT's build, so maybe pointing = me to the best
> help for that and what the differences are would b= e great.

From a basic perspective, you have additional feeds in = feeds.conf,
a specialized config, a special filesystem that does a few= different
things - notably changing the device naming so you can have= multiple
routed interfaces and simplified firewall rules -

So you start by checking out cerofiles. The build_cero.sh script there
is obsolete but can supply clue as to the repos required and ketan hadsome patches to that.

What I do nowadays is mildly different = than the script, in that after
the repos are pulled down, feeds.conf i= s configured, scripts/env new
whatever has been run and cerofiles impo= rted into it, I tend to hammer
at it with

./scripts/feeds i= nstall -p cero `cat env/override.list`
./scripts/feeds install `cat en= v/packages.list`
./scripts/feeds install -p cero `cat env/override.lis= t`

cp env/config-* .config
make defconfig
grep =3Dm .c= onfig | sort -u > t1.txt
grep =3Dm env/config* | sort -u > t2.tx= t

diff t1.txt t2.txt

add the packages via feeds insta= ll (I'm always adding new ones)

the script ALMOST does all that.=


(or I can can get it going for you in 10 minutes)
>
>
>
> I promise I will write the "port" ex= perience up and post it on the Wiki, in
> case others want to port = cerowrt to other "OpenWRT platforms" of sufficient
> capabilities.<= br />>
>
>
> David
>
>
>=
> -----Original Message-----
> From: "Outback Dingo" <o= utbackdingo@gmail.com>
> Sent: Sunday, April 29, 2012 9:42pm
> To: "Dave Taht" <dave.taht@gmail.com>
> Cc: cerowrt-de= vel@lists.bufferbloat.net
> Subject: Re: [Cerowrt-devel] speeding u= p builds
>
> On Sun, Apr 29, 2012 at 8:15 PM, Dave Taht <= ;dave.taht@gmail.com> wrote:
>> I finally acquired a machine = with 32GB of ram, an intel 3930k (6
>> cores), and an SSD.
= >>
>> I put the build_dir, /tmp and /var/tmp on ramdisks, = and...
>>
>> This cut a complete cerowrt build (inclu= ding toolchain) down from >
>> 3.5 hrs down to under 45 minut= es.
>>
>> Without the toolchain rebuild, but after a = make clean (to rebuild the
>> packages and kernel), it's about 2= 8 minutes.
>>
>> I can see that it is possible to par= allelize things more to maybe chop
>> another 30% of of things..= .
>> ...but I'm glad to have 3 hrs of my life back, per build.>>
>> I wanted to figure out to what extent modern har= dware would enhance
>> the existing buildbot system.
>&g= t; Now I know...
>
> odd my laptop will do a full build wit= h tool chain in about an
> hour..... its only a core i3 with 6gb an= d an ssd
>
>>
>>
>> --
>&g= t; Dave T=C3=A4ht
>> SKYPE: davetaht
>> US Tel: 1-239= -829-5608
>> http://www.bufferbloat.net
>> __________= _____________________________________
>> Cerowrt-devel mailing l= ist
>> Cerowrt-devel@lists.bufferbloat.net
>> https:/= /lists.bufferbloat.net/listinfo/cerowrt-devel
> ___________________= ____________________________
> Cerowrt-devel mailing list
>= Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.ne= t/listinfo/cerowrt-devel



--
Dave T=C3=A4ht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.n= et

=0A
 ------=_20120430151544000000_72142--