[Cerowrt-devel] speeding up builds

[Cerowrt-devel] speeding up builds

[Dave Taht]

I finally acquired a machine with 32GB of ram, an intel 3930k (6
cores), and an SSD.

I put the build_dir, /tmp and /var/tmp on ramdisks, and...

This cut a complete cerowrt build (including toolchain) down from >
3.5 hrs down to under 45 minutes.

Without the toolchain rebuild, but after a make clean (to rebuild the
packages and kernel), it's about 28 minutes.

I can see that it is possible to parallelize things more to maybe chop
another 30% of of things...
...but I'm glad to have 3 hrs of my life back, per build.

I wanted to figure out to what extent modern hardware would enhance
the existing buildbot system.
Now I know...


-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Cerowrt-devel] speeding up builds

[Outback Dingo]

On Sun, Apr 29, 2012 at 8:15 PM, Dave Taht <dave.taht@gmail.com> wrote:
> I finally acquired a machine with 32GB of ram, an intel 3930k (6
> cores), and an SSD.
>
> I put the build_dir, /tmp and /var/tmp on ramdisks, and...
>
> This cut a complete cerowrt build (including toolchain) down from >
> 3.5 hrs down to under 45 minutes.
>
> Without the toolchain rebuild, but after a make clean (to rebuild the
> packages and kernel), it's about 28 minutes.
>
> I can see that it is possible to parallelize things more to maybe chop
> another 30% of of things...
> ...but I'm glad to have 3 hrs of my life back, per build.
>
> I wanted to figure out to what extent modern hardware would enhance
> the existing buildbot system.
> Now I know...

odd my laptop will do a full build with tool chain in about an
hour..... its only a core i3 with 6gb and an ssd

>
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://www.bufferbloat.net
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] speeding up builds

[Dave Taht]

On Sun, Apr 29, 2012 at 6:42 PM, Outback Dingo <outbackdingo@gmail.com> wrote:
> On Sun, Apr 29, 2012 at 8:15 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> I finally acquired a machine with 32GB of ram, an intel 3930k (6
>> cores), and an SSD.
>>
>> I put the build_dir, /tmp and /var/tmp on ramdisks, and...
>>
>> This cut a complete cerowrt build (including toolchain) down from >
>> 3.5 hrs down to under 45 minutes.
>>
>> Without the toolchain rebuild, but after a make clean (to rebuild the
>> packages and kernel), it's about 28 minutes.
>>
>> I can see that it is possible to parallelize things more to maybe chop
>> another 30% of of things...
>> ...but I'm glad to have 3 hrs of my life back, per build.
>>
>> I wanted to figure out to what extent modern hardware would enhance
>> the existing buildbot system.
>> Now I know...
>
> odd my laptop will do a full build with tool chain in about an
> hour..... its only a core i3 with 6gb and an ssd

The best box that I had was huchra, a dual quad-core xeon circa 2006,
with 8GB of memory and mirrored drives.

A 'full build' of cero is 578 packages, some of which are rather big,
as well as building the sdk and cross development kit.

For comparison purposes, I just built linux-3.3.4 for ubuntu (so this
includes the kpkg overhead)

real	11m12.286s
user	67m11.076s
sys	7m19.955s

I am puzzled. I end up with only 75MB for disk buffers, according to
top, and I would assume that 25% of memory in this case would be good
for disk buffers.

I do like using ramdisks for this job, (why write to media unless you
have to?) but it seems saner to have the disk cache, caching.

>
>>
>>
>> --
>> Dave Täht
>> SKYPE: davetaht
>> US Tel: 1-239-829-5608
>> http://www.bufferbloat.net
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Cerowrt-devel] speeding up builds

[Dave Taht]

On Sun, Apr 29, 2012 at 6:59 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Sun, Apr 29, 2012 at 6:42 PM, Outback Dingo <outbackdingo@gmail.com> wrote:
>> On Sun, Apr 29, 2012 at 8:15 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>> I finally acquired a machine with 32GB of ram, an intel 3930k (6
>>> cores), and an SSD.
>>>
>>> I put the build_dir, /tmp and /var/tmp on ramdisks, and...
>>>
>>> This cut a complete cerowrt build (including toolchain) down from >
>>> 3.5 hrs down to under 45 minutes.
>>>
>>> Without the toolchain rebuild, but after a make clean (to rebuild the
>>> packages and kernel), it's about 28 minutes.
>>>
>>> I can see that it is possible to parallelize things more to maybe chop
>>> another 30% of of things...
>>> ...but I'm glad to have 3 hrs of my life back, per build.
>>>
>>> I wanted to figure out to what extent modern hardware would enhance
>>> the existing buildbot system.
>>> Now I know...
>>
>> odd my laptop will do a full build with tool chain in about an
>> hour..... its only a core i3 with 6gb and an ssd
>
> The best box that I had was huchra, a dual quad-core xeon circa 2006,
> with 8GB of memory and mirrored drives.
>
> A 'full build' of cero is 578 packages, some of which are rather big,
> as well as building the sdk and cross development kit.
>
> For comparison purposes, I just built linux-3.3.4 for ubuntu (so this
> includes the kpkg overhead)
>
> real    11m12.286s
> user    67m11.076s
> sys     7m19.955s
>
> I am puzzled. I end up with only 75MB for disk buffers, according to
> top, and I would assume that 25% of memory in this case would be good
> for disk buffers.
>
> I do like using ramdisks for this job, (why write to media unless you
> have to?) but it seems saner to have the disk cache, caching.

Ah. I assume that 'cached' here means disk buffers. Maybe.

            total       used       free     shared    buffers     cached
Mem:      32927452   28799604    4127848          0      75600   25122928
-/+ buffers/cache:    3601076   29326376
Swap:     33529852    1527668   32002184

Believe me, after doing the number of builds I've done this year, and
especially in the past two months, finding ways to shave even a few
minutes more off the build(s) would be a godsend.

This particular box can do 64GB of ram, and doing that would add two
channels to the memory controller, assuming I plugged the ram in
wrong...

anyway, a pure kernel build (no kpkg),

time make -j 24

real	7m33.494s
user	73m3.146s
sys	6m31.648s

I see from the phoronix benchmarks that they claim a box of this
caliber can do a kernel build in under 60sec, but I doubt they are
using a kernel of this size.

I've tossed the kernel .deb files, kernel config, script to make it a
deb, and patches here:

http://huchra.bufferbloat.net/~d/debloat/

(note - TOTALLY untested on x86_64 as yet -)

I'd gotten out of the habit of maintaining debloat-testing mostly
because doing a kernel build was taking so bloody long.

>
>>
>>>
>>>
>>> --
>>> Dave Täht
>>> SKYPE: davetaht
>>> US Tel: 1-239-829-5608
>>> http://www.bufferbloat.net
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://www.bufferbloat.net



--
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Cerowrt-devel] speeding up builds

[Dave Taht]

On Sun, Apr 29, 2012 at 7:24 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Sun, Apr 29, 2012 at 6:59 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> On Sun, Apr 29, 2012 at 6:42 PM, Outback Dingo <outbackdingo@gmail.com> wrote:
>>> On Sun, Apr 29, 2012 at 8:15 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>>> I finally acquired a machine with 32GB of ram, an intel 3930k (6
>>>> cores), and an SSD.
>>>>
>>>> I put the build_dir, /tmp and /var/tmp on ramdisks, and...
>>>>
>>>> This cut a complete cerowrt build (including toolchain) down from >
>>>> 3.5 hrs down to under 45 minutes.
>>>>
>>>> Without the toolchain rebuild, but after a make clean (to rebuild the
>>>> packages and kernel), it's about 28 minutes.
>>>>
>>>> I can see that it is possible to parallelize things more to maybe chop
>>>> another 30% of of things...
>>>> ...but I'm glad to have 3 hrs of my life back, per build.
>>>>
>>>> I wanted to figure out to what extent modern hardware would enhance
>>>> the existing buildbot system.
>>>> Now I know...
>>>
>>> odd my laptop will do a full build with tool chain in about an
>>> hour..... its only a core i3 with 6gb and an ssd
>>
>> The best box that I had was huchra, a dual quad-core xeon circa 2006,
>> with 8GB of memory and mirrored drives.
>>
>> A 'full build' of cero is 578 packages, some of which are rather big,
>> as well as building the sdk and cross development kit.
>>
>> For comparison purposes, I just built linux-3.3.4 for ubuntu (so this
>> includes the kpkg overhead)
>>
>> real    11m12.286s
>> user    67m11.076s
>> sys     7m19.955s
>>
>> I am puzzled. I end up with only 75MB for disk buffers, according to
>> top, and I would assume that 25% of memory in this case would be good
>> for disk buffers.
>>
>> I do like using ramdisks for this job, (why write to media unless you
>> have to?) but it seems saner to have the disk cache, caching.
>
> Ah. I assume that 'cached' here means disk buffers. Maybe.
>
>             total       used       free     shared    buffers     cached
> Mem:      32927452   28799604    4127848          0      75600   25122928
> -/+ buffers/cache:    3601076   29326376
> Swap:     33529852    1527668   32002184
>
> Believe me, after doing the number of builds I've done this year, and
> especially in the past two months, finding ways to shave even a few
> minutes more off the build(s) would be a godsend.
>
> This particular box can do 64GB of ram, and doing that would add two
> channels to the memory controller, assuming I plugged the ram in
> wrong...
>
> anyway, a pure kernel build (no kpkg),
>
> time make -j 24
>
> real    7m33.494s
> user    73m3.146s
> sys     6m31.648s
>
> I see from the phoronix benchmarks that they claim a box of this
> caliber can do a kernel build in under 60sec, but I doubt they are
> using a kernel of this size.
>
> I've tossed the kernel .deb files, kernel config, script to make it a
> deb, and patches here:
>
> http://huchra.bufferbloat.net/~d/debloat/
>
> (note - TOTALLY untested on x86_64 as yet -)
>
> I'd gotten out of the habit of maintaining debloat-testing mostly
> because doing a kernel build was taking so bloody long.

And I just did a build right to the ssd, no ramdisk...

real	7m41.516s
user	71m6.395s
sys	6m24.132s

So it looks like, at least at present, with an SSD I/O is not the
bottleneck... Now, from a buildbot perspective I'd really rather not
light up a SSD but use up ram. Although I'm told they have got better.

I still dream of 60s kernel builds tho... hah. the phoronix build test
is available to all...

/me has cpu cycles to burn and is working on something else

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Cerowrt-devel] speeding up builds

[Dave Taht]

On Sun, Apr 29, 2012 at 7:42 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Sun, Apr 29, 2012 at 7:24 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> On Sun, Apr 29, 2012 at 6:59 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>> On Sun, Apr 29, 2012 at 6:42 PM, Outback Dingo <outbackdingo@gmail.com> wrote:
>>>> On Sun, Apr 29, 2012 at 8:15 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>>>> I finally acquired a machine with 32GB of ram, an intel 3930k (6
>>>>> cores), and an SSD.
>>>>>
>>>>> I put the build_dir, /tmp and /var/tmp on ramdisks, and...
>>>>>
>>>>> This cut a complete cerowrt build (including toolchain) down from >
>>>>> 3.5 hrs down to under 45 minutes.
>>>>>
>>>>> Without the toolchain rebuild, but after a make clean (to rebuild the
>>>>> packages and kernel), it's about 28 minutes.
>>>>>
>>>>> I can see that it is possible to parallelize things more to maybe chop
>>>>> another 30% of of things...
>>>>> ...but I'm glad to have 3 hrs of my life back, per build.
>>>>>
>>>>> I wanted to figure out to what extent modern hardware would enhance
>>>>> the existing buildbot system.
>>>>> Now I know...
>>>>
>>>> odd my laptop will do a full build with tool chain in about an
>>>> hour..... its only a core i3 with 6gb and an ssd
>>>
>>> The best box that I had was huchra, a dual quad-core xeon circa 2006,
>>> with 8GB of memory and mirrored drives.
>>>
>>> A 'full build' of cero is 578 packages, some of which are rather big,
>>> as well as building the sdk and cross development kit.
>>>
>>> For comparison purposes, I just built linux-3.3.4 for ubuntu (so this
>>> includes the kpkg overhead)
>>>
>>> real    11m12.286s
>>> user    67m11.076s
>>> sys     7m19.955s
>>>
>>> I am puzzled. I end up with only 75MB for disk buffers, according to
>>> top, and I would assume that 25% of memory in this case would be good
>>> for disk buffers.
>>>
>>> I do like using ramdisks for this job, (why write to media unless you
>>> have to?) but it seems saner to have the disk cache, caching.
>>
>> Ah. I assume that 'cached' here means disk buffers. Maybe.
>>
>>             total       used       free     shared    buffers     cached
>> Mem:      32927452   28799604    4127848          0      75600   25122928
>> -/+ buffers/cache:    3601076   29326376
>> Swap:     33529852    1527668   32002184
>>
>> Believe me, after doing the number of builds I've done this year, and
>> especially in the past two months, finding ways to shave even a few
>> minutes more off the build(s) would be a godsend.
>>
>> This particular box can do 64GB of ram, and doing that would add two
>> channels to the memory controller, assuming I plugged the ram in
>> wrong...
>>
>> anyway, a pure kernel build (no kpkg),
>>
>> time make -j 24
>>
>> real    7m33.494s
>> user    73m3.146s
>> sys     6m31.648s
>>
>> I see from the phoronix benchmarks that they claim a box of this
>> caliber can do a kernel build in under 60sec, but I doubt they are
>> using a kernel of this size.
>>
>> I've tossed the kernel .deb files, kernel config, script to make it a
>> deb, and patches here:
>>
>> http://huchra.bufferbloat.net/~d/debloat/
>>
>> (note - TOTALLY untested on x86_64 as yet -)
>>
>> I'd gotten out of the habit of maintaining debloat-testing mostly
>> because doing a kernel build was taking so bloody long.
>
> And I just did a build right to the ssd, no ramdisk...
>
> real    7m41.516s
> user    71m6.395s
> sys     6m24.132s
>
> So it looks like, at least at present, with an SSD I/O is not the
> bottleneck... Now, from a buildbot perspective I'd really rather not
> light up a SSD but use up ram. Although I'm told they have got better.
>
> I still dream of 60s kernel builds tho... hah. the phoronix build test
> is available to all...
>
> /me has cpu cycles to burn and is working on something else

Phoronix's test suite is pretty neat. 58 seconds for a build with
their kernel config.

phoronix-test-suite benchmark build-linux-kernel

http://openbenchmarking.org/result/1204292-BY-SNAPON08835

So I'm no longer concerned that maybe I goofed on this box. I didn't
realize just how fast the SSD could be, tho, going in.

I could probably do a bit better with a dual socket xeon gulftown,
or maybe with a 48 core opteron, but that drives the price way up.

I used to use things like ccache and dist-cc for stuff like this, too,
but not for embedded, having to have a shared filesystem usually
killed things.


-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

[Cerowrt-devel] Pointers on build setup?

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 2722 bytes --]


Regarding builds - can someone who is building cerowrt from source (including kernel) point me at the recipe for building it completely from scratch?
 
Besides running 3.3-2 on a 3800, I've begun my project to validate that Cerowrt can run on my ubnt RouterStation Pro box, which is preliminary to doing some experimentation on dense scalability that interests me.   I verified that it runs the latest stable Backfire release of OpenWRT this weekend, and it does so quite well.
 
My own private home build server is (essentially) a cluster of two AMD8160 8-core 32 GB servers connected with 40 Gb/s Infiniband and a 6-way RAID5 storage array on one of them that gives quite nice read performance.  I'm not worried about how long it takes to do a build...
 
Anyway - the recipe doesn't have to be perfected for me to use it - and I suspect it's similar to OpenWRT's build, so maybe pointing me to the best help for that and what the differences are would be great.
 
I promise I will write the "port" experience up and post it on the Wiki, in case others want to port cerowrt to other "OpenWRT platforms" of sufficient capabilities.
 
David
 
-----Original Message-----
From: "Outback Dingo" <outbackdingo@gmail.com>
Sent: Sunday, April 29, 2012 9:42pm
To: "Dave Taht" <dave.taht@gmail.com>
Cc: cerowrt-devel@lists.bufferbloat.net
Subject: Re: [Cerowrt-devel] speeding up builds



On Sun, Apr 29, 2012 at 8:15 PM, Dave Taht <dave.taht@gmail.com> wrote:
> I finally acquired a machine with 32GB of ram, an intel 3930k (6
> cores), and an SSD.
>
> I put the build_dir, /tmp and /var/tmp on ramdisks, and...
>
> This cut a complete cerowrt build (including toolchain) down from >
> 3.5 hrs down to under 45 minutes.
>
> Without the toolchain rebuild, but after a make clean (to rebuild the
> packages and kernel), it's about 28 minutes.
>
> I can see that it is possible to parallelize things more to maybe chop
> another 30% of of things...
> ...but I'm glad to have 3 hrs of my life back, per build.
>
> I wanted to figure out to what extent modern hardware would enhance
> the existing buildbot system.
> Now I know...

odd my laptop will do a full build with tool chain in about an
hour..... its only a core i3 with 6gb and an ssd

>
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://www.bufferbloat.net
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

[-- Attachment #2: Type: text/html, Size: 3613 bytes --]

Re: [Cerowrt-devel] Pointers on build setup?

[Dave Taht]

On Mon, Apr 30, 2012 at 8:14 AM,  <dpreed@reed.com> wrote:
> Regarding builds - can someone who is building cerowrt from source
> (including kernel) point me at the recipe for building it completely from
> scratch?

We have most of a build script that has worked at various points in time,
at the moment, I doubt it's correct. Ketan posted some mods recently...

as things stand today, it's kind of fiddly, and requires multiple attempts
at getting the .config right, there's a few tool dependencies, etc.

As soon as I hit feature complete - and had hardware that could turn
around builds fast enough - I'd planned to fix that up.

In the interim...

I can get a correct build setup for you in 10 minutes, if you give me a login.
Certainly having a script that 'just works', and is a one liner, has
been a goal, once things get more stable. Which I would hope be late
this week. Hopefully.

I note I have not checked in all the 'fixes' to the ipv6 alignment
issue (bugs 360, 371, 379) in the past week, as alternatives are being
discussed on the openwrt-devel list.

> Besides running 3.3-2 on a 3800, I've begun my project to validate that
> Cerowrt can run on my ubnt RouterStation Pro box, which is preliminary to
> doing some experimentation on dense scalability that interests me.   I

If you are talking about what I call the 'dense mesh' problem, the predecessor
project to cerowrt had it's original genesis in trying to solve the
meltdown issues olpc had with 802.11s.

"Diversity routing" seemed to be an answer.

www.pps.jussieu.fr/~jch/software/babel/wbmv4.pdf

And the routerstation pro (3 radios! ooh!) is a perfect platform to be
trying that on.
Juliusz was looking into doing experiments on that hardware...

Sadly it shares the unaligned ipv6 problem that we've been struggling to solve,
and the max thruput is less than what those 3 radios could achieve.

> verified that it runs the latest stable Backfire release of OpenWRT this
> weekend, and it does so quite well.
>
>
>
> My own private home build server is (essentially) a cluster of two AMD8160
> 8-core 32 GB servers connected with 40 Gb/s Infiniband and a 6-way RAID5
> storage array on one of them that gives quite nice read performance.  I'm
> not worried about how long it takes to do a build...

I will note that SSDs really rock, and seek times of ~0 do wonders for
the link step.

That said, I was proud of my new hardware, until I read that. :/

>
>
>
> Anyway - the recipe doesn't have to be perfected for me to use it - and I
> suspect it's similar to OpenWRT's build, so maybe pointing me to the best
> help for that and what the differences are would be great.

From a basic perspective, you have additional feeds in feeds.conf,
a specialized config, a special filesystem that does a few different
things - notably changing the device naming so you can have multiple
routed interfaces and simplified firewall rules -

So you start by checking out cerofiles. The build_cero.sh script there
is obsolete but can supply clue as to the repos required and ketan had
some patches to that.

What I do nowadays is mildly different than the script, in that after
the repos are pulled down, feeds.conf is configured, scripts/env new
whatever has been run and cerofiles imported into it, I tend to hammer
at it with

./scripts/feeds install -p cero `cat env/override.list`
./scripts/feeds install `cat env/packages.list`
./scripts/feeds install -p cero `cat env/override.list`

cp env/config-* .config
make defconfig
grep =m .config | sort -u > t1.txt
grep =m env/config* | sort -u > t2.txt

diff t1.txt t2.txt

add the packages via feeds install (I'm always adding new ones)

the script ALMOST does all that.


(or I can can get it going for you in 10 minutes)

>
>
>
> I promise I will write the "port" experience up and post it on the Wiki, in
> case others want to port cerowrt to other "OpenWRT platforms" of sufficient
> capabilities.
>
>
>
> David
>
>
>
> -----Original Message-----
> From: "Outback Dingo" <outbackdingo@gmail.com>
> Sent: Sunday, April 29, 2012 9:42pm
> To: "Dave Taht" <dave.taht@gmail.com>
> Cc: cerowrt-devel@lists.bufferbloat.net
> Subject: Re: [Cerowrt-devel] speeding up builds
>
> On Sun, Apr 29, 2012 at 8:15 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> I finally acquired a machine with 32GB of ram, an intel 3930k (6
>> cores), and an SSD.
>>
>> I put the build_dir, /tmp and /var/tmp on ramdisks, and...
>>
>> This cut a complete cerowrt build (including toolchain) down from >
>> 3.5 hrs down to under 45 minutes.
>>
>> Without the toolchain rebuild, but after a make clean (to rebuild the
>> packages and kernel), it's about 28 minutes.
>>
>> I can see that it is possible to parallelize things more to maybe chop
>> another 30% of of things...
>> ...but I'm glad to have 3 hrs of my life back, per build.
>>
>> I wanted to figure out to what extent modern hardware would enhance
>> the existing buildbot system.
>> Now I know...
>
> odd my laptop will do a full build with tool chain in about an
> hour..... its only a core i3 with 6gb and an ssd
>
>>
>>
>> --
>> Dave Täht
>> SKYPE: davetaht
>> US Tel: 1-239-829-5608
>> http://www.bufferbloat.net
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Cerowrt-devel] Pointers on build setup?

[Dave Taht]

On Mon, Apr 30, 2012 at 8:50 AM, Dave Taht <dave.taht@gmail.com> wrote:
> On Mon, Apr 30, 2012 at 8:14 AM,  <dpreed@reed.com> wrote:
>> Regarding builds - can someone who is building cerowrt from source
>> (including kernel) point me at the recipe for building it completely from
>> scratch?

I have cleaned up build_cero.sh and corrected

http://www.bufferbloat.net/projects/cerowrt/wiki/Building_Cerowrt_on_your_own_machine

As noted elsewhere I have not pushed out the last weeks worth of changes yet.
(if you merge from openwrt head, things will break)

In the long run there will be a packages-3.3 on a github repo, for stability.



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Cerowrt-devel] Pointers on build setup?

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 8117 bytes --]


I'll see about setting a login up for you - it's a bit complicated because the cluster is not exposed at the moment, so I need to secure it and expose it.  I don't want to distract from feature completion...  so it may be a few days (given that I'm busy during the week mostly with my day job at SAP, which involves systems engineering experiments with *really big* computing systems - my personal research toy there is 400 Westmere-EX cores with 5 TB of RAM, and 30 TB of PCIe Flash, and that's on the small end.... my team runs several other, larger systems of the same sort for "production" research  :-)  It runs Linux, of course. )
 
My focus is related to dense mesh stuff, because scalable urban/mobile usage is the big wall for radio internetworking after IPv6 and controlled latency (bufferbloat).  But actually I'm not trying to solve the whole "decentralized mesh" problem - what I'm trying to do is do propagation monitoring and modeling in real home environments on all 2.4 and 5 GHz channels (and maybe create a tool for that).  It's my opinion that we need to understand a lot more about real-world physics *as seen by real radios*, and without that, we can't really design protocols for adaptive, cooperative use of these license-by-rule bands.
 
The physics problem is one that may never be addressable by academic researchers - they don't have the ability to reach into homes, apartments, businesses, etc. to do the experiments.  But an open, modern home router that anyone can buy for at most a couple hundred bucks can provide that platform.  It's my "Internet 3 experiment".  I firmly believe you need to do systems engineering experiments before designing answers.  I can't use smartphones or commercial home routers to do this.
 
Of course, once I've created a way to measure real-time propagation, the next step will be to create cooperation/adaptatation protocols (not just routing, but the complement of routing - where not to transmit given the physical environment, kind of "real-time radio engineering" to maximize utilization among the systems).
 
David
 
 
-----Original Message-----
From: "Dave Taht" <dave.taht@gmail.com>
Sent: Monday, April 30, 2012 11:50am
To: dpreed@reed.com
Cc: "Outback Dingo" <outbackdingo@gmail.com>, cerowrt-devel@lists.bufferbloat.net
Subject: Re: Pointers on build setup?



On Mon, Apr 30, 2012 at 8:14 AM,  <dpreed@reed.com> wrote:
> Regarding builds - can someone who is building cerowrt from source
> (including kernel) point me at the recipe for building it completely from
> scratch?

We have most of a build script that has worked at various points in time,
at the moment, I doubt it's correct. Ketan posted some mods recently...

as things stand today, it's kind of fiddly, and requires multiple attempts
at getting the .config right, there's a few tool dependencies, etc.

As soon as I hit feature complete - and had hardware that could turn
around builds fast enough - I'd planned to fix that up.

In the interim...

I can get a correct build setup for you in 10 minutes, if you give me a login.
Certainly having a script that 'just works', and is a one liner, has
been a goal, once things get more stable. Which I would hope be late
this week. Hopefully.

I note I have not checked in all the 'fixes' to the ipv6 alignment
issue (bugs 360, 371, 379) in the past week, as alternatives are being
discussed on the openwrt-devel list.

> Besides running 3.3-2 on a 3800, I've begun my project to validate that
> Cerowrt can run on my ubnt RouterStation Pro box, which is preliminary to
> doing some experimentation on dense scalability that interests me.   I

If you are talking about what I call the 'dense mesh' problem, the predecessor
project to cerowrt had it's original genesis in trying to solve the
meltdown issues olpc had with 802.11s.

"Diversity routing" seemed to be an answer.

www.pps.jussieu.fr/~jch/software/babel/wbmv4.pdf

And the routerstation pro (3 radios! ooh!) is a perfect platform to be
trying that on.
Juliusz was looking into doing experiments on that hardware...

Sadly it shares the unaligned ipv6 problem that we've been struggling to solve,
and the max thruput is less than what those 3 radios could achieve.

> verified that it runs the latest stable Backfire release of OpenWRT this
> weekend, and it does so quite well.
>
>
>
> My own private home build server is (essentially) a cluster of two AMD8160
> 8-core 32 GB servers connected with 40 Gb/s Infiniband and a 6-way RAID5
> storage array on one of them that gives quite nice read performance.  I'm
> not worried about how long it takes to do a build...

I will note that SSDs really rock, and seek times of ~0 do wonders for
the link step.

That said, I was proud of my new hardware, until I read that. :/

>
>
>
> Anyway - the recipe doesn't have to be perfected for me to use it - and I
> suspect it's similar to OpenWRT's build, so maybe pointing me to the best
> help for that and what the differences are would be great.

From a basic perspective, you have additional feeds in feeds.conf,
a specialized config, a special filesystem that does a few different
things - notably changing the device naming so you can have multiple
routed interfaces and simplified firewall rules -

So you start by checking out cerofiles. The build_cero.sh script there
is obsolete but can supply clue as to the repos required and ketan had
some patches to that.

What I do nowadays is mildly different than the script, in that after
the repos are pulled down, feeds.conf is configured, scripts/env new
whatever has been run and cerofiles imported into it, I tend to hammer
at it with

./scripts/feeds install -p cero `cat env/override.list`
./scripts/feeds install `cat env/packages.list`
./scripts/feeds install -p cero `cat env/override.list`

cp env/config-* .config
make defconfig
grep =m .config | sort -u > t1.txt
grep =m env/config* | sort -u > t2.txt

diff t1.txt t2.txt

add the packages via feeds install (I'm always adding new ones)

the script ALMOST does all that.


(or I can can get it going for you in 10 minutes)

>
>
>
> I promise I will write the "port" experience up and post it on the Wiki, in
> case others want to port cerowrt to other "OpenWRT platforms" of sufficient
> capabilities.
>
>
>
> David
>
>
>
> -----Original Message-----
> From: "Outback Dingo" <outbackdingo@gmail.com>
> Sent: Sunday, April 29, 2012 9:42pm
> To: "Dave Taht" <dave.taht@gmail.com>
> Cc: cerowrt-devel@lists.bufferbloat.net
> Subject: Re: [Cerowrt-devel] speeding up builds
>
> On Sun, Apr 29, 2012 at 8:15 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> I finally acquired a machine with 32GB of ram, an intel 3930k (6
>> cores), and an SSD.
>>
>> I put the build_dir, /tmp and /var/tmp on ramdisks, and...
>>
>> This cut a complete cerowrt build (including toolchain) down from >
>> 3.5 hrs down to under 45 minutes.
>>
>> Without the toolchain rebuild, but after a make clean (to rebuild the
>> packages and kernel), it's about 28 minutes.
>>
>> I can see that it is possible to parallelize things more to maybe chop
>> another 30% of of things...
>> ...but I'm glad to have 3 hrs of my life back, per build.
>>
>> I wanted to figure out to what extent modern hardware would enhance
>> the existing buildbot system.
>> Now I know...
>
> odd my laptop will do a full build with tool chain in about an
> hour..... its only a core i3 with 6gb and an ssd
>
>>
>>
>> --
>> Dave Täht
>> SKYPE: davetaht
>> US Tel: 1-239-829-5608
>> http://www.bufferbloat.net
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

[-- Attachment #2: Type: text/html, Size: 9788 bytes --]

Re: [Cerowrt-devel] Pointers on build setup?

[Maciej Soltysiak]

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

Hi Dave,

I'm following the cleaned up "Building_Cerowrt_on_your_own_machine" and I
only edited one line that wasn't right about cd'ing into a directory.
It's compiling now, will leave it for the night.

However, can you tell me, how far away will I be from the builds that you
are putting up for download if I stick to the instruction and not change a
single .config entry or add/change/remove a package?

Regards,
Maciej

On Mon, Apr 30, 2012 at 6:50 PM, Dave Taht <dave.taht@gmail.com> wrote:

> On Mon, Apr 30, 2012 at 8:50 AM, Dave Taht <dave.taht@gmail.com> wrote:
> > On Mon, Apr 30, 2012 at 8:14 AM,  <dpreed@reed.com> wrote:
> >> Regarding builds - can someone who is building cerowrt from source
> >> (including kernel) point me at the recipe for building it completely
> from
> >> scratch?
>
> I have cleaned up build_cero.sh and corrected
>
>
> http://www.bufferbloat.net/projects/cerowrt/wiki/Building_Cerowrt_on_your_own_machine
>
> As noted elsewhere I have not pushed out the last weeks worth of changes
> yet.
> (if you merge from openwrt head, things will break)
>
> In the long run there will be a packages-3.3 on a github repo, for
> stability.
>
>
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://www.bufferbloat.net
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 2361 bytes --]

Re: [Cerowrt-devel] Pointers on build setup?

[Dave Taht]

On Mon, Apr 30, 2012 at 4:32 PM, Maciej Soltysiak <maciej@soltysiak.com> wrote:
> Hi Dave,
>
> I'm following the cleaned up "Building_Cerowrt_on_your_own_machine" and I
> only edited one line that wasn't right about cd'ing into a directory.


> It's compiling now, will leave it for the night.

I have found that on the 2nd build or later at the moment I have to
run this script due to some flaws in dependency generation (note I
have 6 dual threaded cores)

d@snapon:/home/cero1/src/Cerowrt-3.3$ cat buildit.sh
#!/bin/bash

make -j 12 clean
make -j 12
make V=s
make -j 12 package/e2fsprogs/{clean,compile,install}
make -j 12 package/krb5/{clean,compile,install}
make -j 12 package/pcre/{clean,compile,install}
make -j 12

It might work if you aren't doing -j12
>
> However, can you tell me, how far away will I be from the builds that you
> are putting up for download if I stick to the instruction and not change a
> single .config entry or add/change/remove a package?

aside from the openwrt package set, you'll be 3.3.2-something. I think
the current build was tagged with that. Whenever I get something that
I'm happy with I tag it.

Hopefully in the next week or so it should be possible to get an exact
duplicate,
or at least match up with 3.3.4-something. I'll need to create a
openwrt-packages-3.3 git repo to hold the stable stuff

And I figure half the joy is in adding your own favorite packages!

>
> Regards,
> Maciej
>
> On Mon, Apr 30, 2012 at 6:50 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> On Mon, Apr 30, 2012 at 8:50 AM, Dave Taht <dave.taht@gmail.com> wrote:
>> > On Mon, Apr 30, 2012 at 8:14 AM,  <dpreed@reed.com> wrote:
>> >> Regarding builds - can someone who is building cerowrt from source
>> >> (including kernel) point me at the recipe for building it completely
>> >> from
>> >> scratch?
>>
>> I have cleaned up build_cero.sh and corrected
>>
>>
>> http://www.bufferbloat.net/projects/cerowrt/wiki/Building_Cerowrt_on_your_own_machine
>>
>> As noted elsewhere I have not pushed out the last weeks worth of changes
>> yet.
>> (if you merge from openwrt head, things will break)
>>
>> In the long run there will be a packages-3.3 on a github repo, for
>> stability.
>>
>>
>>
>> --
>> Dave Täht
>> SKYPE: davetaht
>> US Tel: 1-239-829-5608
>> http://www.bufferbloat.net
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net