[Bismark-devel] some thoughts on continuous builds

[Bismark-devel] some thoughts on continuous builds

[Dave Taht]

The advantage of a continuous build and test system is that you catch
new bugs early, as they occur in your external trees, and also fold in
fixes to existing bugs and new features quickly.

A disadvantage is that new bugs occur and invalidate previous testing.

At some point in the near future we will need to freeze bismark,
starting with the kernel and core utilities, do extensive QA, and
ship. Until then, builds should be done weekly (preferably daily)

Up until now I have been doing pulls and builds on a rather arbitrary schedule.

In my judgement there are still sufficient bugs outstanding

http://www.bufferbloat.net/projects/bismark/issues

to be doing continuous builds but I haven't been in the loop this week
(I will be more available after may 1)

So I would like to establish a process whereby we have a trigger point
for doing a new major build, and trigger points for new minor ones.

One nice thing about having multiple builders is that it's possible to
at least test the new stuff before committing to using it on other
people.

For example, last night, on my own tree, in my home dir, I did the usual

cd ~/src/openwrt; git pull

# looked over the logs with git log, saw a few interesting changes

cd ~/src/wndr370

./scripts/feeds update # this updates the external packages and rarely
breaks anything
git pull # in my case this pulls from my private copy of openwrt
make menuconfig # and save. you have to do this fairly often as
options do change
echo make -j 8 | batch

And the latest and greatest stuff did, indeed, build. Now, (due to me
not paying attention this week) I am still a bit out of sync with
what's going on in the core ~bismark tree, and don't have the hardware
to test against right now, but I have a reasonable confidence level
that the main trees could be updated and built against.

So, anyway, whenever y'all get to a good "do a complete build" point,
please do so? The pain you incur now will be saved later.

(I hope to be testing a new build w/hardware saturday)

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

Re: [Bismark-devel] some thoughts on continuous builds

[Nick Feamster]

Hi Dave,

On the topic of builds, I met with some of the OpenFlow folks at Stanford yesterday.  We are going to try to see if we can get this also built as a package:
http://www.openflow.org/wk/index.php/OpenFlow_1.0_for_OpenWRT

at which point, we will see if we can integrate it into the build, as well.

Are you saying below that we should be doing continuous testing and builds, so that we can be certain that the current build runs on the NetGear hardware?  That seems like a good idea.

-Nick

On Apr 26, 2011, at 8:35 AM, Dave Taht wrote:

> The advantage of a continuous build and test system is that you catch
> new bugs early, as they occur in your external trees, and also fold in
> fixes to existing bugs and new features quickly.
> 
> A disadvantage is that new bugs occur and invalidate previous testing.
> 
> At some point in the near future we will need to freeze bismark,
> starting with the kernel and core utilities, do extensive QA, and
> ship. Until then, builds should be done weekly (preferably daily)
> 
> Up until now I have been doing pulls and builds on a rather arbitrary schedule.
> 
> In my judgement there are still sufficient bugs outstanding
> 
> http://www.bufferbloat.net/projects/bismark/issues
> 
> to be doing continuous builds but I haven't been in the loop this week
> (I will be more available after may 1)
> 
> So I would like to establish a process whereby we have a trigger point
> for doing a new major build, and trigger points for new minor ones.
> 
> One nice thing about having multiple builders is that it's possible to
> at least test the new stuff before committing to using it on other
> people.
> 
> For example, last night, on my own tree, in my home dir, I did the usual
> 
> cd ~/src/openwrt; git pull
> 
> # looked over the logs with git log, saw a few interesting changes
> 
> cd ~/src/wndr370
> 
> ./scripts/feeds update # this updates the external packages and rarely
> breaks anything
> git pull # in my case this pulls from my private copy of openwrt
> make menuconfig # and save. you have to do this fairly often as
> options do change
> echo make -j 8 | batch
> 
> And the latest and greatest stuff did, indeed, build. Now, (due to me
> not paying attention this week) I am still a bit out of sync with
> what's going on in the core ~bismark tree, and don't have the hardware
> to test against right now, but I have a reasonable confidence level
> that the main trees could be updated and built against.
> 
> So, anyway, whenever y'all get to a good "do a complete build" point,
> please do so? The pain you incur now will be saved later.
> 
> (I hope to be testing a new build w/hardware saturday)
> 
> -- 
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://the-edge.blogspot.com
> _______________________________________________
> Bismark-devel mailing list
> Bismark-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bismark-devel

Re: [Bismark-devel] some thoughts on continuous builds

[Dave Taht]

On Tue, Apr 26, 2011 at 9:37 AM, Nick Feamster <feamster@cc.gatech.edu> wrote:
> Hi Dave,
>
> On the topic of builds, I met with some of the OpenFlow folks at Stanford yesterday.

I would enjoy meeting with them myself, and will be in California
through at least Wednesday of next week.

We are going to try to see if we can get this also built as a package:
> http://www.openflow.org/wk/index.php/OpenFlow_1.0_for_OpenWRT

Excellent.

> at which point, we will see if we can integrate it into the build, as well.

Openflow sounds invasive but I will defer judgement until I get up to speed.

>
> Are you saying below that we should be doing continuous testing and builds, so that we can be certain that the current build runs on the NetGear hardware?

Yes.

> That seems like a good idea.

It's the best way to manage complexity that I know of.

It is gated by certain things - Doing a major build the day of
planning to also distribute a dozen routers to testers can be
stressful... as is doing it during a major test cycle with those
testers.

That said, it's a good idea for the developers to be working against
the most current trees as possible as often as possible, and the main
tree be updated well prior to doing a big rollout to the testers.

In addition to the somewhat manual, gated by several external factors,
mostly volunteer and "awareness" process I've described thus far, we
have two different automatic continuous build systems under evaluation
- buildbot (which is what the openwrt project uses throughout) and the
newer, sexier "jenkins". That will take care of validating that the
core and packages actually build... but a goal should be to keep the
various build cycles of the components SHORT so that it is easy to
spot and fix where problems occur.

(drawing a picture of how the cycles should work would probably help
more than this textual description)

> -Nick
>
> On Apr 26, 2011, at 8:35 AM, Dave Taht wrote:
>
>> The advantage of a continuous build and test system is that you catch
>> new bugs early, as they occur in your external trees, and also fold in
>> fixes to existing bugs and new features quickly.
>>
>> A disadvantage is that new bugs occur and invalidate previous testing.
>>
>> At some point in the near future we will need to freeze bismark,
>> starting with the kernel and core utilities, do extensive QA, and
>> ship. Until then, builds should be done weekly (preferably daily)
>>
>> Up until now I have been doing pulls and builds on a rather arbitrary schedule.
>>
>> In my judgement there are still sufficient bugs outstanding
>>
>> http://www.bufferbloat.net/projects/bismark/issues
>>
>> to be doing continuous builds but I haven't been in the loop this week
>> (I will be more available after may 1)
>>
>> So I would like to establish a process whereby we have a trigger point
>> for doing a new major build, and trigger points for new minor ones.
>>
>> One nice thing about having multiple builders is that it's possible to
>> at least test the new stuff before committing to using it on other
>> people.
>>
>> For example, last night, on my own tree, in my home dir, I did the usual
>>
>> cd ~/src/openwrt; git pull
>>
>> # looked over the logs with git log, saw a few interesting changes
>>
>> cd ~/src/wndr370
>>
>> ./scripts/feeds update # this updates the external packages and rarely
>> breaks anything
>> git pull # in my case this pulls from my private copy of openwrt
>> make menuconfig # and save. you have to do this fairly often as
>> options do change
>> echo make -j 8 | batch
>>
>> And the latest and greatest stuff did, indeed, build. Now, (due to me
>> not paying attention this week) I am still a bit out of sync with
>> what's going on in the core ~bismark tree, and don't have the hardware
>> to test against right now, but I have a reasonable confidence level
>> that the main trees could be updated and built against.
>>
>> So, anyway, whenever y'all get to a good "do a complete build" point,
>> please do so? The pain you incur now will be saved later.
>>
>> (I hope to be testing a new build w/hardware saturday)
>>
>> --
>> Dave Täht
>> SKYPE: davetaht
>> US Tel: 1-239-829-5608
>> http://the-edge.blogspot.com
>> _______________________________________________
>> Bismark-devel mailing list
>> Bismark-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bismark-devel
>
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

Re: [Bismark-devel] some thoughts on continuous builds

[Dave Taht]

An example of a bug that I would like to see isolated and fixed before
the next big development/Test cycle is:

http://www.bufferbloat.net/issues/88

As it would make it simpler to reflash devices with entirely new
firmware with a good backup.

Can someone take ownership of this one? Determine if it still exists,
and if it does, further bug report it to the openwrt folk at the very
least?

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com