Support for build and test on different systems/platforms

[trounds@…]

We are using Bitten for continuous integration of a Linux-based appliance which runs an embedded Linux on an ARM processor. The appliance has no disk, compilers, and is useless as a build system.

As a result, we are running Bitten on an ordinary PC (Ubuntu 9.04) with an ARM cross-compiler. This works great for building, and has been a huge help to our project, but does not address our testing needs.

Bitten is a wonderful tool, but the assumption that code will be tested on the build machine leaves embedded-systems projects behind. For our project, it can only be used for automated builds in its current form.

Partway through an effort to split build and test such that they can be run by separate bitten-slaves, it became apparent that this is better done by (or at least coordinated with) the Bitten team itself, so I won't have to face a merging nightmare every time I upgrade, particularly when it comes to schema changes.

I am perfectly happy to contribute to any such effort, but this ticket is mostly to make sure it's brought to the attention of the Bitten team, since a search of the ticket system yielded no similar tickets.

[anonymous]

Couldn't you have the slave machine farm the tests out to the embedded machine and then copy the result files back to the slave once the test are done?

If you could outline the approached taken by your patch we can talk about how best to incorporate it into Bitten.

[trounds@…]

Replying to anonymous:

Couldn't you have the slave machine farm the tests out to the embedded machine and then copy the result files back to the slave once the test are done?

If you could outline the approached taken by your patch we can talk about how best to incorporate it into Bitten.

Unfortunately, no. Since the OS is part of what's being built, there needs to be logic running on the device to pull the image and boot it, then start the testing.

Testing is being done, but there's really no way to get the results back to Bitten for processing without having the build machine sit around waiting for the test system, which isn't very efficient.

Essentially, the approach I was trying to take was to expand the phasing mechanism (in this case: build phase, install phase, test phase). Bitten slaves would report their role (build/install/test) in their properties. The Bitten master would keep track of a revision's progress from one phase to the next (I was going to expand IBuildListener for this), and farm the recipes out to the slaves as work progressed. Once the final phase is completed, the final reports can be compiled and published.

This would not only make cross-platform implementation a lot easier, it would also help with single-platform projects with long build or test times on days with heavy code churn. Builds could continue while testing takes place. The option to split build and test between different systems can come in handy in many ways.

None of this can be done without a schema change, changes to (at least) slave, master, web_ui, queue, api, and admin; which is why I think it would be a bad idea for me to just make this modification locally.

Besides, I've only been working with Bitten for nine months or so, and am not familiar enough with the code to know how to implement this in a way that would work best with the rest of the system, and also be backwards-compatible.

At the very least, I'm far better off discussing the design with the team, and offering it as a potential future enhancement.

[anonymous]

It seems that the main issue is really the build slave getting stuck waiting for long running tests (possibly on embedded hardware somewhere)? What about just running multiple slaves? A cronjob which periodically creates a slave and has it perform 0 or 1 builds should work.

Perhaps it might be a good idea to move this discussion onto the mailing list -- it will probably get more readers there.

[Aside: I don't think I've been working seriously with Bitten much longer than about nine months either. :)]

[trounds@…]

Replying to anonymous:

It seems that the main issue is really the build slave getting stuck waiting for long running tests (possibly on embedded hardware somewhere)? What about just running multiple slaves? A cronjob which periodically creates a slave and has it perform 0 or 1 builds should work.

The purpose of filing a ticket was not because I need help figuring out how to get testing done. That's working just fine, as I've said. I already have a workaround.

The issue is support for multiple slaves on multiple machines taking turns with the same build, and aggregating the output from all of them into one final result set. It's the association between the slaves that's missing. If I go the multiple-slave route with the code as it is, each phase of build, install, and test will look like three separate builds in the output.

Perhaps it might be a good idea to move this discussion onto the mailing list -- it will probably get more readers there.

Agreed.

[anonymous]

One multi-slave suggestion has one slave per-build, not one slave per-phase. The time line might look something like:

• Slave process one starts build #123, gets to install / test step and pauses waiting for it to complete.
• Slave process two starts build #124, gets to install / test step and pauses waiting for it to complete.
• Slave process three starts build #125, gets to install / test step and pauses waiting for it to complete.
• Time passes.
• Slave process one completes build #123.
• Slave process two completes build #124.
• Slave process three completes build #125.

I'm really not seeing what is so fundamentally missing from Bitten's build step architecture that you would need to massively change it to suit your goals.

[trounds@…]

What's missing from Bitten's build step architecture is the ability for different steps to be run by different slaves. That's it, really. The assumption is that testing will be run on the same machine the code is built on, and in many projects that's not an option.

Building and testing are performed on different systems, and "pushing" a build to a test system is not an option in this case.

I don't want the build machine to sit around waiting for the tests to complete on another machine. That's not necessary, and would really slow things down.

The workflow I'm looking for is:

• A revision is checked in to source control
• Bitten-master farms the build recipe out to a slave designated for building
• Build completes successfully
• Bitten-master farms out the install recipe to the test machine, to a slave designated as an installer
• Test machine grabs build, installs, signals completion, and reboots
• Bitten-master farms out the test recipe to the test machine, to a slave designated as a tester
• Test results are sent back to bitten-master
• Results, reports, and the timeline look exactly the same for me as they would for any other Bitten user.

The slave on the test machine would have two roles, but role-switching would be automated externally to Bitten.

There are some limitations in this project that need to be better explained, here:

• There is only one build machine, and many test systems
• Some of the compilers are licensed, and only one instance can be running at a time (which is why there's only one build machine)
• The test system is not a PC, it's another kind of device which runs an embedded Linux
• The test systems have no disk, only network
• Every revision is built and tested, and some days have many revisions

If I have to go hunting around three different output pages to see the outcome of one revision, then I'll have exactly what I already have now. I only want to be able to use the output-collection and formatting features in Bitten that everyone else can use, without having our one build machine sitting idle unnecessarily when there's work to be done.

I wouldn't have filed this ticket if I hadn't already exhausted all my other options.

[hodgestar]

What's missing from Bitten's build step architecture is the ability for different steps to be run by different slaves.

As I have explained in some detail this is not necessary to achieve your stated aims.

I don't want the build machine to sit around waiting for the tests to complete on another machine.

In my suggestion, processes wait, not machines. The build machine will be working full-tilt as long as there is work to do.

Building and testing are performed on different systems, and "pushing" a build to a test system is not an option in this case.

If the test systems have to pull the build from somewhere they can just as easily pull it from the build machine as from the Bitten master.

Some of the compilers are licensed, and only one instance can be running at a time (which is why there's only one build machine)

There's no reason why multiple slaves couldn't all wait their turn to use the compiler.

If I have to go hunting around three different output pages to see the outcome of one revision, then I'll have exactly what I already have now.

My suggested solution does not suffer from this problem.

I wouldn't have filed this ticket if I hadn't already exhausted all my other options.

I'm trying to provide more options.

If you have current patches to Bitten I'm more than happy to review them. My gut feeling is that it's going to be tricky to get a clean implementation of this in a generic way but I'm always open to being proved wrong.

While thinking about this ticket I've realised that a check-pointing feature for build recipes might be quite cool -- i.e. some way for a recipe to declare a set of data needed to resume a build recipe from a particular step.

[osimons]

Tickets #90 and #412 requests features that may share some common ground with this ticket.