When the build for *one* revision hangs, the slave hangs forever, and builds aren't triggered anymore

[edgewall.org@…]

Someone in our teams scr*wed up and committed some tests that are hanging the build. We corrected after a few commits, but now, when the slaves do the build, they hang when they get to build the revisions that hang, and we have to restart them by hand when there's a new revision to build...

Unchecking "Build all revisions" doesn't seem to change any of this behavior. Actually, This option isn't really doing what I would expect: it should be called "Trigger a build for every commit, even if it is not on the path for the configuration" or something like that. It would be good that there's a "Only build latest revision" option.

The problem is worse for me, as I build two different configurations: and Bitten tries to build *all the revisions* for one of the configurations, before building the other. Since the slaves hang at some of the revisions for the first configurations, the second configuration is never built - for any revision. I cannot get this configuration to be built, at all.

I'm currently trying to patch 'queue.py', to simply skip the builds that are causing trouble:

queue.

@@ -221 +221 @@
             platforms = []
             for platform, rev, build in collect_changes(repos, config, db):
 
-                if rev > 1710 and rev < 1726:
-                    continue
-
                 if not self.build_all and platform.id in platforms:
                     # We've seen this platform already, so these are older
                     # builds that should only be built if built_all=True

I think it will work, but there may be a cleaner way of doing it ?

The *slave* code should have a way to stop a build if it doesn't finish before the timeout defined in the admin (currently this timeout is only used on the master). I've looked at the code in the slave, it doesn't seem too difficult to implement a control thread that would stop the build. However, i'm not familiar enough with threading in Python to do it...

[dfraser]

Agreed, I've had lots of trouble with this before. Useful things would be:

• Timeout limit for the slave Even nicer, remember the previous execution times and adjust the timeout limit to be N*previous-longest-successful-execution
• The ability to mark certain revisions as not-for-testing (perhaps only on certain platforms)

[wbell]

I don't like the idea of doing a heuristic of how long a build should take-- we have many build slaves of differing speeds, and some slaves take 12 hours for builds that others only take 9.

Anytime a slave stops running a build (as far as the master is concerned), it should make an effort to stop building it, so as not to stay stuck orphaned. One consequence of the current timeout behavior is that if your build does hang and exceeds the timeout, the master happily invalidates it, and assigns it to another slave. The slave processing it continues to run the build (hanging), and a new slave starts, and eventually will hang to repeat the process until all slaves are running the same build, but none are shown as running it as far as the master is concerned.

[osimons]

(Fixed diff formatting in description)

[wbell]

Closed with [830]