Ability to configure build priorities

[mpotter@…]

Having just finished a session of disabling different configurations and adding "Meta.disable ~= 1" rules to different platforms to get things to build in the order desired:

Add the ability to configure priorities for what order the different builds will be performed in.

From what I can tell currently builds are performed in the order (ref​):

• Revision time, most recent first.
• Build Configuration name, alphabetically.

This would mean: Many check-ins on one "branch" could starve another "branch" that had an earlier check-in.

Desired Priority Controls:

• Build Configuration.
• Revision. Either
  • Head - the most recent check-in for the path
  • Other - Another check-in, no the head.
  • Another idea might be:
    • Started - If there has been a build for the check-in in question. Use of this would help keep the statistic more complete. If it started building a given revision try to finish it up before moving on.
• Platform.

Scenario:

• There are four build configurations:
  • Trunk - Builds all the platforms for the trunk
  • branch1 - Builds specific platforms for a specific branch.
  • branch2 - Builds specific platforms for another specific branch.
  • tagAny - Builds all the platforms for any tag. (e.g. #524)

Sample Desired Priority Configuration: Config.Revision.Platform

1. tagAny.*.{Platform1, Platform2, Platform3}
2. tagAny.*.*
3. branch1.Head.*
4. branch2.Head.*
5. Trunk.Head.*

Basically: Tags take priority over branches and branches over trunk. Withing tags, specific platforms take priority over others, no matter what revision. After that the head of the trunk or given branch takes priority over building the old revisions, but branch1 has priority over branch2 over trunk. After that older revisions will be build in reverse order as it currently does.

An additional feature after this (i.e. a new ticket), would be to add buttons (under permission control) which would allow a pending build to have it's priority set to ASAP or Much-Later (first or last).

[Mark Potter <mpotter@…>]

Replying to mpotter@…:

From what I can tell currently builds are performed in the order (ref​):

• Revision time, most recent first.
• Build Configuration name, alphabetically.

Apparently that was the order when I was using SQLite for my database back-end. On changing the back-end to PostgreSQL the "Build Configuration name, alphabetically" became semi-random.

[Mark Potter <mpotter@…>]

Another Scenario: (this real, my current state).

• Branch 603 had a change made.
  • 9 of the 21 targets have finished building.
• Branch 604 then had seven changes made.
  • This branch also has 21 targets.

Therefore Branch 603 has to wait for 147 targets to be built from Branch 604 before any more of 603's targets will be built. This morning it was at around 110, five hours later it is now at 61, therefore it will get to 603 sometime this evening.

The HEAD of any branch should take precedence over back-building.

[Mark Potter <mpotter@…>]

Replying to mpotter@…:

An additional feature after this (i.e. a new ticket), would be to add buttons (under permission control) which would allow a pending build to have it's priority set to ASAP or Much-Later (first or last).

Another Scenario: (yesterday)

• Four different tags were made from three different branches.
• Other changes were made, interspersed, on different branches.
• Was then informed that we need certain targets from these different tags ASAP. In priority order:
  • From tag 1: Targets 1, 2, and 3.
  • From tag 3: Target 4.
  • From tag 4: Everything.
  • From tag 2: Nothing, the tag was made by mistake.

I spent most of the rest of the day manipulating the configurations in order to get the requested targets ASAP:

• Adding and removing "Disabled ~= 1" to different targets and changing the "Last Revision" to the Tag Any configurations.
• Temporarily disabling two Branch configurations.

An ASAP button would have greatly simplified this issue.

Tags, both head and back-building, should have higher precedence then all the other builds.

[Mark Potter <mpotter@…>]

Replying to mpotter@…:

An additional feature after this (i.e. a new ticket), would be to add buttons (under permission control) which would allow a pending build to have it's priority set to ASAP or Much-Later (first or last).

Filed as #832.

[Mark Potter <mpotter@…>]

Replying to Mark Potter <mpotter@…>:

Therefore Branch 603 has to wait for 147 targets to be built from Branch 604 before any more of 603's targets will be built. This morning it was at around 110, five hours later it is now at 61, therefore it will get to 603 sometime this evening.

It is now the next morning. Yesterday 6 additional check-ins where made in various branches and the trunk and 1 additional check-in this morning, thus approximately 175 new builds were all scheduled with a higher priority than those in the 603 branch HEAD. As of right now, the 603 branch's HEAD still has not been built.

The HEAD of any branch should take precedence over back-building.

[Mark Potter <mpotter@…>]

Replying to mpotter@…:

Desired Priority Controls:

• Build Configuration.
• Revision. Either
  • Head - the most recent check-in for the path
  • Other - Another check-in, not the head.
  • Another idea might be:
    • Started - If there has been a build for the check-in in question. Use of this would help keep the statistic more complete. If it started building a given revision try to finish it up before moving on.
• Platform.

For ​KISS sake, strike these for now. Maybe at a later time (i.e. other tickets) these additional features could be addressed.

---

UI Changes

• To "Manage Build Configurations" pages (http://trac-server/admin/bitten/configs/Recipe):
  • Add HEAD Priority: A value from 1 to 5.
  • Add Other Priority: A value from 0 to 5.
    • Restrict setting to be not greater than HEAD Priority.
    • If set to 0, other (non-HEAD) builds will not be performed.
    • Future Feature: Remove Restriction of not greater than HEAD Priority. If set to greater than HEAD Priority builds will be performed in revision order instead of reverse revision order (related to #660).
• To "Manage Build Master" page (http://trac-server/admin/bitten/master):
  • Remove "Build all revisions" checkbox for now covered by setting "Other Priority" above to 0.

DB Changes

• bitten_config
  • Add 'Head_Priority int'
    • Default 3.
  • Add 'Other_Priority int'
    • Default 0 or 2.
    • On upgrade, if "Build all revisions" was set, 2 otherwise 0.
• Where is "Build all revisions" setting? Remove it, or deprecate and ignore it.

Operation

When looking for something to build, select the build with the greatest priority ('bitten_build.config' -> 'bitten_config.Head_Priority' or '.Other_Priority'), then revision date-time ('bitten_build.rev_time').

Sample Desired Priority Configuration

Recipe: Head Priority, Other Priority

1. tagAny: 5, 5
2. branch1: 4, 2
3. branch2: 4, 0
4. Trunk: 3, 2

[Mark Potter <mpotter@…>]

Replying to Mark Potter <mpotter@…>:

Replying to mpotter@…:

• Add HEAD Priority: A value from 1 to 5.
  
• Add Other Priority: A value from 0 to 5.
  

The 1 to 5 range is fairly arbitrary. The way specified any upper limit will do. But a smaller upper limit will lend itself to using a selection list to select the priority as opposed to a text field.

• Future Feature: Remove Restriction of not greater than HEAD Priority. If set to greater than HEAD Priority builds will be performed in revision order instead of reverse revision order (related to #660).

Rereading through #660, that might not be the ticket I was thinking of. I though there was a ticket discussing the desire to have all builds, not just the non-HEAD builds, to be built in order. Oh well, doesn't really matter.