[PATCH] Optionally add slave properties to java:ant tasks

[anonymous]

I'd like the slave properties to be optionally passed on to the java:ant task. I created a patch that does this if the java:ant recipe command has the a new slave_props parameter.

[anonymous]

Adds "slave_props" option to java:ant command

[osimons]

See Documentation/configure.html. The general solution that works for all commands and inputs from recipes is just to add what you need from properties - args="${more.stuff}".

I don't think we'll start passing properties into all commands, but commands that need them should find them in ctxt.config anyway.

Please reopen if there is something important I've not quite understood from your request.

[sahendrickson@…]

Hi osimons,

Thanks for leaving the door open to discussion. Let me describe the situation and see if you have a better approach than what I have thought of :)

I'm using an ant build file to perform actions after each check in. Currently, I'm calling four different ant targets across four different steps from the bitten recipe. I have separated out the calls to the targets in to individual steps so that I can see which succeeds/fails from the build status page.

Each ant target needs access to the slave's properties (i.e. authentication, locations of local resources, local executables, etc.). These targets also utilize bitten properties. For example, the revision number is used during checkout and later during build in order to correctly set the id number of each compiled resource.

I'm currently passing the variables in to each step using the "args" command. It includes the following text: -Declipse.version=${eclipse.version} -Dbitten.slave.location=${bitten.slave.location} -Dbitten.config=${config} -Dbitten.path=${path} -Dbitten.revision=${revision}

There are other properties, but I've just included the above to make it simple. The first parameter is from the slave configuration and the remaining are from bitten (IIRC).

The resulting recipe has those 5 lines repeated 4 times, once for each of the targets called by the recipe. To make matters worse, I have two such recipes, one for commits to the main trunk and a second for commits to a "stable/release" branch. So, those five lines are actually repeated 8 times.

Now, imagine that I want to add a property, say a specification of the specific java runtime to use. I would need to go to the scripts and copy and paste the added property 8 times.

Alternatively, I could specify this as part of the slave properties file and pass the slave properties directly to the targets (as this ticket requests).

NOTE: I think that the supplied patch is not the desired solution. Instead, I would do it differently. It would be better to add two properties to the ant command. One specifying whether the slave properties should be passed to ant and including an optional prefix for those properties, and one for the bitten properties.

The result would look something like this:

<java:ant target="publish" 
    bittenproperties="bitten" 
    slaveproperties="slave"
    args="
-Dsome.custom.property=value"/>

That would indicate that all bitten properties should be passed with the prefix of "bitten" and all slave properties should be passed with the prefix "slave". Optionally, we could omit a prefix such as:

<java:ant target="publish" 
    slaveproperties=""
    args="
-Dsome.custom.property=value"/>

That would mean that the slave properties should be included as-is, without a special prefix, and the bitten properties would not be included at all.

This would greatly improve the ability to tailer builds to the specific slaves that are executing them and ease maintenance of those builds. At the same time, it would allow one to use the ant task as before, without any extra properties other than those specifically indicate through the args parameter.

N.B. one other thing I tried was to pass along the slave properties file and directly include that in ant. However, ant does not allow categories such as:

[category]
property=value

and bitten requires categories. So, one cannot use the same file to specify properties for both tools.

I've reopened the bug. Please let me know what you think.

[osimons]

Thanks for the elaborate reply. Lately I've been working on using subprocess module to execute commands as part of #256. One of the things this new module would bring is support for env= to provide an environment dictionary to the subprocess (cross-platform). Looking at the Ant tutorials, I see that it is possible to map up environment variables.

In my opinion that would provide a general solution for all commands that needs to reference non-commandline input, and we could then support environment both using ${env.property} in recipes as well as include these variables for read by subprocess as any other environment setting.

[env]
property = something

Would such an infrastructure meet your requirements?

[anonymous]

Replying to osimons:

Thanks for the elaborate reply. Lately I've been working on using subprocess module to execute commands as part of #256. One of the things this new module would bring is support for env= to provide an environment dictionary to the subprocess (cross-platform). Looking at the Ant tutorials, I see that it is possible to map up environment variables.

In my opinion that would provide a general solution for all commands that needs to reference non-commandline input, and we could then support environment both using ${env.property} in recipes as well as include these variables for read by subprocess as any other environment setting.

[env]
property = something

Would such an infrastructure meet your requirements?

It sounds like you are saying, "Could we pass the properties to the specific tasks through generic environment variables?".

Ant would then include something like:

<property environment="env" />

That seems like a good solution to me :) One question, would you envision a prefix for the bitten and slave properties?

[osimons]

Replying to anonymous:

One question, would you envision a prefix for the bitten and slave properties?

Not sure yet - just throwing out an idea as my mind passed across documentation... And, of course environment settings are already available for you to use in your Ant scripts today - just set environment before starting bitten-slave, and they should be automatically inherited by subprocess.

To actually extend the use of environment settings in a flexible build environment (a continuously running slave that may build across branches, configurations and even different projects), we'd need to look at some way to map out what should be passed and in what form - preferably specific to the recipe/config name and possibly step id. An example would be to introduce marker '@' in section names, so that if your config 'trunk' has steps 'ant1' and 'ant2' the config could include something like this:

[trunk@]
prop1 = anthill
prop2 = foo

[trunk@ant2]
prop2 = bar
JAVA_HOME = /some/where

More specific (ie config@step) has higher order than the per-config settings (config@). In example above 'ant2' would have {'prop1': 'anthill, 'prop2': 'bar', 'JAVA_HOME': '/some/where'} passed down to subprocess environment. Not having prefixes would allow the settings to override any environment setting. That however, would likely mean we would need to use a case-insensitive version of the parser:

class BittenConfigParser(ConfigParser.SafeConfigParser):
    def optionxform(self, optionstr):
        return optionstr 

Some thinking and testing still needed - and feedback from any with opinions on the matter :-)

Oh, and subprocess is only available default in Python 2.4 (but available as separate install for 2.3 I believe), so that would also be related to ongoing discussion of bumping slave requirements.

[osimons]

Replying to osimons:

An example would be to introduce marker '@' in section names, so that if your config 'trunk' has steps 'ant1' and 'ant2' the config could include something like this:

Just to complete that example, this alternative could also complement the others - setting env variables for a given step ID across configurations:

[@ant1]
prop2 = baz

[dfraser]

Replying to sahendrickson@…:

I'm using an ant build file to perform actions after each check in. Currently, I'm calling four different ant targets across four different steps from the bitten recipe. I have separated out the calls to the targets in to individual steps so that I can see which succeeds/fails from the build status page.

Each ant target needs access to the slave's properties (i.e. authentication, locations of local resources, local executables, etc.). These targets also utilize bitten properties. For example, the revision number is used during checkout and later during build in order to correctly set the id number of each compiled resource.

I'm currently passing the variables in to each step using the "args" command. It includes the following text: -Declipse.version=${eclipse.version} -Dbitten.slave.location=${bitten.slave.location} -Dbitten.config=${config} -Dbitten.path=${path} -Dbitten.revision=${revision}

There are other properties, but I've just included the above to make it simple. The first parameter is from the slave configuration and the remaining are from bitten (IIRC).

The resulting recipe has those 5 lines repeated 4 times, once for each of the targets called by the recipe. To make matters worse, I have two such recipes, one for commits to the main trunk and a second for commits to a "stable/release" branch. So, those five lines are actually repeated 8 times.

Now, imagine that I want to add a property, say a specification of the specific java runtime to use. I would need to go to the scripts and copy and paste the added property 8 times.

Hmmm, if the point is to avoid having to repeat variables, maybe what you really want is macro support in the build recipes.

So basically if we jumped from doing simple interpolation (currently done in bitten.config in Configuration.interpolate) to making the build recipes ​genshi templates ... you could do all kinds of manipulation within the recipe.

[sahendrickson@…]

The underlying problem for me really boils down to the fact that bitten and ant don't speak the same language. My ideal solution? Specify the ant script and indicate which (ant) target outputs you want to capture as (bitten) tasks in the result for viewing results. That would be wonderful! :)

That way, I can run my build commands from bitten or the command line, but still get useful status reports from bitten. Everything would then be unified and passing properties one time for the ant script wouldn't be a big deal. Ant already has macros and extensibility mechanisms that I need and is already widely used by many projects. Although, I have my gripes about ant as well (e.g., no "if" or "try/catch" blocks).

Just my dream :)

For my particular situation, I don't need per configuration settings. I just don't want to have to unnecessarily repeat all properties automatically available for bitten recipes in the ant call.