Improve recipe handling.

[thomas.moschny@…]

It is cumbersome to enter the recipes into the small text field on the buildconfig admin page. So, I'm doing copy-and-paste to/from an (xml-capable) editor. Seems not to be best solution though.

So, as a first step, an upload button would be nice.

But this is still not ideal. The recipe could also be

1. fetched from the repository (from a certain path/rev)
2. taken from a wikipage

(1) is what I'm currently doing anyway: storing the recipe in the svn repos, so my colleagues can see what's happening on the buildbot. (2) would be a nice alternative.

Both solutions would also enable easy re-usage of recipes given they can be customized enough using macros.

[osimons]

#225 also includes a request for alternative storage for recipes, and perhaps combined with xinclude requested in #230 the body of the recipe can be populated from alternative locations.

[thomas.moschny@…]

Meanwhile, I use a scriptlet like this for printing a recipe to stdout:

#
# getrecipe.py: prints recipe text
# Usage: getrecipe.py /path/to/trac-environment foo > foo_recipe.xml
#
from trac.env import Environment
from bitten.model import BuildConfig
import sys

env = Environment(sys.argv[1])
config = BuildConfig.fetch(env, sys.argv[2])

print config.recipe

and another one for updating a recipe with text from stdin:

#
# chrecipe.py: reads new recipe text from stdin
# Usage: chrecipe.py /path/to/trac-environment foo < foo_recipe.xml
#
from trac.env import Environment
from bitten.model import BuildConfig
import sys
from xml.etree import ElementTree

env = Environment(sys.argv[1])
config = BuildConfig.fetch(env, sys.argv[2])

recipe = '\n'.join(line.strip('\r\n') for line in sys.stdin)

# check well-formedness, at least
ElementTree.fromstring(recipe)

config.recipe = recipe
config.update()

Using these, one can easily sync recipes with files in a directory.

[hodgestar]

Patch implementing basic import of parts of build recipes from XML files in repository.

[hodgestar]

I've attached external-step-1.diff with a basic implementation of reading portions of recipes from files in a repository. The mechanism is:

<build>
  <external src="foo/trunk/recipe-header.xml" rev="" />
  <external src="foo/trunk/my-recipe.xml" rev="" />
  <step id="specialsauce">
     <somecmd arg="foo" />
  </step> 
  <external src="foo/trunk/recipe-footer.xml" rev="" />
</build>

An external xml file looks just like an build recipe. External files can be nested. The 'rev' attribute defaults to the same revision as the build.

The outstanding issues with the patch that spring to mind are:

• The patch fiddles with Parsed Element?._node. Should I just added the needed functionality to Parsed Element??
• Namespaces and other attributes on <build> tags all get copied to the final parent build tag but there is currently no protection for clashes and namespaces should probably be handled in a more xi:include type way.
• No protection for missing or bad external XML files.
• It's not a proper xi:include implementation primarily because in order to extract out a set of steps the xi:include implementation would need to support the XPointer Framework (which besides being a fair amount of work is not exactly user friendly).
• We could potentially support an href or url attribute inaddition to a src or rev attribute to regain some xi:include like functionality. This wouldn't be too much effort.

The patch introduces the concept of a master recipe (which is build independent and possibly contains externals) and a slave recipe (which is build dependent and contains only steps).

[hodgestar]

Patch reworked to use xi:include instead of external tags.

[hodgestar]

I've reworked my patch to implement xi:include instead of the external tag hack. The new syntax looks like:

Things to note:

• Build tags may now be nested in recipe XML.
• Steps inside a build tag have ids constructed by joining the ids of their parent builds together with dots, e.g. part3.specialsauce is one of the step ids in the example recipe.
• Any id tag on the root of the recipe XML is ignored.
• Nested build tags allows the whole of the XML from the xi:include href to be pulled in (which removes the need to support the XPointer Framework part of the XInclude spec).
• xi:include tags must have an id attribute. This overrides any id attribute on the included build tag.
• The only href attribute values currently supported are ones of the form repo:///path/file?rev=123. The rev argument is optional and defaults to the revision that triggered the build.
• The additional ParsedElement features needed have been cleaned up.

I don't think this is a change that can go into the 0.6 release but I think that the patch is essentially ready to be committed at the start of the 0.7 release cycle if other people like it.

[hodgestar]

Replying to hodgestar:

The missing syntax example:

<build xmlns:xi="http://www.w3.org/2001/XInclude" >
  <xi:include id="part1" href="repo:///foo/trunk/recipe-header.xml" />
  <xi:include id="part2" href="repo:///foo/trunk/my-recipe.xml?rev=123" />
  <build id="part3">
     <step id="specialsauce">
        <somecmd arg="foo" />
     </step>
  </build> 
  <xi:include id="part4" href="repo:///foo/trunk/recipe-footer.xml" />
</build>

[hodgestar]

The patch will also need some documentation. The XInclude specification can be found at  http://www.w3.org/TR/xinclude/.

[hodgestar]

Patched update to apply against r873

[osimons]

If the <xi:include/> arguments also supports interpolation, that means it will be possible to use for instance ${platform} in href="" which enables easy lookup to include platform-specific versions of the config.

This is a solution to a much requested feature, where for instance the linux and windows builds really have no shared steps or commands.

[hodgestar]

Patch making recipes Genshi templates

[hodgestar]

My new patch provices roughly the same functionality as my previous patch (xiinclude-2.diff) but uses Genshi templates instead of its own xi:include implementation. The major outstanding TODO in the patch is what to pass in as the template context (ideally whatever is passed in also needs to be available in some form when the recipe is validated by the admin console before being saved).

[J. Berger <jeberger@…>]

I have not tried the new patch, but I have seen an issue with the previous one: the “master” build recipe must have at least one step in addition to any includes. I.e the following recipe does not work:

<build xmlns:xi="http://www.w3.org/2001/XInclude">
   <xi:include id="all" href="repo:///trunk/Bitten/UnitTests.xml"/>
</build>

but this works:

<build xmlns:xi="http://www.w3.org/2001/XInclude"
       xmlns:sh="http://bitten.edgewall.org/tools/sh">
   <step id="nop" description="The build must contain at least one step">
      <sh:exec executable="true"/>
   </step>
   <xi:include id="all" href="repo:///trunk/Bitten/UnitTests.xml" />
</build>

The problem apparently comes from source:/trunk/bitten/master.py@950#L274 because recipe.__iter__() is empty so calling next() throws a StopIteration.

[hodgestar]

Replying to J. Berger <jeberger@…>:

I have not tried the new patch, but I have seen an issue with the previous one: the > “master” build recipe must have at least one step in addition to any includes. I.e the following recipe does not work:

That was a minor oversight in the previous patch (with a one-line fix similar to the other cases in master.py). The new patch doesn't suffer from that problem (although I am still trying to figure out what to do about Genshi's ${} syntax clashing with the existing variable substitution syntax).

[anatoly techtonik <techtonik@…>]

This patch seems huge. May take a lot of time to review. Is it possible to split it?

[r.wilczek@…]

I'd really like to see this one in the distribution!

I have like 150 recipes here, and maintaining them via the web-interface is an annoying and error-prone task.

Are there plans to spend any effort in integrating the patches?