Edgewall Software
Modify

Opened 16 years ago

Last modified 11 years ago

#301 assigned enhancement

Improve recipe handling.

Reported by: thomas.moschny@… Owned by: hodgestar
Priority: minor Milestone: 0.7
Component: Administration interface Version: dev
Keywords: Cc: trac@…, mpotter@…, thomas.moschny@…, hodgestar, jeberger@…, ged-edgewall@…, r.wilczek@…, srl@…
Operating System: BSD

Description

It is cumbersome to enter the recipes into the small text field on the buildconfig admin page. So, I'm doing copy-and-paste to/from an (xml-capable) editor. Seems not to be best solution though.

So, as a first step, an upload button would be nice.

But this is still not ideal. The recipe could also be

  1. fetched from the repository (from a certain path/rev)
  2. taken from a wikipage

(1) is what I'm currently doing anyway: storing the recipe in the svn repos, so my colleagues can see what's happening on the buildbot. (2) would be a nice alternative.

Both solutions would also enable easy re-usage of recipes given they can be customized enough using macros.

Attachments (4)

external-step-1.diff (16.4 KB) - added by hodgestar 14 years ago.
Patch implementing basic import of parts of build recipes from XML files in repository.
xiinclude-1.diff (22.6 KB) - added by hodgestar 14 years ago.
Patch reworked to use xi:include instead of external tags.
xiinclude-2.diff (22.8 KB) - added by hodgestar 14 years ago.
Patched update to apply against r873
genshi-recipes-r1004.diff (24.0 KB) - added by hodgestar 13 years ago.
Patch making recipes Genshi templates

Download all attachments as: .zip

Change History (25)

comment:1 Changed 15 years ago by osimons

#225 also includes a request for alternative storage for recipes, and perhaps combined with xinclude requested in #230 the body of the recipe can be populated from alternative locations.

comment:2 Changed 15 years ago by trac@…

  • Cc trac@… added

comment:3 Changed 15 years ago by osimons

  • Milestone changed from 0.6 to 0.7

comment:4 Changed 15 years ago by potter <mpotter@…>

  • Cc mpotter@… added

comment:5 Changed 14 years ago by thomas.moschny@…

  • Cc thomas.moschny@… added

Meanwhile, I use a scriptlet like this for printing a recipe to stdout:

#
# getrecipe.py: prints recipe text
# Usage: getrecipe.py /path/to/trac-environment foo > foo_recipe.xml
#
from trac.env import Environment
from bitten.model import BuildConfig
import sys

env = Environment(sys.argv[1])
config = BuildConfig.fetch(env, sys.argv[2])

print config.recipe

and another one for updating a recipe with text from stdin:

#
# chrecipe.py: reads new recipe text from stdin
# Usage: chrecipe.py /path/to/trac-environment foo < foo_recipe.xml
#
from trac.env import Environment
from bitten.model import BuildConfig
import sys
from xml.etree import ElementTree

env = Environment(sys.argv[1])
config = BuildConfig.fetch(env, sys.argv[2])

recipe = '\n'.join(line.strip('\r\n') for line in sys.stdin)

# check well-formedness, at least
ElementTree.fromstring(recipe)

config.recipe = recipe
config.update()

Using these, one can easily sync recipes with files in a directory.

Changed 14 years ago by hodgestar

Patch implementing basic import of parts of build recipes from XML files in repository.

comment:6 Changed 14 years ago by hodgestar

I've attached external-step-1.diff with a basic implementation of reading portions of recipes from files in a repository. The mechanism is:

<build>
  <external src="foo/trunk/recipe-header.xml" rev="" />
  <external src="foo/trunk/my-recipe.xml" rev="" />
  <step id="specialsauce">
     <somecmd arg="foo" />
  </step> 
  <external src="foo/trunk/recipe-footer.xml" rev="" />
</build>

An external xml file looks just like an build recipe. External files can be nested. The 'rev' attribute defaults to the same revision as the build.

The outstanding issues with the patch that spring to mind are:

  • The patch fiddles with Parsed Element?._node. Should I just added the needed functionality to Parsed Element??
  • Namespaces and other attributes on <build> tags all get copied to the final parent build tag but there is currently no protection for clashes and namespaces should probably be handled in a more xi:include type way.
  • No protection for missing or bad external XML files.
  • It's not a proper xi:include implementation primarily because in order to extract out a set of steps the xi:include implementation would need to support the XPointer Framework (which besides being a fair amount of work is not exactly user friendly).
  • We could potentially support an href or url attribute inaddition to a src or rev attribute to regain some xi:include like functionality. This wouldn't be too much effort.

The patch introduces the concept of a master recipe (which is build independent and possibly contains externals) and a slave recipe (which is build dependent and contains only steps).

Changed 14 years ago by hodgestar

Patch reworked to use xi:include instead of external tags.

comment:7 follow-up: Changed 14 years ago by hodgestar

I've reworked my patch to implement xi:include instead of the external tag hack. The new syntax looks like:

Things to note:

  • Build tags may now be nested in recipe XML.
  • Steps inside a build tag have ids constructed by joining the ids of their parent builds together with dots, e.g. part3.specialsauce is one of the step ids in the example recipe.
  • Any id tag on the root of the recipe XML is ignored.
  • Nested build tags allows the whole of the XML from the xi:include href to be pulled in (which removes the need to support the XPointer Framework part of the XInclude spec).
  • xi:include tags must have an id attribute. This overrides any id attribute on the included build tag.
  • The only href attribute values currently supported are ones of the form repo:///path/file?rev=123. The rev argument is optional and defaults to the revision that triggered the build.
  • The additional ParsedElement features needed have been cleaned up.

I don't think this is a change that can go into the 0.6 release but I think that the patch is essentially ready to be committed at the start of the 0.7 release cycle if other people like it.

comment:8 Changed 14 years ago by hodgestar

  • Owner changed from cmlenz to hodgestar
  • Status changed from new to assigned

comment:9 in reply to: ↑ 7 Changed 14 years ago by hodgestar

Replying to hodgestar:

The missing syntax example:

<build xmlns:xi="http://www.w3.org/2001/XInclude" >
  <xi:include id="part1" href="repo:///foo/trunk/recipe-header.xml" />
  <xi:include id="part2" href="repo:///foo/trunk/my-recipe.xml?rev=123" />
  <build id="part3">
     <step id="specialsauce">
        <somecmd arg="foo" />
     </step>
  </build> 
  <xi:include id="part4" href="repo:///foo/trunk/recipe-footer.xml" />
</build>

comment:10 Changed 14 years ago by hodgestar

The patch will also need some documentation. The XInclude specification can be found at http://www.w3.org/TR/xinclude/.

comment:11 Changed 14 years ago by hodgestar

  • Cc hodgestar added

Changed 14 years ago by hodgestar

Patched update to apply against r873

comment:12 Changed 14 years ago by J. Berger <jeberger@…>

  • Cc jeberger@… added

comment:13 Changed 13 years ago by osimons

If the <xi:include/> arguments also supports interpolation, that means it will be possible to use for instance ${platform} in href="" which enables easy lookup to include platform-specific versions of the config.

This is a solution to a much requested feature, where for instance the linux and windows builds really have no shared steps or commands.

comment:14 Changed 13 years ago by Michael Granger <ged-edgewall@…>

  • Cc ged-edgewall@… added

Changed 13 years ago by hodgestar

Patch making recipes Genshi templates

comment:15 Changed 13 years ago by hodgestar

My new patch provices roughly the same functionality as my previous patch (xiinclude-2.diff) but uses Genshi templates instead of its own xi:include implementation. The major outstanding TODO in the patch is what to pass in as the template context (ideally whatever is passed in also needs to be available in some form when the recipe is validated by the admin console before being saved).

comment:16 follow-up: Changed 13 years ago by J. Berger <jeberger@…>

I have not tried the new patch, but I have seen an issue with the previous one: the “master” build recipe must have at least one step in addition to any includes. I.e the following recipe does not work:

<build xmlns:xi="http://www.w3.org/2001/XInclude">
   <xi:include id="all" href="repo:///trunk/Bitten/UnitTests.xml"/>
</build>

but this works:

<build xmlns:xi="http://www.w3.org/2001/XInclude"
       xmlns:sh="http://bitten.edgewall.org/tools/sh">
   <step id="nop" description="The build must contain at least one step">
      <sh:exec executable="true"/>
   </step>
   <xi:include id="all" href="repo:///trunk/Bitten/UnitTests.xml" />
</build>

The problem apparently comes from source:/trunk/bitten/master.py@950#L274 because recipe.__iter__() is empty so calling next() throws a StopIteration.

comment:17 in reply to: ↑ 16 Changed 13 years ago by hodgestar

Replying to J. Berger <jeberger@…>:

I have not tried the new patch, but I have seen an issue with the previous one: the > “master” build recipe must have at least one step in addition to any includes. I.e the following recipe does not work:

That was a minor oversight in the previous patch (with a one-line fix similar to the other cases in master.py). The new patch doesn't suffer from that problem (although I am still trying to figure out what to do about Genshi's ${} syntax clashing with the existing variable substitution syntax).

comment:18 Changed 12 years ago by anatoly techtonik <techtonik@…>

This patch seems huge. May take a lot of time to review. Is it possible to split it?

comment:19 Changed 12 years ago by Roland Wilczek <r.wilczek@…>

  • Cc r.wilczek@… added

comment:20 Changed 11 years ago by r.wilczek@…

I'd really like to see this one in the distribution!

I have like 150 recipes here, and maintaining them via the web-interface is an annoying and error-prone task.

Are there plans to spend any effort in integrating the patches?

comment:21 Changed 11 years ago by Steven R. Loomis <srl@…>

  • Cc srl@… added

Add Comment

Modify Ticket

Change Properties
Set your email in Preferences
Action
as assigned The owner will remain hodgestar.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.