Use deltas for source transmission

[cmlenz]

For large code bases, it is not efficient to send the entire code base to the slave for every build.

An alternative approach would allow the slave to tell the master that it already has a snapshot of a specific configuration at some revision, and the build master would then only send deltas to get the slave uptodate.

[anonymous]

A very simple approach could be to let the bitten client maintain a svn checkout of the repository and just transmit a revision number. The client could then use svn update to get the deltas for that revision.

[cmlenz]

The suggestion to “just use subversion” comes up every now and then, but I'm not convinced that it's the right approach for a couple of reasons:

• The Bitten slave is currently unaware of any version control system in use. Trac will in the future support other systems in addition to Subversion, and Bitten will simply adopt that support without any changes. Having the slave deal with the version control system itself will take that advantage away.
• Builds should be performed against clean copies of the source. However, a checkout from the version control system can not really be considered a clean copy of the source (it'll contain tons of meta-data files), especially not if it has seen previous builds, and has just been updated to pull in changes. The slave could of course use svn export etc, but that again would increase coupling with the version control system.
• A relatively minor point, but such a change would require that (a) the slave machine has the binaries for the version control system installed, and (b) it can actually access the repository.

So, I am currently leaning towards implementing delta transmission independent of the version control system.

[cmlenz]

Candidates for delta transmission:

• xdelta/vcdiff: See ​RFC 3284 and the ​Xdelta implementation.
• gdiff (General Diff Format): See the ​W3C note

In contrast the the line-based model of the unix diff command, both of the above are treat files as binary streams of octets.

gdiff does seem to be relatively easy to implement, while vcdiff is more complex but provides smaller deltas.

Another resource worth noting here is ​RFC 3229 (Delta encoding in HTTP).

[wiesel@…]

Bitten Client? as SVNClient for Deltas

[wiesel@…]

The previous patch (bitten-r228.diff) is a patch to revision 228 of bitten and uses PySVN to add svn-client support to the bitten-slave, like the first suggestion. Should the client have PySVN installed the bitten-slave offers the transfer method: 'text/x-svnrepos'. Should the bitten-master know a public address of the repository, and sees such an offer, the master merely transfers the revision, the address and repository path and the client does a checkout or an update.

The upsides are: It works. It is simple. It uses deltas, and works with externals. It also has the downsides cmlenz listed (besides being currently being more something of a hack).

However: Having the slave deal with the version control system itself will take that advantage away. That is only true, if it is the sole way the client can transfer data from the master. But with the patch, it is another way to do that. It still can receive tar-balls from, say, a monotone-repository.

Builds should be performed against clean copies of the source Is that always true? I think, it is the task of the build system (make, ant, etc.) to generate correct build from whatever sources are lying around, and however they where modified in the most efficient way. So, it may even be an advantage not to start from a clean checkout. However, should I be interested in a complete rebuild, I still can put a clean step at the beginning of the recipe, or do not use this method of transfer. I think, it shouldn't be mandated by bitten, how a build has to be performed.

A relatively minor point, but such a change would require that... To take advantage of the new transfer method. Yes. However, if everything still works as it did previously, if nothing is installed, is it a downside?

[wiesel@…]

Same as bitten-r228, but applies to Rev. 267

[jornh]

​http://starship.python.net/crew/atuining/cx_bsdiff/index.html might also be worth considering as a candidate for delta transmission. If it's good enough for Firefox binaries auto-update I guess it's also good enough for bitten. Plus it already comes with a python binding.

[bvoigt@…]

Builds should be performed against clean copies of the source -- this can be achieved by using svn update in a staging area and then performing a local copy (via copy-on-write symlinks if available?) with the option of including or excluding the .svn directories.

Then build products won't clutter the next build workspace because they won't ever make it into the staging area.

[wbell]

Bitten doesn't handle the source transmission anymore; users can either do their own checkouts (if bandwidth is free) or do something more complicated (ala #289). Closing.