Changes between Initial Version and Version 1 of Master Slave Protocol Http

Timestamp:

Jul 23, 2007, 10:36:21 PM (17 years ago)

Author:

cmlenz

Comment:

First draft of HTTP-based master/slave protocol

Master Slave Protocol Http

@@ +1 @@
+= Bitten Master/Slave Protocol, using HTTP =
+[[PageOutline(2)]]
+
+This is a proposal for an HTTP-based protocol enabling communication between the build master and various build slaves. The protocol presented here is not final yet, nor is it implemented. Just throwing ideas out there, basically.
+
+== Comparison the Previous BEEP Protocol ==
+
+The BEEP-based protocol currently used by Bitten is described on MasterSlaveProtocol.
+
+The differences can be summarized as follows:
+ * The build master would be a simple HTTP server, implemented as part of the Trac plugin. That means there would no longer be a separate daemon process needed for the master.
+ * The build slaves are simply HTTP clients, probably using [http://bitworking.org/projects/httplib2/ httplib2] and falling back to the httplib or urllib modules in the standard library.
+ * Both SSL and the various authentication methods of HTTP can be used to secure the communication.
+ * Directionality of communication is always from the slave to the master. The master no longer initiates actions on the slave, rather the slave polls the master for pending actions when it is idle.
+ * The build master would no longer be responsible for packaging tarballs and sending them to the slaves; instead, the slaves receive connection details for the repository, and perform a normal checkout. This is a [ticket:79 long standing ticket].
+
+== Build Creation ==
+
+A new client connects to the build master and asks the master whether there are any pending builds it could perform. The slave does this by `POST`ing its profile to the master, which contains information such as:
+ * The platform/architecture of the slave machine,
+ * the operating system,
+ * the product name and version number of each of the dependencies of the project to build (for example, the C compiler or the Python runtime), and
+ * the name and email address of the maintainer of the machine.
+
+{{{
+#!xml
+POST /build/ HTTP/1.1
+Host: example.org
+Content-Type: application/xml; charset=utf-8
+Content-Length: 666
+
+<slave name="lamech">
+  <maintainer>Christopher Lenz &lt;cmlenz@gmx.de&gt;</maintainer>
+  <platform>Power Macintosh</platform>
+  <os family="posix" version="8.1.0">Darwin</os>
+</slave>
+}}}
+
+If the build master finds any pending builds that can be performed by the target platform matching the slave, it would send back a response similar to the following:
+
+{{{
+#!xml
+HTTP/1.1 201 Created
+Location: http://example.org/builds/trunk/123/
+}}}
+
+The response contains the URL to a build recipe as the value of the `Location` header. At this point, the master has allocated a pending build entity in its database that can also be viewed as HTML at the specified URL using any HTTP user agent.
+
+On the other hand, if the master has no work for the slave, it would return a `204 No Content` response:
+
+{{{
+#!xml
+HTTP/1.1 204 No Content
+}}}
+
+== Build Initiation ==
+
+When the slave has received the URL to a build recipe, it can request the build recipe using a simple `GET` request:
+
+{{{
+#!xml
+GET /builds/trunk/123/ HTTP/1.1
+Host: example.org
+Accept: application/x-bitten+xml
+}}}
+
+If the master still has that build in pending state in the database, it will respond with the recipe:
+
+{{{
+#!xml
+HTTP/1.1 200 OK
+
+<build xmlns:python="http://bitten.cmlenz.net/tools/python">
+  <checkout type="svn" url="http://svn.example.org/repos/"
+            revision="1234" />
+  <step id="compile">
+    <python:distutils command="build"/>
+  </step>
+  <step id="dist">
+    <python:distutils command="sdist"/>
+  </step>
+  <upload>
+    <file path="dist/foobar*.tar.gz"/>
+    <file path="dist/foobar*.zip"/>
+  </upload>
+</build>
+}}}
+
+The first element in a build recipe must be the `<checkout>` element containing the information necessary for the slave to perform a checkout from the version control repository.
+
+== Build Reporting ==
+
+As soon as the slave has received the recipe, it should perform the checkout and execute the steps outlined in the build.
+
+  ''Open issue: details of the checkout process should also be reported back the master''
+
+After every completed step, the slave should make a `PUT` requests the `steps` member of build collection:
+
+{{{
+#!xml
+PUT /builds/trunk/123/steps/test/ HTTP/1.1
+Host: example.org
+
+<result status="success" started="2005-06-29T16:41:53" duration="7.61">
+  ...
+</result>
+}}}
+
+The `started` attribute specifies the date and time at which processing of this step was started. The `duration` attribute contains the number of seconds that it took to complete the step (this may include fractions).
+
+The `<result>` element may contain one or more child elements:
+ * `<error></error>` elements indicate errors in the execution of the step,
+ * `<log></log>` elements contain the build log output, and
+ * `<report></report>` elements contain generated [wiki:DataStorage report data].
+
+The build is assumed to be complete after the master has received a request for every step in the recipe.
+
+The server responds with a `201 Created` response.
+
+== Uploading of Build Artifacts ==
+
+If the recipe contains an `<upload>` element at the end of the recipe, the slave is expected to perform file uploads of any of the files specified. This is done using `PUT` requests the the `files` member of the build collection:
+
+{{{
+#!xml
+PUT /builds/trunk/123/files/foobar-r123.tar.gz HTTP/1.1
+Host: example.org
+Content-Type: application/tar
+Content-Encoding: gzip
+Content-Length: 666
+
+...
+}}}
+
+The server responds with a `201 Created` response.