Edgewall Software

Changes between Version 7 and Version 8 of Master Slave Protocol


Ignore:
Timestamp:
Jun 28, 2005, 5:43:44 PM (19 years ago)
Author:
cmlenz
Comment:

Protocol description updated and improved

Legend:

Unmodified
Added
Removed
Modified

  • Master Slave Protocol

    v7 v8  
    55To decouple the master and slave, an application protocol will be defined on top of the meta-protocol BEEP (Blocks Extensible Exchange Protocol, [http://www.faqs.org/rfcs/rfc3080.html RFC 3080]). BEEP was chosen because it provides peer-to-peer communication (so that both the client and the server can send requests to the other) and because of its relative simplicity compared to protocols such as XMPP.
    66
    7 == Why BEEP? ==
     7 '''Why BEEP?'''
    88
    9  I first looked [http://www.jabber.org/ Jabber]/[http://www.xmpp.org/ XMPP], but it seemed to be very complex (with dozens of related specifications), and there are no sufficiently mature implementations for Python. I could live with the complexity, but not if I have to implement the whole stack myself. I didn't look into other IM protocols because I wanted to build on something open/standardized. Note that even if I'd chosen XMPP/etc I would have to design a protocol on top of the provided infrastructure.
     9 ''I first looked [http://www.jabber.org/ Jabber]/[http://www.xmpp.org/ XMPP], but it seemed to be very complex (with dozens of related specifications), and there are no sufficiently mature implementations for Python. I could live with the complexity, but not if I have to implement the whole stack myself. I didn't look into other IM protocols because I wanted to build on something open/standardized. Note that even if I'd chosen XMPP/etc I would have to design a protocol on top of the provided infrastructure.''
    1010
    11  BEEP is simple and flexible, and explicitly designed as a foundation for custom application protocols. While the only Python implementation I found ([http://beepy.sourceforge.net/ BEEPy]) uses [http://twistedmatrix.com/ Twisted] and looks dead, BEEP is really simple enough to be implemented in a basic way in the scope of this project (i.e. minus
    12 the authentication and security features, which ''could'' of course be added later).
     11 ''BEEP is simple and flexible, and explicitly designed as a foundation for custom application protocols. While the only Python implementation I found ([http://beepy.sourceforge.net/ BEEPy]) uses [http://twistedmatrix.com/ Twisted] and looks dead, BEEP is really simple enough to be implemented in a basic way in the scope of this project (i.e. minus
     12the authentication and security features, which ''could'' of course be added later).''
    1313
    1414== Slave Registration ==
    1515
    16 A new client connects to the build master and signals its' availability for executing builds by starting a channel for the Bitten profile.
     16A new client connects to the build master and signals its' availability for executing builds by starting a channel for the Bitten [source:bitten/trunk/doc/orchestration.dtd#latest orchestration profile].
    1717
    1818First, the server needs to query some information about the client for orchestration:
    … …  
    4747  Content-Type: application/beep+xml
    4848 
    49   <build>trunk as of revision 492</build>
     49  <build recipe="path/to/recipe.xml">trunk as of revision 492</build>
    5050  END
    5151}}}
    5252
    53 A client can decline a build request, in which case the build master selects the next available client with the same (or sufficiently similar) configuration.
     53The build request must include the path to the recipe file relative to the root of the code base.
     54
     55A client can decline a build request, in which case the build master selects the next available client with the same (or sufficiently similar) configuration. A build request is declined using a negative reply containing an {{{<error></error>}}} element in the payload:
     56
     57{{{
     58  ERR 1 1 . 0 60
     59  Content-Type: application/beep+xml
     60 
     61  <error code="550">Too busy</error>
     62  END
     63}}}
     64
     65In this case the slave remains in the pool maintained by the master, but the master should attempt to prioritize slaves that accept build requests over those that regularly reject requests, as to avoid constantly polling the latter with requests that will probably be rejected again anyway.
     66
     67 '''TODO''': ''Specify error scenarios and error codes.''
     68
     69== Build execution ==
    5470
    5571If the client accepts a build request by sending a positive reply, the server will transmit a tarball of the code base that is to be built. The client does not need to know which exact revision (or branch) of the project it is building, nor does it need to perform a checkout itself.
    5672
    57 A client accepts a build request by responding with a {{{<download>}}}. for example:
     73A client accepts a build request by responding with a '''{{{RPY}}}''' message containing a {{{<proceed></proceed>}}} element in the payload. The reply must contain a list of archive formats that the slave supports for transmission of the code. For example:
    5874
    5975{{{
    60   RPY 1 1 . 0 60
     76  RPY 1 1 . 0 123
    6177  Content-Type: application/beep+xml
    6278
    63   <download>
    64    <accept type="application/tar" encoding="bzip2"/>
    65    <accept type="application/tar" encoding="gzip"/>
    66   </download>
     79  <proceed>
     80   <accept type="application/tar" encoding="bzip2" />
     81   <accept type="application/tar" encoding="gzip" />
     82  </proceed>
     83  END
    6784}}}
    6885
    69 In this message, the client indicates that it will accept {{{tar}}} archives with {{{bzip2}}} or {{{gzip}}} compression (preferring the former).
     86In this message, the client indicates that it will accept {{{tar}}} archives with {{{bzip2}}} or {{{gzip}}} compression (preferring the former). Another client might specify that it supported only ZIP archives, for example.
    7087
    7188After having received such a reply, the master can proceed by transmitting a snapshot of the code base to the slave:
    … …  
    8097}}}
    8198
    82  '''''Question:''' Should the build recipe be delivered as part of the tarball, or separately? It might make sense to actually put the build recipe in the build initiation message itself; While that would make local testing of the recipe more difficult, it would mean not polluting the tarball with files that are not actually in the repository. This also touches on the subject of where the recipe should be stored in the first place...''
     99The client may respond to this transmission either with a negative reply ('''{{{ERR}}}''' containing an {{{<error></error>}}} element with a description of the error), or by starting a sequence of '''{{{ANS}}}''' replies, terminated by a final '''{{{NUL}}}''' message (see next section).
     100
     101 '''TODO''': ''Specify error scenarios and error codes.''
    83102
    84103== Build Status Reporting ==
    85104
    86 Before and after each step of the [wiki:BuildRecipes build recipe], the client informs the server about its status:
    87  * whether the build has been started, is currently running, or has finished
    88  * which step is currently being processed
     105After having received and upacked the snapshot archive, and having successfully parsed the build recipe, the slave responds with '''{{{ANS}}}''' message containing a {{{<started/>}}} element in the payload:
    89106
    90 After a build step has been completed, the client transmits the data generated (as identified by the {{{<reports>}}} in the recipe) to the server, along with the the output of the build and a success/failure flag.
     107{{{
     108  ANS 1 2 . 0 54 0
     109  Content-Type: application/beep+xml
     110
     111  <started />
     112  END
     113}}}
     114
     115The slave then begins executing the steps in the recipe one-by-one (in the order they appear in the file). After each step of the [wiki:BuildRecipes build recipe], the client informs the server, with '''{{{ANS}}}''' messages containing a {{{<step/>}}} element in the payload, about the step it has processed, and what the outcome was (success or failure):
     116
     117{{{
     118  ANS 1 2 . 0 92 1
     119  Content-Type: application/beep+xml
     120
     121  <step id="test" description="Run all unit tests" result="success" />
     122  END
     123}}}
     124
     125In case of an error, the message should include the primary error message in the body of the {{{<step></step>}}} element:
     126
     127{{{
     128  ANS 1 2 . 0 135 1
     129  Content-Type: application/beep+xml
     130
     131  <step id="test" description="Run all unit tests" result="success">
     132    Could not load command "unittest".
     133  </step>
     134  END
     135}}}
     136
     137 '''TODO''': ''Transmission of build log and generated reports to the master''
     138
     139Furthermore, in case the slave is unexpectedly interrupted while executing a build, it should send an '''{{{ANS}}}''' message containing the element {{{<abort></abort>}}} in the payload:
     140
     141{{{
     142  ANS 1 2 . 0 66 2
     143  Content-Type: application/beep+xml
     144
     145  <aborted>Build cancelled</aborted>
     146  END
     147}}}
     148
     149Usually, the slave will disconnect directly after having aborted a build, but this is not necessary. It should remain in the slave pool maintained by the master until the orchestration channel gets closed.
     150
     151In any case, the slave must finish this exchange by sending a final '''{{{NUL}}}''' message to the master.
     152
     153{{{
     154  NUL 1 2 . 0 0
     155  END
     156}}}
     157
     158At this point, the build is considered completed (or aborted), and the master is free to initiate a new build on that slave.
    91159
    92160== Error Handling ==