Server returned error 409: Conflict (Token mismatch (wrong slave))

[Torsten Landschoff <torsten.landschoff@…>]

Hi again,

Bitten was running fine in our setup for a while. But today I noticed that no further builds are running. AFAIK, nothing has changed wrt. the configuration.

The log is as follows (already using -v):

[INFO    ] Slave launched at 2010-06-16 17:31:40
[DEBUG   ] Enabling authentication with username 'buildbot'
[DEBUG   ] Performing http form authentication
[DEBUG   ] Sending POST request to 'https://foo.bar.baz/trac/login?referer=https%3A%2F%2Ffoo.bar.baz%2Ftrac%2Fbuilds'
[DEBUG   ] Sending POST request to 'https://foo.bar.baz/trac/login?referer=https%3A%2F%2Ffoo.bar.baz%2Ftrac%2Fbuilds'
[DEBUG   ] Server returned error 403: Forbidden (no message available)
[ERROR   ] HTTP Error 403: Forbidden
[DEBUG   ] Removing working directory /tmp/bittenONEiHM

I tried logging in using the buildbots username and passwort using the same address as in the log and the server lets me in to tell me that only POST requests are allowed for the builds directory.

The last lines on the server logs are like this:

2010-06-16 19:30:11,988 Trac[session] DEBUG: Retrieving session for ID u'buildbot'
2010-06-16 19:30:11,990 Trac[chrome] DEBUG: Prepare chrome data for request
2010-06-16 19:30:12,014 Trac[perm] DEBUG: No policy allowed buildbot performing TICKET_CREATE on None
2010-06-16 19:30:12,014 Trac[perm] DEBUG: No policy allowed buildbot performing BROWSER_VIEW on None
2010-06-16 19:30:12,014 Trac[perm] DEBUG: No policy allowed buildbot performing TRAC_ADMIN on None
2010-06-16 19:30:12,015 Trac[perm] DEBUG: No policy allowed buildbot performing TICKET_ADMIN on None
2010-06-16 19:30:12,015 Trac[perm] DEBUG: No policy allowed buildbot performing DISCUSSION_ADMIN on None
2010-06-16 19:30:12,015 Trac[perm] DEBUG: No policy allowed buildbot performing BUILD_ADMIN on None
2010-06-16 19:30:12,016 Trac[perm] DEBUG: No policy allowed buildbot performing BUILD_MODIFY on None
2010-06-16 19:30:12,016 Trac[perm] DEBUG: No policy allowed buildbot performing PERMISSION_GRANT on None
2010-06-16 19:30:12,016 Trac[perm] DEBUG: No policy allowed buildbot performing PERMISSION_REVOKE on None
2010-06-16 19:30:12,021 Trac[perm] DEBUG: No policy allowed anonymous performing WIKI_VIEW on <Resource u'wiki:WikiHome'>
2010-06-16 19:30:12,022 Trac[main] WARNING: 403 Forbidden (WIKI_VIEW privileges are required to perform this operation on WikiHome)

And of course the buildbot has WIKI_VIEW permissions. wtf!?

Any hints? At least the bitten-slave should give a little bit more information especially in the face of https protocol. Interestingly, this error does not happen with http (via a tunnel). Instead, bitten complains about some "token mismatch" as it uses https again to send the job:

[DEBUG   ] Sending POST request to 'http://localhost:8000/trac/builds'
[INFO    ] Build pending at https://foo.bar.baz/trac/builds/425
[DEBUG   ] Sending GET request to 'https://foo.bar.baz/trac/builds/425'
[DEBUG   ] Server returned error 409: Conflict (Token mismatch (wrong slave): slave=25b190f02842ed8031aa4e42, build=69c992dc7ce48b65de7e73990cb1c509)
[ERROR   ] HTTP Error 409: Conflict

[sahendrickson@…]

I believe this is an issue that results from running multiple slaves in parallel.

When I was running them sequentally, this error did not occur. But, when I started run four slaves in parallel, I began getting this error. This is happening for me using http, so I doubt it has anything to do with http vs. https.

[anonymous]

The main issue seems to be:

[DEBUG   ] Server returned error 409: Conflict (Token mismatch (wrong slave): slave=25b190f02842ed8031aa4e42, build=69c992dc7ce48b65de7e73990cb1c509)
[ERROR   ] HTTP Error 409: Conflict

This occurred when I ran multiple slaves (from the same machine) in parallel. I used bitten-slave's "--name" parameter to give each slave a unique name. This seemed to fix the problem (for me at least).

[sahendrickson@…]

Updating the "--name" parameter took care of the mismatch. However, a new error arose where the build_token was empty, which caused the mismatch to reoccur but with "build=None" in the output. The error is not consistently reproducible, but a patch that seems to address it is attached.

[sahendrickson@…]

Patch to create build, if missing