Error processing a build

[nat@…]

When I run a build, I get the following error below in the trac log.

2009-07-06 14:36:26,246 Trac[main] ERROR: Internal Server Error: 
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/Trac-0.11.4-py2.4.egg/trac/web/main.py", line 435, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.4/site-packages/Trac-0.11.4-py2.4.egg/trac/web/main.py", line 205, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib/python2.4/site-packages/Bitten-0.6dev-py2.4.egg/bitten/master.py", line 106, in process_request
    return self._process_build_step(req, config, build)
  File "/usr/lib/python2.4/site-packages/Bitten-0.6dev-py2.4.egg/bitten/master.py", line 225, in _process_build_step
    recipe = Recipe(xmlio.parse(config.recipe))
  File "/usr/lib/python2.4/site-packages/Bitten-0.6dev-py2.4.egg/bitten/recipe.py", line 229, in __init__
    self.ctxt = Context(basedir, config, vars)
  File "/usr/lib/python2.4/site-packages/Bitten-0.6dev-py2.4.egg/bitten/recipe.py", line 54, in __init__
    self.config = config or Configuration()
  File "/usr/lib/python2.4/site-packages/Bitten-0.6dev-py2.4.egg/bitten/build/config.py", line 47, in __init__
    self._merge_sysinfo(parser, properties)
  File "/usr/lib/python2.4/site-packages/Bitten-0.6dev-py2.4.egg/bitten/build/config.py", line 52, in _merge_sysinfo
    system, _, release, version, machine, processor = platform.uname()
  File "/usr/lib64/python2.4/platform.py", line 1007, in uname
    processor = _syscmd_uname('-p','')
  File "/usr/lib64/python2.4/platform.py", line 794, in _syscmd_uname
    rc = f.close()
IOError: [Errno 10] No child processes

I am using the most recent trunk, trac 0.11.4, mysql, and Linux. The slave and master are on the same machine.

[nat@…]

Also, the slave looks like this:

[INFO    ] Build pending at http://www.cominor.com/trac/builds/1
[INFO    ] Executing build step 'a'
[INFO    ] hello world
[INFO    ] Build step a completed successfully
[WARNING ] Server returned error 500: Internal Server Error
[ERROR   ] HTTP Error 500: Internal Server Error

There is only one step.

[me@…]

Hi, can you attach the build configuration page?

[nat@…]

O ya, sry.

<build xmlns:svn="http://bitten.cmlenz.net/tools/svn"
   xmlns:c="http://bitten.cmlenz.net/tools/c"
   xmlns:sh="http://bitten.cmlenz.net/tools/sh"
   xmlns:php="http://bitten.cmlenz.net/tools/php"
   xmlns:java="http://bitten.cmlenz.net/tools/java">

   <step id="a" description="a test">
      <sh:exec executable="echo" args="hello world" />
   </step>
</build>

Is there any possibility that this might be some sort of permission issue. I know I have trac set up correctly, because we've been using it for two years, but I did just upgrade to 0.11.4 from 0.10.4...

[Nat <nat@…>]

The strangest part is that sometimes it works and sometimes it doesn't...

[osimons]

What frontend do you use for Trac? WSGI? What version? Is Python running Apache embedded or in a separate process (daemonized)? I've searched around for the error, and seems to be related to frontend handling as some report this does not happen when running straight tracd frontend. If mod_wsgi is used, try a newer version and see if that helps.

[Nat <nat@…>]

Apache is a seperate process, and we are using mod python (I think).

   SetHandler mod_python
   PythonInterpreter main_interpreter
   PythonHandler trac.web.modpython_frontend 
   PythonOption TracEnv /var/www/trac
   PythonOption TracUriRoot /trac

is what is in our apache config.

So right now, if I restart httpd, builds work for a few minutes, but after that they start getting 500 errors again.

[Nat <nat@…>]

Any suggestions?

[Nat <nat@…>]

This is still happening. Here is the error from the apache access logs.

Jul 15 15:33:57 cominor httpd[27154]: 192.168.41.100 - - [15/Jul/2009:15:33:56 -0700] "POST /trac/builds HTTP/1.1" 401 27181 "-" "Python-urllib/2.4"
Jul 15 15:33:57 cominor httpd[27154]: 192.168.41.100 - nwelch [15/Jul/2009:15:33:57 -0700] "POST /trac/builds HTTP/1.1" 201 13 "-" "Python-urllib/2.4"
Jul 15 15:33:57 cominor httpd[27154]: 192.168.41.100 - - [15/Jul/2009:15:33:57 -0700] "GET /trac/builds/6 HTTP/1.1" 401 27180 "-" "Python-urllib/2.4"
Jul 15 15:33:57 cominor httpd[27154]: 192.168.41.100 - nwelch [15/Jul/2009:15:33:57 -0700] "GET /trac/builds/6 HTTP/1.1" 200 421 "-" "Python-urllib/2.4"
Jul 15 15:33:57 cominor httpd[27154]: 192.168.41.100 - - [15/Jul/2009:15:33:57 -0700] "POST /trac/builds/6/steps/ HTTP/1.1" 401 27319 "-" "Python-urllib/2.4"
Jul 15 15:33:58 cominor httpd[27154]: 192.168.41.100 - nwelch [15/Jul/2009:15:33:57 -0700] "POST /trac/builds/6/steps/ HTTP/1.1" 500 41952 "-" "Python-urllib/2.4"

[anonymous]

is the process running under nwelch instead of as bitten?

[Nat <nat@…>]

The slave is, the master isn't

[Nat <nat@…>]

Also, if I run bitten-slave without username and password, I get this:

Traceback (most recent call last):
  File "/usr/bin/bitten-slave", line 7, in ?
    sys.exit(
  File "/usr/lib/python2.4/site-packages/Bitten-0.6dev-py2.4.egg/bitten/slave.py", line 427, in main
    exit_code = slave.run()
  File "/usr/lib/python2.4/site-packages/Bitten-0.6dev-py2.4.egg/bitten/slave.py", line 182, in run
    job_done = self._create_build(url)
  File "/usr/lib/python2.4/site-packages/Bitten-0.6dev-py2.4.egg/bitten/slave.py", line 225, in _create_build
    resp = self.request('POST', url, body, {
  File "/usr/lib/python2.4/site-packages/Bitten-0.6dev-py2.4.egg/bitten/slave.py", line 156, in request
    resp = self.opener.open(req)
  File "/usr/lib64/python2.4/urllib2.py", line 364, in open
    response = meth(req, response)
  File "/usr/lib64/python2.4/urllib2.py", line 471, in http_response
    response = self.parent.error(
  File "/usr/lib64/python2.4/urllib2.py", line 396, in error
    result = self._call_chain(*args)
  File "/usr/lib64/python2.4/urllib2.py", line 337, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.4/urllib2.py", line 916, in http_error_401
    host, req, headers)
  File "/usr/lib64/python2.4/urllib2.py", line 807, in http_error_auth_reqed
    raise ValueError("AbstractDigestAuthHandler doesn't know "
ValueError: AbstractDigestAuthHandler doesn't know about Basic

[anonymous]

our configuration runs the slave as a cronjob as bitten and authenticated as bitten.

/usr/bin/bitten-slave -s -v -u bitten -p bittenpass -d /bitten-build https://127.0.0.1/trac/builds > /home/bitten/last-build.log 2>&1

dono if that makes a difference. possibly not.

Also, we have a second location in our apache2 config for the bulid:

<Location /trac/builds>
   SetHandler mod_python
   PythonHandler trac.web.modpython_frontend
   PythonOption TracEnv /path/to/trac
   #PythonOptimize On

  AuthType Basic
  AuthName "blah"
  AuthUserFile /path/to/.htpasswds
  Require valid-user
</Location>

hope it helps

[Nat <nat@…>]

Not really, our apache config is the same. I think this might be a python2.4 issue, but I really don't know.

[osimons]

Could you please see "About Trac" as admin, and paste the table of system components ("System Information")?

I'd also suggest using Freenode IRC and visiting #trac and/or #bitten channels, as well as trac-users mailing list for input on this issue - this ticket has very limited audience, and it is not directly Trac or Bitten related (even though the originating call happens in Bitten). Your frontend (Apache/mod_python) makes a call to platform.uname() which in turn opens a process executing uname -p on your system to identify your processor (like 'x86_64'). When the process (from os.popen()) has finished reading the output from the command, it closes the file/pipe - but that is no longer there for some reason that at least is unknown to me.

[osimons]

BTW, Google for os.popen "No child processes" - lots of reports from all kinds of applications and scripts.

[schneider@…]

I get a very similar error on our server. We have two trac projects using bitten. One works correctly, the other gets an Internal Server Error similar to this.

[Mike Schneider <schneider@…>]

This is the contents of my about: trac page:

System Information Trac: 0.11.1 Python: 2.5.2 (r252:60911, Jan 4 2009, 22:17:10) [GCC 4.3.2] setuptools: 0.6c9 psycopg2: 2.0.7 Genshi: 0.5.1 mod_python: 3.3.1 Pygments: 0.10 Subversion: 1.5.1 (r32289) jQuery: 1.2.6 Configuration Section Name Value attachment max_size 99000000 render_unsafe_content false bitten adjust_timestamps yes build_all yes slave_timeout 3600 stabilize_wait 60 browser downloadable_paths /trunk, /branches/*, /tags/* hide_properties svk:merge changeset max_diff_bytes 10000000 max_diff_files 0 wiki_format_messages true components bitten.* enabled ticketdelete.* enabled webadmin.* enabled header_logo alt urbansim height -1 link ​https://svn.ict.usc.edu/urbansim src common/trac_banner.png width -1 logging log_file trac.log log_level DEBUG log_type none mimeviewer enscript_modes text/x-dylan:dylan:4 enscript_path enscript max_preview_size 262144 mime_map text/x-dylan:dylan,text/x-idl:ice,text/x-ada:ads:adb php_path php silvercity_modes tab_width 8 notification always_notify_owner true always_notify_reporter false always_notify_updater true mime_encoding base64 smtp_always_bcc smtp_always_cc smtp_default_domain smtp_enabled true smtp_from svn@… smtp_password smtp_port 25 smtp_replyto svn@… smtp_server localhost smtp_subject_prefix default smtp_user use_public_cc true use_short_addr false use_tls false project descr Subversion Urban Sim? Group footer Visit the Trac open source project at<br /><a href="​http://trac.edgewall.org/">​http://trac.edgewall.org/</a> icon common/trac.ico name urbansim url ​https://svn.ict.usc.edu/urbansim search min_query_length 3 ticket default_component Graphical UI default_milestone Unassigned default_priority major default_type defect default_version restrict_owner true ticket-workflow accept new -> assigned accept.operations set_owner_to_self accept.permissions TICKET_MODIFY leave * -> * leave.default 1 leave.operations leave_status reassign new,assigned,reopened -> new reassign.operations set_owner reassign.permissions TICKET_MODIFY reopen closed -> reopened reopen.operations del_resolution reopen.permissions TICKET_CREATE resolve new,assigned,reopened -> closed resolve.operations set_resolution resolve.permissions TICKET_MODIFY timeline changeset_long_messages false changeset_show_files 0 default_daysback 30 ticket_show_details false trac authz_file authz_module_name base_url check_auth_ip true database postgres://tracurbansim:urbansimtracadmin@localhost/svn_urbansim default_charset iso-8859-15 default_handler Wiki Module? htdocs_location ignore_auth_case false mainnav wiki,timeline,roadmap,browser,tickets,newticket,search metanav login,logout,settings,help,about permission_store Default Permission Store? repository_dir /svn/urbansim repository_type svn timeout 20 wiki ignore_missing_pages false split_page_names false

[Mike Schneider <schneider@…>]

We found a solution to our problem here:

​http://www.exolete.com/bitten-problems-upgrading-trac-sqlite-postgresql

Required updating the primary sequence key on the sql server.