Provide CppUnit output parsing, perhaps as xUnit XML

[Chandler Carruth <chandlerc@…>]

Provide parsing capabilities for ​CppUnit output. Possibly by parsing xml output common to ​CppUnit, ​JUnit, and others.

[cmlenz]

Can you attach an example of the Cpp Unit XML output?

[mgood]

JUnit test processor

[mgood]

I attached a processor for JUnit's XML output. According to the ​changelog for CppUnit 1.11.0 they've included an XSLT stylesheet to transform the Cpp Unit output to JUnit format, which should hopefully allow you to use this same processor.

Note: In the patch I added an "args" attribute to the "ant" task, which I needed to be able to set some custom properties when running ant.

[mgood]

I should also note that the "file" attribute of the "junit" task accepts glob syntax, so you can specify "TEST-*.xml" to process a batch of JUnit test results.

[darkeye@…]

I tried to execute a test using the above patch, but I encountered a failure. my very simple cppunit output was:

<?xml version="1.0" encoding='utf-8' ?>
<TestRun>
  <FailedTests></FailedTests>
  <SuccessfulTests>
    <Test id="1">
      <Name>HelloTest::firstTest</Name>
    </Test>
  </SuccessfulTests>
  <Statistics>
    <Tests>1</Tests>
    <FailuresTotal>0</FailuresTotal>
    <Errors>0</Errors>
    <Failures>0</Failures>
  </Statistics>
</TestRun>

the resulting junit XML file was:

<?xml version="1.0"?>
<testsuite errors="0" failures="0" tests="1" name="from cppunit">
  <testcase classname="HelloTest" name="firstTest"/>
</testsuite>

when the bitten-slave got as far as to process the junit file, I got the following error:

[DEBUG   ] Executing <function junit at 0xb77573ac> with arguments: {'file_': 'junit-output.xml'}
[ERROR   ] 'time'
Traceback (most recent call last):
  File "/usr/lib/python2.3/asyncore.py", line 69, in read
    obj.handle_read_event()
  File "/usr/lib/python2.3/asyncore.py", line 390, in handle_read_event
    self.handle_read()
  File "/usr/lib/python2.3/asynchat.py", line 136, in handle_read
    self.found_terminator()
  File "build/bdist.linux-i686/egg/bitten/util/beep.py", line 278, in found_terminator
  File "build/bdist.linux-i686/egg/bitten/util/beep.py", line 311, in _handle_frame
  File "build/bdist.linux-i686/egg/bitten/util/beep.py", line 465, in handle_data_frame
  File "build/bdist.linux-i686/egg/bitten/slave.py", line 152, in handle_msg
  File "build/bdist.linux-i686/egg/bitten/slave.py", line 176, in execute_build
  File "build/bdist.linux-i686/egg/bitten/recipe.py", line 101, in execute
  File "build/bdist.linux-i686/egg/bitten/recipe.py", line 69, in run
  File "build/bdist.linux-i686/egg/bitten/build/javatools.py", line 94, in junitKeyError: 'time'

[darkeye@…]

it seems that the problem is, that there is no time=".." attribute in the testcase element that is generated by the XSLT transformation from cppunit output to junit.

removing the line in question (line 94 in bitten/build/javatools.py) does straighten things out...

but a native cppunit support would be better, for sure :)

[cmlenz]

Doesn't the Cpp Unit XML output contain information like the file the test is defined in, or timing information?

What about test errors/failures? How does the XML output look for them?

[darkeye@…]

here's an output, two passes, one failure:

<?xml version="1.0" encoding='utf-8' ?>
<TestRun>
  <FailedTests>
    <FailedTest id="2">
      <Name>HelloTest::secondTest</Name>
      <FailureType>Assertion</FailureType>
      <Location>
        <File>HelloTest.cxx</File>
        <Line>95</Line>
      </Location>
      <Message>assertion failed
- Expression: 2 == 3
</Message>
    </FailedTest>
  </FailedTests>
  <SuccessfulTests>
    <Test id="1">
      <Name>HelloTest::firstTest</Name>
    </Test>
    <Test id="3">
      <Name>HelloTest::thirdTest</Name>
    </Test>
  </SuccessfulTests>
  <Statistics>
    <Tests>3</Tests>
    <FailuresTotal>1</FailuresTotal>
    <Errors>0</Errors>
    <Failures>1</Failures>
  </Statistics>
</TestRun>

[darkeye@…]

note that there is some information loss as well: Cpp Unit supplied the line number where the test failed, while this info is not in the junit XML file.

does bitten have a data model to accomodate unit test failure source files and line numbers? (this would actually be useful to log failed builds as well, as the source file in error, and the line number could be pinpointed right away)

[cmlenz]

You can log any attributes you like, really. The Python and (optionally) the JUnit commands also log the file path for successful tests, so that they get linked to the Trac repository browser from the report summary.

If you also provide a line number, Bitten will store it (though not currently display it).

What you should do, is split the test name for the “fixture” name. For example, the test named HelloTest::thirdTest should probably be reported as:

 <test name="thirdTest" fixture="HelloTest"/>

The test results summary is grouped by fixture, so this is rather important.

[cmlenz]

And an example for the failure in the above example:

 <test name="secondTest" fixture="HelloTest" file="HelloTest.cxx" line="95">
  <traceback>assertion failed - Expression: 2 == 3</traceback>
 </test>

(The traceback element is really somewhat of a misnomer, should probably be called reason or something instead. But the others use traceback, so a Cpp Unit command should do the same.)

[Chandler Carruth <chandlerc@…>]

Patch to add Cpp Unit XML output parsing.

[Chandler Carruth <chandlerc@…>]

Newer version... This one is much cleaner, and actually might work.

[cmlenz]

Squeezing into 0.5.

[cmlenz]

Added in [279]. Patch heavily modified, though ;-)