Edgewall Software

Ticket #58 (closed enhancement: fixed)

Opened 4 years ago

Last modified 3 years ago

Provide CppUnit output parsing, perhaps as xUnit XML

Reported by: Chandler Carruth <chandlerc@…> Owned by: cmlenz
Priority: minor Milestone: 0.5
Component: Recipe commands Version: dev
Keywords: Cc:
Operating System:

Description

Provide parsing capabilities for  CppUnit output. Possibly by parsing xml output common to  CppUnit,  JUnit, and others.

Attachments

junit.patch Download (3.7 KB) - added by mgood 4 years ago.
JUnit test processor
ctools.py-cppunit.patch Download (2.4 KB) - added by Chandler Carruth <chandlerc@…> 4 years ago.
Patch to add Cpp Unit XML output parsing.
ctools.py-cppunit.2.patch Download (2.8 KB) - added by Chandler Carruth <chandlerc@…> 4 years ago.
Newer version... This one is much cleaner, and actually might work.

Change History

Changed 4 years ago by cmlenz

Can you attach an example of the Cpp Unit XML output?

Changed 4 years ago by mgood

JUnit test processor

Changed 4 years ago by mgood

I attached a processor for JUnit's XML output. According to the  changelog for CppUnit 1.11.0 they've included an XSLT stylesheet to transform the Cpp Unit output to JUnit format, which should hopefully allow you to use this same processor.

Note: In the patch I added an "args" attribute to the "ant" task, which I needed to be able to set some custom properties when running ant.

Changed 4 years ago by mgood

I should also note that the "file" attribute of the "junit" task accepts glob syntax, so you can specify "TEST-*.xml" to process a batch of JUnit test results.

Changed 4 years ago by darkeye@…

I tried to execute a test using the above patch, but I encountered a failure. my very simple cppunit output was:

<?xml version="1.0" encoding='utf-8' ?>
<TestRun>
  <FailedTests></FailedTests>
  <SuccessfulTests>
    <Test id="1">
      <Name>HelloTest::firstTest</Name>
    </Test>
  </SuccessfulTests>
  <Statistics>
    <Tests>1</Tests>
    <FailuresTotal>0</FailuresTotal>
    <Errors>0</Errors>
    <Failures>0</Failures>
  </Statistics>
</TestRun>

the resulting junit XML file was:

<?xml version="1.0"?>
<testsuite errors="0" failures="0" tests="1" name="from cppunit">
  <testcase classname="HelloTest" name="firstTest"/>
</testsuite>

when the bitten-slave got as far as to process the junit file, I got the following error:

[DEBUG   ] Executing <function junit at 0xb77573ac> with arguments: {'file_': 'junit-output.xml'}
[ERROR   ] 'time'
Traceback (most recent call last):
  File "/usr/lib/python2.3/asyncore.py", line 69, in read
    obj.handle_read_event()
  File "/usr/lib/python2.3/asyncore.py", line 390, in handle_read_event
    self.handle_read()
  File "/usr/lib/python2.3/asynchat.py", line 136, in handle_read
    self.found_terminator()
  File "build/bdist.linux-i686/egg/bitten/util/beep.py", line 278, in found_terminator
  File "build/bdist.linux-i686/egg/bitten/util/beep.py", line 311, in _handle_frame
  File "build/bdist.linux-i686/egg/bitten/util/beep.py", line 465, in handle_data_frame
  File "build/bdist.linux-i686/egg/bitten/slave.py", line 152, in handle_msg
  File "build/bdist.linux-i686/egg/bitten/slave.py", line 176, in execute_build
  File "build/bdist.linux-i686/egg/bitten/recipe.py", line 101, in execute
  File "build/bdist.linux-i686/egg/bitten/recipe.py", line 69, in run
  File "build/bdist.linux-i686/egg/bitten/build/javatools.py", line 94, in junitKeyError: 'time'

Changed 4 years ago by darkeye@…

it seems that the problem is, that there is no time=".." attribute in the testcase element that is generated by the XSLT transformation from cppunit output to junit.

removing the line in question (line 94 in bitten/build/javatools.py) does straighten things out...

but a native cppunit support would be better, for sure :)

Changed 4 years ago by cmlenz

Doesn't the Cpp Unit XML output contain information like the file the test is defined in, or timing information?

What about test errors/failures? How does the XML output look for them?

Changed 4 years ago by darkeye@…

here's an output, two passes, one failure:

<?xml version="1.0" encoding='utf-8' ?>
<TestRun>
  <FailedTests>
    <FailedTest id="2">
      <Name>HelloTest::secondTest</Name>
      <FailureType>Assertion</FailureType>
      <Location>
        <File>HelloTest.cxx</File>
        <Line>95</Line>
      </Location>
      <Message>assertion failed
- Expression: 2 == 3
</Message>
    </FailedTest>
  </FailedTests>
  <SuccessfulTests>
    <Test id="1">
      <Name>HelloTest::firstTest</Name>
    </Test>
    <Test id="3">
      <Name>HelloTest::thirdTest</Name>
    </Test>
  </SuccessfulTests>
  <Statistics>
    <Tests>3</Tests>
    <FailuresTotal>1</FailuresTotal>
    <Errors>0</Errors>
    <Failures>1</Failures>
  </Statistics>
</TestRun>

Changed 4 years ago by darkeye@…

note that there is some information loss as well: Cpp Unit supplied the line number where the test failed, while this info is not in the junit XML file.

does bitten have a data model to accomodate unit test failure source files and line numbers? (this would actually be useful to log failed builds as well, as the source file in error, and the line number could be pinpointed right away)

Changed 4 years ago by cmlenz

You can log any attributes you like, really. The Python and (optionally) the JUnit commands also log the file path for successful tests, so that they get linked to the Trac repository browser from the report summary.

If you also provide a line number, Bitten will store it (though not currently display it).

What you should do, is split the test name for the “fixture” name. For example, the test named HelloTest::thirdTest should probably be reported as:

 <test name="thirdTest" fixture="HelloTest"/>

The test results summary is grouped by fixture, so this is rather important.

Changed 4 years ago by cmlenz

And an example for the failure in the above example:

 <test name="secondTest" fixture="HelloTest" file="HelloTest.cxx" line="95">
  <traceback>assertion failed - Expression: 2 == 3</traceback>
 </test>

(The traceback element is really somewhat of a misnomer, should probably be called reason or something instead. But the others use traceback, so a Cpp Unit command should do the same.)

Changed 4 years ago by Chandler Carruth <chandlerc@…>

Patch to add Cpp Unit XML output parsing.

Changed 4 years ago by Chandler Carruth <chandlerc@…>

Newer version... This one is much cleaner, and actually might work.

Changed 4 years ago by cmlenz

  • status changed from new to assigned
  • milestone set to 0.5

Squeezing into 0.5.

Changed 4 years ago by cmlenz

  • status changed from assigned to closed
  • resolution set to fixed

Added in [279]. Patch heavily modified, though ;-)

Add/Change #58 (Provide CppUnit output parsing, perhaps as xUnit XML)

Author


E-mail address and user name can be saved in the Preferences.


Change Properties
<Author field>
Action
as closed
Next status will be 'reopened'
 
Note: See TracTickets for help on using tickets.