[patch] add support for Qt qtestlib xml data

[david kuehling <dvdkhlng ta gmx tod de>]

hi,

the attached patch adds support for Qt QTestLib test-case data import, similar to the cunit support already provided by ctools.py

This was tested with bitten svn-trunk (r1009). An example recipe is this:

<build xmlns:python="http://bitten.edgewall.org/tools/python"
       xmlns:svn="http://bitten.edgewall.org/tools/svn"
       xmlns:sh="http://bitten.edgewall.org/tools/sh"
       xmlns:c="http://bitten.edgewall.org/tools/c">
  <step id="checkout" description="Checkout source from repository">
    <svn:checkout url="http://localhost/svn/_demo" username="hey" password="secret"
        path="${path}" revision="${revision}" />
  </step>
  <step id="qmake" description="q">
    <sh:exec file="qmake" args="hello.pro" />
  </step>
  <step id="build" description="Compile">
    <c:make target="all"/>
  </step>
  <step id="build_test" description="Compile test-case">
    <sh:exec file="qmake" args="test.pro" />
    <c:make target="all"/>
  </step>
  <step id="run_test" description="Run Tests">
    <sh:exec file="./test" args="-xml -o ./test.xml"/>
    <c:qtestlib file="./*.xml"/>
  </step>
</build>

The corresponding source code for 'qmake_demo' is attached as a tar-ball.

[dvdkhlng <dvdkhlng ta gmx tod de>]

patch adding qtestlib support

[dvdkhlng <dvdkhlng ta gmx tod de>]

example qtestlib project, corresponding to recipe listed above

[hodgestar]

Hi! Thank you very much for the patch (and extra thanks for including a test!). A few quick questions:

• The test named test_empty_summary appears to test something decidedly non-empty. Does the test need a new name?

• Could you explain a bit about the bug in report/testing.py?

• Thanks for picking up the bug in ctools. If you have that fixed, would you mind submitting a bug report with a separate patch? Either way, I think these should land in separate commits.

[David Kühling <dvdkhlng@…>]

Hi, thanks for reviewing the patch.

• The test-case name was copy-pasted from cppunit. Grepping for the string, it looks like it originates from pythontools.py (so cppunit copied from there?). Yes you're right, the name makes no sense here. What about test_xml?
• Which bug in report/testing.py? AFAIK I didn't write about nor touch anything in report/*.
• I just opened a new ticket for the cppunit globbing bug commented about in ctools. As I haven't used cppunit before, I don't feel qualified to fix and test that. Just stumbled over the bug as the qtestlib raw structure was copy-pasted from cppunit, and I needed to make globbing work for qtestlib.

So if you don't object I'll just

• rename the test_empty_summary function
• remove the bug-reporting comment from ctools.cppunit

and resubmit a new patch that includes these 2 changes.

cheers,

David

[hodgestar]

Replying to David Kühling <dvdkhlng@…>:

• The test-case name was copy-pasted from cppunit. Grepping for the string, it looks like it originates from pythontools.py (so cppunit copied from there?). Yes you're right, the name makes no sense here. What about test_xml?

text_xml sounds good.

• Which bug in report/testing.py? AFAIK I didn't write about nor touch anything in report/*.

The comment on line 244 of the patch ends with "all results for fixture need to have the same filename, else report will contain multiple fixture rows (a bug in 'report/testing.py'?)".

• I just opened a new ticket for the cppunit globbing bug commented about in ctools. As I haven't used cppunit before, I don't feel qualified to fix and test that. Just stumbled over the bug as the qtestlib raw structure was copy-pasted from cppunit, and I needed to make globbing work for qtestlib.

Thanks.

So if you don't object I'll just

• rename the test_empty_summary function
• remove the bug-reporting comment from ctools.cppunit

and resubmit a new patch that includes these 2 changes.

Sounds great!

[David Kühling <dvdkhlng@…>]

Replying to hodgestar:

Replying to David Kühling <dvdkhlng@…>:

..

• Which bug in report/testing.py? AFAIK I didn't write about nor touch anything in report/*.

The comment on line 244 of the patch ends with "all results for fixture need to have the same filename, else report will contain multiple fixture rows (a bug in 'report/testing.py'?)".

Now I see what you mean. Didnt't touch this code for about a year, and just forgot about the issue.

The underlying model for test-cases in bitten (or report/testing.py for that matter) is just a list of objects with attributes "fixture, name, file, status, ...". Where "name" is the name of the specific test in the fixture.

However, "file" is not considered a per-test attribute, but treated like part of the fixture name. I.e. if there are two test results with both fixture="myfixture" but one test having file="test1.cpp" while the other test has file="test2.cpp"; then report/testing.py is going to list these two tests like belonging to two different fixtures. I.e. under the "Test Results" tab it's going to show two separate rows reading "myfixture".

For javatools/junit that may not have been a problem, as java has one file per class. Not so in C++ where member-functions of a class may span multiple C files, combined at the linking stage.

In theory for Qtestlib you may have test-fixtures spanning multiple files (altough that may be considered ugly), and in such a case Bitten would consider the results as belonging to two fixtures (having the same name) instead of one fixture.

IIRC this was a problem for qtestlib when trying to add the filenames of assertions failures to the test's file="" attribute (which may occur somewhere else, not directly in the fixture itself). That may have been a deliberate mis-use of the file="" attribe, though.

Maybe in Java it is possible to have distinct test-fixtures in distinct files that share the same fixture name? In this case the current handling of filename-defines-fixture would be correct.

So not sure, whether it's a bug, and whether it needs fixing. Fixing would require touching the SQL statements in report/testing.py which are beyond my understanding :)

[David Kühling <dvdkhlng@…>]

updated version (v3) of the patch

[David <dvdkhlng@…>]

Updated patch (v4): major code cleanup, more robustness against malformed XML

[David Kühling <dvdkhlng@…>]

Slight fix: remove bogus comments about cppunit that had slipped back into the v4 patch