[Patch] Display junit test errors

[krijnschaap@…]

When junit generates an error, the resulting XML contains an <error> tag with a detailed description of the error. It would be nice if those errors could be visible through the trac interface, as in some situations the details aren't included in the logview from the build command (my windows-run ant did show them, the linux variant didn't).

[olemis@…]

Recently ​junitxml project has been created. According to ​the message sent to TiP list, its author states that it includes a very lean junit xml style reporter for pyunit which is (almost ? ) compatible with ​Ant's XML reports for JUnit task.

_{Note: In TiP mailing list there are some threads about language-agnostic reporting tools, protocols for better integration of testing frameworks and other tools (e.g. CI )}

IMO this could be really cool and useful to abstract the way slaves report test results to the master (i.e. a single implementation based on XML and useful for multiple languages and environments), but perhaps this complicates a little the dependencies tree.

_{BTW: Does anybody know if the XML schema is published somewhere out there in cyberspace ? I found ​a reference but I'm in doubt about how much official it is}

[Mat Booth <trac@…>]

Old test report look

[Mat Booth <trac@…>]

New test report look

[Mat Booth <trac@…>]

New test report look expanded

[Mat Booth <trac@…>]

Patch against r708

[Mat Booth <trac@…>]

Old test report look

[Mat Booth <trac@…>]

New test report look

[Mat Booth <trac@…>]

New test report look expanded

[Mat Booth <trac@…>]

Hi,

I've moved this back to 0.6.0 because I'd like you to consider this patch that I've been working on that allows you to see the stack traces generated by unit tests in the test report on the build page.

This is the patch: attachment:ticket:205:bitten.patch​

I have tested this patch with Trac/SQLite and Trac/MySQL and using Junit3 and Junit4 unit tests.

Here's how the test report looks now, in r708:

And here's how my patch will make it look:

As you can, there is a expand handle next to the fixture that contains failed tests. Here's how it looks expanded:

[osimons]

Thanks for the patch! I don't see a problem including a feature that is both useful and quite complete. It also works on my Python unittests.

However, I'm not particularly fond of the visuals in the report summary and would look for some changes there:

1. Patch removes general linking to files in source browser - I'd like to see that stay, most likely by having the collapse/expand icon and name link being separate links. Click on file to browse, click on expand to see details. Or some other variation.
2. Expansion should work the other way around, so that all details display if javascript is disabled, but a jQuery call on the page collapses sections when page loads.
3. I get an all-red background for details on hover. Not nice :-)
4. There should be some more space around each detail to make them visually separate. Perhaps just some padding at top and bottom to separate them and make them more readable.

That's what I spotted on first review at least. It seem to work well - great work! Could you tweak the visuals please, and I'll make some time soon to work through it in detail and get it integrated.

[Mat Booth <trac@…>]

Replying to osimons:

Thanks for the patch! I don't see a problem including a feature that is both useful and quite complete. It also works on my Python unittests.

Nice one, it's good to know I haven't broken other kinds of tests. I really only know what I'm doing when I'm using Junit.

Replying to osimons:

However, I'm not particularly fond of the visuals in the report summary and would look for some changes there:

1. Patch removes general linking to files in source browser - I'd like to see that stay, most likely by having the collapse/expand icon and name link being separate links. Click on file to browse, click on expand to see details. Or some other variation.

Yes, I couldn't think of a clean way (and my CSS-fu isn't great) to include both an expand link and a source link. I made the fixture name into the expand link to maintain consistency with the Trac's admin/general/plugin page, which uses the same expand mechanism. I'll see what I can come up with.

Replying to osimons:

1. Expansion should work the other way around, so that all details display if javascript is disabled, but a jQuery call on the page collapses sections when page loads.

That *is* how it works, at least in FF3.5 in which I tested it. It works the same way as Trac's admin/general/plugin page (uses the same code, in fact). The jQuery call I added to the top of bitten_build.html toggles the "collapsed" style on at start up. (See ​http://trac.edgewall.org/browser/tags/trac-0.11.5/trac/htdocs/js/folding.js)

Replying to osimons:

1. I get an all-red background for details on hover. Not nice :-)

I don't like it either, but it did that before I got here I'm afraid. (See source:/trunk/bitten/htdocs/bitten.css#L162​) I did make it more reliable though because sometimes that style wasn't being applied (there was a logic error detecting failures). I have no problem with changing it. :-)

Replying to osimons:

1. There should be some more space around each detail to make them visually separate. Perhaps just some padding at top and bottom to separate them and make them more readable.

Indeed, I will do that.

Replying to osimons:

That's what I spotted on first review at least. It seem to work well - great work! Could you tweak the visuals please, and I'll make some time soon to work through it in detail and get it integrated.

Thank you very much for the feedback, I should be able to submit another patch this evening.

[osimons]

Replying to Mat Booth <trac@…>:

That *is* how it works, at least in FF3.5 in which I tested it.

You are right - my mistake. Must be something I did when disabling javascript & refreshing - looks correct on second attempt :-)

Look forward to updated patch, and as part of integrating I'll take care of checking tests and adding a new test to check the extraction of error data - unless you you want to take a stab at making a new test as part of the patch :-)

[Mat Booth <trac@…>]

Right then, how's this? What I've done is make the expand handle it's own link and kept the the link to the source on the fixture name, added some padding to the failed test detail and tweaked the colours to make it more readable.

[Mat Booth <trac@…>]

And another screenshot:

[osimons]

Replying to olemis@…:

_{BTW: Does anybody know if the XML schema is published somewhere out there in cyberspace ? I found ​a reference but I'm in doubt about how much official it is}

No idea, but I see that Bitten does not even have a testcase for parsing an example report so I have really no idea other than seeing how the parser iterates and read. Id be grategul if anyone can help out with a small example report + human-readable results roughly according to Bitten presentation of results.

Getting a proper test-case for this may also be reused for #252.

[osimons]

Mat, your patch looks really good. I've added a test-case for it tonight, and will take another read through the code over the weekend and commit it early next week. Thanks for your efforts - really appreciated!

[Mat Booth <trac@…>]

Replying to osimons:

Replying to olemis@…:

_{BTW: Does anybody know if the XML schema is published somewhere out there in cyberspace ? I found ​a reference but I'm in doubt about how much official it is}

No idea, but I see that Bitten does not even have a testcase for parsing an example report so I have really no idea other than seeing how the parser iterates and read. Id be grategul if anyone can help out with a small example report + human-readable results roughly according to Bitten presentation of results.

Getting a proper test-case for this may also be reused for #252.

If it helps, the following attachment is the report for the failed fixture I was using in the screenshots above.

[osimons]

Thanks for the patch Mat - committed as-is (+ my testcase) in [710].

PS! Thanks for the xml test output as well. I've made a note on my todo, and will get around to it someday...

[osimons]

The expand/collapse feature of inline details only really works consistently when using Firefox.

• With Safari 4, no icon is shown and the filename isn't padded either - so even if it is there it isn't shown.
• With IE 8, the padding is there, but no visible icon - it works when clicking the emtpy white area, but not exactly intuitive..

Need to look into this.

[osimons]

This is painful... Quite a few browser-differences here, and I can't yet make it work for all browsers. This patch makes it work reliably in recent/latest versions of all browsers:

bitten/htdocs/bitten.css

@@ -163 +163 @@
 /* collapsible failure details */
 #content.build table.tests tr th p { margin: 0; padding: 0; text-align: left; }
 #content.build table.tests tr th p.details {
-  margin: 0; padding-left: 32px; padding-top: 5px; text-align: left; font-weight: normal;
+  margin: 0; padding-left: 4px; padding-top: 5px; text-align: left; font-weight: normal;
 }
 #content.build table.tests tr th p.details span {
   white-space: pre; font-family: monospace; font-weight: normal; font-size: smaller;
   color: #666;
 }
-#content.build table.tests .fixture { display: inline; }
+#content.build table.tests .fixture { display: inline-block; }
 #content.build table.tests tr.failed th .fixture a { 
-  background: url(../common/expanded.png) 0 50% no-repeat; padding-left: 16px;
+  background: url(../common/expanded.png) 50% 50% no-repeat; padding-left: 16px;
 }
 #content.build table.tests tr.failed th.collapsed .fixture a {
   background-image: url(../common/collapsed.png);

I won't work with IE7 and likely not with FF2 either. Comments or ideas?

[Mat Booth <trac@…>]

Oh man, that's unfortunate. I finally got round to putting Windows on a machine the other week, so I'll take a look this evening.

[osimons]

Did you find anything to improve this, Mat? If not, I think I'll just commit what I have and at least support latest versions of all major browsers - not 'perfect', but at least better than current situation.

[osimons]

I've committed my tweak in [792], and at least it works ok with latest versions. Please reopen if anyone sees ways of improving it.