Edgewall Software

Opened 9 years ago

Closed 9 years ago

#564 closed defect (fixed)

Another py.test patch: Ignore ignored tests

Reported by: Torsten Landschoff <torsten.landschoff@…> Owned by: hodgestar
Priority: minor Milestone: 0.6
Component: Recipe commands Version: 0.6b2
Keywords: Cc:
Operating System: BSD


After #562 went so smooth, here is the next patch :-)

I just see that it does not quite result in what I wanted - the test result does not show these as "ignores" as I would have expected.

Anyway, marking tests as xfail in py.test results in a <skipped/> test result. This is reported as failure by Bitten.

Test case and patch attached.

Attachments (2)

junit_skipped.patch (2.6 KB) - added by Torsten Landschoff <torsten.landschoff@…> 9 years ago.
Do not consider skipping a test a failure.
junit_skipped.2.patch (2.9 KB) - added by hodgestar 9 years ago.
Patch that uses the ignore status for skipped tests.

Download all attachments as: .zip

Change History (10)

Changed 9 years ago by Torsten Landschoff <torsten.landschoff@…>

Do not consider skipping a test a failure.

comment:1 Changed 9 years ago by hodgestar

  • Owner set to hodgestar
  • Status changed from new to assigned

I'll have to confer a bit on what Bitten's general policy is regarding skipped tests but the patch looks good otherwise. Thank you for taking the time to submit them (and keep them coming :).

comment:2 Changed 9 years ago by osimons

Not having looked at details of the patch or its background, I'd think your want to use status 'ignore' instead of 'skipped'. As seen in bitten.report.testing file, a test can be either 'success' or 'ignore' - anything else is considered a failure by the reporting tools. Report table and charting should (!?) accomodate this without any special handling on the tool side.

comment:3 Changed 9 years ago by osimons

Sorry, trying to keep one eye on charting and one eye on table code and failing... Suggestion for use of 'ignore' still stands, as it is one of 4 recognized states: success, failure, error, ignore.

comment:4 Changed 9 years ago by hodgestar

  • Milestone changed from 0.6.1 to 0.6

Changed 9 years ago by hodgestar

Patch that uses the ignore status for skipped tests.

comment:5 Changed 9 years ago by hodgestar

I've added a modified version of the patch that ensure that the py.test statuses are converted to one of success, failure, error or ignore. The failure and error statuses increment the failed counter. I think the patch is ready to commit but I'd like Torsten to confirm that the new patch works for him before I do so.

comment:6 Changed 9 years ago by Torsten Landschoff <torsten.landschoff@…>

Thanks for the updated patch. I installed it and it is in fact better than my patch (as expected).

The test result view now actually shows the xfailed tests as "ignores" and they are counted into the total number of tests. With my patch, ignores would be 0 and the total would not include the skipped tests.

Ship it, if you ask me! :-)

Greetings, Torsten

comment:7 Changed 9 years ago by anatoly techtonik <techtonik@…>

Native regrtest.py from python explicitly says how many tests skipped and displays why they were skipped. Do we have such ability? For example some tests may fail on Windows and should be skipped, but when compatibility is added in the future, they should run ok. The problem is to find skipped tests then.

comment:8 Changed 9 years ago by hodgestar

  • Resolution set to fixed
  • Status changed from assigned to closed

Patch applied in r855 and backported to 0.6.x in r856.

Add Comment

Modify Ticket

Change Properties
Set your email in Preferences
as closed The owner will remain hodgestar.
The resolution will be deleted. Next status will be 'reopened'.

E-mail address and user name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.