Edgewall Software

Opened 10 years ago

Last modified 6 years ago

#490 new defect

"test code coverage" is inaccurate..

Reported by: anonymous Owned by:
Priority: major Milestone: 0.6.1
Component: Build master Version: 0.6b2
Keywords: Cc:
Operating System: Linux


b_set_step_info = True

steps = {
    'recover': {
        'module': 'recover.recover_main',
        'predecessor': None,
        'param': {
#           'param1': '111',

# vim: sw=4 ts=4 bg=dark

When I unittest above code, "trace.py" reports that line is 6 and coverage is 100%.

However, bitten extracts different result from this report.(lines of code is 9 and coverage is 67%)

Attachments (4)

bitten_result.jpg (14.2 KB) - added by anonymous 10 years ago.
bitten_result.GIF (13.6 KB) - added by anonymous 10 years ago.
bitten_result.gif (13.6 KB) - added by anonymous 10 years ago.
trace_result.jpg (19.4 KB) - added by anonymous 10 years ago.

Download all attachments as: .zip

Change History (10)

Changed 10 years ago by anonymous

Changed 10 years ago by anonymous

Changed 10 years ago by anonymous

comment:1 Changed 10 years ago by osimons

Could you please add the step with commands you use (from recipe). As Bitten supports several tools for python coverage (trace, coverage, figleaf) they may well produce different results - and any mismatch between runner and reporter could naturally also affect results.

However the Bitten unittest command should use trace as default, and when using python:trace reporting you should see same results as in your shell-testing.

comment:2 Changed 10 years ago by reporter

<python:distutils command="unittest" options="--xml-output build/test-results.xml --coverage-summary build/test-coverage.txt --coverage-dir build/coverage --test-suite=tests"/>
<python:unittest file="build/test-results.xml"/>
<python:trace summary="build/test-coverage.txt" coverdir="build/coverage" include="src*" />

Unfortunately I use 'trace'..

In my opinion, the comment line(line 8) causes this case.

If I delete it, coverage will be 100%. However, covered line is still different.

P.S. Could you delete the "bitten_result.jpg"? I want to upload another one.

comment:3 Changed 10 years ago by reporter

Sorry.. Not the "bitten_result.jpg", please delete the "trace_result.jpg".

comment:4 Changed 10 years ago by osimons

Attachment gone - you'll upload another with same name, right?

As for the problem... There is no setup.cfg with a [unittest] section that tells it to use some other coverage tool?

Here is how things work:

Together with the code, you can use options for slave to keep the build results. That should allow you to quite easily verify where things go astray. Typically the -k option for bitten-slave to 'keep files', even more useful when combined with --work-dir=DIR to request where you want the build to happen (easier to locate than inside a temp folder). All coverage information should then be retrievable, and comparing on-disk results with your manual run and the information that ends up on master should provide a good starting point.

Changed 10 years ago by anonymous

comment:5 Changed 10 years ago by reporter

"trace_result.jpg" is kept file from bitten-slave's "build/coverage".

  • line 9, 10, 11 are not flagged by _run_with_trace()(= trace.py).
  • trace() method of pythontools.py doesn't count line 9, 10, 11 because of line 8.
    1. line 8 is not a code line. so prev_hits is 0
    2. line 9 is not flagged and prev_hits is 0, so this is not covered.
    3. line 10, 11 is iterated..

comment:6 Changed 10 years ago by osimons

  • Milestone changed from 0.6 to 0.6.1

I won't look more into this before 0.6 final is released, so unless someone makes a working patch for this issue, 0.6.1 is the more likely release for getting this fixed.

Add Comment

Modify Ticket

Change Properties
Set your email in Preferences
as new The ticket will remain with no owner.

E-mail address and user name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.