Using py.test with coverage doesn't include imports
@hynekcer gave me the right idea. But basically the easiest solution lies somewhere else:
Get rid of pytest-cov!
Use
coverage run --source jedi -m py.test
coverage report
instead!!! This way you're just running a coverage on your current py.test configuration, which works perfectly fine! It's also philosophically the right way to go: Make each program do one thing well - py.test runs tests and coverage checks the code coverage.
Now this might sound like a rant, but really. pytest-cov hasn't been working properly for a while now. Some tests were failing, just because we used it.
As of 2014, pytest-cov seems to have changed hands. py.test --cov jedi test seems to be a useful command again (look at the comments). However, you don't need to use it. But in combination with xdist it can speed up your coverage reports.
I fixed the test coverage to 94% by this patch that simplifies import dependencies and by the command:
py.test --cov jedi test # or
py.test --cov jedi test --cov-report=html # + a listing with red uncovered lines
Uncovered lines are only in conditional commands or in some less used functions but all headers are completely covered.
The problem was that the tests configuration test/conftest.py did import prematurely by dependencies almost all files in the project. The conftest file defines also additional command line options and settings that should be set before running the test. Therefore I think that pytest_cov plugin works correctly if it ignores everything that was imported together with this file, although it is a pain. I excluded also __init__.py and settings.py from the report because they are simple and with the complete coverage but they are also imported prematurely in the dependency of conftest.
In my case, all the tests run, but coverage was 0%.
The fix was:
$ export PYTHONPATH="."
After the results were correct.
I had in past few problems with py.test command having problems to import something and setting the PYTHONPATH env var was the solution. It worked for me this time too.
My real example with awslogs
First with PYTHONPATH unset:
$ py.test --cov=awslogs tests/
========================================= test session starts =========================================
platform linux2 -- Python 2.7.9, pytest-2.8.5, py-1.4.31, pluggy-0.3.1
rootdir: /home/javl/sandbox/awslogs/github/awslogs, inifile:
plugins: cov-2.2.0
collected 11 items
tests/test_it.py ...........Coverage.py warning: No data was collected.
--------------------------- coverage: platform linux2, python 2.7.9-final-0 ---------------------------
Name Stmts Miss Cover
-------------------------------------------
awslogs/__init__.py 2 2 0%
awslogs/bin.py 85 85 0%
awslogs/core.py 143 143 0%
awslogs/exceptions.py 12 12 0%
-------------------------------------------
TOTAL 242 242 0%
====================================== 11 passed in 0.38 seconds ======================================
Resulting coverage is 0%.
Then I set the PYTHONPATH:
$ export PYTHONPATH="."
and rerun the test:
$ py.test --cov=awslogs tests/
========================================= test session starts =========================================
platform linux2 -- Python 2.7.9, pytest-2.8.5, py-1.4.31, pluggy-0.3.1
rootdir: /home/javl/sandbox/awslogs/github/awslogs, inifile:
plugins: cov-2.2.0
collected 11 items
tests/test_it.py ...........
--------------------------- coverage: platform linux2, python 2.7.9-final-0 ---------------------------
Name Stmts Miss Cover
-------------------------------------------
awslogs/__init__.py 2 0 100%
awslogs/bin.py 85 9 89%
awslogs/core.py 143 12 92%
awslogs/exceptions.py 12 2 83%
-------------------------------------------
TOTAL 242 23 90%
====================================== 11 passed in 0.44 seconds ======================================
Now is the coverage 90%.
WARNING: Manipulating PYTHONPATH can have strange side effects. Currently I run into problem, that pbr based package is creating egg directory when building distributable and if PYTHONPATH is set to ".", it automatically considers the egg related package as installed. For this reason I stopped using pytest-cov and follow the advice to use coverage tool instead.