Test¶
This page describes the
invoke_tasklib.testing module, which provides tasks to run
doctests, unit/integration/functional tests, and benchmarks with pytest.
Overview¶
| Task | Behavior |
|---|---|
test.doctest |
Runs doctests on source code and markdown files |
test.doctest-src |
Runs doctests on source code |
test.doctest-markdown |
Runs doctests on Python examples in markdown files |
test.unit |
Runs unit tests |
test.integration |
Runs integration tests |
test.functional |
Runs functional tests |
test.all |
Runs all tests (unit, integration, and functional) |
test.coverage-report |
Generates an HTML/terminal report from existing coverage data |
test.benchmark |
Runs performance benchmarks |
All test tasks (except test.benchmark, test.doctest, and test.doctest-markdown) run pytest
with --xdoctest enabled, so doctests embedded in the code under test are also collected. Set
tasklib.test.xdoctest: false if the project does not have the xdoctest package. The test scopes
also run with --timeout (the timeout of the scope, see below); set tasklib.test.timeout: false
if the project does not have the pytest-timeout package.
Running Doctests¶
invoke test.doctest
Runs both test.doctest-src and test.doctest-markdown.
invoke test.doctest-src
Runs doctests against paths.src from the resolved config. It always runs
pytest --xdoctest: tasklib.test.xdoctest only applies to the test scopes, since the task has no
purpose without the plugin. If the xdoctest package is missing, the task stops with a message that
names it; use test.doctest-markdown alone in a project that does not have the package. If
paths.src does not exist (and --paths is not set), the task exits with 2 and names
tasklib.paths.src.
invoke test.doctest-markdown
Finds every *.md file in the project (see below for the files that are left out) and runs, for
each one:
python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS -o REPORT_NDIFF <file>.md
so Python code examples embedded in markdown files (README, docs, ...) stay correct and up to
date. NORMALIZE_WHITESPACE and ELLIPSIS make output comparisons more forgiving (whitespace
differences are ignored, and ... in expected output matches any text), and REPORT_NDIFF gives
a readable diff when an example's output doesn't match. A markdown file with no >>> doctest
blocks is still visited but simply reports zero tests, so it always "passes".
The task reports the number of files it checks: the files of --paths when it is set, otherwise
every markdown file of the project. As for the shell scripts (see Format), the files are
the ones that git lists in a git work tree, so the rules of .gitignore are followed and a package
named build or site is not skipped; outside a git work tree a walk skips the directories of the
environments and of the caches of the tools, and build, dist, site and htmlcov (the
complete list is in Format).
Running Tests by Scope¶
invoke test.unit
invoke test.integration
invoke test.functional
Each targets its own path (paths.unit_tests, paths.integration_tests,
paths.functional_tests), with a longer default timeout (60s vs 10s) for integration and
functional tests, since they typically exercise more of the system. The timeouts are the
test.unit_timeout, test.integration_timeout and test.functional_timeout values of the
config. If the directory of test.integration or
test.functional does not exist, the task prints a message and skips it instead of failing with a
pytest error. test.unit does the same, except that when paths.unit_tests does not exist but
paths.tests does (a flat tests/ layout), it runs the whole paths.tests directory.
invoke test.all
Runs everything in paths.tests in a single pytest invocation, with the largest of the three timeouts
(60s by default, since the integration and functional tests are part of that directory). If
paths.tests does not exist, the task prints a message and skips, like the other scopes.
Coverage¶
Pass --cov to test.unit, test.integration, test.functional, or test.all to generate
coverage reports. The formats are the test.cov_reports value of the config
(html,xml,term by default, each one is a --cov-report of pytest-cov, with an optional argument
such as term-missing:skip-covered or html:cov_html; an empty value leaves the choice to
pytest-cov):
invoke test.unit --cov
A run with --cov replaces the coverage data of the previous run. test.integration and
test.functional have an --append option that adds to the existing data (--cov-append), so
they compose with a prior test.unit --cov run to build up combined coverage across scopes:
invoke test.unit --cov
invoke test.integration --cov --append
invoke test.functional --cov --append
Without --append, invoke test.integration --cov reports only the integration tests, whatever
a previous run left in .coverage. --append without --cov does nothing.
If tasklib.test.cov_fail_under is set, it is passed as --cov-fail-under and the task fails when
the coverage is lower. With --append, the threshold applies to the combined data: start with
test.unit --cov (it replaces the data) or coverage erase, otherwise the result depends on what
was run before.
invoke test.coverage-report
Runs python -m coverage (the interpreter of the tests) to regenerate the HTML report and prints
the terminal summary from whatever coverage data (.coverage) is currently on disk — the data
produced by an earlier --cov run — without re-running the tests. The report goes to the directory
of the html:<dir> value of test.cov_reports if there is one, else to the directory of
[tool.coverage.html] in pyproject.toml, else to htmlcov/.
Pass --open-browser to also open its index.html in your default browser:
invoke test.coverage-report --open-browser
Benchmarks¶
invoke test.benchmark
Runs pytest --benchmark-only --benchmark-enable against paths.benchmarks (--benchmark-enable
measures even if the pytest config has --benchmark-disable in addopts, which this repository
uses so that a normal test run executes each benchmark once). This requires
pytest-benchmark to be installed. The task fails with
the exit code 1 if paths.benchmarks does not exist (and --paths is not given): it is run on
purpose, so a scheduled job does not pass without measuring.
test.benchmark, test.doctest-src and test.doctest-markdown take --args and --paths, and
test.coverage-report takes --args (for example --fail-under=90).
See Also¶
- Config: the
paths.*_testsandpaths.benchmarksconfig used by these tasks. invoke_tasklib.testingreference- Troubleshooting: fixes for "command not found" errors.