Skip to content

Test

📖 This page describes the invoke_tasklib.testing module, which provides tasks to run doctests, unit/integration/functional tests, and benchmarks with pytest.

Overview

Task Behavior
test.doctest Runs doctests on source code and markdown files
test.doctest-src Runs doctests on source code
test.doctest-markdown Runs doctests on Python examples in markdown files
test.unit Runs unit tests
test.integration Runs integration tests
test.functional Runs functional tests
test.all Runs all tests (unit, integration, and functional)
test.coverage-report Generates an HTML/terminal report from existing coverage data
test.benchmark Runs performance benchmarks

All test tasks (except test.benchmark, test.doctest, and test.doctest-markdown) run pytest with --xdoctest enabled, so doctests embedded in the code under test are also collected. Set tasklib.test.xdoctest: false if the project does not have the xdoctest package. The test scopes also run with --timeout (the timeout of the scope, see below); set tasklib.test.timeout: false if the project does not have the pytest-timeout package.

Running Doctests

invoke test.doctest

Runs both test.doctest-src and test.doctest-markdown.

invoke test.doctest-src

Runs doctests against paths.src from the resolved config. It always runs pytest --xdoctest: tasklib.test.xdoctest only applies to the test scopes, since the task has no purpose without the plugin. If the xdoctest package is missing, the task stops with a message that names it; use test.doctest-markdown alone in a project that does not have the package. If paths.src does not exist (and --paths is not set), the task exits with 2 and names tasklib.paths.src.

invoke test.doctest-markdown

Finds every *.md file in the project (see below for the files that are left out) and runs, for each one:

python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS -o REPORT_NDIFF <file>.md

so Python code examples embedded in markdown files (README, docs, ...) stay correct and up to date. NORMALIZE_WHITESPACE and ELLIPSIS make output comparisons more forgiving (whitespace differences are ignored, and ... in expected output matches any text), and REPORT_NDIFF gives a readable diff when an example's output doesn't match. A markdown file with no >>> doctest blocks is still visited but simply reports zero tests, so it always "passes". The task reports the number of files it checks: the files of --paths when it is set, otherwise every markdown file of the project. As for the shell scripts (see Format), the files are the ones that git lists in a git work tree, so the rules of .gitignore are followed and a package named build or site is not skipped; outside a git work tree a walk skips the directories of the environments and of the caches of the tools, and build, dist, site and htmlcov (the complete list is in Format).

Running Tests by Scope

invoke test.unit
invoke test.integration
invoke test.functional

Each targets its own path (paths.unit_tests, paths.integration_tests, paths.functional_tests), with a longer default timeout (60s vs 10s) for integration and functional tests, since they typically exercise more of the system. The timeouts are the test.unit_timeout, test.integration_timeout and test.functional_timeout values of the config. If the directory of test.integration or test.functional does not exist, the task prints a message and skips it instead of failing with a pytest error. test.unit does the same, except that when paths.unit_tests does not exist but paths.tests does (a flat tests/ layout), it runs the whole paths.tests directory.

invoke test.all

Runs everything in paths.tests in a single pytest invocation, with the largest of the three timeouts (60s by default, since the integration and functional tests are part of that directory). If paths.tests does not exist, the task prints a message and skips, like the other scopes.

Coverage

Pass --cov to test.unit, test.integration, test.functional, or test.all to generate coverage reports. The formats are the test.cov_reports value of the config (html,xml,term by default, each one is a --cov-report of pytest-cov, with an optional argument such as term-missing:skip-covered or html:cov_html; an empty value leaves the choice to pytest-cov):

invoke test.unit --cov

A run with --cov replaces the coverage data of the previous run. test.integration and test.functional have an --append option that adds to the existing data (--cov-append), so they compose with a prior test.unit --cov run to build up combined coverage across scopes:

invoke test.unit --cov
invoke test.integration --cov --append
invoke test.functional --cov --append

Without --append, invoke test.integration --cov reports only the integration tests, whatever a previous run left in .coverage. --append without --cov does nothing.

If tasklib.test.cov_fail_under is set, it is passed as --cov-fail-under and the task fails when the coverage is lower. With --append, the threshold applies to the combined data: start with test.unit --cov (it replaces the data) or coverage erase, otherwise the result depends on what was run before.

invoke test.coverage-report

Runs python -m coverage (the interpreter of the tests) to regenerate the HTML report and prints the terminal summary from whatever coverage data (.coverage) is currently on disk — the data produced by an earlier --cov run — without re-running the tests. The report goes to the directory of the html:<dir> value of test.cov_reports if there is one, else to the directory of [tool.coverage.html] in pyproject.toml, else to htmlcov/. Pass --open-browser to also open its index.html in your default browser:

invoke test.coverage-report --open-browser

Benchmarks

invoke test.benchmark

Runs pytest --benchmark-only --benchmark-enable against paths.benchmarks (--benchmark-enable measures even if the pytest config has --benchmark-disable in addopts, which this repository uses so that a normal test run executes each benchmark once). This requires pytest-benchmark to be installed. The task fails with the exit code 1 if paths.benchmarks does not exist (and --paths is not given): it is run on purpose, so a scheduled job does not pass without measuring.

test.benchmark, test.doctest-src and test.doctest-markdown take --args and --paths, and test.coverage-report takes --args (for example --fail-under=90).

See Also