The f🥇rst test is the hardest to write.
Tests are investments and testing over time measures the return on investment. Testing promotes
Often authors defer to notebooks to experiment and test computational ideas. There are testing tools for complete notebooks like informal tests with nbconvert or formal tests with nbval. These tests approaches apply tests to static documents outside of the interactive computing context.
Remember 💭 Restart and run all or it didn't happen.
This essay discusses using formal testing tools during interactive computing contexts in modern computational notebooks. These tools include: doctest, unittest, and pytest.
This topic has been discussed in some of our past presentations.
This document commonly uses assert
statements to indicate a test is happening. Assertions are useful in testing, but not in production. assert
statements can introduce potential security vulnerabilites for optimized python code.
doctest
is the simplest Python testing tool to use in the notebook. doctest
s are tests that live in strings and Python docstrings, and >>>
indicates a place where a test is run.
doctest
discovers tests in the docstrings of functions, classes, modules, and a special dictionary __test__
.
def a_function_with_a_doctest(i):
"""Converts `i` into a string representation
>>> assert all(isinstance(
... a_function_with_a_doctest(object), str) for object in (range(10), 1))
"""
return str(i)
__doc__
the module docstring is tested.
__doc__ = """
For the sake of example, the docstring for the module is created if it doesn't exist
or appended to the original docstring.
Doctest registers the statement below as a test.
>>> assert True
"""
if __name__ == '__main__': print(__import__('doctest').testmod())
TestResults(failed=0, attempted=2)
__test__
is dictionary object that may contain named doctest-able objects like strings, functions, classes, and modules. The key of the dictionary in the test name.
__test__ = {
'a_doc_test': """This is a docstring with tests that will be run.
>>> assert True"""}
Conveniently, __test__
is defined using a private namespace, but this means that a notebook author could easily transport tests within their notebooks and modules without any change to their syntactic choices.
from unittest import TestCase, main as _main
unittest
collects classes subclasses as TestCase
.
class myTest(__import__('unittest').TestCase):
"""This is a unitest class with a docstring. `doctest`"""
def test_assertion(self): assert True
def main():
try: _main(argv=['discover'])
except SystemExit: ...
main()
. ---------------------------------------------------------------------- Ran 1 test in 0.002s OK
The snippet below includes doctest
s within unittest
s.
from doctest import DocTestSuite
def load_tests(loader, tests, ignore):
tests.addTests(DocTestSuite(__import__(__name__)))
return tests
main()
. ---------------------------------------------------------------------- Ran 1 test in 0.002s OK
We can observe that more tests were collection because the load_tests
object exists.
pytest
is one of the most common frameworks in python used for testing. It has an extensive plugin framework and ecosystem that allows non experts to create robust testing frameworks. pytest
generally requires less syntax that traditional doctests especially when combined with pytest.fixtures
.
pytest
discovery is configurable, but by default tests beginning with test_*
will be discovered.
def test_thing(a: (range(10), 1)):
assert isinstance(a_function_with_a_doctest(a), str)
pytest
automated collections unittests and has options for doctesting.
A notebook author will often use IPython magics and other IPython specific API's. These applications require the test be run with an active IPython context. Most examples of pytest
run tests using vanilla pytest by invoking either pytest
or python -m pytest
at the command line. In these situations, the IPython context is unavailable.
The code snippet below shares an application of running pytest
through IPython using
ipython -m doctest --
where any argument following --
import pytest
As stated, pytest
discovery is configurable; notebooks are not python files so we must modify the way tests are found using pytest_collect_file
.
def pytest_collect_file(parent, path):
if path.ext == ".ipynb": return pytest.Module(path, parent)
Run pytest
and collect notebook files.
def _run_pytest_discovery():
pytest.main('--collect-only -p no:pytest-importnb 2018-07-31-Testing-notebooks.ipynb'.split(), [__import__(__name__)])
pytest
¶As far as we know,
pytest
can not run the current context in an IPython session. A Pytest runner is required.
doctest
s run tests in strings and docstrings .unittest
s run tests on objects and may include doctest
.pytest
runs tests on objects and may include doctest
and unittest
tests.In general, we end a lot of our essays with doctest
to promote better documentation and reuse during an interactive session. pytest
is applied to our files or collections of files.