importnb
and nbval
importnb, pidgin
are complimented nicely by nbval
. nbval
+ importnb
performs
normal test discovery AND compares the out results. Doctests may be included also.
with __import__('IPython').utils.capture.capture_output(stdout=True):
if __name__ == '__main__':
no_nbval = !ipython -m pytest -- 2018-11-23-importnb-and-nbval.ipynb --collect-only
with_nbval = !ipython -m pytest -- 2018-11-23-importnb-and-nbval.ipynb --nbval --collect-only
with_nbval_doctests = !ipython -m pytest -- 2018-11-23-importnb-and-nbval.ipynb --doctest-modules --nbval --collect-only
no_nbval_doctests = !ipython -m pytest -- 2018-11-23-importnb-and-nbval.ipynb --doctest-modules --collect-only
for report in (no_nbval, with_nbval, with_nbval_doctests, no_nbval_doctests):
while not report[0].startswith('collected'): report.pop(0)
while report[-1].startswith('==='): report.pop()
while not report[-1].strip(): report.pop()
Include a function that is found with normal pytest
discovery.
def test_function(): """This will not be tested."""
Include a doctest
def a_function_with_a_doctest(): """>>> assert True"""
Don't forget about the underutilized __test__
object
recognized by doctest
.
__test__ = {'another doctest': ">>> assert 100"}
Assert that test is found.
if __name__ == '__main__':
assert '.__test__.' in ''.join(with_nbval_doctests)
with open('2018-11-23-importnb-and-nbval.ipynb') as file: cells = len(
list(object for object in __import__('nbformat').read(file, 4)['cells']
if object['cell_type'] == 'code' and ''.join(object['source']).strip()))
F"""There are {cells} code cells."""
'There are 8 code cells.'
if __name__ == '__main__':
import pandas
df = pandas.DataFrame(list(map(list, """100\n110\n111\n101""".splitlines())), columns=[
'importnb', 'nbval', 'doctests'
], index=[
'importnb', 'nbval', 'all', 'doctests'
]).pipe(lambda df: df.set_index(list(df.columns), append=True))
collected = pandas.concat(dict(zip(df.index.get_level_values(0), [
pandas.Series(object)
for object in (no_nbval, with_nbval, with_nbval_doctests, no_nbval_doctests)
]))).unstack().rename(columns={0:'collected'}).set_index('collected', append=True)
collected = collected.stack(
).str.split(expand=True)[0].str.lstrip('<').to_frame(
'object'
).reset_index(-1, drop=True)['object'].pipe(
lambda df: df.groupby([df.index, df.values]).count()
).unstack().pipe(
lambda df: df.assign(total=df.sum(axis=1))
).fillna('')
collected.pipe(display)
DoctestItem | Function | IPyNbCell | IPyNbFile | NotebookModule | total | |
---|---|---|---|---|---|---|
(all, collected 11 items) | 2 | 1.0 | 8 | 1 | 1.0 | 13.0 |
(doctests, collected 3 items) | 2 | 1.0 | 1.0 | 4.0 | ||
(importnb, collected 1 item) | 1.0 | 1.0 | 2.0 | |||
(nbval, collected 9 items) | 1.0 | 8 | 1 | 1.0 | 11.0 |
F"The difference in the numbers of tests is the {cells} code cells plus the number of modules or files. "
'The difference in the numbers of tests is the 8 code cells plus the number of modules or files. '
--nbval --doctest-modules
& importnb
ensure that a notebook can be tested in a quite a few ways. An author should not be limited in their mode of testing.
It would be great ideal to focus on notebooks as tests rather than experimentation. Early in the ideation process there is a little difference between source code and test code. There are tools in the notebook ecosystem that seamlessly work with a stack of existing notebooks. Using the suite of pytest
notebook testing tools nbval
and importnb
will assist authors in writing more durable notebooks.