Modern notebooks are treated as disposable. The long term value of a notebook is lacking because of programming style, not the technology.
Many scientists and journalists frequently open blank notebooks to informally test ideas. This document discussing the value introducing formal testing into the notebook development process sooner. In this approach, early ideas last longer.
Individual contributors must enjoy the same software engineering benefits as members of organizations. To compete with larger research machines the individual needs:
A successful notebook project will mature notebook source to Python source, but it will try to retain as many notebooks as possible for testing.
The rest of this document discusses tips for successfully maturing notebook projects.
%reload_ext pidgin
* `nbval` will ensure that all of the code cells are executed.
* `importnb` will discover named tests with `pytest`.
* __[.travis.yml]({{name}}/{{name}}.travis.yml)__ will evaluate the tests in continuous integration.
* `nbsphinx` is configured in __conf.py__ so that notebooks are our documentation and may be deployed on
[readthedocs](https://readthedocs.org/).
* __[setup.py]({{name}}/setup.py)__ is configured to make all source within the package usable.
At the beginning of a project, there is no difference between source, tests, and documentation.
Notebooks have the potential to create tests and documentation from the onset of a project.
import nbval, importnb, nbsphinx, nbformat, nbconvert, pytest, pathlib, os
nbval
will ensure that all of the code cells are executed.importnb
will discover named tests with pytest
.nbsphinx
is configured in conf.py so that notebooks are our documentation and may be deployed on
readthedocs.setup.py is configured to make all source within the package usable.
At the beginning of a project, there is no difference between source, tests, and documentation.
Notebooks have the potential to create tests and documentation from the onset of a project.
import nbval, importnb, nbsphinx, nbformat, nbconvert, pytest, pathlib, os
Consider a sample project with the `name` __{{name}}__. _This name is chosen because we are writing
__{{name}}__ plate code._
name = 'boiler'
Consider a sample project with the name
boiler. This name is chosen because we are writing
boiler plate code.
name = 'boiler'
The __{{name}}__ project requires the `packages` are available in the `"src"` directory. Reason for the `"src"` [directory
layout](https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure), redirected from [Good `pytest` integration practices](https://docs.pytest.org/en/latest/goodpractices.html#tests-outside-application-code).
packages = F"""{name}/src/{name}
{name}/src/{name}/docs""".splitlines()
packages
['boiler/src/boiler', 'boiler/src/boiler/docs']
The boiler project requires the packages
are available in the "src"
directory. Reason for the "src"
directory
layout, redirected from Good pytest
integration practices.
packages = F"""{name}/src/{name}
{name}/src/{name}/docs""".splitlines()
packages
For each of the `packages` we should make the corresponding directories.
for package in packages: pathlib.Path(package).mkdir(exist_ok=True, parents=True)
For each of the packages
we should make the corresponding directories.
for package in packages: pathlib.Path(package).mkdir(exist_ok=True, parents=True)
For computational narratives, readme.ipynb is for people as init.py is for Python.
Our __init__.py files will __import__ the readme into its namespace.
_init_file = """with __import__('importnb').Notebook():
from . import readme
from .readme import *
""";
Any code in the default __readme.ipynb__ files will be available within the module.
Our init.py files will import the readme into its namespace.
_init_file = """with __import__('importnb').Notebook():
from . import readme
from .readme import *
""";
Any code in the default readme.ipynb files will be available within the module.
### Configuring the docs
for package in packages:
with pathlib.Path(package, 'readme.ipynb').open('w') as file:
nbformat.write(
nbformat.v4.new_notebook(cells=[nbformat.v4.new_markdown_cell(
"""[readme](readme.ipynb)""", metadata={"nbsphinx-toctree": {}})]), file)
for package in packages:
with pathlib.Path(package, 'readme.ipynb').open('w') as file:
nbformat.write(
nbformat.v4.new_notebook(cells=[nbformat.v4.new_markdown_cell(
"""[readme](readme.ipynb)""", metadata={"nbsphinx-toctree": {}})]), file)
Make the corresponding `"__init__.py"` files.
for package in packages: pathlib.Path(package, '__init__.py').write_text(_init_file)
Make the corresponding "__init__.py"
files.
for package in packages: pathlib.Path(package, '__init__.py').write_text(_init_file)
## Creating the top level readme and docs
import nbformat, nbsphinx, pathlib
`nbsphinx` provides the interface between a notebook based project and __readthedocs__.
top_level_readme = nbformat.v4.new_markdown_cell(F"""
* [Source](src/{name}/readme.ipynb)
* [Docs](src/{name}/docs/readme.ipynb)
""", metadata={"nbsphinx-toctree": {}})
The `top_level_readme['metadata']['nbsphinx-toctree']` object creates the index for our documentation.
> At the beginning of a project there is no different between the project source, tests, and documentation.
with pathlib.Path(name, 'readme.ipynb').open('w') as file:
nbformat.write(nbformat.v4.new_notebook(cells=[top_level_readme]), file)
The top_level_readme['metadata']['nbsphinx-toctree']
object creates the index for our documentation.
At the beginning of a project there is no different between the project source, tests, and documentation.
with pathlib.Path(name, 'readme.ipynb').open('w') as file:
nbformat.write(nbformat.v4.new_notebook(cells=[top_level_readme]), file)
import nbformat, nbsphinx, pathlib
nbsphinx
provides the interface between a notebook based project and readthedocs.
top_level_readme = nbformat.v4.new_markdown_cell(F"""
* [Source](src/{name}/readme.ipynb)
* [Docs](src/{name}/docs/readme.ipynb)
""", metadata={"nbsphinx-toctree": {}})
The top_level_readme['metadata']['nbsphinx-toctree']
object creates the index for our documentation.
At the beginning of a project there is no different between the project source, tests, and documentation.
with pathlib.Path(name, 'readme.ipynb').open('w') as file:
nbformat.write(nbformat.v4.new_notebook(cells=[top_level_readme]), file)
Make a symbollic link between readme and index so the docs build.
os.symlink(pathlib.Path(name, 'readme.ipynb'), pathlib.Path(name, 'index.ipynb'))
--------------------------------------------------------------------------- OSError Traceback (most recent call last) <ipython-input-37-4eb0220860c0> in <module> 1 """Make a symbollic link between readme and index so the docs build.""" 2 ----> 3 os.symlink(pathlib.Path(name, 'readme.ipynb'), pathlib.Path(name, 'index.ipynb')) OSError: symbolic link privilege not held
%%file {name}/.travis.yml
language: python
python: ['3.6']
install: ["python -m pip install ."]
script: ["python setup.py test"]
Overwriting boiler/.travis.yml
versions = '3.6 3.7'
pathlib.Path(name, '.travis.yml').write_text(
language: python
python: {versions.split()}
install: ["python -m pip install ."]
script: ["python setup.py test"]
.format(versions=versions));
versions = ['3.6', '3.7']
pathlib.Path(name, '.travis.yml').write_text(
language: python python: {versions} install: ["python -m pip install ."] script: ["python setup.py test"]
.format(versions=versions));
%%file {name}/requirements.txt
importnb
%%file {name}/src/{name}/tests/requirements.txt
nbval
Overwriting boiler/src/boiler/tests/requirements.txt
_version Hourly and minute versioning
%%file {name}/src/{name}/_version.py
time = __import__('datetime').datetime.now()
__version__ = '.'.join(
str(getattr(time, object))
for object in """
year month day hour minute
""".strip().split())
Overwriting boiler/src/boiler/_version.py
%%file {name}/setup.py
import setuptools, pathlib, nbconvert
name = 'boiler'
here = pathlib.Path(__file__).parent
with (here / 'src' / name / '_version.py').open('r') as file: exec(file.read())
setuptools.setup(
name=name, version=__version__,
long_description=nbconvert.get_exporter('markdown')().from_filename(here / 'readme.ipynb')[0],
long_description_content_type='text/markdown',
packages=setuptools.find_packages(where='src'),
package_dir={'': 'src',},
setup_requires=["pytest-runner", "nbconvert"],
install_requires=(here /'requirements.txt').read_text().splitlines(),
tests_require=(here /'src'/ name / 'tests' / 'requirements.txt').read_text().splitlines(),
include_package_data=True)
Overwriting boiler/setup.py
%%file {name}/MANIFEST.in
include LICENSE readme.md changelog.ipynb
recursive-include src/boiler *.ipynb *.md
Overwriting boiler/MANIFEST.in
%%file {name}/setup.cfg
[tool:pytest]
addopts = --nbval -p no:pytest-pidgin
[aliases]
test=pytest
Overwriting boiler/setup.cfg
python -m pytest --nbval -p no:pytest-pidgin
%%file {name}/_config.yaml
name: {name}
Overwriting boiler/_config.yaml