importlib
, nbconvert
, and nbformat
to import notebooks¶A lot of code is written in Jupyter notebooks, but little of it is reused. Currently, there is ambiguity in how authors design their computational essays.
At deathbeds, we are making an effort to compose our notebooks so they can be reused as modules. This means that our essays can be loaded into Python using the import system. To do this we rely on importnb which is a package dedicated to importing notebooks.
In this post we focus on the most basic steps to add notebooks to the input system. Most demonstrations of custom imports use a meta finder, here we define a path hook. The path hook finder will look for notebooks matching a qualified notebook any where along the sys path. Any notebook in a Python package becomes a potential module.
%reload_ext deathbeds.__The_simplest__import_notebook
from importlib.machinery import SourceFileLoader, FileFinder
from importlib.util import decode_source, cache_from_source
import sys, inspect
from nbconvert.exporters.python import PythonExporter
from nbformat.v4 import reads
Ø = __name__ == '__main__'
The notebook loader is very similar to a Python loader, it just needs to convert the source json into valid Python. The standard source file loader has a source_to_code method just for this purpose. To convert the notebook to python byte code we can use nbconvert.
class NotebookLoader(SourceFileLoader):
def source_to_code(self, data, path=None):
return compile(PythonExporter().from_notebook_node(
reads(decode_source(data))
)[0], path, 'exec')
Using the native Source file loader means that the imported module will be cached and is reloadable.
Creating a new meta finder is easy, adding a new path hook is not. We are going to do some weird shit to discover the existing file finders for the existing Python imports.
def find_finder():
for i, hook in enumerate(sys.path_hooks):
try:
if issubclass(inspect.getclosurevars(hook).nonlocals['cls'],
FileFinder):return i, hook
except: ...
def new_loader(*details):
i, finder = find_finder()
sys.path_hooks[i] = FileFinder.path_hook(
*details,*(
object for object in inspect.getclosurevars(find_finder()[1]).nonlocals['loader_details']
if not any(map('.ipynb'.__eq__, object[1]))))
sys.path_importer_cache.clear()
It is essential to clear the path importer cache otherwise you will enter a waking nightmare while debugging.
def load_ipython_extension(ip, loader=NotebookLoader):
new_loader((loader, ('.ipynb',)))
ip.ast_transformers = [] # This is a precaution for the demo
Ø and load_ipython_extension(get_ipython())
--------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-1-f0f7e257c0c7> in <module>() ----> 1 def load_ipython_extension(ip=None, loader=NotebookLoader): 2 new_loader((loader, ('.ipynb',))) 3 ip.ast_transformers = [] # This is a precaution for the demo 4 Ø and load_ipython_extension(get_ipython()) NameError: name 'NotebookLoader' is not defined
if Ø:
import __style__
assert isinstance(__style__.__loader__, NotebookLoader), "Some other thing is loaded."
assert __import__('pathlib').Path(cache_from_source(__style__.__file__)).exists(), """The module is not cached"""
importnb
attends a solution for a wildcard finders. That is why the deathbeds posts - which start with numbers & contain hyphens - may be imported. if Ø: import disqus