In [ ]:
#hide
#default_exp export
#default_cls_lvl 3
from nbdev.showdoc import show_doc
In [ ]:
#export
from nbdev.imports import *
from fastcore.script import *
from fastcore.foundation import *
from keyword import iskeyword
import nbformat

Export to modules

The functions that transform notebooks in a library

The most important function defined in this module is notebooks2script, so you may want to jump to it before scrolling though the rest, which explain the details behind the scenes of the conversion from notebooks to library. The main things to remember are:

  • put # export on each cell you want exported
  • put # exports on each cell you want exported with the source code shown in the docs
  • put # exporti on each cell you want exported without it being added to __all__, and without it showing up in the docs.
  • one cell should contain # default_exp followed by the name of the module (with points for submodules and without the py extension) everything should be exported in (if one specific cell needs to be exported in a different module, just indicate it after #export: #export special.module)
  • all left members of an equality, functions and classes will be exported and variables that are not private will be put in the __all__ automatically
  • to add something to __all__ if it's not picked automatically, write an exported cell with something like #add2all "my_name"

Examples of export

See these examples on different ways to use #export to export code in notebooks to modules. These include:

  • How to specify a default for exporting cells
  • How to hide code and not export it at all
  • How to export different cells to specific modules

Basic foundations

For bootstrapping nbdev we have a few basic foundations defined in imports, which we test a show here. First, a simple config file class, Config that read the content of your settings.ini file and make it accessible:

In [ ]:
show_doc(Config, title_level=3)

Config[source]

Config(cfg_name='settings.ini')

Reading and writing settings.ini

In [ ]:
create_config("github", "nbdev", user='fastai', path='..', tst_flags='tst', cfg_name='test_settings.ini', recursive='False')
cfg = Config(cfg_name='test_settings.ini')
test_eq(cfg.lib_name, 'nbdev')
test_eq(cfg.git_url, "https://github.com/fastai/nbdev/tree/master/")
# test_eq(cfg.path("lib_path"), Path.cwd().parent/'nbdev')
# test_eq(cfg.path("nbs_path"), Path.cwd())
# test_eq(cfg.path("doc_path"), Path.cwd().parent/'docs')
test_eq(cfg.custom_sidebar, 'False')
test_eq(cfg.recursive, 'False')

Reading a notebook

What's a notebook?

A jupyter notebook is a json file behind the scenes. We can just read it with the json module, which will return a nested dictionary of dictionaries/lists of dictionaries, but there are some small differences between reading the json and using the tools from nbformat so we'll use this one.

In [ ]:
#export
def read_nb(fname):
    "Read the notebook in `fname`."
    with open(Path(fname),'r', encoding='utf8') as f: return nbformat.reads(f.read(), as_version=4)

fname can be a string or a pathlib object.

In [ ]:
test_nb = read_nb('00_export.ipynb')

The root has four keys: cells contains the cells of the notebook, metadata some stuff around the version of python used to execute the notebook, nbformat and nbformat_minor the version of nbformat.

In [ ]:
test_nb.keys()
Out[ ]:
dict_keys(['cells', 'metadata', 'nbformat', 'nbformat_minor'])
In [ ]:
test_nb['metadata']
Out[ ]:
{'jupytext': {'split_at_heading': True},
 'kernelspec': {'display_name': 'Python 3',
  'language': 'python',
  'name': 'python3'},
 'language_info': {'codemirror_mode': {'name': 'ipython', 'version': 3},
  'file_extension': '.py',
  'mimetype': 'text/x-python',
  'name': 'python',
  'nbconvert_exporter': 'python',
  'pygments_lexer': 'ipython3',
  'version': '3.8.3'}}
In [ ]:
f"{test_nb['nbformat']}.{test_nb['nbformat_minor']}"
Out[ ]:
'4.4'

The cells key then contains a list of cells. Each one is a new dictionary that contains entries like the type (code or markdown), the source (what is written in the cell) and the output (for code cells).

In [ ]:
test_nb['cells'][0]
Out[ ]:
{'cell_type': 'code',
 'execution_count': None,
 'metadata': {'hide_input': False},
 'outputs': [],
 'source': '#hide\n#default_exp export\n#default_cls_lvl 3\nfrom nbdev.showdoc import show_doc'}

Finding patterns

The following functions are used to catch the flags used in the code cells.

In [ ]:
#export
def check_re(cell, pat, code_only=True):
    "Check if `cell` contains a line with regex `pat`"
    if code_only and cell['cell_type'] != 'code': return
    if isinstance(pat, str): pat = re.compile(pat, re.IGNORECASE | re.MULTILINE)
    return pat.search(cell['source'])

pat can be a string or a compiled regex. If code_only=True, this function ignores non-code cells, such as markdown.

In [ ]:
cell = test_nb['cells'][1].copy()
assert check_re(cell, '#export') is not None
assert check_re(cell, re.compile('#export')) is not None
assert check_re(cell, '# bla') is None
cell['cell_type'] = 'markdown'
assert check_re(cell, '#export') is None
assert check_re(cell, '#export', code_only=False) is not None
In [ ]:
#export
def check_re_multi(cell, pats, code_only=True):
    "Check if `cell` contains a line matching any regex in `pats`, returning the first match found"
    return L(pats).map_first(partial(check_re, cell, code_only=code_only))
In [ ]:
cell = test_nb['cells'][0].copy()
cell['source'] = "a b c"
assert check_re(cell, 'a') is not None
assert check_re(cell, 'd') is None
# show that searching with patterns ['d','b','a'] will match 'b'
# i.e. 'd' is not found and we don't search for 'a'
assert check_re_multi(cell, ['d','b','a']).span() == (2,3)
In [ ]:
#export
def _mk_flag_re(body, n_params, comment):
    "Compiles a regex for finding nbdev flags"
    assert body!=True, 'magics no longer supported'
    prefix = r"\s*\#\s*"
    param_group = ""
    if n_params == -1: param_group = r"[ \t]+(.+)"
    if n_params == 1: param_group = r"[ \t]+(\S+)"
    if n_params == (0,1): param_group = r"(?:[ \t]+(\S+))?"
    return re.compile(rf"""
# {comment}:
^            # beginning of line (since re.MULTILINE is passed)
{prefix}
{body}
{param_group}
[ \t]*       # any number of spaces and/or tabs
$            # end of line (since re.MULTILINE is passed)
""", re.MULTILINE | re.VERBOSE)

This function returns a regex object that can be used to find nbdev flags in multiline text

  • body regex fragment to match one or more flags,
  • n_params number of flag parameters to match and catch (-1 for any number of params; (0,1) for 0 for 1 params),
  • comment explains what the compiled regex should do.
In [ ]:
#hide
re_blank_test = _mk_flag_re('export[si]?', 0, "test")
re_mod_test = _mk_flag_re('export[si]?', 1, "test")
re_opt_test = _mk_flag_re('export[si]?', (0,1), "test")
for f in ['export', 'exports', 'exporti']:
    cell = nbformat.v4.new_code_cell(f'#{f}  \n some code')
    assert check_re(cell, re_blank_test) is not None
    assert check_re(cell, re_mod_test) is None
    assert check_re(cell, re_opt_test) is not None
    test_eq(check_re(cell, re_opt_test).groups()[0], None)
    cell.source = f'#{f} special.module \n some code'
    assert check_re(cell, re_blank_test) is None
    assert check_re(cell, re_mod_test) is not None
    test_eq(check_re(cell, re_mod_test).groups()[0], 'special.module')
    assert check_re(cell, re_opt_test) is not None
    test_eq(check_re(cell, re_opt_test).groups()[0], 'special.module')
In [ ]:
#export
_re_blank_export = _mk_flag_re("export[si]?", 0,
    "Matches any line with #export, #exports or #exporti without any module name")
In [ ]:
#export
_re_mod_export = _mk_flag_re("export[si]?", 1,
    "Matches any line with #export, #exports or #exporti with a module name and catches it in group 1")
In [ ]:
#export
_re_internal_export = _mk_flag_re("exporti", (0,1),
    "Matches any line with #exporti with or without a module name")
In [ ]:
#exporti
def _is_external_export(tst):
    "Check if a cell is an external or internal export. `tst` is an re match"
    return _re_internal_export.search(tst.string) is None
In [ ]:
#export
def is_export(cell, default):
    "Check if `cell` is to be exported and returns the name of the module to export it if provided"
    tst = check_re(cell, _re_blank_export)
    if tst:
        if default is None:
            print(f"No export destination, ignored:\n{cell['source']}")
        return default, _is_external_export(tst)
    tst = check_re(cell, _re_mod_export)
    if tst: return os.path.sep.join(tst.groups()[0].split('.')), _is_external_export(tst)
    else: return None

is_export returns;

  • a tuple of ("module name", "external boolean" (False for an internal export)) if cell is to be exported or
  • None if cell will not be exported.

The cells to export are marked with #export/#exporti/#exports, potentially with a module name where we want it exported. The default module is given in a cell of the form #default_exp bla inside the notebook (usually at the top), though in this function, it needs the be passed (the final script will read the whole notebook to find it).

  • a cell marked with #export/#exporti/#exports will be exported to the default module
  • an exported cell marked with special.module appended will be exported in special.module (located in lib_name/special/module.py)
  • a cell marked with #export will have its signature added to the documentation
  • a cell marked with #exports will additionally have its source code added to the documentation
  • a cell marked with #exporti will not show up in the documentation, and will also not be added to __all__.
In [ ]:
cell = test_nb['cells'][1].copy()
test_eq(is_export(cell, 'export'), ('export', True))
cell['source'] = "# exports"
test_eq(is_export(cell, 'export'), ('export', True))
cell['source'] = "# exporti"
test_eq(is_export(cell, 'export'), ('export', False))
cell['source'] = "# export mod"
test_eq(is_export(cell, 'export'), ('mod', True))
In [ ]:
#hide
cell['source'] = "# export mod.file"
test_eq(is_export(cell, 'export'), (f'mod{os.path.sep}file', True))
cell['source'] = "# exporti mod.file"
test_eq(is_export(cell, 'export'), (f'mod{os.path.sep}file', False))
cell['source'] = "# expt mod.file"
assert is_export(cell, 'export') is None
cell['source'] = "# exportmod.file"
assert is_export(cell, 'export') is None
cell['source'] = "# exportsmod.file"
assert is_export(cell, 'export') is None
cell['source'] = "# exporti mod file"
assert is_export(cell, 'export') is None
In [ ]:
#export
_re_default_exp = _mk_flag_re('default_exp', 1, "Matches any line with #default_exp with a module name")
In [ ]:
#export
def find_default_export(cells):
    "Find in `cells` the default export module."
    res = L(cells).map_first(check_re, pat=_re_default_exp)
    return res.groups()[0] if res else None

Stops at the first cell containing # default_exp (if there are several) and returns the value behind. Returns None if there are no cell with that code.

In [ ]:
test_eq(find_default_export(test_nb['cells']), 'export')
assert find_default_export(test_nb['cells'][2:]) is None
In [ ]:
#hide
mods = [f'mod{i}' for i in range(3)]
cells = [{'cell_type': 'code', 'source': f'#default_exp {mod}'} for mod in mods]
for i, mod in enumerate(mods): test_eq(mod, find_default_export(cells[i:]))

Listing all exported objects

The following functions make a list of everything that is exported to prepare a proper __all__ for our exported module.

In [ ]:
#export
_re_patch_func = re.compile(r"""
# Catches any function decorated with @patch, its name in group 1 and the patched class in group 2
@patch         # At any place in the cell, something that begins with @patch
(?:\s*@.*)*    # Any other decorator applied to the function
\s*def         # Any number of whitespace (including a new line probably) followed by def
\s+            # One whitespace or more
([^\(\s]+)     # Catch a group composed of anything but whitespace or an opening parenthesis (name of the function)
\s*\(          # Any number of whitespace followed by an opening parenthesis
[^:]*          # Any number of character different of : (the name of the first arg that is type-annotated)
:\s*           # A column followed by any number of whitespace
(?:            # Non-catching group with either
([^,\s\(\)]*)  #    a group composed of anything but a comma, a parenthesis or whitespace (name of the class)
|              #  or
(\([^\)]*\)))  #    a group composed of something between parenthesis (tuple of classes)
\s*            # Any number of whitespace
(?:,|\))       # Non-catching group with either a comma or a closing parenthesis
""", re.VERBOSE)
In [ ]:
tst = _re_patch_func.search("""
@patch
@log_args(a=1)
def func(obj:Class):""")
tst, tst.groups()
Out[ ]:
(<re.Match object; span=(1, 42), match='@patch\[email protected]_args(a=1)\ndef func(obj:Class)'>,
 ('func', 'Class', None))
In [ ]:
#hide
tst = _re_patch_func.search("""
@patch
def func(obj:Class):""")
test_eq(tst.groups(), ("func", "Class", None))
tst = _re_patch_func.search("""
@patch
def func (obj:Class, a)""")
test_eq(tst.groups(), ("func", "Class", None))
tst = _re_patch_func.search("""
@patch
def func (obj:(Class1, Class2), a)""")
test_eq(tst.groups(), ("func", None, "(Class1, Class2)"))
tst = _re_patch_func.search("""
@patch
def func (obj:(Class1, Class2), a:int)->int:""")
test_eq(tst.groups(), ("func", None, "(Class1, Class2)"))
tst = _re_patch_func.search("""
@patch
@log_args(but='a,b')
@funcs_kwargs
def func (obj:(Class1, Class2), a:int)->int:""")
test_eq(tst.groups(), ("func", None, "(Class1, Class2)"))
tst = _re_patch_func.search("""
@patch
@contextmanager
def func (obj:Class, a:int)->int:""")
test_eq(tst.groups(), ("func", "Class", None))
In [ ]:
#export
_re_typedispatch_func = re.compile(r"""
# Catches any function decorated with @typedispatch
(@typedispatch  # At any place in the cell, catch a group with something that begins with @typedispatch
\s*def          # Any number of whitespace (including a new line probably) followed by def
\s+             # One whitespace or more
[^\(]+          # Anything but whitespace or an opening parenthesis (name of the function)
\s*\(           # Any number of whitespace followed by an opening parenthesis
[^\)]*          # Any number of character different of )
\)[\s\S]*:)     # A closing parenthesis followed by any number of characters and whitespace (type annotation) and :
""", re.VERBOSE)
In [ ]:
#hide
assert _re_typedispatch_func.search("@typedispatch\ndef func(a, b):").groups() == ('@typedispatch\ndef func(a, b):',)
assert (_re_typedispatch_func.search("@typedispatch\ndef func(a:str, b:bool)->int:").groups() ==
                                    ('@typedispatch\ndef func(a:str, b:bool)->int:',))
In [ ]:
#export
_re_class_func_def = re.compile(r"""
# Catches any 0-indented function or class definition with its name in group 1
^              # Beginning of a line (since re.MULTILINE is passed)
(?:async\sdef|def|class)  # Non-catching group for def or class
\s+            # One whitespace or more
([^\(\s]+)     # Catching group with any character except an opening parenthesis or a whitespace (name)
\s*            # Any number of whitespace
(?:\(|:)       # Non-catching group with either an opening parenthesis or a : (classes don't need ())
""", re.MULTILINE | re.VERBOSE)
In [ ]:
#hide
test_eq(_re_class_func_def.search("class Class:").groups(), ('Class',))
test_eq(_re_class_func_def.search("def func(a, b):").groups(), ('func',))
test_eq(_re_class_func_def.search("def func(a:str, b:bool)->int:").groups(), ('func',))
test_eq(_re_class_func_def.search("async def func(a, b):").groups(), ('func',))
In [ ]:
#export
_re_obj_def = re.compile(r"""
# Catches any 0-indented object definition (bla = thing) with its name in group 1
^                          # Beginning of a line (since re.MULTILINE is passed)
([_a-zA-Z]+[a-zA-Z0-9_\.]*)  # Catch a group which is a valid python variable name
\s*                        # Any number of whitespace
(?::\s*\S.*|)=  # Non-catching group of either a colon followed by a type annotation, or nothing; followed by an =
""", re.MULTILINE | re.VERBOSE)
In [ ]:
#hide
test_eq(_re_obj_def.search("a = 1").groups(), ('a',))
test_eq(_re_obj_def.search("a.b = 1").groups(), ('a.b',))
test_eq(_re_obj_def.search("_aA1=1").groups(), ('_aA1',))
test_eq(_re_obj_def.search("a : int =1").groups(), ('a',))
test_eq(_re_obj_def.search("a:f(':=')=1").groups(), ('a',))
assert _re_obj_def.search("@abc=2") is None
assert _re_obj_def.search("a a=2") is None
In [ ]:
#export
def _not_private(n):
    for t in n.split('.'):
        if (t.startswith('_') and not t.startswith('__')) or t.startswith('@'): return False
    return '\\' not in t and '^' not in t and '[' not in t and t != 'else'

def export_names(code, func_only=False):
    "Find the names of the objects, functions or classes defined in `code` that are exported."
    #Format monkey-patches with @patch
    def _f(gps):
        nm, cls, t = gps.groups()
        if cls is not None: return f"def {cls}.{nm}():"
        return '\n'.join([f"def {c}.{nm}():" for c in re.split(', *', t[1:-1])])

    code = _re_typedispatch_func.sub('', code)
    code = _re_patch_func.sub(_f, code)
    names = _re_class_func_def.findall(code)
    if not func_only: names += _re_obj_def.findall(code)
    return [n for n in names if _not_private(n) and not iskeyword(n)]

This function only picks the zero-indented objects on the left side of an =, functions or classes (we don't want the class methods for instance) and excludes private names (that begin with _) but no dunder names. It only returns func and class names (not the objects) when func_only=True.

To work properly with fastai added python functionality, this function ignores function decorated with @typedispatch (since they are defined multiple times) and unwraps properly functions decorated with @patch.

In [ ]:
test_eq(export_names("def my_func(x):\n  pass\nclass MyClass():"), ["my_func", "MyClass"])
In [ ]:
#hide
#Indented funcs are ignored (funcs inside a class)
test_eq(export_names("  def my_func(x):\n  pass\nclass MyClass():"), ["MyClass"])

#Private funcs are ignored, dunder are not
test_eq(export_names("def _my_func():\n  pass\nclass MyClass():"), ["MyClass"])
test_eq(export_names("__version__ = 1:\n  pass\nclass MyClass():"), ["MyClass", "__version__"])

#trailing spaces
test_eq(export_names("def my_func ():\n  pass\nclass MyClass():"), ["my_func", "MyClass"])

#class without parenthesis
test_eq(export_names("def my_func ():\n  pass\nclass MyClass:"), ["my_func", "MyClass"])

#object and funcs
test_eq(export_names("def my_func ():\n  pass\ndefault_bla=[]:"), ["my_func", "default_bla"])
test_eq(export_names("def my_func ():\n  pass\ndefault_bla=[]:", func_only=True), ["my_func"])

#Private objects are ignored
test_eq(export_names("def my_func ():\n  pass\n_default_bla = []:"), ["my_func"])

#Objects with dots are privates if one part is private
test_eq(export_names("def my_func ():\n  pass\ndefault.bla = []:"), ["my_func", "default.bla"])
test_eq(export_names("def my_func ():\n  pass\ndefault._bla = []:"), ["my_func"])

#Monkey-path with @patch are properly renamed
test_eq(export_names("@patch\ndef my_func(x:Class):\n  pass"), ["Class.my_func"])
test_eq(export_names("@patch\ndef my_func(x:Class):\n  pass", func_only=True), ["Class.my_func"])
test_eq(export_names("some code\n@patch\ndef my_func(x:Class, y):\n  pass"), ["Class.my_func"])
test_eq(export_names("some code\n@patch\ndef my_func(x:(Class1,Class2), y):\n  pass"), ["Class1.my_func", "Class2.my_func"])

#Check delegates
test_eq(export_names("@delegates(keep=True)\nclass someClass:\n  pass"), ["someClass"])

#Typedispatch decorated functions shouldn't be added
test_eq(export_names("@patch\ndef my_func(x:Class):\n  pass\n@typedispatch\ndef func(x: TensorImage): pass"), ["Class.my_func"])

#try, except and other keywords should not be picked up (these can look like object def with type annotation)
test_eq(export_names("try:\n    a=1\nexcept:\n    b=2"), [])
test_eq(export_names("try:\n    this_might_work\nexcept:\n    b=2"), [])
In [ ]:
#export
_re_all_def   = re.compile(r"""
# Catches a cell with defines \_all\_ = [\*\*] and get that \*\* in group 1
^_all_   #  Beginning of line (since re.MULTILINE is passed)
\s*=\s*  #  Any number of whitespace, =, any number of whitespace
\[       #  Opening [
([^\n\]]*) #  Catching group with anything except a ] or newline
\]       #  Closing ]
""", re.MULTILINE | re.VERBOSE)

#Same with __all__
_re__all__def = re.compile(r'^__all__\s*=\s*\[([^\]]*)\]', re.MULTILINE)
In [ ]:
#export
def extra_add(flags, code):
    "Catch adds to `__all__` required by a cell with `_all_=`"
    m = check_re({'source': code}, _re_all_def, False)
    if m:
        code = m.re.sub('#nbdev_' + 'comment \g<0>', code)
        code = re.sub(r'([^\n]|^)\n*$', r'\1', code)
    if not m: return [], code
    def clean_quotes(s):
        "Return `s` enclosed in single quotes, removing double quotes if needed"
        if s.startswith("'") and s.endswith("'"): return s
        if s.startswith('"') and s.endswith('"'): s = s[1:-1]
        return f"'{s}'"
    return [clean_quotes(s) for s in parse_line(m.group(1))], code

Sometimes objects are not picked to be automatically added to the __all__ of the module so you will need to add them manually. To do so, create an exported cell with the following code _all_ = ["name", "name2"]

In [ ]:
#hide
for code, expected in [
        ['_all_ = ["func", "func1", "func2"]', 
         (["'func'", "'func1'", "'func2'"],'#nbdev_comment _all_ = ["func", "func1", "func2"]')],
        ['_all_=[func, func1, func2]', 
         (["'func'", "'func1'", "'func2'"],'#nbdev_comment _all_=[func, func1, func2]')],
        ["_all_ = ['func', 'func1', 'func2']", 
         (["'func'", "'func1'", "'func2'"],"#nbdev_comment _all_ = ['func', 'func1', 'func2']")],
        ['_all_ = ["func",   "func1" , "func2"]', 
         (["'func'", "'func1'", "'func2'"],'#nbdev_comment _all_ = ["func",   "func1" , "func2"]')],
        ["_all_ = ['func','func1', 'func2']\n", 
         (["'func'", "'func1'", "'func2'"],"#nbdev_comment _all_ = ['func','func1', 'func2']")],
        ["_all_ = ['func']\n_all_ = ['func1', 'func2']\n", 
         (["'func'"],"#nbdev_comment _all_ = ['func']\n#nbdev_comment _all_ = ['func1', 'func2']")],
        ['code\n\n_all_ = ["func", "func1", "func2"]', 
         (["'func'", "'func1'", "'func2'"],'code\n\n#nbdev_comment _all_ = ["func", "func1", "func2"]')],
        ['code\n\n_all_ = [func]\nmore code', 
         (["'func'"],'code\n\n#nbdev_comment _all_ = [func]\nmore code')]]:
    test_eq(extra_add('', code), expected)
    
# line breaks within the list of names means _all_ is ignored
test_eq(extra_add('', "_all_ = ['func',\n'func1', 'func2']\n"), ([],"_all_ = ['func',\n'func1', 'func2']\n"))
In [ ]:
#export
_re_from_future_import = re.compile(r"^from[ \t]+__future__[ \t]+import.*$", re.MULTILINE)

def _from_future_import(fname, flags, code, to_dict=None):
    "Write `__future__` imports to `fname` and return `code` with `__future__` imports commented out"
    from_future_imports = _re_from_future_import.findall(code)
    if from_future_imports: code = _re_from_future_import.sub('#nbdev' + '_comment \g<0>', code)
    else: from_future_imports = _re_from_future_import.findall(flags)
    if not from_future_imports or to_dict is not None: return code
    with open(fname, 'r', encoding='utf8') as f: text = f.read()
    start = _re__all__def.search(text).start()
    with open(fname, 'w', encoding='utf8') as f:
        f.write('\n'.join([text[:start], *from_future_imports, '\n', text[start:]]))
    return code

If you need a from __future__ import in your library, you can export your cell with special comments:

#export
from __future__ import annotations
class ...

Notice that #export is after the __future__ import. Because __future__ imports must occur at the beginning of the file, nbdev allows __future__ imports in the flags section of a cell.

In [ ]:
#hide
txt = """
# AUTOHEADER ... File to edit: mod.ipynb (unless otherwise specified).

__all__ = [my_file, MyClas]
# Cell
def valid_code(): pass"""
expected_txt = """
# AUTOHEADER ... File to edit: mod.ipynb (unless otherwise specified).


from  __future__ import  annotations    
from __future__ import generator_stop


__all__ = [my_file, MyClas]
# Cell
def valid_code(): pass"""
flags="# export"
code = """
# comment
from  __future__ import  annotations    
valid_code = False # but _from_future_import will work anyway
from __future__ import generator_stop
 from __future__ import not_zero_indented
valid_code = True
"""
expected_code = """
# comment
#nbdev_comment from  __future__ import  annotations    
valid_code = False # but _from_future_import will work anyway
#nbdev_comment from __future__ import generator_stop
 from __future__ import not_zero_indented
valid_code = True
"""
In [ ]:
def _run_from_future_import_test():
    fname = 'test_from_future_import.txt'
    with open(fname, 'w', encoding='utf8') as f: f.write(txt)

    actual_code=_from_future_import(fname, flags, code, {})
    test_eq(expected_code, actual_code)
    with open(fname, 'r', encoding='utf8') as f: test_eq(f.read(), txt)

    actual_code=_from_future_import(fname, flags, code)
    test_eq(expected_code, actual_code)
    with open(fname, 'r', encoding='utf8') as f: test_eq(f.read(), expected_txt)

    os.remove(fname)

_run_from_future_import_test()

flags="""from  __future__ import  annotations    
from __future__ import generator_stop
#export"""
code = ""
expected_code = ""
fname = 'test_from_future_import.txt'

_run_from_future_import_test()
In [ ]:
#export
def _add2all(fname, names, line_width=120):
    if len(names) == 0: return
    with open(fname, 'r', encoding='utf8') as f: text = f.read()
    tw = TextWrapper(width=120, initial_indent='', subsequent_indent=' '*11, break_long_words=False)
    re_all = _re__all__def.search(text)
    start,end = re_all.start(),re_all.end()
    text_all = tw.wrap(f"{text[start:end-1]}{'' if text[end-2]=='[' else ', '}{', '.join(names)}]")
    with open(fname, 'w', encoding='utf8') as f: f.write(text[:start] + '\n'.join(text_all) + text[end:])
In [ ]:
#hide
fname = 'test_add.txt'
with open(fname, 'w', encoding='utf8') as f: f.write("Bla\n__all__ = [my_file, MyClas]\nBli")
_add2all(fname, ['new_function'])
with open(fname, 'r', encoding='utf8') as f: 
    test_eq(f.read(), "Bla\n__all__ = [my_file, MyClas, new_function]\nBli")
_add2all(fname, [f'new_function{i}' for i in range(10)])
with open(fname, 'r', encoding='utf8') as f: 
    test_eq(f.read(), """Bla
__all__ = [my_file, MyClas, new_function, new_function0, new_function1, new_function2, new_function3, new_function4,
           new_function5, new_function6, new_function7, new_function8, new_function9]
Bli""")
os.remove(fname)
In [ ]:
#export
def relative_import(name, fname):
    "Convert a module `name` to a name relative to `fname`"
    mods = name.split('.')
    splits = str(fname).split(os.path.sep)
    if mods[0] not in splits: return name
    i=len(splits)-1
    while i>0 and splits[i] != mods[0]: i-=1
    splits = splits[i:]
    while len(mods)>0 and splits[0] == mods[0]: splits,mods = splits[1:],mods[1:]
    return '.' * (len(splits)) + '.'.join(mods)

When we say from

from lib_name.module.submodule import bla

in a notebook, it needs to be converted to something like

from .module.submodule import bla

or from .submodule import bla depending on where we are. This function deals with those imports renaming.

Note that import of the form

import lib_name.module

are left as is as the syntax import module does not work for relative imports.

In [ ]:
test_eq(relative_import('nbdev.core', Path.cwd()/'nbdev'/'data.py'), '.core')
test_eq(relative_import('nbdev.core', Path('nbdev')/'vision'/'data.py'), '..core')
test_eq(relative_import('nbdev.vision.transform', Path('nbdev')/'vision'/'data.py'), '.transform')
test_eq(relative_import('nbdev.notebook.core', Path('nbdev')/'data'/'external.py'), '..notebook.core')
test_eq(relative_import('nbdev.vision', Path('nbdev')/'vision'/'learner.py'), '.')
In [ ]:
#export
_re_import = ReLibName(r'^(\s*)from (LIB_NAME\.\S*) import (.*)$')
In [ ]:
#export
def _deal_import(code_lines, fname):
    def _replace(m):
        sp,mod,obj = m.groups()
        return f"{sp}from {relative_import(mod, fname)} import {obj}"
    return [_re_import.re.sub(_replace,line) for line in code_lines]
In [ ]:
#hide
lines = ["from nbdev.core import *", 
         "nothing to see", 
         "  from nbdev.vision import bla1, bla2", 
         "from nbdev.vision import models",
         "import nbdev.vision"]
test_eq(_deal_import(lines, Path.cwd()/'nbdev'/'data.py'), [
    "from .core import *", 
    "nothing to see", 
    "  from .vision import bla1, bla2", 
    "from .vision import models",
    "import nbdev.vision"
])

Create the library

Saving an index

To be able to build back a correspondence between functions and the notebooks they are defined in, we need to store an index. It's done in the private module _nbdev inside your library, and the following function are used to define it.

In [ ]:
#export
_re_index_custom = re.compile(r'def custom_doc_links\(name\):(.*)$', re.DOTALL)
In [ ]:
#export
def reset_nbdev_module():
    "Create a skeleton for <code>_nbdev</code>"
    fname = Config().path("lib_path")/'_nbdev.py'
    fname.parent.mkdir(parents=True, exist_ok=True)
    sep = '\n'* (int(Config().get('cell_spacing', '1'))+1)
    if fname.is_file():
        with open(fname, 'r') as f: search = _re_index_custom.search(f.read())
    else: search = None
    prev_code = search.groups()[0] if search is not None else ' return None\n'
    with open(fname, 'w') as f:
        f.write(f"# AUTOGENERATED BY NBDEV! DO NOT EDIT!")
        f.write('\n\n__all__ = ["index", "modules", "custom_doc_links", "git_url"]')
        f.write('\n\nindex = {}')
        f.write('\n\nmodules = []')
        f.write(f'\n\ndoc_url = "{Config().doc_host}{Config().doc_baseurl}"')
        f.write(f'\n\ngit_url = "{Config().git_url}"')
        f.write(f'{sep}def custom_doc_links(name):{prev_code}')
In [ ]:
#export
class _EmptyModule():
    def __init__(self):
        self.index,self.modules = {},[]
        self.doc_url,self.git_url = f"{Config().doc_host}{Config().doc_baseurl}",Config().git_url

    def custom_doc_links(self, name): return None
In [ ]:
#export
def get_nbdev_module():
    "Reads <code>_nbdev</code>"
    try:
        spec = importlib.util.spec_from_file_location(f"{Config().lib_name}._nbdev", Config().path("lib_path")/'_nbdev.py')
        mod = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(mod)
        return mod
    except: return _EmptyModule()
In [ ]:
#export
_re_index_idx = re.compile(r'index\s*=\s*{[^}]*}')
_re_index_mod = re.compile(r'modules\s*=\s*\[[^\]]*\]')
In [ ]:
#export
def save_nbdev_module(mod):
    "Save `mod` inside <code>_nbdev</code>"
    fname = Config().path("lib_path")/'_nbdev.py'
    with open(fname, 'r') as f: code = f.read()
    t = r',\n         '.join([f'"{k}": "{v}"' for k,v in mod.index.items()])
    code = _re_index_idx.sub("index = {"+ t +"}", code)
    t = r',\n           '.join(['"' + f.replace('\\','/') + '"' for f in mod.modules])
    code = _re_index_mod.sub(f"modules = [{t}]", code)
    with open(fname, 'w') as f: f.write(code)
In [ ]:
#hide
ind,ind_bak = Config().path("lib_path")/'_nbdev.py',Config().path("lib_path")/'_nbdev.bak'
if ind.exists(): shutil.move(ind, ind_bak)
try:
    reset_nbdev_module()
    mod = get_nbdev_module()
    test_eq(mod.index, {})
    test_eq(mod.modules, [])

    mod.index = {'foo':'bar'}
    mod.modules.append('lala.bla')
    save_nbdev_module(mod)

    mod = get_nbdev_module()
    test_eq(mod.index, {'foo':'bar'})
    test_eq(mod.modules, ['lala.bla'])
finally:
    if ind_bak.exists(): shutil.move(ind_bak, ind)

Create the modules

In [ ]:
#export
def split_flags_and_code(cell, return_type=list):
    "Splits the `source` of a cell into 2 parts and returns (flags, code)"
    code_lines = cell['source'].split('\n')
    split_pos = 0 if code_lines[0].strip().startswith('#') else -1
    for i, line in enumerate(code_lines):
        if not line.startswith('#') and line.strip() and not _re_from_future_import.match(line): break
    split_pos+=1
    res = code_lines[:split_pos], code_lines[split_pos:]
    if return_type is list: return res
    return tuple('\n'.join(r) for r in res)

return_type tells us if the tuple returned will contain lists of lines or strings with line breaks.

We treat the first comment line as a flag

split_flags_and_code example

In [ ]:
def _test_split_flags_and_code(expected_flags, expected_code):
    cell = nbformat.v4.new_code_cell('\n'.join(expected_flags + expected_code))
    test_eq((expected_flags, expected_code), split_flags_and_code(cell))
    expected=('\n'.join(expected_flags), '\n'.join(expected_code))
    test_eq(expected, split_flags_and_code(cell, str))
    
_test_split_flags_and_code([
    '#export'],
    ['# TODO: write this function',
    'def func(x): pass'])
In [ ]:
#export
def create_mod_file(fname, nb_path, bare=False):
    "Create a module file for `fname`."
    bare = str(Config().get('bare', bare)) == 'True'
    fname.parent.mkdir(parents=True, exist_ok=True)
    file_path = os.path.relpath(nb_path, Config().config_file.parent).replace('\\', '/')
    with open(fname, 'w') as f:
        if not bare: f.write(f"# AUTOGENERATED! DO NOT EDIT! File to edit: {file_path} (unless otherwise specified).")
        f.write('\n\n__all__ = []')

A new module filename is created each time a notebook has a cell marked with #default_exp. In your collection of notebooks, you should only have one notebook that creates a given module since they are re-created each time you do a library build (to ensure the library is clean). Note that any file you create manually will never be overwritten (unless it has the same name as one of the modules defined in a #default_exp cell) so you are responsible to clean up those yourself.

fname is the notebook that contained the #default_exp cell.

In [ ]:
#export
def create_mod_files(files, to_dict=False, bare=False):
    "Create mod files for default exports found in `files`"
    modules = []
    for f in sorted(files):
        fname = Path(f)
        nb = read_nb(fname)
        default = find_default_export(nb['cells'])
        if default is not None:
            default = os.path.sep.join(default.split('.'))
            modules.append(default)
            if not to_dict:
                create_mod_file(Config().path("lib_path")/f'{default}.py', Config().path("nbs_path")/f'{fname}', bare=bare)
    return modules

Create module files for all #default_export flags found in files and return a list containing the names of modules created.

Note: The number if modules returned will be less that the number of files passed in if files do not #default_export.

By creating all module files before calling _notebook2script, the order of execution no longer matters - so you can now export to a notebook that is run "later".

You might still have problems when

  • converting a subset of notebooks or
  • exporting to a module that does not have a #default_export yet

in which case _notebook2script will print warnings like;

Warning: Exporting to "core.py" but this module is not part of this build

If you see a warning like this

  • and the module file (e.g. "core.py") does not exist, you'll see a FileNotFoundError
  • if the module file exists, the exported cell will be written - even if the exported cell is already in the module file
In [ ]:
#export
def _notebook2script(fname, modules, silent=False, to_dict=None, bare=False):
    "Finds cells starting with `#export` and puts them into a module created by `create_mod_files`"
    bare = str(Config().get('bare', bare)) == 'True'
    if os.environ.get('IN_TEST',0): return  # don't export if running tests
    sep = '\n'* (int(Config().get('cell_spacing', '1'))+1)
    fname = Path(fname)
    nb = read_nb(fname)
    default = find_default_export(nb['cells'])
    if default is not None:
        default = os.path.sep.join(default.split('.'))
    mod = get_nbdev_module()
    exports = [is_export(c, default) for c in nb['cells']]
    cells = [(i,c,e) for i,(c,e) in enumerate(zip(nb['cells'],exports)) if e is not None]
    for i,c,(e,a) in cells:
        if e not in modules: print(f'Warning: Exporting to "{e}.py" but this module is not part of this build')
        fname_out = Config().path("lib_path")/f'{e}.py'
        if bare: orig = "\n"
        else: orig = (f'# {"" if a else "Internal "}C' if e==default else f'# Comes from {fname.name}, c') + 'ell\n'
        flag_lines,code_lines = split_flags_and_code(c)
        code_lines = _deal_import(code_lines, fname_out)
        code = sep + orig + '\n'.join(code_lines)
        names = export_names(code)
        flags = '\n'.join(flag_lines)
        extra,code = extra_add(flags, code)
        code = _from_future_import(fname_out, flags, code, to_dict)
        if a:
            if to_dict is None: _add2all(fname_out, [f"'{f}'" for f in names if '.' not in f and len(f) > 0] + extra)
        mod.index.update({f: fname.name for f in names})
        code = re.sub(r' +$', '', code, flags=re.MULTILINE)
        if code != sep + orig[:-1]:
            if to_dict is not None: to_dict[e].append((i, fname, code))
            else:
                with open(fname_out, 'a', encoding='utf8') as f: f.write(code)
        if f'{e}.py' not in mod.modules: mod.modules.append(f'{e}.py')
    save_nbdev_module(mod)

    if not silent: print(f"Converted {fname.name}.")
    return to_dict
In [ ]:
#hide
if not os.environ.get('IN_TEST',0):
    modules = create_mod_files(glob.glob('00_export.ipynb'))
    _notebook2script('00_export.ipynb', modules)
Converted 00_export.ipynb.
In [ ]:
#hide
with open(Config().path("lib_path")/('export.py')) as f: l = f.readline()
test_eq(l, '# AUTOGENERATED! DO NOT EDIT! File to edit: nbs/00_export.ipynb (unless otherwise specified).\n')
In [ ]:
#export
def add_init(path):
    "Add `__init__.py` in all subdirs of `path` containing python files if it's not there already"
    for p,d,f in os.walk(path):
        for f_ in f:
            if f_.endswith('.py'):
                if not (Path(p)/'__init__.py').exists(): (Path(p)/'__init__.py').touch()
                break
In [ ]:
with tempfile.TemporaryDirectory() as d:
    os.makedirs(Path(d)/'a', exist_ok=True)
    (Path(d)/'a'/'f.py').touch()
    os.makedirs(Path(d)/'a/b', exist_ok=True)
    (Path(d)/'a'/'b'/'f.py').touch()
    add_init(d)
    assert not (Path(d)/'__init__.py').exists()
    for e in [Path(d)/'a', Path(d)/'a/b']:
        assert (e/'__init__.py').exists()
In [ ]:
#export
_re_version = re.compile('^__version__\s*=.*$', re.MULTILINE)
In [ ]:
#export
def update_version():
    "Add or update `__version__` in the main `__init__.py` of the library"
    fname = Config().path("lib_path")/'__init__.py'
    if not fname.exists(): fname.touch()
    version = f'__version__ = "{Config().version}"'
    with open(fname, 'r') as f: code = f.read()
    if _re_version.search(code) is None: code = version + "\n" + code
    else: code = _re_version.sub(version, code)
    with open(fname, 'w') as f: f.write(code)
In [ ]:
#export
_re_baseurl = re.compile('^baseurl\s*:.*$', re.MULTILINE)
In [ ]:
#export
def update_baseurl():
    "Add or update `baseurl` in `_config.yml` for the docs"
    fname = Config().path("doc_path")/'_config.yml'
    if not fname.exists(): return
    with open(fname, 'r') as f: code = f.read()
    if _re_baseurl.search(code) is None: code = code + f"\nbaseurl: {Config().doc_baseurl}"
    else: code = _re_baseurl.sub(f"baseurl: {Config().doc_baseurl}", code)
    with open(fname, 'w') as f: f.write(code)
In [ ]:
#export
def nbglob(fname=None, recursive=False, extension='.ipynb', config_key='nbs_path') -> L:
    "Find all files in a directory matching an extension given a `config_key`. Ignores hidden directories and filenames starting with `_`"
    fname = Config().path(config_key) if fname is None else Path(fname)
    if fname.is_dir(): fname = f'{fname.absolute()}/**/*{extension}' if recursive else f'{fname.absolute()}/*{extension}'
    fls = L(glob.glob(str(fname), recursive=recursive)).filter(lambda x: '/.' not in x).map(Path)
    return fls.filter(lambda x: not x.name.startswith('_') and x.name.endswith(extension))
In [ ]:
#hide
with tempfile.TemporaryDirectory() as d:
    os.makedirs(Path(d)/'a', exist_ok=True)
    (Path(d)/'a'/'a.ipynb').touch()
    (Path(d)/'a'/'fake_a.ipynb').touch()
    os.makedirs(Path(d)/'a/b', exist_ok=True)
    (Path(d)/'a'/'b'/'fake_b.ipynb').touch()
    os.makedirs(Path(d)/'a/b/c', exist_ok=True)
    (Path(d)/'a'/'b'/'c'/'fake_c.ipynb').touch()
    (Path(d)/'a'/'b'/'c'/'foo_c.ipynb').touch()
    
    if sys.platform != "win32":
        assert len(nbglob(str(Path(f'{str(d)}/**/foo*')), recursive=True)) == 1
    assert len(nbglob(d, recursive=True)) == 5
    assert len(nbglob(d)) == 0
    assert len(nbglob(str(Path(f'{d}/a')))) == 2
In [ ]:
#hide
if sys.platform != "win32":
    assert len(nbglob('*')) > 1
    assert len(nbglob('*')) >  len(nbglob('0*'))
In [ ]:
assert not nbglob().filter(lambda x: '.ipynb_checkpoints' in str(x))
In [ ]:
#hide
fnames = nbglob()
test_eq(len(fnames) > 0, True)

fnames = nbglob(fnames[0])
test_eq(len(fnames), 1)

Optionally you can pass a config_key to dictate which directory you are pointing to. By default it's nbs_path as without any parameters passed in, it will check for notebooks. To have it instead find library files simply pass in lib_path instead.

Note: it will only search for paths in Config().path

In [ ]:
#hide
fnames = nbglob(extension='.py', config_key='lib_path')
test_eq(len(fnames) > 1, True)
In [ ]:
#export
def notebook2script(fname=None, silent=False, to_dict=False, bare=False, recursive=False):
    "Convert notebooks matching `fname` to modules"
    # initial checks
    if os.environ.get('IN_TEST',0): return  # don't export if running tests
    if fname is None:
        reset_nbdev_module()
        update_version()
        update_baseurl()
    files = nbglob(fname=fname, recursive=recursive)
    d = collections.defaultdict(list) if to_dict else None
    modules = create_mod_files(files, to_dict, bare=bare)
    for f in sorted(files): d = _notebook2script(f, modules, silent=silent, to_dict=d, bare=bare)
    if to_dict: return d
    else: add_init(Config().path("lib_path"))

Finds cells starting with #export and puts them into the appropriate module. If fname is not specified, this will convert all notebook not beginning with an underscore in the nb_folder defined in setting.ini. Otherwise fname can be a single filename or a glob expression.

silent makes the command not print any statement and to_dict is used internally to convert the library to a dictionary.

In [ ]:
#export
class DocsTestClass:
    "for tests only"
    def test(): pass
In [ ]:
#hide
#exporti
#for tests only
def update_lib_with_exporti_testfn(): pass

Export -

In [ ]:
#hide
notebook2script()
Converted 00_export.ipynb.
Converted 01_sync.ipynb.
Converted 02_showdoc.ipynb.
Converted 03_export2html.ipynb.
Converted 04_test.ipynb.
Converted 05_merge.ipynb.
Converted 06_cli.ipynb.
Converted 07_clean.ipynb.
Converted 99_search.ipynb.
Converted example.ipynb.
Converted index.ipynb.
Converted nbdev_comments.ipynb.
Converted tutorial.ipynb.
Converted tutorial_colab.ipynb.
In [ ]: