In [ ]:
from nbdev import *
%nbdev_default_export export
%nbdev_default_class_level 3
Cells will be exported to nbdev.export,
unless a different module is specified after an export flag: `%nbdev_export special.module`
In [ ]:
#export
from nbdev.imports import *
from fastcore.script import *
from keyword import iskeyword
from nbdev.flags import parse_line

Export to modules

The functions that transform notebooks in a library

The most important function defined in this module is notebooks2script, so you may want to jump to it before scrolling though the rest, which explain the details behind the scenes of the conversion from notebooks to library. The main things to remember are:

  • put an # export or %nbdev_export flag on each cell you want exported
  • put an # exports or %nbdev_export_and_show flag on each cell you want exported with the source code shown in the docs
  • put an # exporti or %nbdev_export_internal flag on each cell you want exported without it being added to __all__, and without it showing up in the docs.
  • one cell should contain a # default_exp or %nbdev_default_export flag followed by the name of the module (with points for submodules and without the py extension) everything should be exported in (if one specific cell needs to be exported in a different module, just indicate it after the %nbdev_export flag: %nbdev_export special.module)
  • all left members of an equality, functions and classes will be exported and variables that are not private will be put in the __all__ automatically
  • to add something to __all__ if it's not picked automatically, write an exported cell with something like %nbdev_add2all "my_name"

Basic foundations

For bootstrapping nbdev we have a few basic foundations defined in imports, which we test a show here. First, a simple config file class, Config that read the content of your settings.ini file and make it accessible:

In [ ]:
show_doc(Config, title_level=3)

Config[source]

Config(cfg_name='settings.ini')

Store the basic information for nbdev to work

In [ ]:
create_config("github", "nbdev", user='fastai', path='..', tst_flags='tst', cfg_name='test_settings.ini')
cfg = Config(cfg_name='test_settings.ini')
test_eq(cfg.lib_name, 'nbdev')
test_eq(cfg.git_url, "https://github.com/fastai/nbdev/tree/master/")
test_eq(cfg.lib_path, Path.cwd().parent/'nbdev')
test_eq(cfg.nbs_path, Path.cwd())
test_eq(cfg.doc_path, Path.cwd().parent/'docs')
test_eq(cfg.custom_sidebar, 'False')

We derive some useful variables to check what environment we're in:

In [ ]:
if not os.environ.get("IN_TEST", None):
    assert IN_NOTEBOOK
    assert not IN_COLAB
    assert IN_IPYTHON
In [ ]:
#hide
for py, expected in [['ipython', 'True'], ['python', 'False']]:
    in_ipython_test_result = !$py ../tests/print_in_ipython.py
    # don't fail this test if we see just one element (could mean !python is not available)
    if len(in_ipython_test_result) == 2:
        test_eq(expected, in_ipython_test_result[1])

Then we have a few util functions.

In [ ]:
show_doc(last_index)

last_index[source]

last_index(x, o)

Finds the last index of occurence of x in o (returns -1 if no occurence)

In [ ]:
test_eq(last_index(1, [1,2,1,3,1,4]), 4)
test_eq(last_index(2, [1,2,1,3,1,4]), 1)
test_eq(last_index(5, [1,2,1,3,1,4]), -1)
In [ ]:
show_doc(compose)

compose[source]

compose(*funcs, order=None)

Create a function that composes all functions in funcs, passing along remaining *args and **kwargs to all

In [ ]:
f1 = lambda o,p=0: (o*2)+p
f2 = lambda o,p=1: (o+1)/p
test_eq(f2(f1(3)), compose(f1,f2)(3))
test_eq(f2(f1(3,p=3),p=3), compose(f1,f2)(3,p=3))
test_eq(f2(f1(3,  3),  3), compose(f1,f2)(3,  3))
In [ ]:
show_doc(parallel)

parallel[source]

parallel(f, items, *args, n_workers=64, total=None, progress=None, pause=0, **kwargs)

Applies func in parallel to items, using n_workers

In [ ]:
import time,random
In [ ]:
def add_one(x, a=1): 
    time.sleep(random.random()/100)
    return x+a

inp,exp = range(50),range(1,51)
test_eq(parallel(add_one, inp, n_workers=2), list(exp))
test_eq(parallel(add_one, inp, n_workers=0), list(exp))
test_eq(parallel(add_one, inp, n_workers=1, a=2), list(range(2,52)))
test_eq(parallel(add_one, inp, n_workers=0, a=2), list(range(2,52)))
In [ ]:
#export
def first(x):
    "First element of `x`, or None if missing"
    try: return next(iter(x))
    except StopIteration: return None

Reading a notebook

What's a notebook?

A jupyter notebook is a json file behind the scenes. We can just read it with the json module, which will return a nested dictionary of dictionaries/lists of dictionaries, but there are some small differences between reading the json and using the tools from nbformat so we'll use this one.

In [ ]:
#export
def read_nb(fname):
    "Read the notebook in `fname`."
    with open(Path(fname),'r', encoding='utf8') as f: return nbformat.reads(f.read(), as_version=4)

fname can be a string or a pathlib object.

In [ ]:
test_nb = read_nb('00_export.ipynb')

The root has four keys: cells contains the cells of the notebook, metadata some stuff around the version of python used to execute the notebook, nbformat and nbformat_minor the version of nbformat.

In [ ]:
test_nb.keys()
Out[ ]:
dict_keys(['cells', 'metadata', 'nbformat', 'nbformat_minor'])
In [ ]:
test_nb['metadata']
Out[ ]:
{'jupytext': {'split_at_heading': True},
 'kernelspec': {'display_name': 'Python 3',
  'language': 'python',
  'name': 'python3'},
 'language_info': {'codemirror_mode': {'name': 'ipython', 'version': 3},
  'file_extension': '.py',
  'mimetype': 'text/x-python',
  'name': 'python',
  'nbconvert_exporter': 'python',
  'pygments_lexer': 'ipython3',
  'version': '3.7.7'},
 'toc': {'base_numbering': 1,
  'nav_menu': {},
  'number_sections': False,
  'sideBar': True,
  'skip_h1_title': True,
  'title_cell': 'Table of Contents',
  'title_sidebar': 'Contents',
  'toc_cell': False,
  'toc_position': {},
  'toc_section_display': True,
  'toc_window_display': False}}
In [ ]:
f"{test_nb['nbformat']}.{test_nb['nbformat_minor']}"
Out[ ]:
'4.4'

The cells key then contains a list of cells. Each one is a new dictionary that contains entries like the type (code or markdown), the source (what is written in the cell) and the output (for code cells).

In [ ]:
test_nb['cells'][0]
Out[ ]:
{'cell_type': 'code',
 'execution_count': None,
 'metadata': {'hide_input': True},
 'outputs': [{'name': 'stdout',
   'output_type': 'stream',
   'text': 'Cells will be exported to nbdev.export,\nunless a different module is specified after an export flag: `%nbdev_export special.module`\n'}],
 'source': 'from nbdev import *\n%nbdev_default_export export\n%nbdev_default_class_level 3'}

Finding patterns

The following functions are used to catch the flags used in the code cells.

In [ ]:
#export
def check_re(cell, pat, code_only=True):
    "Check if `cell` contains a line with regex `pat`"
    if code_only and cell['cell_type'] != 'code': return
    if isinstance(pat, str): pat = re.compile(pat, re.IGNORECASE | re.MULTILINE)
    return pat.search(cell['source'])

pat can be a string or a compiled regex, if code_only=True, this function ignores markdown cells.

In [ ]:
cell = test_nb['cells'][1].copy()
assert check_re(cell, '#export') is not None
assert check_re(cell, re.compile('#export')) is not None
assert check_re(cell, '# bla') is None
cell['cell_type'] = 'markdown'
assert check_re(cell, '#export') is None
assert check_re(cell, '#export', code_only=False) is not None
In [ ]:
#export
def check_re_multi(cell, pats, code_only=True):
    "Check if `cell` contains a line matching any regex in `pats`, returning the first match found"
    for pat in pats:
        tst = check_re(cell, pat, code_only)
        if tst: return tst
In [ ]:
cell = test_nb['cells'][0].copy()
cell['source'] = "a b c"
assert check_re(cell, 'a') is not None
assert check_re(cell, 'd') is None
# show that searching with paterns ['d','b','a'] will match 'b'
# i.e. 'd' is not found and we don't search for 'a'
assert check_re_multi(cell, ['d','b','a']).span() == (2,3)
In [ ]:
#export
def _mk_flag_re(magic_flag, body, n_params, comment):
    "Compiles a regex for finding nbdev flags"
    prefix = r"%nbdev_" if magic_flag else r"\s*\#\s*"
    param_group = ""
    if n_params == -1: param_group = r"[ \t]+(.+)"
    if n_params == 1: param_group = r"[ \t]+(\S+)"
    if n_params == (0,1): param_group = r"(?:[ \t]+(\S+))?"
    return re.compile(rf"""
# {comment}:
^            # beginning of line (since re.MULTILINE is passed)
{prefix}
{body}
{param_group}
[ \t]*       # any number of spaces and/or tabs
$            # end of line (since re.MULTILINE is passed)
""", re.MULTILINE | re.VERBOSE if magic_flag else re.IGNORECASE | re.MULTILINE | re.VERBOSE)

This function returns a regex object that can be used to find nbdev flags in multiline text

  • magic_flag pass True to find magic flags, False to find comment flags,
  • body regex fragment to match one or more flags,
  • n_params number of flag parameters to match and catch (-1 for any number of params),
  • comment explains what the compiled regex should do.
In [ ]:
#hide
re_blank_test = _mk_flag_re(False, 'export[si]?', 0, "test")
re_mod_test = _mk_flag_re(False, 'export[si]?', 1, "test")
re_opt_test = _mk_flag_re(False, 'export[si]?', (0,1), "test")
for f in ['export', 'Exports', 'exportI']:
    cell = nbformat.v4.new_code_cell(f'#{f}  \n some code')
    assert check_re(cell, re_blank_test) is not None
    assert check_re(cell, re_mod_test) is None
    assert check_re(cell, re_opt_test) is not None
    test_eq(check_re(cell, re_opt_test).groups()[0], None)
    cell.source = f'#{f} special.module \n some code'
    assert check_re(cell, re_blank_test) is None
    assert check_re(cell, re_mod_test) is not None
    test_eq(check_re(cell, re_mod_test).groups()[0], 'special.module')
    assert check_re(cell, re_opt_test) is not None
    test_eq(check_re(cell, re_opt_test).groups()[0], 'special.module')
In [ ]:
#export
_re_blank_export = _mk_flag_re(False, "export[si]?", 0,
    "Matches any line with #export, #exports or #exporti without any module name")
_re_blank_export_magic = _mk_flag_re(True, "export(?:|_and_show|_internal)", 0,
    "Matches any line with any kind of export magic without any module name")
# any kind of export magic = %nbdev_export, %nbdev_export_and_show or %nbdev_export_internal
In [ ]:
#export
_re_mod_export = _mk_flag_re(False, "export[si]?", 1,
    "Matches any line with #export, #exports or #exporti with a module name and catches it in group 1")
_re_mod_export_magic = _mk_flag_re(True, "export(?:|_and_show|_internal)", 1,
    "Matches any line with any kind of export magic with a module name and catches it in group 1")
In [ ]:
#export
_re_internal_export = _mk_flag_re(False, "exporti", (0,1),
    "Matches any line with #exporti with or without a module name")
_re_internal_export_magic = _mk_flag_re(True, "export_internal", (0,1),
    "Matches any line with export internal magic with or without a module name")
In [ ]:
%nbdev_export_internal
def _is_external_export(tst):
    "Check if a cell is an external or internal export. `tst` is an re match"
    if _re_internal_export.search(tst.string) is not None: return False
    if _re_internal_export_magic.search(tst.string) is not None: return False
    return True
In [ ]:
#export
def is_export(cell, default):
    "Check if `cell` is to be exported and returns the name of the module to export it if provided"
    tst = check_re_multi(cell, [_re_blank_export, _re_blank_export_magic])
    if tst:
        if default is None:
            print(f"This cell doesn't have an export destination and was ignored:\n{cell['source'][1]}")
        return default, _is_external_export(tst)
    tst = check_re_multi(cell, [_re_mod_export, _re_mod_export_magic])
    if tst: return os.path.sep.join(tst.groups()[0].split('.')), _is_external_export(tst)
    else: return None

is_export returns;

  • a tuple of ("module name", "external boolean") if cell is to be exported or
    • "external boolean" will be False for an internal export
  • None if cell will not be exported.

The cells to export are marked with an %nbdev_export, %nbdev_export_and_show or %nbdev_export_internal flag, potentially with a module name where we want it exported. The default module is given in a cell of the form %nbdev_default_export bla inside the notebook (usually at the top), though in this function, it needs the be passed (the final script will read the whole notebook to find it).

  • a cell marked with %nbdev_export, %nbdev_export_and_show or %nbdev_export_internal will be exported to the default module
  • a cell marked with %nbdev_export special.module, %nbdev_export_and_show special.module or %nbdev_export_internal special.module will be exported in special.module (located in lib_name/special/module.py)
  • a cell marked with %nbdev_export will have it's signature added to the documentation
  • a cell marked with %nbdev_export_and_show will additionally have it's source code added to the documentation
  • a cell marked with %nbdev_export_internal will not show up in the documentation, and will also not be added to __all__.
In [ ]:
cell = test_nb['cells'][1].copy()
test_eq(is_export(cell, 'export'), ('export', True))
cell['source'] = "%nbdev_export\nfrom nbdev.imports import *"
test_eq(is_export(cell, 'export'), ('export', True))
cell['source'] = "# exports"
test_eq(is_export(cell, 'export'), ('export', True))
cell['source'] = "%nbdev_export_and_show"
test_eq(is_export(cell, 'export'), ('export', True))
cell['source'] = "# exporti"
test_eq(is_export(cell, 'export'), ('export', False))
cell['source'] = "%nbdev_export_internal"
test_eq(is_export(cell, 'export'), ('export', False))
cell['source'] = "# export mod"
test_eq(is_export(cell, 'export'), ('mod', True))
cell['source'] = "%nbdev_export mod\nfrom nbdev.imports import *"
test_eq(is_export(cell, 'export'), ('mod', True))
cell['source'] = "# export mod.file"
test_eq(is_export(cell, 'export'), (f'mod{os.path.sep}file', True))
cell['source'] = "%nbdev_export mod.file"
test_eq(is_export(cell, 'export'), (f'mod{os.path.sep}file', True))
cell['source'] = "# exporti mod.file"
test_eq(is_export(cell, 'export'), (f'mod{os.path.sep}file', False))
cell['source'] = "%nbdev_export_internal mod.file"
test_eq(is_export(cell, 'export'), (f'mod{os.path.sep}file', False))
cell['source'] = "# expt mod.file"
assert is_export(cell, 'export') is None
cell['source'] = "# exportmod.file"
assert is_export(cell, 'export') is None
cell['source'] = "# exportsmod.file"
assert is_export(cell, 'export') is None
cell['source'] = "%nbdev_export_and_showmod.file"
assert is_export(cell, 'export') is None
cell['source'] = "%nbdev_export_internalmod.file"
assert is_export(cell, 'export') is None
cell['source'] = "# exporti mod file"
assert is_export(cell, 'export') is None
cell['source'] = "%nbdev_export_internal mod file"
assert is_export(cell, 'export') is None
In [ ]:
#export
_re_default_exp = _mk_flag_re(False, 'default_exp', 1,
    "Matches any line with #default_exp with a module name and catches it in group 1")
_re_default_exp_magic = _mk_flag_re(True, 'default_export', 1,
    "Matches any line with %nbdev_default_export with a module name and catches it in group 1")
In [ ]:
#export
def find_default_export(cells):
    "Find in `cells` the default export module."
    for cell in cells:
        tst = check_re_multi(cell, [_re_default_exp, _re_default_exp_magic])
        if tst: return tst.groups()[0]

Stops at the first cell containing a # default_exp or %nbdev_default_export flag (if there are several) and returns the value behind. Returns None if there are no cell with that code.

In [ ]:
test_eq(find_default_export(test_nb['cells']), 'export')
assert find_default_export(test_nb['cells'][2:]) is None
In [ ]:
#hide
mods = [f'mod{i}' for i in range(6)]
cells = [{'cell_type': 'code', 'source': f'{"#default_exp" if i%2==0 else "%nbdev_default_export"} {mod}'} 
         for i, mod in enumerate(mods)]
for i, mod in enumerate(mods):
    test_eq(mod, find_default_export(cells[i:]))

Note: ReTstFlags isn't used during export but is needed by export2html and test.

In [ ]:
#export
class ReTstFlags():
    "Provides test flag matching regular expressions"
    def __init__(self, all_flag):
        "`all_flag` tells us to match flags applied to all cells (`True`) or individual cells (`False`)"
        self.all_flag = all_flag

    def _deferred_init(self):
        "Compile at first use but not before since patterns need `Config().tst_flags`"
        if not hasattr(self, '_re'):
            tst_flags = Config().get('tst_flags', '')
            _re_all, _re_magic_all = ('all_', '[ \t]+all') if self.all_flag else ('', '')
            self._re = _mk_flag_re(False, f"{_re_all}({tst_flags})", 0,
                "Matches any line with a test flag and catches it in a group")
            self._re_magic = _mk_flag_re(True, f"({tst_flags})_test{_re_magic_all}", 0,
                "Matches any line with a magic test flag and catches it in a group")

    def findall(self, source):
        "Return all test flags found in `source`"
        self._deferred_init()
        return self._re.findall(source) + self._re_magic.findall(source)

    def search(self, source):
        "Return a match object for the first test flag found in `source`"
        self._deferred_init()
        for pat in [self._re, self._re_magic]:
            tst = pat.search(source)
            if tst: return tst
In [ ]:
%nbdev_show_doc ReTstFlags * default_class_level=3

class ReTstFlags[source]

ReTstFlags(all_flag)

Provides test flag matching regular expressions

ReTstFlags.__init__[source]

ReTstFlags.__init__(all_flag)

all_flag tells us to match flags applied to all cells (True) or individual cells (False)

ReTstFlags.findall[source]

ReTstFlags.findall(source)

Return all test flags found in source

ReTstFlags.search[source]

ReTstFlags.search(source)

Return a match object for the first test flag found in source

Listing all exported objects

The following functions make a list of everything that is exported to prepare a proper __all__ for our exported module.

In [ ]:
#export
_re_patch_func = re.compile(r"""
# Catches any function decorated with @patch, its name in group 1 and the patched class in group 2
@patch         # At any place in the cell, something that begins with @patch
(?:\s*@.*)*    # Any other decorator applied to the function
\s*def         # Any number of whitespace (including a new line probably) followed by def
\s+            # One whitespace or more
([^\(\s]+)     # Catch a group composed of anything but whitespace or an opening parenthesis (name of the function)
\s*\(          # Any number of whitespace followed by an opening parenthesis
[^:]*          # Any number of character different of : (the name of the first arg that is type-annotated)
:\s*           # A column followed by any number of whitespace
(?:            # Non-catching group with either
([^,\s\(\)]*)  #    a group composed of anything but a comma, a parenthesis or whitespace (name of the class)
|              #  or
(\([^\)]*\)))  #    a group composed of something between parenthesis (tuple of classes)
\s*            # Any number of whitespace
(?:,|\))       # Non-catching group with either a comma or a closing parenthesis
""", re.VERBOSE)
In [ ]:
tst = _re_patch_func.search("""
@patch
@log_args(a=1)
def func(obj:Class):""")
tst, tst.groups()
Out[ ]:
(<re.Match object; span=(1, 42), match='@patch\[email protected]_args(a=1)\ndef func(obj:Class)'>,
 ('func', 'Class', None))
In [ ]:
#hide
tst = _re_patch_func.search("""
@patch
def func(obj:Class):""")
test_eq(tst.groups(), ("func", "Class", None))
tst = _re_patch_func.search("""
@patch
def func (obj:Class, a)""")
test_eq(tst.groups(), ("func", "Class", None))
tst = _re_patch_func.search("""
@patch
def func (obj:(Class1, Class2), a)""")
test_eq(tst.groups(), ("func", None, "(Class1, Class2)"))
tst = _re_patch_func.search("""
@patch
def func (obj:(Class1, Class2), a:int)->int:""")
test_eq(tst.groups(), ("func", None, "(Class1, Class2)"))
tst = _re_patch_func.search("""
@patch
@log_args(but='a,b')
@funcs_kwargs
def func (obj:(Class1, Class2), a:int)->int:""")
test_eq(tst.groups(), ("func", None, "(Class1, Class2)"))
tst = _re_patch_func.search("""
@patch
@contextmanager
def func (obj:Class, a:int)->int:""")
test_eq(tst.groups(), ("func", "Class", None))
In [ ]:
#export
_re_typedispatch_func = re.compile(r"""
# Catches any function decorated with @typedispatch
(@typedispatch  # At any place in the cell, catch a group with something that begins with @typedispatch
\s*def          # Any number of whitespace (including a new line probably) followed by def
\s+             # One whitespace or more
[^\(]+          # Anything but whitespace or an opening parenthesis (name of the function)
\s*\(           # Any number of whitespace followed by an opening parenthesis
[^\)]*          # Any number of character different of )
\)[\s\S]*:)     # A closing parenthesis followed by any number of characters and whitespace (type annotation) and :
""", re.VERBOSE)
In [ ]:
#hide
assert _re_typedispatch_func.search("@typedispatch\ndef func(a, b):").groups() == ('@typedispatch\ndef func(a, b):',)
assert (_re_typedispatch_func.search("@typedispatch\ndef func(a:str, b:bool)->int:").groups() ==
                                    ('@typedispatch\ndef func(a:str, b:bool)->int:',))
In [ ]:
#export
_re_class_func_def = re.compile(r"""
# Catches any 0-indented function or class definition with its name in group 1
^              # Beginning of a line (since re.MULTILINE is passed)
(?:async\sdef|def|class)  # Non-catching group for def or class
\s+            # One whitespace or more
([^\(\s]+)     # Catching group with any character except an opening parenthesis or a whitespace (name)
\s*            # Any number of whitespace
(?:\(|:)       # Non-catching group with either an opening parenthesis or a : (classes don't need ())
""", re.MULTILINE | re.VERBOSE)
In [ ]:
#hide
test_eq(_re_class_func_def.search("class Class:").groups(), ('Class',))
test_eq(_re_class_func_def.search("def func(a, b):").groups(), ('func',))
test_eq(_re_class_func_def.search("def func(a:str, b:bool)->int:").groups(), ('func',))
test_eq(_re_class_func_def.search("async def func(a, b):").groups(), ('func',))
In [ ]:
#export
_re_obj_def = re.compile(r"""
# Catches any 0-indented object definition (bla = thing) with its name in group 1
^                          # Beginning of a line (since re.MULTILINE is passed)
([_a-zA-Z]+[a-zA-Z0-9_\.]*)  # Catch a group which is a valid python variable name
\s*                        # Any number of whitespace
(?::\s*\S.*|)=  # Non-catching group of either a colon followed by a type annotation, or nothing; followed by an =
""", re.MULTILINE | re.VERBOSE)
In [ ]:
#hide
test_eq(_re_obj_def.search("a = 1").groups(), ('a',))
test_eq(_re_obj_def.search("a.b = 1").groups(), ('a.b',))
test_eq(_re_obj_def.search("_aA1=1").groups(), ('_aA1',))
test_eq(_re_obj_def.search("a : int =1").groups(), ('a',))
test_eq(_re_obj_def.search("a:f(':=')=1").groups(), ('a',))
assert _re_obj_def.search("@abc=2") is None
assert _re_obj_def.search("a a=2") is None
In [ ]:
#export
def _not_private(n):
    for t in n.split('.'):
        if (t.startswith('_') and not t.startswith('__')) or t.startswith('@'): return False
    return '\\' not in t and '^' not in t and '[' not in t and t != 'else'

def export_names(code, func_only=False):
    "Find the names of the objects, functions or classes defined in `code` that are exported."
    #Format monkey-patches with @patch
    def _f(gps):
        nm, cls, t = gps.groups()
        if cls is not None: return f"def {cls}.{nm}():"
        return '\n'.join([f"def {c}.{nm}():" for c in re.split(', *', t[1:-1])])

    code = _re_typedispatch_func.sub('', code)
    code = _re_patch_func.sub(_f, code)
    names = _re_class_func_def.findall(code)
    if not func_only: names += _re_obj_def.findall(code)
    return [n for n in names if _not_private(n) and not iskeyword(n)]

This function only picks the zero-indented objects on the left side of an =, functions or classes (we don't want the class methods for instance) and excludes private names (that begin with _) but no dunder names. It only returns func and class names (not the objects) when func_only=True.

To work properly with fastai added python functionality, this function ignores function decorated with @typedispatch (since they are defined multiple times) and unwraps properly functions decorated with @patch.

In [ ]:
test_eq(export_names("def my_func(x):\n  pass\nclass MyClass():"), ["my_func", "MyClass"])

#Indented funcs are ignored (funcs inside a class)
test_eq(export_names("  def my_func(x):\n  pass\nclass MyClass():"), ["MyClass"])

#Private funcs are ignored, dunder are not
test_eq(export_names("def _my_func():\n  pass\nclass MyClass():"), ["MyClass"])
test_eq(export_names("__version__ = 1:\n  pass\nclass MyClass():"), ["MyClass", "__version__"])

#trailing spaces
test_eq(export_names("def my_func ():\n  pass\nclass MyClass():"), ["my_func", "MyClass"])

#class without parenthesis
test_eq(export_names("def my_func ():\n  pass\nclass MyClass:"), ["my_func", "MyClass"])

#object and funcs
test_eq(export_names("def my_func ():\n  pass\ndefault_bla=[]:"), ["my_func", "default_bla"])
test_eq(export_names("def my_func ():\n  pass\ndefault_bla=[]:", func_only=True), ["my_func"])

#Private objects are ignored
test_eq(export_names("def my_func ():\n  pass\n_default_bla = []:"), ["my_func"])

#Objects with dots are privates if one part is private
test_eq(export_names("def my_func ():\n  pass\ndefault.bla = []:"), ["my_func", "default.bla"])
test_eq(export_names("def my_func ():\n  pass\ndefault._bla = []:"), ["my_func"])

#Monkey-path with @patch are properly renamed
test_eq(export_names("@patch\ndef my_func(x:Class):\n  pass"), ["Class.my_func"])
test_eq(export_names("@patch\ndef my_func(x:Class):\n  pass", func_only=True), ["Class.my_func"])
test_eq(export_names("some code\n@patch\ndef my_func(x:Class, y):\n  pass"), ["Class.my_func"])
test_eq(export_names("some code\n@patch\ndef my_func(x:(Class1,Class2), y):\n  pass"), ["Class1.my_func", "Class2.my_func"])

#Check delegates
test_eq(export_names("@delegates(keep=True)\nclass someClass:\n  pass"), ["someClass"])

#Typedispatch decorated functions shouldn't be added
test_eq(export_names("@patch\ndef my_func(x:Class):\n  pass\n@typedispatch\ndef func(x: TensorImage): pass"), ["Class.my_func"])

#try, except and other keywords should not be picked up (these can look like object def with type annotation)
test_eq(export_names("try:\n    a=1\nexcept:\n    b=2"), [])
test_eq(export_names("try:\n    this_might_work\nexcept:\n    b=2"), [])
In [ ]:
#export
_re_all_def   = re.compile(r"""
# Catches a cell with defines \_all\_ = [\*\*] and get that \*\* in group 1
^_all_   #  Beginning of line (since re.MULTILINE is passed)
\s*=\s*  #  Any number of whitespace, =, any number of whitespace
\[       #  Opening [
([^\n\]]*) #  Catching group with anything except a ] or newline
\]       #  Closing ]
""", re.MULTILINE | re.VERBOSE)

#Same with __all__
_re__all__def = re.compile(r'^__all__\s*=\s*\[([^\]]*)\]', re.MULTILINE)

_re_all_def_magic = _mk_flag_re(True, 'add2all', -1,
    "# Catches a cell with %nbdev_add2all \*\* and get that \*\* in group 1")

hide

To be consistent with _all_, extra_add allows %nbdev_add2all to be used in either the flags section ↓

#export
%nbdev_add2all ["_not_included_by_default"]

or code section ↓

#export
def _not_included_by_default(): pass
%nbdev_add2all ["_not_included_by_default"]

Note: sub #nbdev_comment \g<0> means that if we find _all_ or %nbdev_add2all in the code section, it'll be written to the library but prefixed with #nbdev_comment - sync#_script2notebook knows to remove this special comment when writing back to notebooks.

In [ ]:
#export
def extra_add(flags, code):
    "Catch adds to `__all__` required by a cell with `_all_=` or `%nbdev_add2all`"
    m = check_re_multi({'source': code}, [_re_all_def, _re_all_def_magic], False)
    if m:
        code = m.re.sub('#nbdev_' + 'comment \g<0>', code)
        code = re.sub(r'([^\n]|^)\n*$', r'\1', code)
    if not m: m = check_re({'source': flags}, _re_all_def_magic, False)
    if not m: return [], code
    def clean_quotes(s):
        "Return `s` enclosed in single quotes, removing double quotes if needed"
        if s.startswith("'") and s.endswith("'"): return s
        if s.startswith('"') and s.endswith('"'): s = s[1:-1]
        return f"'{s}'"
    return [clean_quotes(s) for s in parse_line(m.group(1))], code

Sometimes objects are not picked to be automatically added to the __all__ of the module so you will need to add them manually. To do so, create an exported cell with the following code %nbdev_add2all "name", "name2"

Please note:

  • elements in %nbdev_add2all can be space and/or comma separated and don't have to be quoted
    # 'func', 'func2' and 'func3' will be added to `__all__`
    %nbdev_add2all func, func2 func3
    # you'll see warnings if any unquoted function names can't be found
    
  • elements on a new line will not be added to __all__
    # 'func2' won't get added to `__all__`
    %nbdev_add2all 'func',
          'func2'
    
  • only the first %nbdev_add2all in a cell will get picked up
    %nbdev_add2all ['func']
    # `func2` won't get added to `__all__`
    %nbdev_add2all ['func2']
    
  • but you can have any number of %nbdev_add2alls in a notebook by putting them in different cells.
In [ ]:
#hide
# most of what you can do with _all_ works the same with %nbdev_add2all
for code, expected in [
        ['_all_ = ["func", "func1", "func2"]', 
         (["'func'", "'func1'", "'func2'"],'#nbdev_comment _all_ = ["func", "func1", "func2"]')],
        ['_all_=[func, func1, func2]', 
         (["'func'", "'func1'", "'func2'"],'#nbdev_comment _all_=[func, func1, func2]')],
        ["_all_ = ['func', 'func1', 'func2']", 
         (["'func'", "'func1'", "'func2'"],"#nbdev_comment _all_ = ['func', 'func1', 'func2']")],
        ['_all_ = ["func",   "func1" , "func2"]', 
         (["'func'", "'func1'", "'func2'"],'#nbdev_comment _all_ = ["func",   "func1" , "func2"]')],
        ["_all_ = ['func','func1', 'func2']\n", 
         (["'func'", "'func1'", "'func2'"],"#nbdev_comment _all_ = ['func','func1', 'func2']")],
        ["_all_ = ['func']\n_all_ = ['func1', 'func2']\n", 
         (["'func'"],"#nbdev_comment _all_ = ['func']\n#nbdev_comment _all_ = ['func1', 'func2']")],
        ['code\n\n_all_ = ["func", "func1", "func2"]', 
         (["'func'", "'func1'", "'func2'"],'code\n\n#nbdev_comment _all_ = ["func", "func1", "func2"]')],
        ['code\n\n_all_ = [func]\nmore code', 
         (["'func'"],'code\n\n#nbdev_comment _all_ = [func]\nmore code')]]:
    test_eq(extra_add('', code), expected)
    code = code.replace('_all_ =', '%nbdev_add2all')
    expected = (expected[0], expected[1].replace('_all_ =', '%nbdev_add2all'))
    test_eq(extra_add('', code), expected)
    
# line breaks within the list of names means _all_ is ignored
test_eq(extra_add('', "_all_ = ['func',\n'func1', 'func2']\n"), ([],"_all_ = ['func',\n'func1', 'func2']\n"))
# same thing with %nbdev_add2all makes a mess but ...
# it can be good to fail fast and you will have seen a UsageError in the nb
test_eq(extra_add('', "%nbdev_add2all ['func',\n'func1', 'func2']"), 
        (["'['func''"],"#nbdev_comment %nbdev_add2all ['func',\n'func1', 'func2']"))
# without square brakets, you won't see any errors (if they have line breaks in the list of names)
test_eq(extra_add('', "%nbdev_add2all 'func',\n'func1', 'func2'"), 
        (["'func'"],"#nbdev_comment %nbdev_add2all 'func',\n'func1', 'func2'"))
test_eq(extra_add('', "%nbdev_add2all func,\nfunc1, func2"), 
        (["'func'"],"#nbdev_comment %nbdev_add2all func,\nfunc1, func2"))
# comma separation is optional
test_eq(extra_add('', "%nbdev_add2all [ func func1 func2 ]"), 
        (["'func'", "'func1'", "'func2'"],'#nbdev_comment %nbdev_add2all [ func func1 func2 ]'))
# these are the neatest/recommended ways to use %nbdev_add2all
test_eq(extra_add('%nbdev_export', "%nbdev_add2all func func1 func2"), 
        (["'func'", "'func1'", "'func2'"],'#nbdev_comment %nbdev_add2all func func1 func2'))
test_eq(extra_add('#export', "%nbdev_add2all func,func1,func2"), 
        (["'func'", "'func1'", "'func2'"],'#nbdev_comment %nbdev_add2all func,func1,func2'))
test_eq(extra_add('', "%nbdev_add2all func, func1, func2"), 
        (["'func'", "'func1'", "'func2'"],'#nbdev_comment %nbdev_add2all func, func1, func2'))
test_eq(extra_add("%nbdev_add2all func func1 func2",'some code'), (["'func'", "'func1'", "'func2'"],'some code'))
test_eq(extra_add("%nbdev_add2all func,func1,func2",'some code'), (["'func'", "'func1'", "'func2'"],'some code'))
test_eq(extra_add("%nbdev_add2all func, func1, func2",'some code'), (["'func'", "'func1'", "'func2'"],'some code'))
In [ ]:
#export
_re_from_future_import = re.compile(r"^from[ \t]+__future__[ \t]+import.*$", re.MULTILINE)

def _from_future_import(fname, flags, code, to_dict=None):
    "Write `__future__` imports to `fname` and return `code` with `__future__` imports commented out"
    from_future_imports = _re_from_future_import.findall(code)
    if from_future_imports: code = _re_from_future_import.sub('#nbdev' + '_comment \g<0>', code)
    else: from_future_imports = _re_from_future_import.findall(flags)
    if not from_future_imports or to_dict is not None: return code
    with open(fname, 'r', encoding='utf8') as f: text = f.read()
    start = _re__all__def.search(text).start()
    with open(fname, 'w', encoding='utf8') as f:
        f.write('\n'.join([text[:start], *from_future_imports, '\n', text[start:]]))
    return code

If you need a from __future__ import in your library, you can export your cell with special comments:

#export
from __future__ import annotations
class ...

or magic flags:

from __future__ import annotations
#export
class ...

Notice that %nbdev_export is after the __future__ import. Because __future__ imports must occur at the beginning of the file, nbdev allows __future__ imports in the flags section of a cell.

In [ ]:
#hide
txt = """
# AUTOHEADER ... File to edit: mod.ipynb (unless otherwise specified).

__all__ = [my_file, MyClas]
# Cell
def valid_code(): pass"""
expected_txt = """
# AUTOHEADER ... File to edit: mod.ipynb (unless otherwise specified).


from  __future__ import  annotations    
from __future__ import generator_stop


__all__ = [my_file, MyClas]
# Cell
def valid_code(): pass"""
flags="# export"
code = """
# comment
from  __future__ import  annotations    
valid_code = False # but _from_future_import will work anyway
from __future__ import generator_stop
 from __future__ import not_zero_indented
valid_code = True
"""
expected_code = """
# comment
#nbdev_comment from  __future__ import  annotations    
valid_code = False # but _from_future_import will work anyway
#nbdev_comment from __future__ import generator_stop
 from __future__ import not_zero_indented
valid_code = True
"""

def _run_from_future_import_test():
    fname = 'test_from_future_import.txt'
    with open(fname, 'w', encoding='utf8') as f: f.write(txt)

    actual_code=_from_future_import(fname, flags, code, {})
    test_eq(expected_code, actual_code)
    with open(fname, 'r', encoding='utf8') as f: test_eq(f.read(), txt)

    actual_code=_from_future_import(fname, flags, code)
    test_eq(expected_code, actual_code)
    with open(fname, 'r', encoding='utf8') as f: test_eq(f.read(), expected_txt)

    os.remove(fname)

_run_from_future_import_test()

flags="""from  __future__ import  annotations    
from __future__ import generator_stop
#export"""
code = ""
expected_code = ""
fname = 'test_from_future_import.txt'

_run_from_future_import_test()
In [ ]:
#export
def _add2all(fname, names, line_width=120):
    if len(names) == 0: return
    with open(fname, 'r', encoding='utf8') as f: text = f.read()
    tw = TextWrapper(width=120, initial_indent='', subsequent_indent=' '*11, break_long_words=False)
    re_all = _re__all__def.search(text)
    start,end = re_all.start(),re_all.end()
    text_all = tw.wrap(f"{text[start:end-1]}{'' if text[end-2]=='[' else ', '}{', '.join(names)}]")
    with open(fname, 'w', encoding='utf8') as f: f.write(text[:start] + '\n'.join(text_all) + text[end:])
In [ ]:
#hide
fname = 'test_add.txt'
with open(fname, 'w', encoding='utf8') as f: f.write("Bla\n__all__ = [my_file, MyClas]\nBli")
_add2all(fname, ['new_function'])
with open(fname, 'r', encoding='utf8') as f: 
    test_eq(f.read(), "Bla\n__all__ = [my_file, MyClas, new_function]\nBli")
_add2all(fname, [f'new_function{i}' for i in range(10)])
with open(fname, 'r', encoding='utf8') as f: 
    test_eq(f.read(), """Bla
__all__ = [my_file, MyClas, new_function, new_function0, new_function1, new_function2, new_function3, new_function4,
           new_function5, new_function6, new_function7, new_function8, new_function9]
Bli""")
os.remove(fname)
In [ ]:
#export
def relative_import(name, fname):
    "Convert a module `name` to a name relative to `fname`"
    mods = name.split('.')
    splits = str(fname).split(os.path.sep)
    if mods[0] not in splits: return name
    i=len(splits)-1
    while i>0 and splits[i] != mods[0]: i-=1
    splits = splits[i:]
    while len(mods)>0 and splits[0] == mods[0]: splits,mods = splits[1:],mods[1:]
    return '.' * (len(splits)) + '.'.join(mods)

When we say from

from lib_name.module.submodule import bla

in a notebook, it needs to be converted to something like

from .module.submodule import bla

or from .submodule import bla depending on where we are. This function deals with those imports renaming.

Note that import of the form

import lib_name.module

are left as is as the syntax import module does not work for relative imports.

In [ ]:
test_eq(relative_import('nbdev.core', Path.cwd()/'nbdev'/'data.py'), '.core')
test_eq(relative_import('nbdev.core', Path('nbdev')/'vision'/'data.py'), '..core')
test_eq(relative_import('nbdev.vision.transform', Path('nbdev')/'vision'/'data.py'), '.transform')
test_eq(relative_import('nbdev.notebook.core', Path('nbdev')/'data'/'external.py'), '..notebook.core')
test_eq(relative_import('nbdev.vision', Path('nbdev')/'vision'/'learner.py'), '.')
In [ ]:
#export
_re_import = ReLibName(r'^(\s*)from (LIB_NAME\.\S*) import (.*)$')
In [ ]:
#export
def _deal_import(code_lines, fname):
    def _replace(m):
        sp,mod,obj = m.groups()
        return f"{sp}from {relative_import(mod, fname)} import {obj}"
    return [_re_import.re.sub(_replace,line) for line in code_lines]
In [ ]:
#hide
lines = ["from nbdev.core import *", 
         "nothing to see", 
         "  from nbdev.vision import bla1, bla2", 
         "from nbdev.vision import models",
         "import nbdev.vision"]
test_eq(_deal_import(lines, Path.cwd()/'nbdev'/'data.py'), [
    "from .core import *", 
    "nothing to see", 
    "  from .vision import bla1, bla2", 
    "from .vision import models",
    "import nbdev.vision"
])

Create the library

Saving an index

To be able to build back a correspondence between functions and the notebooks they are defined in, we need to store an index. It's done in the private module _nbdev inside your library, and the following function are used to define it.

In [ ]:
#export
_re_index_custom = re.compile(r'def custom_doc_links\(name\):(.*)$', re.DOTALL)
In [ ]:
#export
def reset_nbdev_module():
    "Create a skeletton for <code>_nbdev</code>"
    fname = Config().lib_path/'_nbdev.py'
    fname.parent.mkdir(parents=True, exist_ok=True)
    sep = '\n'* (int(Config().get('cell_spacing', '1'))+1)
    if fname.is_file():
        with open(fname, 'r') as f: search = _re_index_custom.search(f.read())
    else: search = None
    prev_code = search.groups()[0] if search is not None else ' return None\n'
    with open(fname, 'w') as f:
        f.write(f"# AUTOGENERATED BY NBDEV! DO NOT EDIT!")
        f.write('\n\n__all__ = ["index", "modules", "custom_doc_links", "git_url"]')
        f.write('\n\nindex = {}')
        f.write('\n\nmodules = []')
        f.write(f'\n\ndoc_url = "{Config().doc_host}{Config().doc_baseurl}"')
        f.write(f'\n\ngit_url = "{Config().git_url}"')
        f.write(f'{sep}def custom_doc_links(name):{prev_code}')
In [ ]:
#export
class _EmptyModule():
    def __init__(self):
        self.index,self.modules = {},[]
        self.doc_url,self.git_url = f"{Config().doc_host}{Config().doc_baseurl}",Config().git_url

    def custom_doc_links(self, name): return None
In [ ]:
#export
def get_nbdev_module():
    "Reads <code>_nbdev</code>"
    try:
        spec = importlib.util.spec_from_file_location(f"{Config().lib_name}._nbdev", Config().lib_path/'_nbdev.py')
        mod = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(mod)
        return mod
    except: return _EmptyModule()
In [ ]:
#export
_re_index_idx = re.compile(r'index\s*=\s*{[^}]*}')
_re_index_mod = re.compile(r'modules\s*=\s*\[[^\]]*\]')
In [ ]:
#export
def save_nbdev_module(mod):
    "Save `mod` inside <code>_nbdev</code>"
    fname = Config().lib_path/'_nbdev.py'
    with open(fname, 'r') as f: code = f.read()
    t = r',\n         '.join([f'"{k}": "{v}"' for k,v in mod.index.items()])
    code = _re_index_idx.sub("index = {"+ t +"}", code)
    t = r',\n           '.join(['"' + f.replace('\\','/') + '"' for f in mod.modules])
    code = _re_index_mod.sub(f"modules = [{t}]", code)
    with open(fname, 'w') as f: f.write(code)
In [ ]:
#hide
ind,ind_bak = Config().lib_path/'_nbdev.py',Config().lib_path/'_nbdev.bak'
if ind.exists(): shutil.move(ind, ind_bak)
try:
    reset_nbdev_module()
    mod = get_nbdev_module()
    test_eq(mod.index, {})
    test_eq(mod.modules, [])

    mod.index = {'foo':'bar'}
    mod.modules.append('lala.bla')
    save_nbdev_module(mod)

    mod = get_nbdev_module()
    test_eq(mod.index, {'foo':'bar'})
    test_eq(mod.modules, ['lala.bla'])
finally:
    if ind_bak.exists(): shutil.move(ind_bak, ind)

Create the modules

In [ ]:
#export
def split_flags_and_code(cell, return_type=list):
    "Splits the `source` of a cell into 2 parts and returns (flags, code)"
    code_lines = cell['source'].split('\n')
    split_pos = 0 if code_lines[0].strip().startswith('#') else -1
    for i, line in enumerate(code_lines):
        if line.startswith('%nbdev_'): split_pos=i
        elif not line.startswith('#') and line.strip() and not _re_from_future_import.match(line): break
    split_pos+=1
    res = code_lines[:split_pos], code_lines[split_pos:]
    if return_type is list: return res
    return tuple('\n'.join(r) for r in res)

return_type tells us if the tuple returned will contain lists of lines or strings with line breaks.

If no magic flags are found, treat the first comment line as a flag

split_flags_and_code example

If magic flags are found, the flags part can contain multiple lines

split_flags_and_code example

hide

When magic flags are found:

  • We consider any line, in a code cell, that starts with %nbdev_ to be a flag
  • We consider any non-blank line that does not start with %nbdev_, from __future__ import or # to be code
  • The flags section can contain comments and/or blank lines
  • The last line of a flags section is always a flag
    • Unless there are no flags at all, in which case the flags section is empty
  • If flags appear after the start of the code section (after while in this case) they will be included in the code section
    • So we never have code in a flags section but
    • We might find flags in a code section
  • Using %nbdev_ flags in the code section is probably a mistake but will be exported as-is
    • This will probably cause your module to fail at runtime
In [ ]:
def _test_split_flags_and_code(expected_flags, expected_code):
    cell = nbformat.v4.new_code_cell('\n'.join(expected_flags + expected_code))
    test_eq((expected_flags, expected_code), split_flags_and_code(cell))
    expected=('\n'.join(expected_flags), '\n'.join(expected_code))
    test_eq(expected, split_flags_and_code(cell, str))
    
_test_split_flags_and_code([
    '#export'],
    ['# TODO: write this function',
    'def func(x): pass'])
In [ ]:
#hide
_test_split_flags_and_code([
    ],
    ['class DocsTestClass:',
    '    def test(): pass',
    '%nbdev_export in.the.wrong.place'])

_test_split_flags_and_code([
    '#hide',
    '%nbdev_export'],
    ['',
    '#some_kind_of_test',
    '# for tests only',
    ' # another_kind_of_test',
    '  #another kind of comment',
    'class DocsTestClass:',
    '    def test(): pass'])

_test_split_flags_and_code([
    'from __future__ import annotations',
    '%nbdev_export'],
    [])
In [ ]:
#export
def create_mod_file(fname, nb_path):
    "Create a module file for `fname`."
    fname.parent.mkdir(parents=True, exist_ok=True)
    file_path = os.path.relpath(nb_path, Config().config_file.parent).replace('\\', '/')
    with open(fname, 'w') as f:
        f.write(f"# AUTOGENERATED! DO NOT EDIT! File to edit: {file_path} (unless otherwise specified).")
        f.write('\n\n__all__ = []')

A new module filename is created each time a notebook has a cell marked with %nbdev_default_export. In your collection of notebooks, you should only have one notebook that creates a given module since they are re-created each time you do a library build (to ensure the library is clean). Note that any file you create manually will never be overwritten (unless it has the same name as one of the modules defined in a %nbdev_default_export cell) so you are responsible to clean up those yourself.

fname is the notebook that contained the %nbdev_default_export cell.

In [ ]:
#export
def create_mod_files(files, to_dict=False):
    "Create mod files for default exports found in `files`"
    modules = []
    for f in files:
        fname = Path(f)
        nb = read_nb(fname)
        default = find_default_export(nb['cells'])
        if default is not None:
            default = os.path.sep.join(default.split('.'))
            modules.append(default)
            if not to_dict:
                create_mod_file(Config().lib_path/f'{default}.py', Config().nbs_path/f'{fname}')
    return modules

Create module files for all %nbdev_default_export flags found in files and return a list containing the names of modules created.

Note: The number if modules returned will be less that the number of files passed in if files do not %nbdev_default_export.

By creating all module files before calling _notebook2script, the order of execution no longer matters - so you can now export to a notebook that is run "later".

You might still have problems when

  • converting a subset of notebooks or
  • exporting to a module that does not have a %nbdev_default_export yet

in which case _notebook2script will print warnings like;

Warning: Exporting to "core.py" but this module is not part of this build

If you see a warning like this

  • and the module file (e.g. "core.py") does not exist, you'll see a FileNotFoundError
  • if the module file exists, the exported cell will be written - even if the exported cell is already in the module file
In [ ]:
#export
def _notebook2script(fname, modules, silent=False, to_dict=None):
    "Finds cells starting with `#export` and puts them into a module created by `create_mod_files`"
    if os.environ.get('IN_TEST',0): return  # don't export if running tests
    sep = '\n'* (int(Config().get('cell_spacing', '1'))+1)
    fname = Path(fname)
    nb = read_nb(fname)
    default = find_default_export(nb['cells'])
    if default is not None:
        default = os.path.sep.join(default.split('.'))
    mod = get_nbdev_module()
    exports = [is_export(c, default) for c in nb['cells']]
    cells = [(i,c,e) for i,(c,e) in enumerate(zip(nb['cells'],exports)) if e is not None]
    for i,c,(e,a) in cells:
        if e not in modules: print(f'Warning: Exporting to "{e}.py" but this module is not part of this build')
        fname_out = Config().lib_path/f'{e}.py'
        orig = (f'# {"" if a else "Internal "}C' if e==default else f'# Comes from {fname.name}, c') + 'ell\n'
        flag_lines,code_lines = split_flags_and_code(c)
        code_lines = _deal_import(code_lines, fname_out)
        code = sep + orig + '\n'.join(code_lines)
        names = export_names(code)
        flags = '\n'.join(flag_lines)
        extra,code = extra_add(flags, code)
        code = _from_future_import(fname_out, flags, code, to_dict)
        if a:
            if to_dict is None: _add2all(fname_out, [f"'{f}'" for f in names if '.' not in f and len(f) > 0] + extra)
        mod.index.update({f: fname.name for f in names})
        code = re.sub(r' +$', '', code, flags=re.MULTILINE)
        if code != sep + orig[:-1]:
            if to_dict is not None: to_dict[e].append((i, fname, code))
            else:
                with open(fname_out, 'a', encoding='utf8') as f: f.write(code)
        if f'{e}.py' not in mod.modules: mod.modules.append(f'{e}.py')
    save_nbdev_module(mod)

    if not silent: print(f"Converted {fname.name}.")
    return to_dict
In [ ]:
#hide
if not os.environ.get('IN_TEST',0):
    modules = create_mod_files(glob.glob('00_export.ipynb'))
    _notebook2script('00_export.ipynb', modules)
Converted 00_export.ipynb.
In [ ]:
#hide
with open(Config().lib_path/('export.py')) as f: l = f.readline()
test_eq(l, '# AUTOGENERATED! DO NOT EDIT! File to edit: nbs/00_export.ipynb (unless otherwise specified).\n')
In [ ]:
#export
def add_init(path):
    "Add `__init__.py` in all subdirs of `path` containing python files if it's not there already"
    for p,d,f in os.walk(path):
        for f_ in f:
            if f_.endswith('.py'):
                if not (Path(p)/'__init__.py').exists(): (Path(p)/'__init__.py').touch()
                break
In [ ]:
with tempfile.TemporaryDirectory() as d:
    os.makedirs(Path(d)/'a', exist_ok=True)
    (Path(d)/'a'/'f.py').touch()
    os.makedirs(Path(d)/'a/b', exist_ok=True)
    (Path(d)/'a'/'b'/'f.py').touch()
    add_init(d)
    assert not (Path(d)/'__init__.py').exists()
    for e in [Path(d)/'a', Path(d)/'a/b']:
        assert (e/'__init__.py').exists()
In [ ]:
#export
_re_version = re.compile('^__version__\s*=.*$', re.MULTILINE)
In [ ]:
#export
def update_version():
    "Add or update `__version__` in the main `__init__.py` of the library"
    fname = Config().lib_path/'__init__.py'
    if not fname.exists(): fname.touch()
    version = f'__version__ = "{Config().version}"'
    with open(fname, 'r') as f: code = f.read()
    if _re_version.search(code) is None: code = version + "\n" + code
    else: code = _re_version.sub(version, code)
    with open(fname, 'w') as f: f.write(code)
In [ ]:
#export
_re_baseurl = re.compile('^baseurl\s*:.*$', re.MULTILINE)
In [ ]:
#export
def update_baseurl():
    "Add or update `baseurl` in `_config.yml` for the docs"
    fname = Config().doc_path/'_config.yml'
    if not fname.exists(): return
    with open(fname, 'r') as f: code = f.read()
    if _re_baseurl.search(code) is None: code = code + f"\nbaseurl: {Config().doc_baseurl}"
    else: code = _re_baseurl.sub(f"baseurl: {Config().doc_baseurl}", code)
    with open(fname, 'w') as f: f.write(code)
In [ ]:
#export
def notebook2script(fname=None, silent=False, to_dict=False):
    "Convert notebooks matching `fname` to modules"
    # initial checks
    if os.environ.get('IN_TEST',0): return  # don't export if running tests
    if fname is None:
        reset_nbdev_module()
        update_version()
        update_baseurl()
        files = [f for f in Config().nbs_path.glob('*.ipynb') if not f.name.startswith('_')]
    else: files = glob.glob(fname)
    d = collections.defaultdict(list) if to_dict else None
    modules = create_mod_files(files, to_dict)
    for f in sorted(files): d = _notebook2script(f, modules, silent=silent, to_dict=d)
    if to_dict: return d
    else: add_init(Config().lib_path)

Finds cells starting with %nbdev_export and puts them into the appropriate module. If fname is not specified, this will convert all notebook not beginning with an underscore in the nb_folder defined in setting.ini. Otherwise fname can be a single filename or a glob expression.

silent makes the command not print any statement and to_dict is used internally to convert the library to a dictionary.

In [ ]:
#export
class DocsTestClass:
    "for tests only"
    def test(): pass
In [ ]:
#hide
%nbdev_export_internal
#for tests only
def update_lib_with_exporti_testfn(): pass

Export -

In [ ]:
#hide
from nbdev.export import *
notebook2script()
Converted 00_export.ipynb.
Converted 01_sync.ipynb.
Converted 02_showdoc.ipynb.
Converted 03_export2html.ipynb.
Converted 04_test.ipynb.
Converted 05_merge.ipynb.
Converted 05a_conda.ipynb.
Converted 06_cli.ipynb.
Converted 07_clean.ipynb.
Converted 99_search.ipynb.
Converted index.ipynb.
Converted magic_flags.ipynb.
Converted nbdev_callbacks.ipynb.
Converted tutorial.ipynb.
In [ ]: