Tutorial for Box-X

Tool-box for Efficient Build and Debug in Python. Especially for Scientific Computing and Computer Vision.

We use Binder to run this notebooks in an executable interactive online environment. That mean you can run those cells rightnow in your browser without download repository.

This tutorial is divided into 2 parts by wether the tool is general:

  1. General Python Tool. The tools could be used anywhere in Python
  2. Scientific Computing and Computer Vision Tool. Those tools only be useful in Scientific Computing and Computer Vision field

P.S. This notebook compatible with Python 2/3

1. General Python Tool

p is a better way to do print


1. p/x will print(x) and return x

In [1]:
from boxx import p 

s = 'p/x will print(x) and return x'
p/s
p/x will print(x) and return x
Out[1]:
'p/x will print(x) and return x'
In [2]:
from boxx import p 
from random import randint

s = 'ABCD'
print('the output of randint(0, 3) is :')

sample = s[p/randint(0, 3)]

sample
the output of randint(0, 3) is :
1
Out[2]:
'B'

As you see, p/x is easy to print value in expression while debugging.

💡 Note:

p/randint(0, 3) print the value of randint(0, 3) and return the value itself, which won't influences the program.


↓ Use pow operator for highest evaluation order.

In [ ]:
# try run this cell online
from boxx import p 
from random import randint 
tenx = 10 * p**randint(0,9) 
tenx

2. p() to pretty print all variables in function or module with thier name

In [3]:
from boxx import p 

def f(arg=517):
    l = [1, 2]
    p()
f()
File: "<ipython-input-3-ed27d16b8559>", line 3, in f
Stacks: f <-ipython-input
Locals: 
└── /: dict  2
    ├── l: list  2
    │   ├── 0: 1
    │   └── 1: 2
    └── arg: 517

p() will pretty print all variables in locals() and some infomation about the frame.

BTW, import boxx.p has the same effect.

3. with p: will pretty print mulit variables under "with statement"

Only interested variables are printed which is under "with statement"

In [4]:
from boxx import p
from random import randint
def f():
    other_vars = "No need to pay attention"
    with p:
        a = randint(1, 9)
        l = [a, a*2]
    others = "No need to pay attention"
f()
withprint from File: "<ipython-input-4-cdc830ddfc96>", line 3, in f
All Vars's Values :
└── /: dict  2
    ├── l: list  2
    │   ├── 0: 1
    │   └── 1: 2
    └── a: 1

g and gg could transport variable to Python interactive console


1. Use g.name=x or g.name/x to transport variable that in function or module to console.

The meaning of g,gg are "to Global", "to Global and log"

In [5]:
from boxx import g

def f():
    listt = [1,2]
    g.l = listt # `listt` is transported to console as `l`
f()

l
Out[5]:
[1, 2]

g.l = listt create new var l In Python interactive console and transport listt assign to l.

💡 Note: if variable name exists in console before, the variable's value will be covered by new value.


gg is same usage as g, but gg will print the transported variable.
Use g.name/x to convenient transport value in expression.

In [6]:
from boxx import g, gg

def f():
    listt = [1,2]
    gg.l = listt
    return g.by_div/listt

listt = f()

# l, by_div are transported to console
(listt, l, by_div, l is listt, by_div is listt)
gg.l:"[1, 2]"
Out[6]:
([1, 2], [1, 2], [1, 2], True, True)

💡 Note:

  • In Python interactive console, variable l, by_div are created.

  • All of they are listt has the same id.

2. g() to transport all variables that in the function to Python interactive console

g() in a function ,can transport all variables that in the function (or module) to console. It's a useful tool for debugging.

In [7]:
from boxx import g
def f(arg=517):
    l = [1, 2]
    g()
f()

# transport all variables in function to console 
arg, l
Out[7]:
(517, [1, 2])

💡 Note:

  • g() only transport the locals() to console, the globals() will save to boxx.p

  • gg() is a print version of g, gg() will pretty print all variable with thier name and some infomation about the frame.

  • import boxx.g is convenient way to use g() instead of from boxx import g;g()(import boxx.gg is avaliable too)

In [ ]:
# try run this cell online

def f(arg):
    a = 2
    import boxx.gg
    
inp = [5 ,1 , 7]
f(inp)

# gg will pretty print all variables in f
# and `a` and `arg` are transported to console 
a, arg, arg is inp

3. with g: will transport mulit variables under "with statement"

with g will transport the interested variables to Python interactive console under "with statement"(with gg: is avaliable too)

In [8]:
from boxx import g
from random import randint
def f():
    other_vars = "No need to pay attention"
    with g:  # only transport a, l
        a = randint(1, 9)
        l = [a, a*2]
    others = "No need to pay attention"
f()

print('In console:',a , l, 'others' in locals())
In console: 9 [9, 18] False

💡 Note:

1 . with p, with g, with gg only act on assignment variables under "with statement".

2 . If variable's name exists in locals() before and id(variable) not change ,variable may not be detected Especially following cases:

1. var is int and < 256
2. `id(var)` not change

▶ Summary for debug tools


boxx debug tool matrix

How many vars \ Operation print transport print & transport
Single variable p/x g.name/x gg.name/x
Multi variables with p: with g: with gg:
All locals() p() g() gg()
All locals()_2 import boxx.p import boxx.g import boxx.gg

💡 Note:

  • transport mean "transport variable to Python interactive console"
  • All locals() mean operation will act on all variables in the function or module
  • All locals()_2 : when boxx are not imported, import boxx.{operation} is a convenient way to execution operation

timeit is convenient timing tool

In [9]:
from boxx import timeit
from time import sleep

with timeit():
    sleep(0.01) # simulation timing code

with timeit(name='sleep'):
    sleep(0.1) # simulation timing code
"timeit" spend time: 0.01011705
"sleep" spend time: 0.1004236

timeit will timing code block under "with statement" and print spend time in blue color.

mapmt is Multi Threading version of map

mapmt is the meaning of "MAP for Multi Threading", has almost same usage as map

In [10]:
from boxx import mapmt, timeit
from time import sleep

def io_block(x): # simulation io block
    sleep(0.1)
    return x
xs = range(10)

with timeit('map'):
    resoult_1 = list(map(io_block, xs))
with timeit('mapmt'):
    resoult_2 = mapmt(io_block, xs, pool=10)
    # pool=10 mean 10 threadings

resoult_1 == resoult_2
"map" spend time: 1.00472
"mapmt" spend time: 0.204875
Out[10]:
True

mapmp is Multi Process version of map

mapmp is the meaning of "MAP for Multi Process", has the same usage as map and mapmt but faster.

In [11]:
from boxx import mapmp, timeit
def bad_fibonacci(x): # simulation Complex calculations
    return x<=1 or x*bad_fibonacci(x-1)

xs = [800]*10000

if __name__ == '__main__':
    with timeit('map'):
        resoult_1 = list(map(bad_fibonacci, xs))

    with timeit('mapmp'):
        resoult_2 = mapmp(bad_fibonacci, xs)
    
    resoult_1 == resoult_2
    # the time printed below is run on a Intel i5 CPU on Ubuntu
"map" spend time: 2.870245
"mapmp" spend time: 1.291457

💡 Note:

mapmp and mapmt has same usage, they both support two parameters

pool : int, default None

the number of Process or Threading, the default is the number of CPUs in the system

printfreq : int or float, default None

the meaning of print frequent, auto print program progress in mapmt and mapmp
if printfreq < 1 then printfreq = len(iterables[0])*printfreq


In multi process programs, display processing progress is troublesome.
printfreq parameter in mapmp can handle this problem

In [ ]:
# try run this cell
from boxx import mapmp
from operator import add

xs = list(range(100))
double_xs = mapmp(add, xs, xs, pool=2, printfreq=.2)

double_xs

x_ to quick build function without lambda x:

In [12]:
from boxx import x_
f = x_**2

f(1), f(2), f(3)
Out[12]:
(1, 4, 9)

x_ often used with map, reduce, filter

In [ ]:
# try run this cell
xs = range(5)
powx = map(x_**x_, xs, xs)
list(powx)

mf to quick add magic method to function

mf is the meaning of "Magic Method", to wrap the function that often used while debugging.

In [13]:
from boxx import mf

l = mf(list)
tuplee = (5, 1, 7)
print('- :', l-tuplee)
print('* :', l*tuplee)
print('**:', l**tuplee)
print('/ :', l/tuplee)
- : [5, 1, 7]
* : [5, 1, 7]
**: [5, 1, 7]
/ : (5, 1, 7)

💡 Note:

  • when -, *, ** as magic method: do f(x) and return f(x)

  • when / as magic metho: do f(x) but return x

  • Functions that wraps by mf in boxx: stdout, log, logc, printt, pblue, pred, pdanger, perr, pinfo, typestr, getfathers, getfather, nextiter, mf, plot, show, showb, shows, loga, tree, treem, treea, dira, what, wtf, tprgb, torgb, normalizing, norma, npa, histEqualize, boolToIndex

tree to visualization complex struct in tree format

In [14]:
from boxx import tree

complex_struct = dict(key=[0, 'str', ('in_tuple', None)], tree=tree)

tree(complex_struct)
└── /: dict  2
    ├── key: list  3
    │   ├── 0: 0
    │   ├── 1: str
    │   └── 2: tuple 2
    │       ├── 0: in_tuple
    │       └── 1: None
    └── tree: FunAddMagicMethod(<function tree at 0x7f2c...

Like tree command in shell, boxx.tree could visualization any struct in tree format.

Support types include list, tuple, dict, numpy, torch.tensormxnet.ndarray, PIL.Image.etc

dira(x) to show x's all attribute

dira(x) is the meaning of "dir Attribute".

In [15]:
from boxx import dira

dira(LookupError)
Classes: 
└── Type of LookupError <-Exception <-BaseException <-object
Attrs: 
└── type: 31 attrs, Base class for lookup errors.
    ├── __cause__: <attribute '__cause__' of 'BaseException' objects>
    ├── __class__: <class 'type'>
    ├── __context__: <attribute '__context__' of 'BaseException' obj...
    ├── __delattr__: <slot wrapper '__delattr__' of 'BaseException' ...
    ├── __dict__: mappingproxy  3
    ├── __dir__: <method '__dir__' of 'object' objects>
    ├── __doc__: Base class for lookup errors.
    ├── __eq__: <slot wrapper '__eq__' of 'object' objects>
    ├── __format__: <method '__format__' of 'object' objects>
    ├── __ge__: <slot wrapper '__ge__' of 'object' objects>
    ├── __getattribute__: <slot wrapper '__getattribute__' of 'BaseExcept...
    ├── __gt__: <slot wrapper '__gt__' of 'object' objects>
    ├── __hash__: <slot wrapper '__hash__' of 'object' objects>
    ├── __init__: <slot wrapper '__init__' of 'LookupError' objects>
    ├── __init_subclass__: builtin-method : This method is called whe...
    ├── __le__: <slot wrapper '__le__' of 'object' objects>
    ├── __lt__: <slot wrapper '__lt__' of 'object' objects>
    ├── __ne__: <slot wrapper '__ne__' of 'object' objects>
    ├── __new__: builtin-method : Create and return a new o...
    ├── __reduce__: <method '__reduce__' of 'BaseException' objects>
    ├── __reduce_ex__: <method '__reduce_ex__' of 'object' objects>
    ├── __repr__: <slot wrapper '__repr__' of 'BaseException' obj...
    ├── __setattr__: <slot wrapper '__setattr__' of 'BaseException' ...
    ├── __setstate__: <method '__setstate__' of 'BaseException' objects>
    ├── __sizeof__: <method '__sizeof__' of 'object' objects>
    ├── __str__: <slot wrapper '__str__' of 'BaseException' obje...
    ├── __subclasshook__: builtin-method : Abstract classes can over...
    ├── __suppress_context__: <member '__suppress_context__' of 'BaseExceptio...
    ├── __traceback__: <attribute '__traceback__' of 'BaseException' o...
    ├── args: <attribute 'args' of 'BaseException' objects>
    └── with_traceback: <method 'with_traceback' of 'BaseException' obj...

dira(x) will pretty print x's all attribute in tree format.
And dira(x) will print x's Father Classes too.

what to know "What's this?"

In [16]:
from boxx import what

from boxx import ylsys
what(ylsys)
To Str: 
└── "<module 'boxx.ylsys' from '/home/yanglei/mygit/Box-X/boxx/ylsys.py'>"

Classes: 
└── Instance of module <-object

Document: 
└── 
     A module provide system info and Python Info for boxx
     
     @author: yanglei
     


Attrs: 
└── module: 31 attrs, A module provide system info and Python...
    ├── PythonInfo: <class 'boxx.ylsys.PythonInfo'>
    ├── SystemInfo: <class 'boxx.ylsys.SystemInfo'>
    ├── __TmpboxxWithCall: <class 'boxx.ylsys.__TmpboxxWithCall'>
    ├── __builtins__: 【builtins-dict 152 omitted】
    ├── __cached__: /home/yanglei/mygit/Box-X/boxx/__pycache__/ylsy...
    ├── __doc__: ↳A module provide system info and Python Info f...
    ├── __file__: /home/yanglei/mygit/Box-X/boxx/ylsys.py
    ├── __loader__: <_frozen_importlib_external.SourceFileLoader ob...
    ├── __module: <module 'multiprocessing' from '/home/yanglei/m...
    ├── __name__: boxx.ylsys
    ├── __package__: boxx
    ├── __spec__: ModuleSpec(name='boxx.ylsys', loader=<_frozen_i...
    ├── cloud: False
    ├── cpun: 4
    ├── cuda: False
    ├── environ: environ({'STY': '3620.pts-1.yanglei', 'TERM': '...
    ├── homeYl: /home/yanglei/
    ├── jupyterNotebookOrQtConsole: <function jupyterNotebookOrQtConsole at 0x7f2cc...
    ├── linuxYl: True
    ├── os: <module 'os' from '/home/yanglei/miniconda3/env...
    ├── osxYl: False
    ├── py2: False
    ├── py3: True
    ├── pyi: └── boxx.ylsys.PythonInfo: 12 att...
    ├── pyv: 3
    ├── sys: <module 'sys' (built-in)>
    ├── sysi: └── boxx.ylsys.SystemInfo: 12 att...
    ├── tmpYl: /tmp/
    ├── tmpboxx: /tmp/boxxTmp/
    ├── usecuda: auto
    └── winYl: False

----------end of what("<module 'boxx.ylsys' from '...")----------

what(x) will show "what is x?" by pretty print it's Document, Father Classes, Inner Struct and Attributes. It is a supplement of help(x)

💡 Note:

boxx.what is a useful tool when learn a new module or package.It reduce the time to check the API document.

wtf is the short of what, use wtf-x for convenience.

In [ ]:
# try run this cell
from collections import defaultdict
from boxx import wtf

ddict = defaultdict(lambda x:'boxx', Starman='Bowie')
wtf-ddict

logc to pretty print expression by show every variable's value in expression

logc is the meaning of "Log Code"

In [17]:
from random import random
from boxx import logc
a = random()
b = random()
logc("mean = (a + b) / 2", exe=True) # exe=True mean exec(code)
Code:  mean   = (   a    +    b   ) / 2
  └── 0.20618 = (0.11511 + 0.29724) / 2

heatmap to show the time heat map of your code

In [18]:
%matplotlib inline
from boxx import heatmap

heatmap('./yllab.py')

heatmap also support python code string.

In [19]:
%matplotlib inline
from boxx import heatmap

code = '''
def bad_fibonacci(x): # simulation Complex calculations
    if x<=1 :
        return 1
    return x*bad_fibonacci(x-1)
bad_fibonacci(3)
'''
heatmap(code)

performance could statistic function calls and visualize code performance

In [ ]:
from boxx import performance

performance('./yllab.py') 
# broswer will open a web page to visualization code perfomance if possible

💡 Note: if you are runing this Notebook on Binder, Browser won't open the web page. Please see demo here performance demo.gif


performance also support python code string.

In [ ]:
code = '''
def bad_fibonacci(x): # simulation Complex calculations
    if x<=1 :
        return 1
    return x*bad_fibonacci(x-1)
bad_fibonacci(5)
'''
performance(code) 
# broswer will open a web page to visualization code perfomance if possible

dicto is a convenient version of dict

dicto is the meaning of "dict that like Object"

In [20]:
from boxx import dicto
d = {'a':22}

dd = dicto(d)
print(dd.a)

dd.b = 517
dd
22
Out[20]:
{'a': 22, 'b': 517}

💡 Note: dicto is sub-class of dict that is easy to use, allows to get and set dict values as attributes.

BTW, boxx.cf is a dicto instance that could save your global config, and it could be used at all your .py files by from boxx import cf

ll is a convenient tool for list

ll is the meaning of "List tooL"

In [21]:
from boxx import ll

print(ll * 5) # instead of list(range(5))

print(ll/zip([0, 1])) # quick way to do `list(x)` when x iterable

ll # BTW, ll self is a list 
[0, 1, 2, 3, 4]
[(0,), (1,)]
Out[21]:
[0, 1]

sysi include many infomation about operating environment

In [22]:
from boxx import dira, sysi

dira(sysi, pattern='^[^_]')
Classes: 
└── Instance of boxx.ylsys.SystemInfo <-object
Attrs: 
Filter by pattern: "^[^_]"
└── boxx.ylsys.SystemInfo: 12 attrs, sys info
    ├── cpun: 4
    ├── cuda: False
    ├── display: :0
    ├── gui: True
    ├── host: yanglei
    ├── linux: True
    ├── os: linux
    ├── osx: False
    ├── pyv: 3
    ├── tmp: /tmp/
    ├── user: yanglei
    └── win: False

Use sysi.cpun, sysi.user, sysi.host to let code know wether the environment is local or remote.

2. Scientific Computing and Computer Vision Tool

The tools introduced in General Python Tool are also useful in Scientific Computing and Computer Vision(SC&CV) field.

In this section we will introduce tools that only uesed in SC&CV field.

BTW. Those tools support many array-like types include numpy, torch.tensormxnet.ndarray, PIL.Image.etc

loga for visualization matrix and tensor

loga is the meaning of "log array"

In [23]:
%matplotlib inline
import numpy as np
array = np.random.normal(size=(5,3, 244, 244))

from boxx import loga
loga(array)
shape:(5, 3, 244, 244) type:(float64 of numpy.ndarray) max: 4.9569, min: -4.4856, mean: -6.047e-4

💡 Note:

  • loga analysis the numpy.ndarray by it's shape, max, min, mean, and distribute.

  • loga support other array-like types include list, numpy, torch.tensor, mxnet.ndarray, PIL.Image.etc

  • loga will tell you how many nan, inf in the array if array include nan, inf:

In [24]:
array[...,:10] = np.inf
array[...,-10:] = -np.inf
array[...,:10,:] = np.nan
loga(array)
/home/yanglei/miniconda3/envs/boxx/lib/python3.6/site-packages/numpy/core/_methods.py:70: RuntimeWarning: invalid value encountered in reduce
  ret = umr_sum(arr, axis, dtype, out, keepdims)
shape:(5, 3, 244, 244) type:(float64 of numpy.ndarray) max: nan, min: nan, mean: nan

Notice: "nan":36600 (4.10%), "inf":70200 (7.86%),  finite max: 4.9569, finite min: -4.4856, finite mean: -9.357e-4

tree to visualization complex struct for Scientific Computing

In [25]:
# prepare images
import numpy as np
from skimage.io import imread
image_path = 'test/imgForTest/img.jpg'
ground_truth_path = 'test/imgForTest/gt_seg.png'
Lenna = imread('test/imgForTest/Lenna.jpg')

image = imread(image_path)
ground_truth = imread(ground_truth_path)

# complex struct
batch = dict(
    path=(image_path, ground_truth_path),
    img=image, 
    gt=ground_truth, 
    listt=[
        np.append(image, ground_truth[..., None], -1),
        np.array([Lenna, Lenna]),
    ],
)

from boxx import  tree
print('visualization the struct:')
tree(batch)
visualization the struct:
└── /: dict  4
    ├── path: tuple 2
    │   ├── 0: test/imgForTest/img.jpg
    │   └── 1: test/imgForTest/gt_seg.png
    ├── img: (300, 400, 3)uint8
    ├── gt: (300, 400)uint8
    └── listt: list  2
        ├── 0: (300, 400, 4)uint8
        └── 1: (2, 256, 256, 3)uint8

Like tree command in shell, boxx.tree could visualization complex struct (like a batch of data) in tree format.
💡 Note:

  • Support types include list, tuple, dict, numpy, torch.tensormxnet.ndarray, PIL.Image.etc

  • Support sample a batch from torch.Dataset, torch.DataLoader. then visualization the batch's struct.

show is easy to do imshow, even images are in complex struct

In [26]:
%matplotlib inline
from skimage.io import imread
Lenna = imread('test/imgForTest/Lenna.jpg')

from boxx import show
show(Lenna)

show could find every image in complex struct and plt.imshow they.

In [27]:
%matplotlib inline
# prepare images
import numpy as np
from skimage.io import imread
image_path = 'test/imgForTest/img.jpg'
ground_truth_path = 'test/imgForTest/gt_seg.png'
Lenna = imread('test/imgForTest/Lenna.jpg')

image = imread(image_path)
ground_truth = imread(ground_truth_path)

# complex struct
batch = dict(
    path=(image_path, ground_truth_path),
    img=image, 
    gt=ground_truth, 
    listt=[
        np.append(image, ground_truth[..., None], -1),
        np.array([Lenna, Lenna]),
    ],
)

from boxx import show, tree
print('the struct of batch:')
tree(batch)
print('show all images in batch:')
show(batch)
the struct of batch:
└── /: dict  4
    ├── path: tuple 2
    │   ├── 0: test/imgForTest/img.jpg
    │   └── 1: test/imgForTest/gt_seg.png
    ├── img: (300, 400, 3)uint8
    ├── gt: (300, 400)uint8
    └── listt: list  2
        ├── 0: (300, 400, 4)uint8
        └── 1: (2, 256, 256, 3)uint8
show all images in batch:

💡 Note:

  • Support image types include numpy, torch.tensormxnet.ndarray, PIL.Image.etc

  • And Support sample a batch from torch.Dataset, torch.DataLoader, then plt.imshow the batch.

npa transform other array-like object to numpy in one way

npa is the meaning of "numpy.array", use magic method to quick transform other numpy like object to numpy, suport torch.tensor, mxnet.ndarray, PIL.Image, list, tuple .etc

In [28]:
from boxx import npa

print(npa-range(3))

import numpy as np
r = npa-range(3)
npa**[r, r]
[0 1 2]
Out[28]:
array([[0, 1, 2],
       [0, 1, 2]])