Performance analysis in Python

Timing

The timeit module provides a simple way to time small bits of Python code. It has both a Command-Line Interface as well as a callable one. It avoids a number of common traps for measuring execution times. It is also available as an IPython magic command: %timeit.

In [1]:
%%file primes.py

def init_s(n):
    return range(3, n + 1, 2)

def calc_j(m):
    return (m * m - 3) / 2

def get_primes(n):
    if n == 2:
        return [2]
    elif n < 2:
        return []
    s = init_s(n)
    mroot = n ** 0.5
    half = (n + 1) / 2 - 1
    i = 0
    m = 3
    while m <= mroot:
        if s[i]:
            j = calc_j(m)
            s[j] = 0
            while j < half:
                s[j] = 0
                j += m
        i += 1
        m = 2 * i + 3
    return [2] + [x for x in s if x]

if __name__ == "__main__":
    get_primes(1000000)
Overwriting primes.py
In [2]:
from primes import get_primes
%timeit -n 5 -r 2 get_primes(1000000)
5 loops, best of 2: 208 ms per loop

Tracing program execution

The trace module allows you to trace program execution, generate annotated statement coverage listings, print caller/callee relationships and list functions executed during a program run. It can be used in another program or from the command line.

In [3]:
!python -m trace -T primes.py # the -T switch displays the calling relationships exposed by running the program.
calling relationships:

*** /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/trace.py ***
    trace.Trace.runctx -> trace._unsettrace
  --> primes.py
    trace.Trace.runctx -> primes.<module>

*** primes.py ***
    primes.<module> -> primes.get_primes
    primes.get_primes -> primes.calc_j
    primes.get_primes -> primes.init_s

Call graph visualizations

The pycallgraph module creates call graph visualizations for Python applications.

In [4]:
!pycallgraph graphviz --output-file=primes.png -- primes.py
In [5]:
from IPython.display import Image
Image(filename="primes.png")
Out[5]:

Built-in Profiler

%prun runs code using the built-in Python profiler, the cProfile module.

In [9]:
%prun get_primes(1000000)
 

Line Profiling

In [10]:
%load_ext line_profiler
The line_profiler extension is already loaded. To reload it, use:
  %reload_ext line_profiler
In [11]:
%lprun -f get_primes get_primes(10000) # interacts badly with the notebook

Profiling memory usage

In [12]:
%load_ext memory_profiler
In [15]:
%memit get_primes(1000000)
peak memory: 43.50 MiB, increment: 2.39 MiB
In [16]:
%mprun -f get_primes get_primes(100000)
('',)