This notebook/presentation has been prepared for the 2017 edition of http://python.g-node.org, the renowned Advanced Scientific Programming in Python summer school (Happy 10th Anniversary!!). I gratefully acknowledge the efforts of the entire Python community, which produced great documentation I largely consumed to create this notebook; a list of which can be found at the end of the notebook. If I have missed anyone, apologies, let me know and I'll add you to the list!

Although you should be able to run the notebook straight out of the box, bear in mind that it was designed to work with Python3, in conjunction with the following nbextensions:

The repository also contains exercises, with and without solutions, which I borrowed from last year's edition of the summer school.

I hope you enjoy it! By all means get in touch! :)

Etienne

In [1]:
import sys
print('Python version ' + sys.version)

import time
from IPython.display import display, Image
from IPython.core.display import HTML

def countdown(t, display_picture=False):
"""Displays countdown.

Keyword arguments:
t -- the amount of time to countdown in seconds
"""
while t:
mins, secs = divmod(t, 60)
timeformat = '{:02d}:{:02d} left'.format(mins, secs)
print(timeformat, end='\r')
time.sleep(1)
t -= 1
print('Hands off of keyboards now!')
if display_picture:
display(Image(filename="./picts/aspp2017.png", width=400))

Python version 3.6.1 |Anaconda 4.4.0 (x86_64)| (default, May 11 2017, 13:04:09)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]


# Iterators, generators, decorators, and context managers¶

### .. and all these little things you have used without really knowing what is going on¶

Etienne B. Roesch | University of Reading

http://etienneroes.ch

## I ...¶

• am/was an old fashioned software engineer
• am a cognitive scientist, and passionate interdisciplinarian
• visual perception (psychophysics) and experience (attention, emotion)
• methods for neuroimaging: coupled EEG-fMRI, ERG
• modelling at various scales: cells, brain areas, networks of areas
• increasingly into open science practices
• increasingly bigger data person (soon Google Cloud Platform certified)

## Take-home message¶

• Iterators are arcane mechanisms that support loops, and everything else;

• Generators are kinds of iterators that provide a level of optimisation and interactivity;

• Decorators are a mechanism to incrementally power-up existing code;

• Context managers are semantically related to decorators, to manage resources properly.

## Iterators¶

An iterator is any Python type that can be used with a for loop.

They implement the iterator protocol, which describes implicit methods, like __init__(), to iterate in sets of objects. In Python 3, you find them everywhere, e.g. files, i/o streams, etc.

https://docs.python.org/3.6/whatsnew/2.2.html#pep-234-iterators

In [2]:
import numpy as np
nums = np.arange(2)    # ndarray contains [0, 1]

In [3]:
for x in nums:
print(x, end=" ")

0 1
In [4]:
iter(nums)             # ndarray is an iterable

Out[4]:
<iterator at 0x1040d00b8>
In [5]:
it = iter(nums)
it.__next__()          # One way to iterate

Out[5]:
0
In [6]:
next(it)    # Another way to iterate

Out[6]:
1
In [7]:
next(it)    # Raises StopIteration exception

---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
----> 1 next(it)    # Raises StopIteration exception

StopIteration: 

http://www.scipy-lectures.org/intro/language/exceptions.html#exceptions

Leonardo Filius Bonacci (1175-1245), aka Leonardo Fibonacci, defines the recurrence relation that now bears his name and fuels conspiracy theorists.

$F_{n} = F_{n-1} + F_{n-2}$ given $F_{0} = 0, F_{1} = 1$
![noimg](picts/FibonacciSpiral.png)
In [8]:
class Fib:
'''Iterator Class to calculate the Fibonacci series'''

def __init__(self, max):
self.max = max

def __iter__(self):    # defines initial conditions
self.a = 0
self.b = 1
return self        # returns a handle to the object

def __next__(self):    # defines behaviour for next()
fib = self.a
if fib > self.max:
raise StopIteration # is caught when in _for_ loop
temp_b = self.a + self.b
self.a = self.b
self.b = temp_b
return fib         # F_n = F_n-1 + F_n+2

# 33rd degree in Freemason Antient & Accepted Scottish Rite
for i in Fib(33):
print(i, end=' ')   # literally calls the __next__() method

0 1 1 2 3 5 8 13 21

## Generators¶

Generators (generator-iterators as they are called) is a mechanism to simplify this process.

Python provides the yield keyword to define generators, which takes care of __iter__() and __next__() for you.

https://www.python.org/dev/peps/pep-0255/

In [9]:
def fib_without_iterator_protocol(max):
numbers = []          # Needs to return an array of values
a, b = 0, 1           # a = 0  and  b = 1
while a < max:
numbers.append(a)
a, b = b, a + b   # Evalute right-hand side first and assign
return numbers        # Returns full list of numbers

for i in fib_without_iterator_protocol(33):
print(i, end=" ")     # iterates through array of values

0 1 1 2 3 5 8 13 21

In real life problems, this way of doing things is problematic because it forces us to compute all numbers in turn and to store everything in one go.

yield expression_list

yield does something similar to return:

• return gives back control to the caller function, and returns some content;
• yield freezes execution temporarily, stores current context, and returns some content to .__next__()'s caller;

yield saves local state and variables, instruction pointer and internal evaluation stack; i.e. enough information so that .__next__() behaves like an external call.

In [10]:
def fib_with_yield(max_limit):
'''fib function using yield'''
a, b = 0, 1          # a = 0  and  b = 1
while a < max_limit:
yield a          # freezes execution, returns current a
a, b = b, a + b  # a = b  and  b = a + b

for i in fib_with_yield(33):
print(i, end=" ")

0 1 1 2 3 5 8 13 21
In [11]:
my_masonic_secret = fib_with_yield(33)
my_masonic_secret

Out[11]:
<generator object fib_with_yield at 0x104157620>
In [12]:
next(my_masonic_secret)

Out[12]:
0
In [13]:
next(my_masonic_secret)

Out[13]:
1
In [14]:
next(my_masonic_secret)

Out[14]:
1
In [15]:
next(my_masonic_secret)

Out[15]:
2

... and so on.

## Exercise¶

Write a function that uses yield to draw numbers for the lottery--with replacement is fine! That's six numbers between 1-40; and if you feel ambitious, add one number between 1-15.

In [16]:
countdown(1)

Hands off of keyboards now!


## Solution¶

In [17]:
import random

def super_million_lottery():
# returns 6 numbers between 1 and 40
for i in range(6):
yield random.randint(1, 40)

# returns a 7th number between 1 and 15
yield random.randint(1,15)

for i in super_million_lottery():
print(i, end=" ")

4 18 9 35 33 16 3

Python's list comprehension, with [..], computes everything at once and can take a lot of memory.

In [18]:
squares = [i**2 for i in range(10)]
squares

Out[18]:
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Generator expressions, with (..), are computed on demand.

In [19]:
squares = (i**2 for i in range(10))
squares

Out[19]:
<generator object <genexpr> at 0x1041575c8>

https://www.python.org/dev/peps/pep-0289/

On-demand calculation is important for the streamed processing of big amount of data; where the size of the data is uncertain, values of parameters are changing, etc, or when the processing steps might take a long time, yields errors or enter infinite loops.

Generators are also an easier way to handle callbacks, and can be used to simulate concurrency.

https://www.python.org/dev/peps/pep-0342/

Bash pipeline to count the number of characters, omitting whitespaces, per line, in a given file:

In [20]: