Using functional programming in Python like a boss: Generators, Iterators and Decorators

Features of functions

  1. First-class functions are objects and thus:
    • Can be assigned to variables
    • Can be stored in data structures
    • Can be used as parameters
    • Can be used as a return value
  2. Higher order functions:
    • Accept a function as an argument and/or return a function as a value.
    • Create composite functions from simpler functions.
    • Modify the behavior of existing functions.
  3. Pure functions:
    • Do not depend on hidden state, or equivalently depend only on their input.
    • Evaluation of the function does not cause side effects
In [1]:
# A pure function
def my_min(x, y):
    if x < y:
        return x
    else:
        return y


# An impure function
# 1) Depends on global variable, 2) Changes its input
exponent = 2

def my_powers(L):
    for i in range(len(L)):
        L[i] = L[i]**exponent
    return L

What can act as a function in Python?

  1. A function object, created with the def statement.
  2. A lambda anonymous function, restricted to an expression in single line.
  3. An instance of a class implementing the __call__ function.
In [2]:
def min_def(x, y):
    return x if x < y else y

min_lambda = lambda x, y: x if x < y else y

class MinClass:
    def __call__(self, x, y):
        return x if x < y else y

min_class = MinClass()
print(min_def(2,3) == min_lambda(2, 3) == min_class(2,3))
True

Common gotchas 1: Mutable Default Arguments

  • Python's default arguments are evaluated once when the function is defined and not each time the function is called (like say, in Ruby).
  • If you use a mutable default argument and mutate it, you will and have mutated that object for all future calls to the function as well.
  • Sometimes you can specifically "exploit" (read: use as intended) this behavior to maintain state between calls of a function. This is often done when writing a caching function.
In [3]:
def append_to(element, to=[]):
    to.append(element)
    return to

my_list = append_to(12)
print("my_list:", my_list)
my_other_list = append_to(42)
print("my_other_list:", my_other_list)

def append_to2(element, to=None):
    if to is None:
        to = []
    to.append(element)
    return to

my_list2 = append_to2(12)
print("my_list2:", my_list2)
my_other_list2 = append_to2(42)
print("my_other_list2:", my_other_list2)
my_list: [12]
my_other_list: [12, 42]
my_list2: [12]
my_other_list2: [42]

Common gotchas 2: Late Binding Closures

  1. A closure occurs when a function has access to a local variable from an enclosing scope that has finished its execution.
  2. Python's closures are late binding.
  3. Values of variables used in closures are looked up at the time the inner function is called and not when it is defined.
In [4]:
def create_multipliers():
    multipliers = []

    for i in range(5):
        def multiplier(x):
            return i * x
        multipliers.append(multiplier)

    return multipliers

for multiplier in create_multipliers():
    print(multiplier(2))
8
8
8
8
8

Higher order functions and decorators

  • Python functions are objects.
  • Can be defined in functions.
  • Can be assigned to variables.
  • Can be used as function parameters or returned from functions.
  • Decorators are syntactic sugar for higher order functions.

In [5]:
# Higher order functions

def makebold(fn):
    def wrapped():
        return "<b>" + fn() + "</b>"
    return wrapped

def hello():
    return "hello world"

print(hello())
hello = makebold(hello)
print(hello())
hello world
<b>hello world</b>
In [6]:
# Decorated function with *args and **kewargs

def makebold(fn):
    def wrapped(*args, **kwargs):
        return "<b>" + fn(*args, **kwargs) + "</b>"
    return wrapped

@makebold  # hello = makebold(hello)
def hello(*args, **kwargs):
    return "Hello. args: {}, kwargs: {}".format(args, kwargs)

print(hello('world', 'pythess', where='soho'))
<b>Hello. args: ('world', 'pythess'), kwargs: {'where': 'soho'}</b>
In [7]:
# Decorators can be combined

def makeitalic(fn):
    def wrapped(*args, **kwargs):
        return "<i>" + fn(*args, **kwargs) + "</i>"
    return wrapped

def makebold(fn):
    def wrapped(*args, **kwargs):
        return "<b>" + fn(*args, **kwargs) + "</b>"
    return wrapped

@makeitalic
@makebold  # hello = makeitalic(makebold(hello))
def hello(*args, **kwargs):
    return "Hello. args: {}, kwargs: {}".format(args, kwargs)

print(hello('world', 'pythess', where='soho'))
<i><b>Hello. args: ('world', 'pythess'), kwargs: {'where': 'soho'}</b></i>
In [8]:
# Decorators can be instances of callable classes

class BoldMaker:
    def __init__(self, fn):
        self.fn = fn
    def __call__(self, *args, **kwargs):
        return "<b>" + self.fn(*args, **kwargs) + "</b>"

@BoldMaker  # hello = Bookmaker(hello)
def hello(*args, **kwargs):
    return "Hello. args: {}, kwargs: {}".format(args, kwargs)

# hello.__call__(*args, **kwargs)
print(hello('world', 'pythess', where='soho'))
<b>Hello. args: ('world', 'pythess'), kwargs: {'where': 'soho'}</b>
In [9]:
# Decorators can take arguments

def enclose_in_tags(opening_tag, closing_tag):  # returns a decorator
    def make_with_tags(fn):  # returns a decorated function
        def wrapped():  # the function to be decorated (modified)
            return opening_tag + fn() + closing_tag
        return wrapped
    return make_with_tags

# decorator function make_with_tags with the arguments in closure
heading_decorator = enclose_in_tags('<h1>', '</h1>')
paragraph_decorator = enclose_in_tags('<p>', '</p>')

def hello():
    return "hello world"

h1_hello = heading_decorator(hello)
p_hello = paragraph_decorator(hello)
h1_p_hello = heading_decorator(paragraph_decorator(hello))
print(h1_hello())
print(p_hello())
print(h1_p_hello())
<h1>hello world</h1>
<p>hello world</p>
<h1><p>hello world</p></h1>
In [10]:
# Decorators with arguments combined

def enclose_in_tags(opening_tag, closing_tag):
    def make_with_tags(fn):
        def wrapped():
            return opening_tag + fn() + closing_tag
        return wrapped
    return make_with_tags

# hello = enclose_in_tags('<h1>', '</h1>')(hello)
@enclose_in_tags('<h1>', '</h1>') 
def hello():
    return "hello world"

print(hello())

# hello = enclose_in_tags('<p>', '</p>')(hello)
@enclose_in_tags('<p>', '</p>')
def hello():
    return "hello world"

print(hello())

# hello = enclose_in_tags('<h1>', '</h1>')(enclose_in_tags('<p>', '</p>')(hello))
@enclose_in_tags('<h1>', '</h1>')
@enclose_in_tags('<p>', '</p>')
def hello():
    return "hello world"

print(hello())
<h1>hello world</h1>
<p>hello world</p>
<h1><p>hello world</p></h1>
In [11]:
# Decorators with arguments as instances of callable classes

class TagEncloser:
    def __init__(self, opening_tag, closing_tag):
        self.opening_tag = opening_tag
        self.closing_tag = closing_tag
    def __call__(self, fn):
        def wrapped():
            return self.opening_tag + fn() + self.closing_tag
        return wrapped

tag_h1 = TagEncloser('<h1>', '</h1>')
tag_p = TagEncloser('<p>', '</p>')

@tag_h1
@tag_p
def hello():  # hello = tag_h1(tag_p(hello))
    return "hello world"

print(hello())
<h1><p>hello world</p></h1>

Iterables and Iterators

  1. Iteration is a general term for taking each item of something, one after another. Any time you use a loop, explicit or implicit, to go over a group of items, that is iteration.
  2. An iterable is an anything that can be looped over. It either:
    • Has an __iter__ method which returns an iterator for that object when you call iter() on it, or implicitly in a for loop.
    • Defines a __getitem__ method that can take sequential indexes starting from zero (and raises an IndexError when the indexes are no longer valid).
  3. An iterator is:
    • A stateful helper object which defines a __next__ method and will produce the next value when you call next() on it. If there are no further items, it raises the StopIteration exception.
    • An object that is self-iterable (meaning that it has an __iter__ method that returns self).

Therefore: An iterable is an object from which we can get an iterator. An iterator is always an iterable. An iterable is not always an iterator but will always return an iterator.

In [12]:
x = [1, 2, 3]  # this is an iterable
y = iter(x)  # an iterator of this iterable
z = iter(x)  # another iterator of this iterable
print(next(y))
print(next(y))
print(next(z))
print(type(x))
print(type(y))
1
2
1
<class 'list'>
<class 'list_iterator'>
  • Here, x is the iterable, while y and z are two individual instances of an iterator, producing values from the iterable x. Both y and z hold state, as you can see from the example. In this example, x is a data structure (a list), but that is not a requirement.
  • Often, for pragmatic reasons, iterable classes will implement both __iter__() and __next__() in the same class, and have __iter__() return self, which makes the class both an iterable and its own iterator. It is perfectly fine to return a different object as the iterator, though.

When you write:

x = [1, 2, 3]
for elem in x:
    ...

This is what actually happens:

In [13]:
s = 'cat'  # s is an ITERABLE
           # s is a str object that is immutable
           # s has no state
           # s has a __getitem__() method 

t = iter(s)    # t is an ITERATOR
               # t has state (it starts by pointing at the "c")
               # t has a __next__() method and an __iter__() method
try:
    print(next(t))        # the next() function returns the next value and advances the state
    print(next(t))        # the next() function returns the next value and advances
    print(next(t))        # the next() function returns the next value and advances
    print(next(t))        # next() raises StopIteration to signal that iteration is complete
except StopIteration as e:
    print("StopIteration raised")
c
a
t
StopIteration raised
In [14]:
class FibonacciIterator:
    """
    Produces an arbitrary number of the Fibonacci numbers.
    Is an both an iterable and an iterator.
    """
    def __init__(self):
        self.prev = 0
        self.curr = 1
    
    def __iter__(self):
        return self
    
    def __next__(self):
        self.prev, self.curr = self.curr, self.prev + self.curr
        return self.prev

class Countdown:
    """A simple iterable, NOT an iterator"""
    def __iter__(self):
        return iter([5, 4, 3, 2, 1, 'launch'])
    
f = FibonacciIterator()
print([next(f) for _i in range(10)])
c = Countdown()
print([i for i in c])
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]
[5, 4, 3, 2, 1, 'launch']

Generators

A generator is a factory object that lazily produces values (i.e. generates values on demand). A generator is either:

  1. A function that contains the yield keyword (yield expression).
    • When this function is called, it does not execute, but returns a generator object.
    • The first time that next() is called, when this function encounters the yield keyword it suspends execution at that point, saves its context and returns to the caller along with any value in expression.
    • When the caller invokes next() again, execution of the function continues till another yield or return is encountered or end of function is reached.
  2. A generator expression, i.e. a syntactic construct for creating an anonymous generator object, following the form of mathematical set-builder notation. These are like list comprehensions but enclosed in () instead of [].
    • The semantics of a generator expression are equivalent to creating an anonymous generator function and calling it.

A generator is always an iterator, but not vice versa (iterator is a more general concept).

In [15]:
# generator expression
g = (x ** 2 for x in range(10))
print(next(g))
print(next(g))
print([i for i in g])

# generator function
def _gen(exp):
    for x in exp:
        yield x ** 2
g = _gen(range(10))
print(next(g))
print(next(g))
print([i for i in g])
0
1
[4, 9, 16, 25, 36, 49, 64, 81]
0
1
[4, 9, 16, 25, 36, 49, 64, 81]
In [16]:
# generator
def it_gen(text):
    for ch in text:
        yield ch.upper()

# generator expression
def it_genexp(text):
    return (ch.upper() for ch in text)

# iterator protocol (__iter__)
class ItIter():
    def __init__(self, text):
        self.text = text
        self.index = 0
    def __iter__(self):
        return self
    def __next__(self):
        try:
            result = self.text[self.index].upper()
        except IndexError:
            raise StopIteration
        self.index += 1
        return result

# iterator protocol (__getitem__)
class ItGetItem():
    def __init__(self, text):
        self.text = text
    def __getitem__(self, index):
        result = self.text[index].upper()
        return result

# an iterable of iterables (see what i did there? :P)
for iterator in (it_gen, it_genexp, ItIter, ItGetItem):
    for ch in iterator('abcde'):
        print(ch, end='')
    print('\n')
ABCDE

ABCDE

ABCDE

ABCDE

In [17]:
import collections

def flatten(container):
    """Flatten an iterable of arbitrary depth."""
    for item in container:
        if isinstance(item, collections.Iterable) and not isinstance(item, (str, bytes)):
            for element in flatten(item):
                yield element
        else:
            yield item

d3 = [[[i for i in range(2)] for j in range(3)] for k in range(4)]
print(d3)
print([k for i in d3 for j in i for k in j])
print([i for i in flatten(d3)])
[[[0, 1], [0, 1], [0, 1]], [[0, 1], [0, 1], [0, 1]], [[0, 1], [0, 1], [0, 1]], [[0, 1], [0, 1], [0, 1]]]
[0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1]
[0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1]

More cool stuff

  • Partial functions, currying
  • Coroutines, asyncio, yield from
  • functools, itertools (standard library)
  • toolz (cool functional library)