Notebook
A typical function definition looks like the below. In this case, we return two values as a tuple. We use this function by calling it with (). In Python, functions are generalised as "callables" and support the __call__ protocol. Many different things are callables (classes, C-functions, generators, &c.)
A typical generator definition looks like the below. We swap out the `return` statement with a `yield statement` and provide two values. We use this generator by instantiating it with () then iterating over it. In Python, generators are generalised as "iterables" and support the __iter__/__next__ protocols. Many different things are iterable (lists, tuples, dictionaries, generators, &c.)
We may note that a generator defined with `yield`-syntax appears to support both the __call__ and __iter__/__next__ protocols. In this case, we __call__ the generator to create an instance, then we iterate over the instance. A generator expression is one way we can construct a single instance inline.
A generator defined with `yield`-syntax is not strictly the same as, but can be considered equivalent with a generator expression that is constructed dynamically behind a lambda.
Callables are the Python generalisation of functions. In Python a callable is just some object that supports the __call__ protocol. __call__ corresponds to () or apply()
What if __call__ is a staticmethod?
Iterables are the Python generalisation of lists, tuples, &c. In Python an iterable is just some object that supports the __iter__ protocol and returns an object which supports the __next__ protocol. __iter__ corresponds to iter() and __next__ corresponds to next() Note that in Python 2, the __next__ protocol is provided by next. This was fixed in Python 3 (which we should all be using.)
Note that, in the above, we have an instance member called `self.state` that tracks the state. An iterator is anything that can be iterated over. An iterable is merely some object that tracks the state of the iteration.
A generator is merely a much lower-level expression of the above. Instead of explicitly tracking the state with some instance member, the state is tracked by a frame object which tracks the last line of code executed.
The generator below can `yield` values. This is in correspondence to how a function can `return` values.
We can also send values into a generator at any point during the iteration. This is what makes a generator a coroutine. In fact, generators support a rich protocol including: .next: get the next value .throw: raise an exception .send: send a value in .close: terminate iteration
next(g) is the same as g.next() These are the same as g.send(None) or g.send() g.send() sends a value back into the generator at the point of `yield`
One annoyance of generators is that we retrieve a value sent into the generator on the left hand side of a `yield`. This means that we must `yield` a value before we can accept a value back in. As a result, we have the following problem.
Itertools is an extremely useful collection of utilities in the standard library. It contains very efficient generalisations, helpers, and algorithms that allow us to work very effectively with generators. The contents of itertools are themselves not generators; they are written in C and are mostly standard Python C-functions and C-objects.
One reason to use generators is that they can efficiently represent certain constructs.
For algorithms that require the use of only a subset of the entire return value, generators can prove far more memory efficient than their functional equivalents.
Note that this can be written very tersely and very generically by combining helpers from `itertools`
Generators tend to be fairly efficient in practice: <performance/bythrees> vs <performance/bythrees.py>
Generators are opaque building-blocks, so they can be swapped out very easily with more efficient representations: <nwise_main.py>
Someone asked me about numpy arrays versus generators. Numpy arrays are both efficient in practice and also very convenient to use. While generators can be very efficient and convenient, their real draw is in the ability to better model problems.
Generators allow us to model programmes with a stream processing or pipeline conceptualisation.
The showcase is one of the main attractions of `The Price is Right.` 70+ games. Drew Carey has lost a lot of weight. Who remembers Bob Barker? The `Price is Right` is a global phenomenon: 購物街 (高博) Showcase is the last game; spin the wheel.
Lets add our own pseudo-random number generator.
How can we better simulate the wheel?
Can we simulate this more simply?
Notice that we have more control over the wheel.
Notice how easy it is to test.
We can further modularise this.
All together...