Effective Python - 59 Specific Ways to Write Better Python.

Chapter 3 - Classes and Inheritance

Book by Brett Slatkin. Summary notes by Tyler Banks.

22: Prefer Helper Classes Over Bookkeping with Dictionaries and Tuples

  • Avoid making dictionaries with values that are other dictionaries or long tuples
  • Use namedtuple for lightweight, immutable data containers before classes
  • Move bookkeeping code to use multiple helper classes when internal state dictionaries get complicated
In [3]:
#Named Tuple example
import collections
Grade = collections.namedtuple('Grade', ('score', 'weight'))


  • You can't specify default arguments for namedtuple classes.
  • The attribute values of namedtuple are accessible using indexes and iteration
  • If you're not in control of the usage of your namedtuple instances, use your own classes.

Item 23: Accept Functions for Simple Interfaces Instead of Classes

  • You can pass function definitions as arguments to be called
  • Example: list type's sort method takes optional key argument
In [5]:
names = ['Socrates', 'Archimedes', 'Plato', 'Aristotle']
names.sort(key=lambda x: -len(x))
['Archimedes', 'Aristotle', 'Socrates', 'Plato']
  • The __call__ method allows for classes to be called just like functions
  • It also allows for the callable built-in function to return True for the instance
  • Ex: (default dict allows for a function to be called when an item is added)
In [15]:
from collections import defaultdict
class BetterCountMissing(object):
    def __init__(self):
        self.added = 0
    def __call__(self):
        self.added += 1
        return 0

counter = BetterCountMissing()
assert callable(counter)
In [16]:
counter = BetterCountMissing()
current = {'greed': 12, 'blau': 5}
increments = [
    ('red', 2),
    ('greed', 23)
result = defaultdict(counter, current) # Relies on __call__
for key, amount in increments:
    result[key] += amount
assert counter.added == 1

Item 24: Use @classmethod Polymorphism to Construct Objects Generically

To see how we can bind classes together more cohesively, see the following example:

In [19]:
class GenericInputData(object):
    def read(self):
        raise NotImplementedError
    def generate_inputs(cls, config):
        raise NotImplementedError

class PathInputData(GenericInputData):
    def __init__(self, path):
        self.path = path
    def read(self):
        return open(self.path).read()
    def generate_inputs(cls, config):
        data_dir = config['data_dir']
        for name in os.listdir(data_dir):
            yield cls(os.path.join(data_dir, name))

class GenericWorker(object):
    # …
    def map(self):
        raise NotImplementedError
    def reduce(self, other):
        raise NotImplementedError
    def create_workers(cls, input_class, config):
        workers = []
        for input_data in input_class.generate_inputs(config):
        return workers
def mapreduce(worker_class, input_class, config):
    workers = worker_class.create_workers(input_class, config)
    return execute(workers)

Item 25: Initialize Parent Classes with super

  • Old way of initializing a parent class from a child class was to call parent's __init__
  • Ex:
In [20]:
class MyBaseClass(object):
    def __init__(self, value):
        self.value = value
class MyChildClass(MyBaseClass):
    def __init__(self):
        MyBaseClass.__init__(self, 5)
  • If you were to have classes with multiple inheritance you could end up with odd results
In [21]:
class TimesTwo(object):
    def __init__(self):
        self.value *= 2
class PlusFive(object):
    def __init__(self):
        self.value += 5
class OneWay(MyBaseClass, TimesTwo, PlusFive):
    def __init__(self, value):
        MyBaseClass.__init__(self, value)
foo = OneWay(5)
print('First ordering is (5 * 2) + 5 =', foo.value)

class AnotherWay(MyBaseClass, PlusFive, TimesTwo):
    def __init__(self, value):
        MyBaseClass.__init__(self, value)

bar = AnotherWay(5)
print('Second ordering still is', bar.value)
First ordering is (5 * 2) + 5 = 15
Second ordering still is 15

Diamond inherticance happens when two parent classes have the same parent

In [22]:
class TimesFive(MyBaseClass):
    def __init__(self, value):
        MyBaseClass.__init__(self, value)
        self.value *= 5
class PlusTwo(MyBaseClass):
    def __init__(self, value):
        MyBaseClass.__init__(self, value)
        self.value += 2
class ThisWay(TimesFive, PlusTwo):
    def __init__(self, value):
        TimesFive.__init__(self, value)
        PlusTwo.__init__(self, value)
foo = ThisWay(5)
print('Should be (5 * 5) + 2 = 27 but is', foo.value)
Should be (5 * 5) + 2 = 27 but is 7

Because PlusTwo.__init__ causes self.value to be reset back to 5 when MyBaseClass.__init__ gets called a second time.

  • This is solved with the super built in keyword, which defines the method resolution order
In [23]:
class Explicit(MyBaseClass):
    def __init__(self, value):
        super(__class__, self).__init__(value * 2)
class Implicit(MyBaseClass):
    def __init__(self, value):
        super().__init__(value * 2)
assert Explicit(10).value == Implicit(10).value
In [24]:
#Use MRO method to see resolution order
class TimesFiveCorrect(MyBaseClass):
    def __init__(self, value):
        super(TimesFiveCorrect, self).__init__(value)
        self.value *= 5
class PlusTwoCorrect(MyBaseClass):
    def __init__(self, value):
        super(PlusTwoCorrect, self).__init__(value)
        self.value += 2
class GoodWay(TimesFiveCorrect, PlusTwoCorrect):
    def __init__(self, value):
        super(GoodWay, self).__init__(value)
foo = GoodWay(5)
print('Should be 5 * (5 + 2) = 35 and is ', foo.value)
Should be 5 * (5 + 2) = 35 and is  35
In [25]:
from pprint import pprint
[<class '__main__.GoodWay'>,
 <class '__main__.TimesFiveCorrect'>,
 <class '__main__.PlusTwoCorrect'>,
 <class '__main__.MyBaseClass'>,
 <class 'object'>]

Item 26: Use Multiple Inheritance Only for Mix-in Utility Classes

  • A mix-in is a small class that only defines a set of additional methods that a class should provide.
  • Mix-in classes don't define their own instance attributes or require __init__ to be called.
  • Following example is a generic mix-in to convert a Python object from in-memory representation to a dictionary ready for serialization.
In [3]:
class ToDictMixin(object):
    def to_dict(self):
        return self._traverse_dict(self.__dict__)
    def _traverse_dict(self, instance_dict):
        output = {}
        for key, value in instance_dict.items():
            output[key] = self._traverse(key, value)
        return output
    def _traverse(self, key, value):
        if isinstance(value, ToDictMixin):
            return value.to_dict()
        elif isinstance(value, dict):
            return self._traverse_dict(value)
        elif isinstance(value, list):
            return [self._traverse(key, i) for i in value]
        elif hasattr(value, '__dict__'):
            return self._traverse_dict(value.__dict__)
            return value

Following uses dynamic attribute access using hasattr, dynamic type inspection with isinstance and accessing the instance dictionary __dict__.

In [4]:
class BinaryTree(ToDictMixin):
    def __init__(self, value, left=None, right=None):
        self.value = value
        self.left = left
        self.right = right
In [5]:
tree = BinaryTree(10,
    left=BinaryTree(7, right=BinaryTree(9)),
    right=BinaryTree(13, left=BinaryTree(11)))
{'value': 10, 'left': {'value': 7, 'left': None, 'right': {'value': 9, 'left': None, 'right': None}}, 'right': {'value': 13, 'left': {'value': 11, 'left': None, 'right': None}, 'right': None}}
  • Avoid multiple inheritance if mix-ins can achive same outcome
  • Use pluggable behaviors at instance level to provide per-class customization
  • Compose mix-ins to create complex function from simple behavior

Item 27: Prefer Public Attributes Over Private Ones

  • Python has public and private fields
  • private fields can be set by prefacing variable names with a double underscore (__)
In [6]:
class MyObject(object):
    def __init__(self):
        self.public_field = 5
        self.__private_field = 10
    def get_private_field(self):
        return self.__private_field
In [7]:
foo = MyObject()
assert foo.public_field == 5
assert foo.get_private_field() == 10
AttributeError                            Traceback (most recent call last)
<ipython-input-7-1fc23a9aeac8> in <module>()
      2 assert foo.public_field == 5
      3 assert foo.get_private_field() == 10
----> 4 foo.__private_field

AttributeError: 'MyObject' object has no attribute '__private_field'
  • Class methods can access private fields
In [9]:
class MyOtherObject(object):
    def __init__(self):
        self.__private_field = 71
    def get_private_field_of_instance(cls, instance):
        return instance.__private_field
bar = MyOtherObject()
assert MyOtherObject.get_private_field_of_instance(bar) == 71
  • Subclasses can't access its parent class's private fields
In [10]:
class MyParentObject(object):
    def __init__(self):
        self.__private_field = 71
class MyChildObject(MyParentObject):
    def get_private_field(self):
        return self.__private_field
baz = MyChildObject()
AttributeError                            Traceback (most recent call last)
<ipython-input-10-38fbdb394643> in <module>()
      6         return self.__private_field
      7 baz = MyChildObject()
----> 8 baz.get_private_field()

<ipython-input-10-38fbdb394643> in get_private_field(self)
      4 class MyChildObject(MyParentObject):
      5     def get_private_field(self):
----> 6         return self.__private_field
      7 baz = MyChildObject()
      8 baz.get_private_field()

AttributeError: 'MyChildObject' object has no attribute '_MyChildObject__private_field'
  • To minimize damage PEP 8 style indicates that any variable prefaced with a single underscore(_) should be accessed with caution.
  • Private attributes are NOT rigerously enforced by the Python compiler. Can access with _classname__privatevar
  • Use docs for protected fields to guide subclasses

Item 28: Inherit from collections.abc for Custom Container Types

  • Most python classes contain data and describe how objects held within relate to one another.
  • Extend built in collection types to retain functionality
  • Ex:
In [11]:
class FrequencyList(list):
    def __init__(self, members):
    def frequency(self):
        counts = {}
        for item in self:
            counts.setdefault(item, 0)
            counts[item] += 1
        return counts
  • Implement methods like __getitem__ to take advantage of list-like features
In [19]:
class BinaryNode(object):
    def __init__(self, value, left=None, right=None):
        self.value = value
        self.left = left
        self.right = right

class IndexableNode(BinaryNode):
    def _search(self, count, index):
        # …
        # Returns (found, count)
    def __getitem__(self, index):
        found, _ = self._search(0, index)
        if not found:
            raise IndexError('Index out of range')
        return found.value
tree = IndexableNode(
            6, right=IndexableNode(7))),
        15, left=IndexableNode(11)))

print('LRR =', tree.left.right.right.value)
#print('Index 0 =', tree[0])
#print('Index 1 =', tree[1])
#print('11 in the tree?', 11 in tree)
#print('17 in the tree?', 17 in tree)
#print('Tree is', list(tree))
LRR = 7
  • Watch out for large number of methods you need to implement to create a custom container correctly
In [ ]: