The decorators they won't tell you about

You know, decorators are one of the features that makes Python great, but there are a lot of people who either haven't been exposed to them or - worse - have been exposed to them (either in the wild or as part of a tutorial) but never "gotten" them.

If you fit in the latter category, you'll almost necessarily have had spouted reassuringly at you: "Decorators are simple, they're just functions that take functions and return other functions!" You'll have seen a blog post that taught you how to make decorators that add one to a function's output, or print when it's called, or implement caching around it - as if these were insurmountable problems that only a decorator could solve. You'll have extensively used flask's @app.route, but have tried very carefully not to wonder too hard about it, for fear of its becoming disobedient and surly.

In short, you'll know of decorators, but not know what they're good for, or why they are the way they are. Practice makes perfect, but abstention breeds apathy; like someone who has been taught the violin exclusively as an instrument of classical music, and who couldn't care less about classical music, you are unlikely to become a virtuoso without discovering the fiddle.

This brief guide aims to dispell the myths you've heard about decorators and instead show you the good stuff, the decorators they won't tell you about, the decorators that do things you didn't know decorators could do.

Your mandatory refresher

State law requires that, before we go any further, I remind you of the basic principles of decorators.

Functions in Python

The first thing you must understand is that all functions in Python are first-class citizens, which is to say, are objects just like any other object you might encounter.

They have attributes.

In [1]:
def f():
    """Print something to standard out."""
    print('something')

print(dir(f))
['__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__doc__', '__format__', '__get__', '__getattribute__', '__globals__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']

You can assign them to variables.

In [2]:
g = f
g()
something

You can use them as arguments to other functions.

In [3]:
def func_name(function):
    return function.__name__

func_name(f)
Out[3]:
'f'

You can throw them into data structures.

In [4]:
function_collection = [f, g]
for function in function_collection:
    function()
something
something

The possibilities are endless, inasmuch as they are endless with any other object.

Decorators

Decorators are often described as "functions which take functions and return functions", a description which is notable in that, technically speaking, not a single word of it is true. What is true is the following:

  • Decorators are applied once, at function definition time.
  • Annotating a function definition x with a decorator @d is equivalent to defining x, then, immediately afterward, having x = d(x).
  • Decorating a function with @d and @e, in that order, is equivalent to performing x = d(e(x)) after the function's definition.

The second of these principles is demonstrated here:

In [5]:
def print_when_called(function):
    def new_function(*args, **kwargs):
        print("{} was called".format(function.__name__))
        return function(*args, **kwargs)
    return new_function

def one():
    return 1
one = print_when_called(one)

@print_when_called
def one_():
    return 1

[one(), one_(), one(), one_()]
one was called
one_ was called
one was called
one_ was called
Out[5]:
[1, 1, 1, 1]

One immediate point of clarification: although I just said decorators are applied at function definition time, you might notice that messages in the example above are being printed at function call time. This is because print_when_called returns new_function, which itself prints before calling one or one_. As I said earler, decorators themselves are applied at function definition time:

In [6]:
def print_when_applied(function):
    print("print_when_applied was applied to {}".format(function.__name__))
    return function

@print_when_applied
def never_called():
    import os
    os.system('rm -rf /')
print_when_applied was applied to never_called

never_called is, aptly, never called, but the message from the print_when_applied decorator is printed regardless.

Finally, a demonstration of decorator ordering:

In [7]:
@print_when_applied
@print_when_called
def this_name_will_be_printed_when_called_but_not_at_definition_time():
    pass

this_name_will_be_printed_when_called_but_not_at_definition_time()
print_when_applied was applied to new_function
this_name_will_be_printed_when_called_but_not_at_definition_time was called

print_when_called returns a function named new_function, and it is this function that print_when_applied is called upon.

Decorator myths

To really understand the potential of decorators, it is not sufficient to learn what you do not yet know - you must also unlearn what you know already.

Myth the first - decorators return functions

Earlier, I claimed that applying a decorator d to a function x was the same as writing the definition of x, then x = d(x).

But you might have noticed - who says that d has to return a function? In fact, func_name (which I defined earlier) returns a string, but works just fine as a decorator.

In [8]:
@func_name
def a_named_function():
    return

a_named_function
Out[8]:
'a_named_function'

What is important about this? It means that functions can be more than callable objects, but instead can be little isolated scopes for doing... whatever you'd like. For instance, if you want to process a list, you could do:

In [9]:
def process_list(list_):
    def decorator(function):
        return function(list_)
    return decorator

unprocessed_list = [0, 1, 2, 3]
special_var = "don't touch me please"

@process_list(unprocessed_list)
def processed_list(items):
    special_var = 1
    return [item for item in items if item > special_var]

(processed_list, special_var)
Out[9]:
([2, 3], "don't touch me please")

processed_list is a list, and special_var has remained unchanged thanks to Python's function scoping rules. This example is a bit silly - there are much more sensible ways to do the same thing - but the principle is very useful. A more conventional exploitation of it might look like:

In [10]:
class FunctionHolder(object):
    def __init__(self, function):
        self.func = function
        self.called_count = 0
    def __call__(self, *args, **kwargs):
        try:
            return self.func(*args, **kwargs)
        finally:
            self.called_count += 1
            
def held(function):
    return FunctionHolder(function)

@held
def i_am_counted():
    pass

i_am_counted()
i_am_counted()
i_am_counted()
i_am_counted.called_count
Out[10]:
3

Myth the second - decorators are functions

Nothing in x = d(x) necessitates that d is a function - d just has to be callable! The above example can just as well be written:

In [11]:
@FunctionHolder
def i_am_also_counted(val):
    print(val)
    
i_am_also_counted('a')
i_am_also_counted('b')
i_am_also_counted.called_count
a
b
Out[11]:
2

In fact, i_am_also_counted, which is a FunctionHolder instance, not a function, can also be used as a decorator:

In [12]:
@i_am_also_counted
def about_to_be_printed():
    pass

i_am_also_counted.called_count
<function about_to_be_printed at 0x7f73f678aaa0>
Out[12]:
3

Myth the third - decorators take functions

Granted - Python's syntax means that the @decorator notation can't be used just anywhere, but that doesn't mean they exclusively take functions as arguments. For instance, here's len, which operates on sequences (not functions), being used as a decorator.

In [13]:
@len
@func_name
def nineteen_characters():
    """are in this function's name"""
    pass

nineteen_characters
Out[13]:
19

You can, in fact, apply essentially any function you'd like as a decorator:

In [14]:
mappings = {'correct': 'good', 'incorrect': 'bad'}

@list
@str.upper
@mappings.get
@func_name
def incorrect():
    pass

incorrect
Out[14]:
['B', 'A', 'D']

Decorators can also be applied to class definitions:

In [15]:
import re

def constructor(type_):
    def decorator(method):
        method.constructs_type = type_
        return method
    return decorator

def register_constructors(cls):
    for item_name in cls.__dict__:
        item = getattr(cls, item_name)
        if hasattr(item, 'constructs_type'):
            cls.constructors[item.constructs_type] = item
    return cls
            
@register_constructors
class IntStore(object):
    constructors = {}
    
    def __init__(self, value):
        self.value = value
        
    @classmethod
    @constructor(int)
    def from_int(cls, x):
        return cls(x)
    
    @classmethod
    @constructor(float)
    def from_float(cls, x):
        return cls(int(x))
    
    @classmethod
    @constructor(str)
    def from_string(cls, x):
        match = re.search(r'\d+', x)
        if match is None:
            return cls(0)
        return cls(int(match.group()))
    
    @classmethod
    def from_auto(cls, x):
        constructor = cls.constructors[type(x)]
        return constructor(x)
    
IntStore.from_auto('at the 11th hour').value == IntStore.from_auto(11.1).value
Out[15]:
True

Decorator toolbox

So now that we've established that decorators aren't "functions that take functions and return functions", but instead "(callable) objects that take objects and return objects", the question is not what you can do with them, but what they're good for.

One answer is that, yes, they're still good for taking functions and wrapping them in other functions and changing their behavior and yadda yadda yadda, but there are a million other decorator tutorials out there that will gladly show you how to do that - doesn't mean it's not true.

But I'm here to show you the uses they won't tell you about. Here are the things I've found decorators useful for that I don't see other people talking about very often. Some of these are distilled versions of things I've already demonstrated (to some extent) above. All are things that can also be done without using decorators - the point being that decorators make them easy, meaning that you'll be much more likely to do them, which is important in creating maintainable systems.

Decorators for annotation

Decorators can add annotations to functions when they're declared. For instance, suppose we want to label two types of functions, "red" functions and "blue" functions:

In [16]:
def red(fn):
    fn.color = 'red'
    return fn

def blue(fn):
    fn.color = 'blue'
    return fn

@red
def combine(a, b):
    result = []
    result.extend(a)
    result.extend(b)
    return result

@blue
def unsafe_combine(a, b):
    a.extend(b)
    return a

@blue
def combine_and_save(a, b):
    result = a + b
    with open('combined') as f:
        f.write(repr(result))
    return combined

Now we have more information at run-time about each of these functions, which lets us make decisions we couldn't've before:

In [17]:
def combine_using(fn, a, b):
    if hasattr(fn, 'color') and fn.color == 'blue':
        print("Sorry, only red functions allowed here!")
        return combine(a, b)  # fall back to default implementation
    return fn(a, b)

a = [1, 2]
b = [3, 4]
print(combine_using(unsafe_combine, a, b))
a
Sorry, only red functions allowed here!
[1, 2, 3, 4]
Out[17]:
[1, 2]

And while we could have equally well have done:

In [18]:
def combine(a, b):
    return a + b
combine.color = 'red'

the red and blue decorators offer the benefits of being:

  • highly and immediately visible
  • inherently closely attached to/above the function definition
  • consistent and foolproof (no room for typos).

If you've ever used pytest, this is what @pytest.mark.parametrize, @pytest.mark.skip, @pytest.mark.[etc] are doing - simply setting attributes on your test function, (some of) which are later used by the framework to dictate how the test is to be run.

Decorators for registration

Sometimes, we want to have a centralized means of discovering a number of different functions. Decorators are a convenient means of doing this.

In [19]:
FUNCTION_REGISTRY = []

def registered(fn):
    FUNCTION_REGISTRY.append(fn)
    return fn

@registered
def step_1():
    print("Hello")
    
@registered
def step_2():
    print("world!")

This gives us the capability to, for instance, iterate and execute all registered functions:

In [20]:
def run_all():
    for function in FUNCTION_REGISTRY:
        function()
        
run_all()
Hello
world!

Again, while we could have accomplished the same thing by doing:

In [21]:
def step_1():
    print("Hello")
    
def step_2():
    print("world!")
    
FUNCTION_REGISTRY = [step_1, step_2]

This means that, looking at the definition of step_1 in isolation, we don't know that it's been registered and, further, if we want to figure out how or why it's being run, we need to first identify that it's included in FUNCTION_REGISTRY, and then see that FUNCTION_REGISTRY is used in run_all. Additionally, if we add a step_3 - possibly referencing our other step functions - we have to remember to add it to FUNCTION_REGISTRY when we're done. This is harder to forget when you see step_1 and step_2 very visibly decorated with @registered.

Decorators for verification

Discovering bugs at runtime is a big bummer. Even moreso when they pop up at the end of a very long-running script or program, or in code running in production. Because decorators are evaluated at function definition time, we can use them to give us "compile"-time assurances immediately when a module is imported.

For instance, pretty frequently you'll want to use other languages or DSLs within Python: regular expressions, SQL, XPath, etc. The problem is that these are almost always represented as strings, not code, meaning that you can't benefit from syntax checking (although it doesn't have to be this way). Using a decorator, we can at least be alerted when the strings in our function have mismatched brackets - regardless of if or when the function is run:

In [22]:
def brackets_balanced(s):
    brackets = {
        opening: closing
        for opening, closing in 
        '() {} []'.split()
    }
    closing = set(brackets.values())
    stack = []
    for char in s:
        if char not in closing:
            if char in brackets:
                stack.append(brackets[char])
            continue
        try:
            expected = stack.pop()
        except IndexError:
            return False
        if char != expected:
            return False
    return not stack

def ensure_brackets_balanced(fn):
    for const in fn.__code__.co_consts:
        if not isinstance(const, str) or brackets_balanced(const):
            continue
        print(
            "WARNING - {.__name__} contains unbalanced brackets: {}".format(
                fn, const
            )
        )
    return fn

@ensure_brackets_balanced
def get_root_div_paragraphs(xml_element):
    return xml_element.xpath("//div[not(ancestor::div]/p")
WARNING - get_root_div_paragraphs contains unbalanced brackets: //div[not(ancestor::div]/p

For ensuring more advanced properties about the structure of the function's code itself, you'd want to choose a tool that can provide you with a function's abstract syntax tree - or write your own linters using something similar.

Decorators for dispatch

It's often very convenient to not explicitly, yourself, decide what functions should be run under what circumstances on which inputs, but to instead simply indicate the necessary circumstances for a function, and then let the computer decide what function to run by using that information. Decorators are a clean way of establishing these mappings between input conditions and handling strategy.

For instance, consider the IntStore example above. The constructor decorator annotates each method, the register_constructors class decorator registers each of these on the class, and the from_auto method uses this information to dispatch on the input type.

This can be taken even further by creating a suite of strategies, each with their own preconditions, available for use by the program at run time. Giving your program options like this can give you a robustness and flexibility that is very desirable for some applications - for instance, web scraping, in which the document you're parsing can vary (on a single site) depending on a huge number of factors.

Anyway, here's an example:

In [23]:
STRATEGIES = []

def precondition(cond):
    def decorator(fn):
        fn.precondition_met = lambda **kwargs: eval(cond, kwargs)
        STRATEGIES.append(fn)
        return fn
    return decorator

@precondition("s.startswith('The year is ')")
def parse_year_from_declaration(s):
    return int(s[-4:])

@precondition("any(substr.isdigit() for substr in s.split())")
def parse_year_from_word(s):
    for substr in s.split():
        try:
            return int(substr)
        except Exception:
            continue
            
@precondition("'-' in s")
def parse_year_from_iso(s):
    from dateutil import parser
    return parser.parse(s).year

def parse_year(s):
    for strategy in STRATEGIES:
        if strategy.precondition_met(s=s):
            return strategy(s)
        
parse_year("It's 2017 bro.")
Out[23]:
2017

Decorators for metaprogramming

Metaprogramming is a little outside the scope of this article, so I'll just briefly say that AST inspection and manipulation can be a very powerful tool. pytest, for instance, heavily uses metaprogramming to, for instance, rewrite assertions to provide more useful error messages. astoptimizer uses it to speed up your programs. patterns uses it to provide convenient pattern matching. All these applications hinge, to some extent, on the useful fact that function bodies are only checked for syntactic correctness - ignoring, e.g., whether the variable names used can be resolved. Metaprogramming Beyond Decency gives an excellent introductory overview of decorators for metaprogramming, for those interested.

A final caveat

As we all know, "with great power comes great responsibility". Decorators are powerful, and must always be used wisely and with good taste. As I've demonstrated, decorators can make code unpredictable - you never know what decorated function does, or, indeed, whether it's even still a function. Decorators are also for forever - you can't shuck a decorator from a function to get back to the original, un-decorated version; the decorator is an inherent part of the function's definition. Be careful and use decorators sensibly.

About the author