You know, decorators are one of the features that makes Python great, but there are a lot of people who either haven't been exposed to them or - worse - have been exposed to them (either in the wild or as part of a tutorial) but never "gotten" them.
If you fit in the latter category, you'll almost necessarily have had spouted reassuringly at you: "Decorators are simple, they're just functions that take functions and return other functions!" You'll have seen a blog post that taught you how to make decorators that add one to a function's output, or print when it's called, or implement caching around it - as if these were insurmountable problems that only a decorator could solve. You'll have extensively used flask's @app.route
, but have tried very carefully not to wonder too hard about it, for fear of its becoming disobedient and surly.
In short, you'll know of decorators, but not know what they're good for, or why they are the way they are. Practice makes perfect, but abstention breeds apathy; like someone who has been taught the violin exclusively as an instrument of classical music, and who couldn't care less about classical music, you are unlikely to become a virtuoso without discovering the fiddle.
This brief guide aims to dispell the myths you've heard about decorators and instead show you the good stuff, the decorators they won't tell you about, the decorators that do things you didn't know decorators could do.
State law requires that, before we go any further, I remind you of the basic principles of decorators.
The first thing you must understand is that all functions in Python are first-class citizens, which is to say, are objects just like any other object you might encounter.
They have attributes.
def f():
"""Print something to standard out."""
print('something')
print(dir(f))
['__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__doc__', '__format__', '__get__', '__getattribute__', '__globals__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']
You can assign them to variables.
g = f
g()
something
You can use them as arguments to other functions.
def func_name(function):
return function.__name__
func_name(f)
'f'
You can throw them into data structures.
function_collection = [f, g]
for function in function_collection:
function()
something something
The possibilities are endless, inasmuch as they are endless with any other object.
Decorators are often described as "functions which take functions and return functions", a description which is notable in that, technically speaking, not a single word of it is true. What is true is the following:
x
with a decorator @d
is equivalent to defining x
, then, immediately afterward, having x = d(x)
.@d
and @e
, in that order, is equivalent to performing x = d(e(x))
after the function's definition.The second of these principles is demonstrated here:
def print_when_called(function):
def new_function(*args, **kwargs):
print("{} was called".format(function.__name__))
return function(*args, **kwargs)
return new_function
def one():
return 1
one = print_when_called(one)
@print_when_called
def one_():
return 1
[one(), one_(), one(), one_()]
one was called one_ was called one was called one_ was called
[1, 1, 1, 1]
One immediate point of clarification: although I just said decorators are applied at function definition time, you might notice that messages in the example above are being printed at function call time. This is because print_when_called
returns new_function
, which itself prints before calling one
or one_
. As I said earler, decorators themselves are applied at function definition time:
def print_when_applied(function):
print("print_when_applied was applied to {}".format(function.__name__))
return function
@print_when_applied
def never_called():
import os
os.system('rm -rf /')
print_when_applied was applied to never_called
never_called
is, aptly, never called, but the message from the print_when_applied
decorator is printed regardless.
Finally, a demonstration of decorator ordering:
@print_when_applied
@print_when_called
def this_name_will_be_printed_when_called_but_not_at_definition_time():
pass
this_name_will_be_printed_when_called_but_not_at_definition_time()
print_when_applied was applied to new_function this_name_will_be_printed_when_called_but_not_at_definition_time was called
print_when_called
returns a function named new_function
, and it is this function that print_when_applied
is called upon.
To really understand the potential of decorators, it is not sufficient to learn what you do not yet know - you must also unlearn what you know already.
Earlier, I claimed that applying a decorator d
to a function x
was the same as writing the definition of x
, then x = d(x)
.
But you might have noticed - who says that d
has to return a function? In fact, func_name
(which I defined earlier) returns a string, but works just fine as a decorator.
@func_name
def a_named_function():
return
a_named_function
'a_named_function'
What is important about this? It means that functions can be more than callable objects, but instead can be little isolated scopes for doing... whatever you'd like. For instance, if you want to process a list, you could do:
def process_list(list_):
def decorator(function):
return function(list_)
return decorator
unprocessed_list = [0, 1, 2, 3]
special_var = "don't touch me please"
@process_list(unprocessed_list)
def processed_list(items):
special_var = 1
return [item for item in items if item > special_var]
(processed_list, special_var)
([2, 3], "don't touch me please")
processed_list
is a list, and special_var
has remained unchanged thanks to Python's function scoping rules. This example is a bit silly - there are much more sensible ways to do the same thing - but the principle is very useful. A more conventional exploitation of it might look like:
class FunctionHolder(object):
def __init__(self, function):
self.func = function
self.called_count = 0
def __call__(self, *args, **kwargs):
try:
return self.func(*args, **kwargs)
finally:
self.called_count += 1
def held(function):
return FunctionHolder(function)
@held
def i_am_counted():
pass
i_am_counted()
i_am_counted()
i_am_counted()
i_am_counted.called_count
3
Nothing in x = d(x)
necessitates that d
is a function - d
just has to be callable! The above example can just as well be written:
@FunctionHolder
def i_am_also_counted(val):
print(val)
i_am_also_counted('a')
i_am_also_counted('b')
i_am_also_counted.called_count
a b
2
In fact, i_am_also_counted
, which is a FunctionHolder
instance, not a function, can also be used as a decorator:
@i_am_also_counted
def about_to_be_printed():
pass
i_am_also_counted.called_count
<function about_to_be_printed at 0x7f73f678aaa0>
3
Granted - Python's syntax means that the @decorator
notation can't be used just anywhere, but that doesn't mean they exclusively take functions as arguments. For instance, here's len
, which operates on sequences (not functions), being used as a decorator.
@len
@func_name
def nineteen_characters():
"""are in this function's name"""
pass
nineteen_characters
19
You can, in fact, apply essentially any function you'd like as a decorator:
mappings = {'correct': 'good', 'incorrect': 'bad'}
@list
@str.upper
@mappings.get
@func_name
def incorrect():
pass
incorrect
['B', 'A', 'D']
Decorators can also be applied to class definitions:
import re
def constructor(type_):
def decorator(method):
method.constructs_type = type_
return method
return decorator
def register_constructors(cls):
for item_name in cls.__dict__:
item = getattr(cls, item_name)
if hasattr(item, 'constructs_type'):
cls.constructors[item.constructs_type] = item
return cls
@register_constructors
class IntStore(object):
constructors = {}
def __init__(self, value):
self.value = value
@classmethod
@constructor(int)
def from_int(cls, x):
return cls(x)
@classmethod
@constructor(float)
def from_float(cls, x):
return cls(int(x))
@classmethod
@constructor(str)
def from_string(cls, x):
match = re.search(r'\d+', x)
if match is None:
return cls(0)
return cls(int(match.group()))
@classmethod
def from_auto(cls, x):
constructor = cls.constructors[type(x)]
return constructor(x)
IntStore.from_auto('at the 11th hour').value == IntStore.from_auto(11.1).value
True
So now that we've established that decorators aren't "functions that take functions and return functions", but instead "(callable) objects that take objects and return objects", the question is not what you can do with them, but what they're good for.
One answer is that, yes, they're still good for taking functions and wrapping them in other functions and changing their behavior and yadda yadda yadda, but there are a million other decorator tutorials out there that will gladly show you how to do that - doesn't mean it's not true.
But I'm here to show you the uses they won't tell you about. Here are the things I've found decorators useful for that I don't see other people talking about very often. Some of these are distilled versions of things I've already demonstrated (to some extent) above. All are things that can also be done without using decorators - the point being that decorators make them easy, meaning that you'll be much more likely to do them, which is important in creating maintainable systems.
Decorators can add annotations to functions when they're declared. For instance, suppose we want to label two types of functions, "red" functions and "blue" functions:
def red(fn):
fn.color = 'red'
return fn
def blue(fn):
fn.color = 'blue'
return fn
@red
def combine(a, b):
result = []
result.extend(a)
result.extend(b)
return result
@blue
def unsafe_combine(a, b):
a.extend(b)
return a
@blue
def combine_and_save(a, b):
result = a + b
with open('combined', 'w') as f:
f.write(repr(result))
return result
Now we have more information at run-time about each of these functions, which lets us make decisions we couldn't've before:
def combine_using(fn, a, b):
if hasattr(fn, 'color') and fn.color == 'blue':
print("Sorry, only red functions allowed here!")
return combine(a, b) # fall back to default implementation
return fn(a, b)
a = [1, 2]
b = [3, 4]
print(combine_using(unsafe_combine, a, b))
a
Sorry, only red functions allowed here! [1, 2, 3, 4]
[1, 2]
And while we could have equally well have done:
def combine(a, b):
return a + b
combine.color = 'red'
the red
and blue
decorators offer the benefits of being:
If you've ever used pytest
, this is what @pytest.mark.parametrize
, @pytest.mark.skip
, @pytest.mark.[etc]
are doing - simply setting attributes on your test function, (some of) which are later used by the framework to dictate how the test is to be run.
Sometimes, we want to have a centralized means of discovering a number of different functions. Decorators are a convenient means of doing this.
FUNCTION_REGISTRY = []
def registered(fn):
FUNCTION_REGISTRY.append(fn)
return fn
@registered
def step_1():
print("Hello")
@registered
def step_2():
print("world!")
This gives us the capability to, for instance, iterate and execute all registered functions:
def run_all():
for function in FUNCTION_REGISTRY:
function()
run_all()
Hello world!
Again, while we could have accomplished the same thing by doing:
def step_1():
print("Hello")
def step_2():
print("world!")
FUNCTION_REGISTRY = [step_1, step_2]
This means that, looking at the definition of step_1
in isolation, we don't know that it's been registered and, further, if we want to figure out how or why it's being run, we need to first identify that it's included in FUNCTION_REGISTRY
, and then see that FUNCTION_REGISTRY
is used in run_all
. Additionally, if we add a step_3
- possibly referencing our other step functions - we have to remember to add it to FUNCTION_REGISTRY
when we're done. This is harder to forget when you see step_1
and step_2
very visibly decorated with @registered
.
Discovering bugs at runtime is a big bummer. Even moreso when they pop up at the end of a very long-running script or program, or in code running in production. Because decorators are evaluated at function definition time, we can use them to give us "compile"-time assurances immediately when a module is imported.
For instance, pretty frequently you'll want to use other languages or DSLs within Python: regular expressions, SQL, XPath, etc. The problem is that these are almost always represented as strings, not code, meaning that you can't benefit from syntax checking (although it doesn't have to be this way). Using a decorator, we can at least be alerted when the strings in our function have mismatched brackets - regardless of if or when the function is run:
def brackets_balanced(s):
brackets = {
opening: closing
for opening, closing in
'() {} []'.split()
}
closing = set(brackets.values())
stack = []
for char in s:
if char not in closing:
if char in brackets:
stack.append(brackets[char])
continue
try:
expected = stack.pop()
except IndexError:
return False
if char != expected:
return False
return not stack
def ensure_brackets_balanced(fn):
for const in fn.__code__.co_consts:
if not isinstance(const, str) or brackets_balanced(const):
continue
print(
"WARNING - {.__name__} contains unbalanced brackets: {}".format(
fn, const
)
)
return fn
@ensure_brackets_balanced
def get_root_div_paragraphs(xml_element):
return xml_element.xpath("//div[not(ancestor::div]/p")
WARNING - get_root_div_paragraphs contains unbalanced brackets: //div[not(ancestor::div]/p
For ensuring more advanced properties about the structure of the function's code itself, you'd want to choose a tool that can provide you with a function's abstract syntax tree - or write your own linters using something similar.
It's often very convenient to not explicitly, yourself, decide what functions should be run under what circumstances on which inputs, but to instead simply indicate the necessary circumstances for a function, and then let the computer decide what function to run by using that information. Decorators are a clean way of establishing these mappings between input conditions and handling strategy.
For instance, consider the IntStore
example above. The constructor
decorator annotates each method, the register_constructors
class decorator registers each of these on the class, and the from_auto
method uses this information to dispatch on the input type.
This can be taken even further by creating a suite of strategies, each with their own preconditions, available for use by the program at run time. Giving your program options like this can give you a robustness and flexibility that is very desirable for some applications - for instance, web scraping, in which the document you're parsing can vary (on a single site) depending on a huge number of factors.
Anyway, here's an example:
STRATEGIES = []
def precondition(cond):
def decorator(fn):
fn.precondition_met = lambda **kwargs: eval(cond, kwargs)
STRATEGIES.append(fn)
return fn
return decorator
@precondition("s.startswith('The year is ')")
def parse_year_from_declaration(s):
return int(s[-4:])
@precondition("any(substr.isdigit() for substr in s.split())")
def parse_year_from_word(s):
for substr in s.split():
try:
return int(substr)
except Exception:
continue
@precondition("'-' in s")
def parse_year_from_iso(s):
from dateutil import parser
return parser.parse(s).year
def parse_year(s):
for strategy in STRATEGIES:
if strategy.precondition_met(s=s):
return strategy(s)
parse_year("It's 2017 bro.")
2017
Metaprogramming is a little outside the scope of this article, so I'll just briefly say that AST inspection and manipulation can be a very powerful tool. pytest
, for instance, heavily uses metaprogramming to, for instance, rewrite assertions to provide more useful error messages. astoptimizer
uses it to speed up your programs. patterns
uses it to provide convenient pattern matching. All these applications hinge, to some extent, on the useful fact that function bodies are only checked for syntactic correctness - ignoring, e.g., whether the variable names used can be resolved. Metaprogramming Beyond Decency gives an excellent introductory overview of decorators for metaprogramming, for those interested.
As we all know, "with great power comes great responsibility". Decorators are powerful, and must always be used wisely and with good taste. As I've demonstrated, decorators can make code unpredictable - you never know what decorated function does, or, indeed, whether it's even still a function. Decorators are also for forever - you can't shuck a decorator from a function to get back to the original, un-decorated version; the decorator is an inherent part of the function's definition. Be careful and use decorators sensibly.