#!/usr/bin/env python
# coding: utf-8

# # The decorators they won't tell you about

# You know, decorators are one of the features that makes Python great, but there are a lot of people who either haven't been exposed to them or - worse - have been exposed to them (either in the wild or as part of a tutorial) but never "gotten" them. 
# 
# If you fit in the latter category, you'll almost necessarily have had spouted reassuringly at you: "Decorators are simple, they're just functions that take functions and return other functions!" You'll have seen a blog post that taught you how to make decorators that add one to a function's output, or print when it's called, or implement caching around it - as if these were insurmountable problems that only a decorator could solve. You'll have extensively used flask's `@app.route`, but have tried very carefully not to wonder too hard about it, for fear of its becoming disobedient and surly.
# 
# In short, you'll know _of_ decorators, but not know what they're _good_ for, or why they are the way they are. Practice makes perfect, but abstention breeds apathy; like someone who has been taught the violin exclusively as an instrument of classical music, and who couldn't care less about classical music, you are unlikely to become a virtuoso without discovering the fiddle.
# 
# This brief guide aims to dispell the myths you've heard about decorators and instead show you the _good_ stuff, the decorators they _won't_ tell you about, the decorators that do things you didn't know decorators could _do_.

# ## Your mandatory refresher
# State law requires that, before we go any further, I remind you of the basic principles of decorators.

# ### Functions in Python
# The first thing you must understand is that all functions in Python are [first-class citizens](https://en.wikipedia.org/wiki/First-class_citizen), which is to say, are objects just like any other object you might encounter.
# 
# They have attributes.

# In[1]:


def f():
    """Print something to standard out."""
    print('something')

print(dir(f))


# You can assign them to variables.

# In[2]:


g = f
g()


# You can use them as arguments to other functions.

# In[3]:


def func_name(function):
    return function.__name__

func_name(f)


# You can throw them into data structures.

# In[4]:


function_collection = [f, g]
for function in function_collection:
    function()


# The possibilities are endless, inasmuch as they are endless with any other object.

# ### Decorators
# Decorators are often described as "functions which take functions and return functions", a description which is notable in that, technically speaking, not a single word of it is true. What _is_ true is the following:
# - Decorators are applied once, at function definition time.
# - Annotating a function definition `x` with a decorator `@d` is equivalent to defining `x`, then, immediately afterward, having `x = d(x)`.
# - Decorating a function with `@d` and `@e`, in that order, is equivalent to performing `x = d(e(x))` after the function's definition.
#     
# The second of these principles is demonstrated here:

# In[5]:


def print_when_called(function):
    def new_function(*args, **kwargs):
        print("{} was called".format(function.__name__))
        return function(*args, **kwargs)
    return new_function

def one():
    return 1
one = print_when_called(one)

@print_when_called
def one_():
    return 1

[one(), one_(), one(), one_()]


# One immediate point of clarification: although I just said decorators are applied at function definition time, you might notice that messages in the example above are being printed at function _call_ time. This is because `print_when_called` returns `new_function`, which itself prints before calling `one` or `one_`. As I said earler, decorators themselves are applied at function definition time:

# In[6]:


def print_when_applied(function):
    print("print_when_applied was applied to {}".format(function.__name__))
    return function

@print_when_applied
def never_called():
    import os
    os.system('rm -rf /')


# `never_called` is, aptly, never called, but the message from the `print_when_applied` decorator is printed regardless.
# 
# Finally, a demonstration of decorator ordering:

# In[7]:


@print_when_applied
@print_when_called
def this_name_will_be_printed_when_called_but_not_at_definition_time():
    pass

this_name_will_be_printed_when_called_but_not_at_definition_time()


# `print_when_called` returns a function named `new_function`, and it is _this_ function that `print_when_applied` is called upon.

# ## Decorator myths
# To really understand the potential of decorators, it is not sufficient to learn what you do not yet know - you must also unlearn what you know already.

# ### Myth the first - decorators return functions
# Earlier, I claimed that applying a decorator `d` to a function `x` was the same as writing the definition of `x`, then `x = d(x)`. 
# 
# But you might have noticed - who says that `d` has to return a function? In fact, `func_name` (which I defined earlier) returns a string, but works just fine as a decorator.

# In[8]:


@func_name
def a_named_function():
    return

a_named_function


# What is important about this? It means that functions can be more than callable objects, but instead can be little isolated scopes for doing... whatever you'd like. For instance, if you want to process a list, you could do:

# In[9]:


def process_list(list_):
    def decorator(function):
        return function(list_)
    return decorator

unprocessed_list = [0, 1, 2, 3]
special_var = "don't touch me please"

@process_list(unprocessed_list)
def processed_list(items):
    special_var = 1
    return [item for item in items if item > special_var]

(processed_list, special_var)


# `processed_list` is a list, and `special_var` has remained unchanged thanks to Python's function scoping rules. This example is a bit silly - there are much more sensible ways to do the same thing - but the principle is very useful. A more conventional exploitation of it might look like:

# In[10]:


class FunctionHolder(object):
    def __init__(self, function):
        self.func = function
        self.called_count = 0
    def __call__(self, *args, **kwargs):
        try:
            return self.func(*args, **kwargs)
        finally:
            self.called_count += 1
            
def held(function):
    return FunctionHolder(function)

@held
def i_am_counted():
    pass

i_am_counted()
i_am_counted()
i_am_counted()
i_am_counted.called_count


# ### Myth the second - decorators are functions
# Nothing in `x = d(x)` necessitates that `d` is a function - `d` just has to be callable! The above example can just as well be written:

# In[11]:


@FunctionHolder
def i_am_also_counted(val):
    print(val)
    
i_am_also_counted('a')
i_am_also_counted('b')
i_am_also_counted.called_count


# In fact, `i_am_also_counted`, which is a `FunctionHolder` instance, not a function, can _also_ be used as a decorator:

# In[12]:


@i_am_also_counted
def about_to_be_printed():
    pass

i_am_also_counted.called_count


# ### Myth the third - decorators take functions
# Granted - Python's syntax means that the `@decorator` notation can't be used just anywhere, but that doesn't mean they exclusively take functions as arguments. For instance, here's `len`, which operates on sequences (not functions), being used as a decorator.

# In[13]:


@len
@func_name
def nineteen_characters():
    """are in this function's name"""
    pass

nineteen_characters


# You can, in fact, apply essentially any function you'd like as a decorator:

# In[14]:


mappings = {'correct': 'good', 'incorrect': 'bad'}

@list
@str.upper
@mappings.get
@func_name
def incorrect():
    pass

incorrect


# Decorators can also be applied to class definitions:

# In[15]:


import re

def constructor(type_):
    def decorator(method):
        method.constructs_type = type_
        return method
    return decorator

def register_constructors(cls):
    for item_name in cls.__dict__:
        item = getattr(cls, item_name)
        if hasattr(item, 'constructs_type'):
            cls.constructors[item.constructs_type] = item
    return cls
            
@register_constructors
class IntStore(object):
    constructors = {}
    
    def __init__(self, value):
        self.value = value
        
    @classmethod
    @constructor(int)
    def from_int(cls, x):
        return cls(x)
    
    @classmethod
    @constructor(float)
    def from_float(cls, x):
        return cls(int(x))
    
    @classmethod
    @constructor(str)
    def from_string(cls, x):
        match = re.search(r'\d+', x)
        if match is None:
            return cls(0)
        return cls(int(match.group()))
    
    @classmethod
    def from_auto(cls, x):
        constructor = cls.constructors[type(x)]
        return constructor(x)
    
IntStore.from_auto('at the 11th hour').value == IntStore.from_auto(11.1).value


# ## Decorator toolbox
# So now that we've established that decorators aren't "functions that take functions and return functions", but instead "(callable) objects that take objects and return objects", the question is not what you _can_ do with them, but what they're _good_ for. 
# 
# One answer is that, yes, they're still good for taking functions and wrapping them in other functions and changing their behavior and yadda yadda yadda, but there are a million other decorator tutorials out there that will gladly show you how to do that - doesn't mean it's not true.
# 
# But I'm here to show you the uses they won't tell you about. Here are the things I've found decorators useful for that I don't see other people talking about very often. Some of these are distilled versions of things I've already demonstrated (to some extent) above. _All_ are things that can also be done _without_ using decorators - the point being that decorators make them _easy_, meaning that you'll be much more _likely_ to do them, which is [important in creating maintainable systems](http://www.haskellforall.com/2016/04/worst-practices-should-be-hard.html). 

# ### Decorators for annotation
# Decorators can add annotations to functions when they're declared. For instance, suppose we want to label two types of functions, "red" functions and "blue" functions: 

# In[16]:


def red(fn):
    fn.color = 'red'
    return fn

def blue(fn):
    fn.color = 'blue'
    return fn

@red
def combine(a, b):
    result = []
    result.extend(a)
    result.extend(b)
    return result

@blue
def unsafe_combine(a, b):
    a.extend(b)
    return a

@blue
def combine_and_save(a, b):
    result = a + b
    with open('combined', 'w') as f:
        f.write(repr(result))
    return result


# Now we have more information at run-time about each of these functions, which lets us make decisions we couldn't've before:

# In[17]:


def combine_using(fn, a, b):
    if hasattr(fn, 'color') and fn.color == 'blue':
        print("Sorry, only red functions allowed here!")
        return combine(a, b)  # fall back to default implementation
    return fn(a, b)

a = [1, 2]
b = [3, 4]
print(combine_using(unsafe_combine, a, b))
a


# And while we could have equally well have done:

# In[18]:


def combine(a, b):
    return a + b
combine.color = 'red'


# the `red` and `blue` decorators offer the benefits of being:
# - highly and immediately visible
# - inherently closely attached to/above the function definition
# - consistent and foolproof (no room for typos).
# 
# If you've ever used [`pytest`](https://docs.pytest.org/en/latest/), this is what `@pytest.mark.parametrize`, `@pytest.mark.skip`, `@pytest.mark.[etc]` are doing - simply setting attributes on your test function, (some of) which are later used by the framework to dictate how the test is to be run.

# ### Decorators for registration
# Sometimes, we want to have a centralized means of discovering a number of different functions. Decorators are a convenient means of doing this.

# In[19]:


FUNCTION_REGISTRY = []

def registered(fn):
    FUNCTION_REGISTRY.append(fn)
    return fn

@registered
def step_1():
    print("Hello")
    
@registered
def step_2():
    print("world!")


# This gives us the capability to, for instance, iterate and execute all registered functions: 

# In[20]:


def run_all():
    for function in FUNCTION_REGISTRY:
        function()
        
run_all()


# Again, while we could have accomplished the same thing by doing:

# In[21]:


def step_1():
    print("Hello")
    
def step_2():
    print("world!")
    
FUNCTION_REGISTRY = [step_1, step_2]


# This means that, looking at the definition of `step_1` in isolation, we don't _know_ that it's been registered and, further, if we want to figure out how or why it's being run, we need to first identify that it's included in `FUNCTION_REGISTRY`, and then see that `FUNCTION_REGISTRY` is used in `run_all`. Additionally, if we add a `step_3` - possibly referencing our other step functions - we have to remember to add it to `FUNCTION_REGISTRY` when we're done. This is harder to forget when you see `step_1` and `step_2` very visibly decorated with `@registered`.

# ### Decorators for verification
# Discovering bugs at runtime is a big bummer. Even moreso when they pop up at the end of a very long-running script or program, or in code running in production. Because decorators are evaluated at function definition time, we can use them to give us "compile"-time assurances immediately when a module is imported.
# 
# For instance, pretty frequently you'll want to use other languages or DSLs within Python: regular expressions, SQL, XPath, etc. The problem is that these are almost always represented as strings, not code, meaning that you can't benefit from syntax checking (although it [doesn't have to be this way](https://github.com/hchasestevens/xpyth)). Using a decorator, we can at least be alerted when the strings in our function have mismatched brackets - regardless of if or when the function is run:

# In[22]:


def brackets_balanced(s):
    brackets = {
        opening: closing
        for opening, closing in 
        '() {} []'.split()
    }
    closing = set(brackets.values())
    stack = []
    for char in s:
        if char not in closing:
            if char in brackets:
                stack.append(brackets[char])
            continue
        try:
            expected = stack.pop()
        except IndexError:
            return False
        if char != expected:
            return False
    return not stack

def ensure_brackets_balanced(fn):
    for const in fn.__code__.co_consts:
        if not isinstance(const, str) or brackets_balanced(const):
            continue
        print(
            "WARNING - {.__name__} contains unbalanced brackets: {}".format(
                fn, const
            )
        )
    return fn

@ensure_brackets_balanced
def get_root_div_paragraphs(xml_element):
    return xml_element.xpath("//div[not(ancestor::div]/p")


# For ensuring more advanced properties about the structure of the function's code itself, you'd want to choose a [tool](https://github.com/hchasestevens/asttools) that can provide you with a function's [abstract syntax tree](https://greentreesnakes.readthedocs.io/en/latest/) - or write your own linters using [something similar](https://github.com/hchasestevens/astpath).

# ### Decorators for dispatch
# 
# It's often very convenient to not explicitly, yourself, decide what functions should be run under what circumstances on which inputs, but to instead simply indicate the necessary circumstances for a function, and then let the computer decide what function to run by using that information. Decorators are a clean way of establishing these mappings between input conditions and handling strategy.
# 
# For instance, consider the `IntStore` example above. The `constructor` decorator _annotates_ each method, the `register_constructors` class decorator _registers_ each of these on the class, and the `from_auto` method uses this information to _dispatch_ on the input type. 
# 
# This can be taken even further by creating a suite of strategies, each with their own preconditions, available for use by the program at run time. Giving your program _options_ like this can give you a _robustness_ and _flexibility_ that is very desirable for some applications - for instance, web scraping, in which the document you're parsing can vary (on a single site) depending on a huge number of factors.
# 
# Anyway, here's an example:

# In[23]:


STRATEGIES = []

def precondition(cond):
    def decorator(fn):
        fn.precondition_met = lambda **kwargs: eval(cond, kwargs)
        STRATEGIES.append(fn)
        return fn
    return decorator

@precondition("s.startswith('The year is ')")
def parse_year_from_declaration(s):
    return int(s[-4:])

@precondition("any(substr.isdigit() for substr in s.split())")
def parse_year_from_word(s):
    for substr in s.split():
        try:
            return int(substr)
        except Exception:
            continue
            
@precondition("'-' in s")
def parse_year_from_iso(s):
    from dateutil import parser
    return parser.parse(s).year

def parse_year(s):
    for strategy in STRATEGIES:
        if strategy.precondition_met(s=s):
            return strategy(s)
        
parse_year("It's 2017 bro.")


# ### Decorators for metaprogramming
# Metaprogramming is a little outside the scope of this article, so I'll just briefly say that AST inspection and manipulation can be a very powerful tool. `pytest`, for instance, heavily uses metaprogramming to, for instance, [rewrite](http://pybites.blogspot.com/2011/07/behind-scenes-of-pytests-new-assertion.html) [assertions](https://docs.pytest.org/en/latest/assert.html) to provide more useful error messages. [`astoptimizer`](https://pypi.python.org/pypi/astoptimizer) uses it to speed up your programs. [`patterns`](https://github.com/Suor/patterns) uses it to provide convenient pattern matching. All these applications hinge, to some extent, on the useful fact that function bodies are only checked for syntactic correctness - ignoring, e.g., whether the variable names used can be resolved.  [Metaprogramming Beyond Decency](http://hackflow.com/blog/2015/03/29/metaprogramming-beyond-decency/) gives an excellent introductory overview of decorators for metaprogramming, for those interested.

# ## A final caveat
# As we all know, "with great power comes great responsibility". Decorators are powerful, and must always be used wisely and with good taste. As I've demonstrated, decorators can make code unpredictable - you never know what decorated function does, or, indeed, whether it's even still a function. Decorators are also for forever - you can't shuck a decorator from a function to get back to the original, un-decorated version; the decorator is an inherent part of the function's definition. Be careful and use decorators sensibly.

# ## About the author
# * Name: [H. Chase Stevens](http://www.chasestevens.com)
# * Github: [hchasestevens](https://github.com/hchasestevens)
# * Twitter: [@hchasestevens](https://twitter.com/hchasestevens)
#