Python Decorators (and Digressions)

Decorators are a bit of meta-programming that allows one to add behaviour to functions and classes. The original function or class is wrapped in another function or class, and its name is pointed to that wrapper. Any use of that name will refer to the wrapper, not the original. This happens at “compile” time, not at runtime

The mechanics treat any function or class name prefixed by an “@”¹ as a decorator and apply it to the following function, class, or method definition. The compiler takes the completed definition and passes it as an argument to the decorator, and the decorator returns a function or class, which the original token (name) points to

@a_decorator
class Foo(object):  #  "Foo" = a_decorator(class formerly-known-as Foo)
    ...

@a_decorator        #  "Bar" = a_decorator(function fka Bar)
def Bar:
    ...

rab = Bar()         #  rab is pointing to the result of
                          #  a_decorator, as it both wraps and 
                          # acts like the Bar function

class Fob(object):
    @a_decorator
    def bof(self):    # same with methods
        ... 

The added behaviour from the wrapper is usually logging or validation of the arguments rather than changing the actual behaviour of the original function or class itself; the decorator acts like a proxy for the wrapped function or class rather than replacing it. One can think of it as a virtualized override of the visibly-defined object

Decorator Naming #

The implementation of a decorator is either a function or a class whose name is used to invoke the decorator behaviour. Note that a decorator implemented by a class can decorate a function, and a decorator implemented by a function can decorate a class (sort of),
A decorator implemented as a class might have a lower-case name instead of CorrectCase name as it normally would to follow the naming idiom of decorators

    class log_exceptions(object):  #  lower-case class name
        ...

    @log_exceptions
    def do_something(arg1):
        ...

The Decorator Mechanism #

Clearly, the way decorators work involves what would be called in other languages “compiler magic”. As Python modules get imported by other modules (or explicitly executed), the interpreter/compiler acts on it. Most source code defines classes, functions, etc.; it does not perform behaviour immediately (that would be script code). Decorators are part of the process that changes the definition of a function or class into a bit of behaviour referenced by a token (a callable reference)

The simple case is merely wrapping the callable reference in another callable reference and pointing the token at the wrapper, but not all cases are simple.
Because the Decorator Mechanism behaves according to convention instead of explicit direction, its behaviour has hidden issues. The Decorator Mechanism changes how it interacts with decorator implementations based on if the user applies the decorator with or without arguments

Decorator Modes #

Mode 1: The decorator is specified (called) with no arguments, not even empty parentheses

@decorator
def foo:
    ...

Mode 2: The decorator is specified (called) with parentheses, empty or with arguments

@decorator()
def foo:
    ...

Mode 1 #

The Decorator Mechanism calls the decorator function or creates an instance of the decorator class, passing in the object-to-be-decorated as the sole argument.
The result is a callable object² that will act like a function or a class, depending on the object-to-be-decorated

Mode 2 #

The Decorator Mechanism calls the decorator function or creates an instance of the decorator class, passing in the decorator arguments specified by the user. The result is a callable object which it will call immediately, passing in the object-to-be-decorated as the sole argument.
The result of this second call is a callable object that will act like a function or a class, depending on the object-to-be-decorated

Summary of Modes #

If a decorator is not robust enough to handle calls in both modes [e.g. @decorator(…) vs @decorator ], it should fail quickly and loudly.
The easiest way to detect the wrong mode is to have required parameters for one mode’s calls and not the other, however this does not produce intuitive error messages³

Another is to examine the initial call’s arguments and raise an exception if it has the wrong type or number. This requires that the first expected decorator parameter is not a function (e.g. a logging function) to be able to differentiate

Simple Decorators (Mode 1) #

In the simplest case, the decorator takes no parameters itself and returns a reference to a function for the outside code to call while hiding the wrapped function (i.e. with the same parameter signature). The wrapper might add logging, caching, or validation, but almost always calls the wrapped function to get the real result, passing the supplied call-parameters (if any), and returning the result to the caller.
The decorator “remembers” (i.e. as a closure) the wrapped function as either a persistent field of a class instance or as a function closure

Class implementation #

    class decorator:
        def __init__(self, a_function):
            self._wrapped_function = a_function

        def  __call__(self, *args, **kwargs):
        #  do before-stuff here
            result = self._wrapped_function(*args, **kwargs)
        #  do after-stuff here
            return result

Function implementation #

    def decorator(a_function):
        def wrapper(*args, **kwargs):
        #  do before-stuff here
            result = a_function(*args, **kwargs)
        #  do after-stuff here
            return result)

        return wrapper

Decorators with Parameters (Mode 2) #

When the Decorator Mechanism calls a decorator with arguments (independent of if the wrapped function or wrapped class constructor has parameters), it behaves differently.
It passes the arguments to the decorator function/class, and it expects the return value to be either:

• a function reference, which it will immediately call to get the wrapper object
• a class instance, upon which it will immediately call (i.e. the __call__ method) to get the wrapper object

Class implementation #

class decorator:
        def __init__(self, decorator_param):
            self.param = decorator_param

        def  __call__(self, a_function):
            def wrapper(*args, **kwargs):
            #  do before-stuff here
                result = self._wrapped_function(*args, **kwargs)
            # do after-stuff here
                return result

            return wrapper

Function implementation #

    def decorator(decorator_param):
        def outer_wrapper(a_function):
            def inner_wrapper(*args, **kwargs):
            #  do before-stuff here
                result = a_function(*args, **kwargs)
            #  do after-stuff here
                return result

            return inner_wrapper

        return outer_wrapper

Decorating Classes #

When decorating a class, rather than a function or method, the decorator result will be an instance of something that looks and works like a class.

Digression #

All Python objects, including classes, class instances, and function, are instances of types. All objects have a lot in common, and different types of objects get treated differently only when some code decides to discriminate based on type

As long as an object is callable, it does not have to be a function or method to stand in for a function or method. As long as an object has the needed behaviours of a class (e.g. returns new instances from a direct call, host static methods, etc.), then it does not matter what sort of object it actually is (like duck-typing)

Wrapping the Class #

Decorating a function is (for Python) initiative: the wrapper might log uses of the function; handle exceptions raised by the function; in cases approaching the pathological, execute the functions in threads. Decorating a class has different opportunities to act

• when a class is defined
• when something calls a class method or static method
• when something instantiates a class
• when something operates on a class instantiation, such as
  • accessing or mutating an attribute
  • calling an instance method
  • calling the built-in __call__ method

In order to decorate an entire class, this class-wrapper will have embed an instance of the original, decorated class, and it might proxy all the classmethods and staticmethods of that class (which is tricky). The only thing it must do is create new instances when the user calls ClassName() . If the decorator returns a class instance as a wrapper, then that instance can override its built-in call-handler to return new instances

A function can return class instances. In any case, the class-wrapper must be able to return instances of a (wrapped) decorated class. That object must execute the behaviors of the wrapped class: running the __init__ method; getting and setting attributes; and allowing the default __call__ method

Decorator Class Interface #

    class decorator:
        def __init__(self, a_class):
            self._wrapped_class = a_class

        def  __call__(self):
            return ClassWrapper(self._wrapped_class)

Decorator Function Interface #

    def decorator(a_class):
        return ClassWrapper(a_class)

Class-Wrapper Implementation #

    class ClassWraper:
        def __init__(self, a_class):
            self._wrapped_class = a_class

    # this is the "constructor" for instances of the wrapped class, 
    # with creation parameters
        def __call__(self, *args, **kwargs):
            wrapped_instance = self._wrapped_class(*args, **kwargs)
            result = InstanceWrapper(wrapped_instance)

        return result

Instance-Wrapper Implementation #

    class InstanceWrapper:
        def __init__(self, wrapped_class_instance):
            self._wrapped_instance = wrapped_class_instance
            self._wrapper_attributes = ["_wrapped_instance", 
                                        "_wrapper_attributes"]

        def __call__(self, *args, **kwargs):

        #  do stuff here
            return self._wrapped_instance(*args, **kwargs)

        def __getattr__(self, attribute_name):
            if attribute_name in self._wrapper_attributes:
                return self.__dict__[attribute_name]
            else:
            #  do stuff here
                return getattr(self._wrapped_instance, attribute_name)

        def __setattr__(self, attribute_name, value):
            if attribute_name in self._wrapper_attributes:
                self.__dict__[attribute_name] = value
            else:
            #  do stuff here
                setattr(self._wrapped_instance, attribute_name, value)

Class Decorator Lifecycle #

Given this code:

@decorator
class Foo:
    def __init__(flag, count):
        ...

the Decorator Mechanism will

1. create an instance of ClassWrapper
2. return the instance of ClassWrapper so the label “Foo” points to it

my_foo = Foo("bof", 42)

1. gets the arguments in ClassWrapper.__call__(self, *args, **kwargs) as it is working as a constructor
2. create an instance of the real Foo class with the arguments
3. create an instance of InstanceWrapper with a new Foo-instance inside of it
4. returns the InstanceWrapper instance so “x” points to it

Super Decorators #

If one wants a robust decorator that users can use on functions, methods, and classes, with or without decorator parameters, one must create a decorator that can tell what it is decorating and what mode the Decorator Mechanism is using.
If the first, let alone sole, decorator parameter is a function reference, your code will have little way of determining if the Decorator Mechanism called it in Mode 1 or Mode 2. Even if it has no practical use, the decorator class’ __init__ or the decorator function might take an unnecessary argument just so it can differentiate the mode. If it can decorate both classes and functions, it will want to do so for each one in different ways

Likewise, the object passed from the Decorator Mechanism might not be something the decorator wants to wrap. A robust decorator must detect when it’s applied to something it cannot decorate and fail gracefully

Code #

    class decorator:
        def __init__(self, a_value = None):   
            #  a_value might be a parameter or the object to decorate
            if isclass(a_value):
                self._wrapped_class = a_value

            else if callable(a_value):
                self._wrapped_function = a_value

            else if a_value is not None:
                self._value = a_value

        def  __call__(self, *args, **kwargs):    
            if self._wrapped_function is not None:   
            #  do before-stuff here
                result = self._wrapped_function(*args, **kwargs)
            #  do after-stuff here
                return result

            if self._wrapped_class is not None:
                return ClassWrapper(self._wrapped_class, self._value)

            if isclass(args[0]):
                return ClassWrapper(args[0], self._value)

            if callable(args[0]):
                def wrapper(*args, **kwargs):
                #    do before-stuff here
                    result = args[0](*args, **kwargs)
                #    do after-stuff here
                    return result

                return wrapper

    class ClassWraper:
        def __init__(self, a_class, a_value):
            self._wrapped_class = a_class
            self._value = a_value
            self._wrapper_attributes = ["_wrapped_class", 
                                        "_wrapper_attributes", "_value"]

        # this is the "constructor" of the wrapped class
        def __call__(self, *args, **kwargs):
            wrapped_instance = self._wrapped_class(*args, **kwargs)
            result = InstanceWrapper(wrapped_instance)
            return result

        #  access the class’ attributes
        def __getattr__(self, attribute_name):
            if attribute_name in self._wrapper_attributes:
                return self.__dict__[attribute_name]
            else:
            #  do stuff here
                return getattr(self._wrapped_class, attribute_name)

        def __setattr__(self, attribute_name, value):
            if attribute_name in self._wrapper_attributes:
                self.__dict__[attribute_name] = value
            else:
            #  do stuff here
                setattr(self._wrapped_class, attribute_name, value)

    class InstanceWrapper:
        def __init__(self, wrapped_class_instance):
            self._wrapped_instance = wrapped_class_instance
            self._wrapper_attributes = ["_wrapped_instance", 
                                        "_wrapper_attributes"]

        def __call__(self, *args, **kwargs):
            return self._wrapped_instance(*args, **kwargs) 

        #  access the instance’s attributes
        def __getattr__(self, attribute_name):
            if attribute_name in self._wrapper_attributes:
                return self.__dict__[attribute_name]
            else:
            #  do stuff here
                return getattr(self._wrapped_instance, attribute_name)

        def __setattr__(self, attribute_name, value):
            if attribute_name in self._wrapper_attributes:
                self.__dict__[attribute_name] = value
            else:
            #  do stuff here
                setattr(self._wrapped_instance, attribute_name, value)

Appendix #

Wrapping Class and Static Methods #

One way is to iterate the attributes of the original class and find the class methods and create attributes in the wrapper that point to the original (using a partial to pass in the cls parameter). This works for both class wrappers and instance wrappers.

    if getattr(attribute_value, 'im_class', None) == type(object):
        base_function = getattr(attribute_value, 'im_func', None)
        if base_function is None:
            raise TypeError("Problem with " + str(original_class) + 
                                                 "." + attribute_name)

        wrapper.__dict__[attribute_name] = 
                      functools.partial(base_function, original_class)

Static methods are like module functions in a class’ namespace

    if getattr(attribute_value, 'func_name', None) is not None:
        base_function = getattr(attribute_value, '__call__', None)
        if base_function is None:
            raise TypeError("Problem with " + str(original_class) + 
                                                   "." + attribute_name)

        wrapper.__dict__[attribute_name] = base_function

Mocking Decorators #

Decoration happens at compile time. It replaces the object that your name points to. How can one patch a decorator for testing if the test initialization happens after the decorator has done its job and sits securely wrapped around the function one wants to test?
Enter the imp library, specifically its reload function. It allows one to patch the decorator itself (remember, decorators are just callable objects like functions), and reload the module to allow one’s patch to be the decorating object.
As we are doing compilish things at run time, we have a little extra housekeeping to do. If one wants to wrap the original decorator or control the original decorator’s parameters, then one saves a pointer to that original decorator before patching it (sensible when one thinks about it, but not intuitive to me, at least)
This example uses setupModule, but one could use setupClass and addCleanup

import imp
from mock import patch

...

original_decorator = decorator_module.decorator
decorator_mock = None

def mock_decorator(arguments, for, original, decorator):
    # do whatever with the arguments or return a completely new wrapper
    return original_decorator(arguments, for, original, decorator)

def setupModule():  # magic name that runs once for the module
    decorator_mock = 
                mock.patch('decorator_module.decorator', mock_decorator)
    decorator_mock.start()

    # Reload the module to apply the new decorator
    imp.reload(module_using_decorator)  

def teardownModule():   # magic name that runs once for the module
    decorator_mock.stop()

    # Reload the module to restore the original decorator
    imp.reload(module_using_decorator)  

See Also #

https://stackoverflow.com/questions/739654/how-to-make-a-chain-of-function-decorators/159448
https://wiki.python.org/moin/PythonDecoratorLibrary
https://python-3-patterns-idioms-test.readthedocs.io/en/latest/PythonDecorators.html