Python Decorators (and Digressions)
Decorators are a bit of meta-programming that allows one to add behaviour to functions and classes. The original function or class is wrapped in another function or class, and its name is pointed to that wrapper. Any use of that name will refer to the wrapper, not the original. This happens at “compile” time, not at runtime
The mechanics treat any function or class name prefixed by an “@”[1] as a decorator and apply it to the following function, class, or method definition. The compiler takes the completed definition and passes it as an argument to the decorator, and the decorator returns a function or class, which the original token (name) points to
@a_decorator
class Foo(object): # "Foo" = a_decorator(class formerly-known-as Foo)
...
@a_decorator # "Bar" = a_decorator(function fka Bar)
def Bar:
...
rab = Bar() # rab is pointing to the result of
# a_decorator, as it both wraps and
# acts like the Bar function
class Fob(object):
@a_decorator
def bof(self): # same with methods
...
The added behaviour from the wrapper is usually logging or validation of the arguments rather than changing the actual behaviour of the original function or class itself; the decorator acts like a proxy for the wrapped function or class rather than replacing it. One can think of it as a virtualized override of the visibly-defined object
Decorator Naming #
The implementation of a decorator is either a function or a class whose name is used to invoke the decorator behaviour. Note that a decorator implemented by a class can decorate a function, and a decorator implemented by a function can decorate a class (sort of),
A decorator implemented as a class might have a lower-case name instead of CorrectCase name as it normally would to follow the naming idiom of decorators
class log_exceptions(object): # lower-case class name
...
@log_exceptions
def do_something(arg1):
...
The Decorator Mechanism #
Clearly, the way decorators work involves what would be called in other languages “compiler magic”. As Python modules get imported by other modules (or explicitly executed), the interpreter/compiler acts on it. Most source code defines classes, functions, etc.; it does not perform behaviour immediately (that would be script code). Decorators are part of the process that changes the definition of a function or class into a bit of behaviour referenced by a token (a callable reference)
The simple case is merely wrapping the callable reference in another callable reference and pointing the token at the wrapper, but not all cases are simple.
Because the Decorator Mechanism behaves according to convention instead of explicit direction, its behaviour has hidden issues. The Decorator Mechanism changes how it interacts with decorator implementations based on if the user applies the decorator with or without arguments
Decorator Modes #
Mode 1: The decorator is specified (called) with no arguments, not even empty parentheses
@decorator
def foo:
...
Mode 2: The decorator is specified (called) with parentheses, empty or with arguments
@decorator()
def foo:
...
| The caller determines which mode the Decorator Mechanism will use The expectation of parameters on the implementing decorator function/class, or lack thereof MAKES NO DIFFERENCE |
Mode 1 #
The Decorator Mechanism calls the decorator function or creates an instance of the decorator class, passing in the object-to-be-decorated as the sole argument.
The result is a callable object[2] that will act like a function or a class, depending on the object-to-be-decorated
Mode 2 #
The Decorator Mechanism calls the decorator function or creates an instance of the decorator class, passing in the decorator arguments specified by the user. The result is a callable object which it will call immediately, passing in the object-to-be-decorated as the sole argument.
The result of this second call is a callable object that will act like a function or a class, depending on the object-to-be-decorated
Summary of Modes #
| Mode 1 | Mode 2 | |
| decorator function |
|
|
| decorator class |
|
|
If a decorator is not robust enough to handle calls in both modes [e.g. @decorator(…) vs @decorator ], it should fail quickly and loudly.
The easiest way to detect the wrong mode is to have required parameters for one mode’s calls and not the other, however this does not produce intuitive error messages[3]
Another is to examine the initial call’s arguments and raise an exception if it has the wrong type or number. This requires that the first expected decorator parameter is not a function (e.g. a logging function) to be able to differentiate
Simple Decorators (Mode 1) #
In the simplest case, the decorator takes no parameters itself and returns a reference to a function for the outside code to call while hiding the wrapped function (i.e. with the same parameter signature). The wrapper might add logging, caching, or validation, but almost always calls the wrapped function to get the real result, passing the supplied call-parameters (if any), and returning the result to the caller.
The decorator “remembers” (i.e. as a closure) the wrapped function as either a persistent field of a class instance or as a function closure
Class implementation #
class decorator:
def __init__(self, a_function):
self._wrapped_function = a_function
def __call__(self, *args, **kwargs):
# do before-stuff here
result = self._wrapped_function(*args, **kwargs)
# do after-stuff here
return result
Function implementation #
def decorator(a_function):
def wrapper(*args, **kwargs):
# do before-stuff here
result = a_function(*args, **kwargs)
# do after-stuff here
return result)
return wrapper
Decorators with Parameters (Mode 2) #
When the Decorator Mechanism calls a decorator with arguments (independent of if the wrapped function or wrapped class constructor has parameters), it behaves differently.
It passes the arguments to the decorator function/class, and it expects the return value to be either:
- a function reference, which it will immediately call to get the wrapper object
- a class instance, upon which it will immediately call (i.e. the
__call__method) to get the wrapper object
Class implementation #
class decorator:
def __init__(self, decorator_param):
self.param = decorator_param
def __call__(self, a_function):
def wrapper(*args, **kwargs):
# do before-stuff here
result = self._wrapped_function(*args, **kwargs)
# do after-stuff here
return result
return wrapper
Function implementation #
def decorator(decorator_param):
def outer_wrapper(a_function):
def inner_wrapper(*args, **kwargs):
# do before-stuff here
result = a_function(*args, **kwargs)
# do after-stuff here
return result
return inner_wrapper
return outer_wrapper
Decorating Classes #
When decorating a class, rather than a function or method, the decorator result will be an instance of something that looks and works like a class.
Digression #
All Python objects, including classes, class instances, and function, are instances of types. All objects have a lot in common, and different types of objects get treated differently only when some code decides to discriminate based on type
As long as an object is callable, it does not have to be a function or method to stand in for a function or method. As long as an object has the needed behaviours of a class (e.g. returns new instances from a direct call, host static methods, etc.), then it does not matter what sort of object it actually is (like duck-typing)
Wrapping the Class #
Decorating a function is (for Python) initiative: the wrapper might log uses of the function; handle exceptions raised by the function; in cases approaching the pathological, execute the functions in threads. Decorating a class has different opportunities to act
- when a class is defined
- when something calls a class method or static method
- when something instantiates a class
- when something operates on a class instantiation, such as
- accessing or mutating an attribute
- calling an instance method
- calling the built-in
__call__method
In order to decorate an entire class, this class-wrapper will have embed an instance of the original, decorated class, and it might proxy all the classmethods and staticmethods of that class (which is tricky). The only thing it must do is create new instances when the user calls ClassName() . If the decorator returns a class instance as a wrapper, then that instance can override its built-in call-handler to return new instances
A function can return class instances. In any case, the class-wrapper must be able to return instances of a (wrapped) decorated class. That object must execute the behaviors of the wrapped class: running the __init__ method; getting and setting attributes; and allowing the default __call__ method
Decorator Class Interface #
class decorator:
def __init__(self, a_class):
self._wrapped_class = a_class
def __call__(self):
return ClassWrapper(self._wrapped_class)
Decorator Function Interface #
def decorator(a_class):
return ClassWrapper(a_class)
Class-Wrapper Implementation #
class ClassWraper:
def __init__(self, a_class):
self._wrapped_class = a_class
# this is the "constructor" for instances of the wrapped class,
# with creation parameters
def __call__(self, *args, **kwargs):
wrapped_instance = self._wrapped_class(*args, **kwargs)
result = InstanceWrapper(wrapped_instance)
return result
Instance-Wrapper Implementation #
class InstanceWrapper:
def __init__(self, wrapped_class_instance):
self._wrapped_instance = wrapped_class_instance
self._wrapper_attributes = ["_wrapped_instance",
"_wrapper_attributes"]
def __call__(self, *args, **kwargs):
# do stuff here
return self._wrapped_instance(*args, **kwargs)
def __getattr__(self, attribute_name):
if attribute_name in self._wrapper_attributes:
return self.__dict__[attribute_name]
else:
# do stuff here
return getattr(self._wrapped_instance, attribute_name)
def __setattr__(self, attribute_name, value):
if attribute_name in self._wrapper_attributes:
self.__dict__[attribute_name] = value
else:
# do stuff here
setattr(self._wrapped_instance, attribute_name, value)
Class Decorator Lifecycle #
Given this code:
@decorator
class Foo:
def __init__(flag, count):
...
the Decorator Mechanism will
- create an instance of ClassWrapper
- return the instance of ClassWrapper so the label “Foo” points to it
my_foo = Foo("bof", 42)
- gets the arguments in ClassWrapper.
__call__(self, *args, **kwargs)as it is working as a constructor - create an instance of the real Foo class with the arguments
- create an instance of InstanceWrapper with a new Foo-instance inside of it
- returns the InstanceWrapper instance so “x” points to it
Super Decorators #
If one wants a robust decorator that users can use on functions, methods, and classes, with or without decorator parameters, one must create a decorator that can tell what it is decorating and what mode the Decorator Mechanism is using.
If the first, let alone sole, decorator parameter is a function reference, your code will have little way of determining if the Decorator Mechanism called it in Mode 1 or Mode 2. Even if it has no practical use, the decorator class’ __init__ or the decorator function might take an unnecessary argument just so it can differentiate the mode. If it can decorate both classes and functions, it will want to do so for each one in different ways
Likewise, the object passed from the Decorator Mechanism might not be something the decorator wants to wrap. A robust decorator must detect when it’s applied to something it cannot decorate and fail gracefully
Code #
class decorator:
def __init__(self, a_value = None):
# a_value might be a parameter or the object to decorate
if isclass(a_value):
self._wrapped_class = a_value
else if callable(a_value):
self._wrapped_function = a_value
else if a_value is not None:
self._value = a_value
def __call__(self, *args, **kwargs):
if self._wrapped_function is not None:
# do before-stuff here
result = self._wrapped_function(*args, **kwargs)
# do after-stuff here
return result
if self._wrapped_class is not None:
return ClassWrapper(self._wrapped_class, self._value)
if isclass(args[0]):
return ClassWrapper(args[0], self._value)
if callable(args[0]):
def wrapper(*args, **kwargs):
# do before-stuff here
result = args[0](*args, **kwargs)
# do after-stuff here
return result
return wrapper
class ClassWraper:
def __init__(self, a_class, a_value):
self._wrapped_class = a_class
self._value = a_value
self._wrapper_attributes = ["_wrapped_class",
"_wrapper_attributes", "_value"]
# this is the "constructor" of the wrapped class
def __call__(self, *args, **kwargs):
wrapped_instance = self._wrapped_class(*args, **kwargs)
result = InstanceWrapper(wrapped_instance)
return result
# access the class’ attributes
def __getattr__(self, attribute_name):
if attribute_name in self._wrapper_attributes:
return self.__dict__[attribute_name]
else:
# do stuff here
return getattr(self._wrapped_class, attribute_name)
def __setattr__(self, attribute_name, value):
if attribute_name in self._wrapper_attributes:
self.__dict__[attribute_name] = value
else:
# do stuff here
setattr(self._wrapped_class, attribute_name, value)
class InstanceWrapper:
def __init__(self, wrapped_class_instance):
self._wrapped_instance = wrapped_class_instance
self._wrapper_attributes = ["_wrapped_instance",
"_wrapper_attributes"]
def __call__(self, *args, **kwargs):
return self._wrapped_instance(*args, **kwargs)
# access the instance’s attributes
def __getattr__(self, attribute_name):
if attribute_name in self._wrapper_attributes:
return self.__dict__[attribute_name]
else:
# do stuff here
return getattr(self._wrapped_instance, attribute_name)
def __setattr__(self, attribute_name, value):
if attribute_name in self._wrapper_attributes:
self.__dict__[attribute_name] = value
else:
# do stuff here
setattr(self._wrapped_instance, attribute_name, value)
Appendix #
Wrapping Class and Static Methods #
One way is to iterate the attributes of the original class and find the class methods and create attributes in the wrapper that point to the original (using a partial to pass in the cls parameter). This works for both class wrappers and instance wrappers.
if getattr(attribute_value, 'im_class', None) == type(object):
base_function = getattr(attribute_value, 'im_func', None)
if base_function is None:
raise TypeError("Problem with " + str(original_class) +
"." + attribute_name)
wrapper.__dict__[attribute_name] =
functools.partial(base_function, original_class)
Static methods are like module functions in a class’ namespace
if getattr(attribute_value, 'func_name', None) is not None:
base_function = getattr(attribute_value, '__call__', None)
if base_function is None:
raise TypeError("Problem with " + str(original_class) +
"." + attribute_name)
wrapper.__dict__[attribute_name] = base_function
Mocking Decorators #
Decoration happens at compile time. It replaces the object that your name points to. How can one patch a decorator for testing if the test initialization happens after the decorator has done its job and sits securely wrapped around the function one wants to test?
Enter the imp library, specifically its reload function. It allows one to patch the decorator itself (remember, decorators are just callable objects like functions), and reload the module to allow one’s patch to be the decorating object.
As we are doing compilish things at run time, we have a little extra housekeeping to do. If one wants to wrap the original decorator or control the original decorator’s parameters, then one saves a pointer to that original decorator before patching it (sensible when one thinks about it, but not intuitive to me, at least)
This example uses setupModule, but one could use setupClass and addCleanup
import imp
from mock import patch
...
original_decorator = decorator_module.decorator
decorator_mock = None
def mock_decorator(arguments, for, original, decorator):
# do whatever with the arguments or return a completely new wrapper
return original_decorator(arguments, for, original, decorator)
def setupModule(): # magic name that runs once for the module
decorator_mock =
mock.patch('decorator_module.decorator', mock_decorator)
decorator_mock.start()
# Reload the module to apply the new decorator
imp.reload(module_using_decorator)
def teardownModule(): # magic name that runs once for the module
decorator_mock.stop()
# Reload the module to restore the original decorator
imp.reload(module_using_decorator)
See Also #
https://stackoverflow.com/questions/739654/how-to-make-a-chain-of-function-decorators/159448
https://wiki.python.org/moin/PythonDecoratorLibrary
https://python-3-patterns-idioms-test.readthedocs.io/en/latest/PythonDecorators.html
[1]: derived from the latin preposition “ad” (at). Giorgio Stabile, a professor of history in Rome, has traced the symbol back to the Italian Renaissance in a Roman mercantile document signed by Francesco Lapi on 1536-05-04. In Dutch it is called “apestaartje” (little ape-tail), in German “affenschwanz” (ape tail). The French name is “arobase”. In Spain and Portugal it denotes a weight of about 25 pounds, the weight and the symbol are called “arroba”. Italians call it “chiocciola” (snail)
[2]: callable object is a function (which one can call) or a class instance (with its call method)
[3]: “compile-time” error messages (i.e. errors when a module loads) are very disconcerting, unintuitive, and difficult to debug