Python introduced decorators very early-in PEP-318, as a mechanism to simplify the way functions and methods are defined, these functions and methods must be modified after the initial definition.
One of the original motivations for this is to use functions such as classmethod and staticmethod to transform the original definition of the method, but they require an extra line of code to modify the initial definition of the function.
Generally speaking, every time a conversion must be applied to a function, we must call it with the modifier function and then reassign it to the name of the function when it was originally defined.
For example, suppose there is a function called original, on which there is a function (called modifier) that changes the behavior of original, then we must write:
def original(...):
...
original = modifier(original)Please note how we changed the function and reassigned it to the same name. This is confusing, error-prone (assuming someone forgets to reassign the function, or reassigns the function, but not in the line after the function definition, but farther away), and it is troublesome. For this reason, the Python language has added some syntax support.
The previous example can be rewritten as follows:
@modifier
def original(...):
...This means that the decorator is just syntactic sugar, and the content after the decorator is called is used as the first parameter of the decorator itself, and the result will be the content returned by the decorator.
In order to be consistent with Python's terminology, in our example modifier is called a decorator, and original is a decorated function, usually also called a wrapper object.
Although this feature was originally thought to be used for methods and functions, the actual syntax allows it to decorate any type of object, so we will study decorators applied to functions, methods, generators, and classes.
The last thing to note is that although the name of the decorator is correct (after all, the decorator is actually changing, extending, or processing the wrapper function), don't confuse it with the decorator design pattern.
5.1.1 Decorator function
A function is probably the simplest representation of a Python object that can be decorated. We can use decorators on functions to apply various logic-we can verify parameters, check preconditions, completely change behavior, modify its signature, cache results (create a memory version of the original function), etc.
For example, we will create a basic decorator that implements the retry mechanism to control a specific domain-level exception and retry a certain number of times:
# decorator_function_1.py
class ControlledException(Exception):
"""A generic exception on the program's domain."""
def retry(operation):
@wraps(operation)
def wrapped(*args, **kwargs):
last_raised = None
RETRIES_LIMIT = 3
for _ in range(RETRIES_LIMIT):
try:
return operation(*args, **kwargs)
except ControlledException as e:
logger.info("retrying %s", operation.__qualname__)
last_raised = e
raise last_raised
return wrappedThe use of @wrap can be ignored for now, as it will be discussed in another section. The use of "_" in a for loop means that this number is assigned to a variable that we are not currently interested in because it is not used in a for loop (in Python, it is common to name the ignored value "_" idiomatic usage).
The retry decorator does not receive any parameters, so it can be easily applied to any function, as shown below:
@retry
def run_operation(task):
"""Run a particular task, simulating some failures on its execution."""
return task.run()As explained at the beginning, the definition of @retry above run_operation is just syntactic sugar provided by Python for the actual execution of run_operation = retry(run_operation).
In this limited example, we can see how to use the decorator to create a general retry operation, under certain certain conditions (in this example, expressed as an exception that may be related to timeout), the operation will allow more The decorated code is called for the second time.
5.1.2 Decoration
Classes can also be decorated (PEP-3129), and the decoration method is the same as that of the syntax function. The only difference is that when writing code for the decorator, we must take into account that the received is a class, not a function.
Some practitioners may think that decorating a class is quite complicated. Such a scenario may harm readability, because we will declare some properties and methods in the class, but behind the scenes, the decorator may apply some changes to render A completely different class.
This assessment is correct, but only in the case of serious abuse of decorative technology. Objectively speaking, this is no different from decorative functions; after all, classes and functions are just a type of object in the Python ecosystem. In Section 5.4, we will re-examine the advantages and disadvantages of this problem, but here we only explore the advantages of decorators, especially those applicable to classes.
(1) Reuse code and all the benefits of the DRY principle. An effective case for class decorators is to force multiple classes to conform to a specific interface or standard (by checking only once in the decorator that will be applied to multiple classes).
(2) It is possible to create smaller or simpler classes-these classes will be enhanced by decorators later.
(3) If you use a decorator, the conversion logic that needs to be applied to a specific class will be easier to maintain, instead of using more complex (usually discouraged) methods, such as metaclasses.
Among all the possible applications of decorators, we will explore a simple example to understand where decorators can be used. Remember, this is not the only application type for class decorators, and the code given can have many other solutions. All these solutions have advantages and disadvantages, and the reason why decorators are chosen is to illustrate their usefulness.
Looking back at the event system used to monitor the platform, it is now necessary to transform the data of each event and send it to the external system. However, when choosing how to send data, each type of event may have its own peculiarities.
In particular, login events may contain sensitive information, such as credentials we wish to hide. Fields in other fields such as timestamp may also need some conversion, because we want to display them in a specific format. The first attempt to meet these requirements is as simple as having a class mapped to each specific event and knowing how to serialize it:
class LoginEventSerializer:
def __init__(self, event):
self.event = event
def serialize(self) -> dict:
return {
"username": self.event.username,
"password": "**redacted**",
"ip": self.event.ip,
"timestamp": self.event.timestamp.strftime("%Y-%m-%d
%H:%M"),
}
class LoginEvent:
SERIALIZER = LoginEventSerializer
def __init__(self, username, password, ip, timestamp):
self.username = username
self.password = password
self.ip = ip
self.timestamp = timestamp
def serialize(self) -> dict:
return self.SERIALIZER(self).serialize()Here, we declare a class. This class will be directly mapped to the login event, which contains some of its logic-hide the password field, and format the timestamp as needed.
Although this is feasible and may seem like a good choice at first, over time, if you want to expand the system, you will find some problems.
(1) Too many classes . As the number of events increases, the number of serialized classes will increase by the same order of magnitude because they are mapped one by one.
(2) The solution is not flexible enough . If we need to reuse some components (for example, we need to hide the password in another type of event that also has similar requirements), we have to extract it into a function, but also call it from multiple classes, which means We did not reuse so much code.
(3) Template documents . The serialize() method must appear in all event classes and call the same code at the same time. Although we can extract it into another class (create a mixin), this does not seem to use inheritance well.
Another solution is to be able to construct an object dynamically: given a set of filters (transition functions) and an event instance, the object can serialize it by applying the filter to its fields. Then, we only need to define a function to convert each field type, and create a serializer by combining these functions.
Once we have this object, we can decorate the class to add the serialize() method. This method will only call these serialized objects themselves:
def hide_field(field) -> str:
return "**redacted**"
def format_time(field_timestamp: datetime) -> str:
return field_timestamp.strftime("%Y-%m-%d %H:%M")
def show_original(event_field):
return event_field
class EventSerializer:
def __init__(self, serialization_fields: dict) -> None:
self.serialization_fields = serialization_fields
def serialize(self, event) -> dict:
return {
field: transformation(getattr(event, field))
for field, transformation in
self.serialization_fields.items()
}
class Serialization:
def __init__(self, **transformations):
self.serializer = EventSerializer(transformations)
def __call__(self, event_class):
def serialize_method(event_instance):
return self.serializer.serialize(event_instance)
event_class.serialize = serialize_method
return event_class
@Serialization(
username=show_original,
password=hide_field,
ip=show_original,
timestamp=format_time,
)
class LoginEvent:
def __init__(self, username, password, ip, timestamp):
self.username = username
self.password = password
self.ip = ip
self.timestamp = timestampNote that the decorator makes it easier for you to know how to handle each field without having to look at the code of another class. Just by reading the parameters passed to the class decorator, we know that the username and IP address will remain the same, the password will be hidden, and the timestamp will be formatted.
Now, the code of the class does not need to define the serialize() method, nor does it need to extend from the mixin class that implements it, because these will be added by the decorator. In fact, this may be the only reason to create a class decorator, because if it is not the case, the serialized object may be a class attribute of LoginEvent, but it changes the class by adding a new method to the class, which makes the class created Decorator becomes impossible.
We can also use another class decorator to implement the logic of the init method by defining the properties of the class, but this is beyond the scope of this example.
By using this class decorator (PEP-557) in Python 3.7+, the previous example can be rewritten in a more concise way without using the boilerplate code of init, as shown below:
from dataclasses import dataclass
from datetime import datetime
@Serialization(
username=show_original,
password=hide_field,
ip=show_original,
timestamp=format_time,
)
@dataclass
class LoginEvent:
username: str
password: str
ip: str
timestamp: datetime5.1.3 Other types of decorators
Now that we know the actual meaning of the @ grammar of the decorator, we can draw the conclusion that it is not only functions, methods, or classes that can be decorated; in fact, anything that can be defined (such as generators, coroutines, or even Decorated objects) can be decorated, which means that decorators can be stacked.
The previous example shows how to link decorators. We first define the class, and then apply @dataclass to the class-it transforms the class into a data class, acting as a container for these attributes. After that, the logic is applied to the class through @Serialization, thereby generating a new class, which adds a new serialize() method.
Another good use of decorators is for generators that should be used as coroutines. We will explore the details of generators and coroutines in Chapter 7. The main idea is that before sending any data to a newly created generator, the latter must be advanced to the next yield statement by calling next(). This is a manual process that every user must remember, so it's easy to make mistakes. We can easily create a decorator to receive the generator as a parameter, call next(), and then return to the generator.
5.1.4 Passing parameters to the decorator
So far, we have considered decorators as a powerful tool in Python. If we can pass parameters to the decorator to make its logic more abstract, its function may be more powerful.
There are several ways to implement decorators to receive parameters, but we will only discuss the most common methods in the following. The first method is to create the decorator as a nested function with a new level of indirection, so that everything in the decorator goes one level deep. The second method is to use a class for the decorator.
Generally, the second method is more readable, because it is easier from an object point of view than 3 or more nested functions that use closures. However, for the sake of completeness, we will discuss these two methods so that you can choose the method that best suits the current problem.
1. Decorator with nested functions
Roughly speaking, the basic idea of decorators is to create a function that returns a function (usually called a higher-order function). The internal function defined in the main body of the decorator will be the actually called function.
Now, if you want to pass parameters to it, you need another level of indirection. The first function will receive parameters, in this function, we will define a new function (it will be a decorator), and this new function will define another new function, the function returned by the decorating process. This means that we will have at least 3 levels of nested functions.
If you don't understand the meaning of the above content so far, don't worry, you will understand after reviewing the examples given below.
The first example is that the decorator implements a retry function on some functions. This is a good idea, but there is a problem: the implementation does not allow specifying the number of retries, only a fixed number of times is allowed in the decorator.
Now, we want to be able to indicate how many retries each example has, and maybe even add a default value for this parameter. In order to achieve this function, we need to use another layer of nested functions-first for the parameters, and then for the decorator itself.
This is because of the following code:
@retry(arg1, arg2,... )The decorator must be returned because the @ syntax will apply the result of the calculation to the object to be decorated. Semantically speaking, it can be translated into the following content:
<original_function> = retry(arg1, arg2, ....)(<original_function>)In addition to the required number of retries, we can also specify the type of exception we wish to control. The new version of the code that supports the new requirements may look like this:
RETRIES_LIMIT = 3
def with_retry(retries_limit=RETRIES_LIMIT, allowed_exceptions=None):
allowed_exceptions = allowed_exceptions or (ControlledException,)
def retry(operation):
@wraps(operation)
def wrapped(*args, **kwargs):
last_raised = None
for _ in range(retries_limit):
try:
return operation(*args, **kwargs)
except allowed_exceptions as e:
logger.info("retrying %s due to %s", operation, e)
last_raised = e
raise last_raised
return wrapped
return retryHere are some examples of how this decorator can be applied to functions, showing the different options it receives:
# decorator_parametrized_1.py
@with_retry()
def run_operation(task):
return task.run()
@with_retry(retries_limit=5)
def run_with_custom_retries_limit(task):
return task.run()
@with_retry(allowed_exceptions=(AttributeError,))
def run_with_custom_exceptions(task):
return task.run()
@with_retry(
retries_limit=4, allowed_exceptions=(ZeroDivisionError, AttributeError)
)
def run_with_custom_parameters(task):
return task.run()2. Decorator object
The previous example requires 3 levels of nested functions. First, this will be a parameter for receiving the decorator we want to use. In this function, the rest of the functions are closures that use these parameters and decorator logic.
A more concise way is to use a class to define the decorator. In this case, we can pass parameters in the __init__ method, and then implement the decorator logic on the magic method named __call__.
The decorator code is as follows:
class WithRetry:
def __init__(self, retries_limit=RETRIES_LIMIT,
allowed_exceptions=None):
self.retries_limit = retries_limit
self.allowed_exceptions = allowed_exceptions or
(ControlledException,)
def __call__(self, operation):
@wraps(operation)
def wrapped(*args, **kwargs):
last_raised = None
for _ in range(self.retries_limit):
try:
return operation(*args, **kwargs)
except self.allowed_exceptions as e:
logger.info("retrying %s due to %s", operation, e)
last_raised = e
raise last_raised
return wrappedThis decorator can be applied as before, like this:
@WithRetry(retries_limit=5)
def run_with_custom_retries_limit(task):
return task.run()It is important to note how the Python syntax works here. First, we create the object, so that before applying the @ operation, the object has been created, and its parameters are passed to it, initialize the object with these parameters, as defined in the init method. After this, we will call the @ operation so that the object will wrap a function named run_with_custom_reries_limit, which means it will be passed to the magic method call.
In the magic method of call, we define the logic of the decorator, just as we usually do—wrap the original function and return a new function that contains the required logic.
5.1.5 Make full use of decorators
This section introduces some common patterns that make the most of decorators. It is a very good choice to use decorators in some common scenarios.
There are countless decorators that can be used in applications, and here are just a few of the most common or relevant ones.
(1) Conversion parameters . Change the signature of the function to expose a better API, while encapsulating detailed information on how to process and transform parameters.
(2) Tracking code . Record the execution of the function and its parameters.
(3) Verify the parameters .
(4) Realize the retry operation .
(5) Simplify the class by moving some (repetitive) logic into the decorator .
Next, discuss the first two applications in detail.
1. Conversion parameters
As mentioned earlier, decorators can be used to verify parameters (even to force some pre-conditions or post-conditions under the concept of DbC), so you may have learned that this is a common way to use decorators when processing or manipulating parameters .
In particular, in some cases, we will find ourselves repeatedly creating similar objects or applying similar transformations, and we want to abstract away these transformations. Most of the time, we can achieve this by simply using decorators.
2. Tracking code
When discussing tracking in this section , we will mention some more general content, which is related to the execution of the function to be monitored, specifically:
(1) Actually track the execution of the function (for example, by recording the line executed by the function);
(2) Some indicators of the monitoring function (such as CPU usage or memory usage);
(3) The running time of the measurement function;
(4) The log when the function is called, and the parameters passed to it.
We will analyze a simple decorator example in Section 5.2, which records the execution of the function, including the function name and running time.
This article is excerpted from "Writing Clean Python Code"
This book introduces the main practices and principles of Python software engineering, and aims to help readers write more maintainable and cleaner code. The book consists of 10 chapters: Chapter 1 introduces the basic knowledge of the Python language and the main tools needed to build a Python development environment; Chapter 2 describes Python style code and introduces the first idiom in Python; Chapter 3 summarizes the good code General features, review the general principles in software engineering; Chapter 4 introduces a set of object-oriented software design principles, namely SOLID principles; Chapter 5 introduces decorators, which are one of the most important features of Python; Chapter 6 discusses descriptions Characters, introduces how to obtain more information from objects through descriptors; Chapters 7 and 8 introduce generators and related content of unit testing and refactoring; Chapter 9 reviews the most common design patterns in Python; Chapter 10 Chapter once again emphasizes that code cleanliness is the basis for a good architecture.
This book is suitable for all Python programming enthusiasts, people interested in programming, and other software engineering practitioners who want to learn more about Python.