Python's advanced source code analysis: how to turn a class method into multiple methods?
Previous article " How to implement parameterized testing in Python?" In ", I mentioned several libraries that implement parameterized testing in Python, and left a question:
How do they turn a method into multiple methods, and bind each method with the corresponding parameters?
Let's distill it again, the original question is equal to: In a class, how to use a decorator to turn a class method into multiple class methods (or produce a similar effect)?
# 带有一个方法的测试类
class TestClass:
def test_func(self):
pass
# 使用装饰器，生成多个类方法
class TestClass:
def test_func1(self):
pass
def test_func2(self):
pass
def test_func3(self):
pass
The essence of decorators in Python is to move from one to another, replacing the decorated method with a new method. In the process of implementing parameterization, what means / secret weapon did the several libraries we introduced use?
1. How does ddt achieve parameterization?
First review the writing method of the ddt library in the previous article:
import unittest
from ddt import ddt,data,unpack
@ddt
class MyTest(unittest.TestCase):
@data((3, 1), (-1, 0), (1.2, 1.0))
@unpack
def test(self, first, second):
pass
ddt can provide 4 decorators: 1 @ddt added to the class, and 3 @data, @unpack and @file_data added to the class method (not mentioned above).
First look at the role of the three decorators added to the class method:
# ddt 版本（win）：1.2.1
def data(*values):
global index_len
index_len = len(str(len(values)))
return idata(values)
def idata(iterable):
def wrapper(func):
setattr(func, DATA_ATTR, iterable)
return func
return wrapper
def unpack(func):
setattr(func, UNPACK_ATTR, True)
return func
def file_data(value):
def wrapper(func):
setattr(func, FILE_ATTR, value)
return func
return wrapper
Their common role is to add attributes to setattr () on class methods. As for when these attributes are used? Let's take a look at the @ddt decorator source code added to the class:
The first layer of for loop traverses all the class methods, and then two branches of if / elif, corresponding to DATA_ATTR / FILE_ATTR, that is, two sources of corresponding parameters: data (@data) and file (@file_data).
The elif branch has the logic to parse the file, and then it is similar to the data processing, so we skip it, mainly looking at the previous if branch. The logic of this part is very clear, the main tasks completed are as follows:
- Parameter key value pairs of traversal class methods
- Create a new method name based on the original method and parameter pair
- Get the document string of the original method
- Unpack parameters of tuple and list types
- Add a new test method on the test class, and bind parameters and document strings
Analyzing the source code, we can see that the three decorators @data, @unpack and @file_data mainly set attributes and pass parameters, and the @ddt decorator is the core processing logic.
This scheme of distributing decorators (added separately to classes and class methods) and then combining them is not elegant. Why can't they be used together? Later, we will analyze its hidden meaning, first press the table to see what other implementations are like?
2. How to realize parameterization?
First review the writing of the parameterized library in the previous article:
import unittest
from parameterized import parameterized
class MyTest(unittest.TestCase):
@parameterized.expand([(3,1), (-1,0), (1.5,1.0)])
def test_values(self, first, second):
self.assertTrue(first > second)
It provides a decorator class @parameterized, the source code is as follows (version 0.7.1), mainly to do some initial verification and parameter analysis, not the focus of our attention, skip it.
We mainly focus on the expand () method of this decorator class, which is written in the documentation comments:
A "brute force" method of parameterizing test cases. Creates new test cases and injects them into the namespace that the wrapped function is being defined in. Useful for parameterizing tests in subclasses of 'UnitTest', where Nose test generators don't work.
The two key actions are: "creates new test cases" and "inject them into the namespace ...".
Regarding the first point, it is similar to ddt, except for some differences in naming styles, as well as different parameter parsing and binding, which is not worthy of too much attention.
The most different is how to make the new test method effective?
parameterized uses an "injection" approach:
inspect Is a powerful standard library, used here to obtain information about the program call stack. The purpose of the first three lines of code is to take out f_locals, which means "local namespace seen by this frame", where f_locals refers to the local namespace of the class.
Speaking of local namespaces, you may think of locals (), but we have mentioned in previous articles "read and write problems of locals () and globals ()". The code uses f_locals.
3. How does pytest achieve parameterization?
First look at the writing in the previous article according to the convention:
import pytest
@pytest.mark.parametrize("first,second", [(3,1), (-1,0), (1.5,1.0)])
def test_values(first, second):
assert(first > second)
First see "mark", some tags are built into pytest, such as parametrize, timeout, skipif, xfail, tryfirst, trylast, etc., also support user-defined tags, you can set execution conditions, group filter execution, and modify the original test behavior and many more.
The usage is also very simple, however, the source code can be much more complicated. We only focus on parametrize here, first look at the core piece of code:
According to the incoming parameter pairs, it copies the calling information of the original test method and stores it in the list to be called. Unlike the two libraries analyzed earlier, it does not create new test methods here, but reuses existing methods. Look up the Metafunc class to which the parametrize () belongs, and you can track where the _calls list is used:
Finally, it is executed in the Function class:
What's interesting is that here we can see a few lines of god notes ...
Reading (superficially) the source code of pytest is really asking for trouble ... However, it can be roughly seen that when it is parameterized, it uses the generator's scheme, and traversing a parameter calls the test method once. In the previous ddt and parameterized, all parameters are parsed at once, and n new test methods are generated, and then handed over to the test framework for scheduling.
In comparison, the ideas of the first two libraries are very clear, and because their design is purely for parameterization, unlike pytest, there are no tags and too many abstract designs, so it is easier to read and understand. The first two libraries take advantage of Python's dynamic features, setting class attributes or injecting local namespaces, and pytest seems to be a clumsy way of borrowing ideas from what static language.
4. The final summary
Back to the question in the title "How do I change a method into multiple methods?" In addition to parameterized testing, I wonder what other scenarios will have this appeal? Welcome to leave a message to discuss.
This article analyzes the implementation of the decorator of the three test libraries. By reading the source code, we can find that each has its own advantages. This discovery itself is quite interesting. When using decorators, they are not very different on the surface, but the details of the real kung fu are hidden underneath.
The significance of source code analysis is to explore why it is. What can readers gain in this exploration journey? Let's talk together! (PS: Send "learning group" in the background of the "Python cat" public account to obtain the password of the group.)
