A complete tutorial on pytest for people who don’t often write Python

Sharpen your tools first

Virtual environment and pytest configuration

First you need to prepare pycharm and install the python environment. (Mac basically comes with python, and versions 3.7.8 and later are available). Then follow the steps below to configure the virtual environment.

This document is a tutorial and will not involve specific business logic and code. But it may be recommended that you use local debugging to understand some of the knowledge. Therefore, it is recommended to establish a learning project. For example, my learning project is called MyXXX (such as MyPython). Before starting the following content, you only need to create a folder and then open it with pycharm.
This article does not cover the syntax of python3 (unless some advanced features are helpful for using pytest), but the syntax of python is very simple, and you can write it if you follow it. If you want to learn systematically, I recommend this tutorial: www.liaoxuefeng.com/wiki /101695…

At this point, the virtual environment has been created, and we can see an orange-red venv folder in the root directory.

Golang will download all packages to the same directory, then use the package name + version to uniquely specify a package, and read go.mod during compilation to obtain the corresponding package + version. Python is different. In the Python package directory, a package will only have one folder, and there will not be multiple versions coexisting. Therefore, the virtual environment helps you create a folder that is only available for this project and downloads third-party packages.

Then we create a requirements.txt file and a pytest.ini file

The requirements.txt file is similar to golang's go.mod file and is used to specify dependencies and versions. However, requirements.txt is a file for people to see, and Python will not check requirements.txt when running.

allure-pytest==2.8.6
allure-python-commons==2.8.6
pytest==6.2.3
pytest-assume==2.2.1
pytest-cov==2.8.1
pytest-cover==3.0.0
pytest-coverage==0.0
pytest-dependency==0.5.1
pytest-forked==1.4.0
pytest-pythonpath==0.7.4
pytest-ordering==0.6
pytest-repeat==0.9.1
pytest-rerunfailures==11.0
pytest-xdist==1.30.0
python-dateutil==2.8.2
retry==0.9.2
retrying==1.3.4

This is the running configuration file of pytest. pytest will automatically find the file in the running directory and read the configuration. There are some preset configurations here, which can save us the need to type configurations on the command line later.

[pytest]
markers =
    p0: 优先级标 marks tests as p0
    p1: 优先级标 marks tests as p1
    p2: 优先级标 marks tests as p2

python_paths = .
addopts = -v -s --alluredir=reports/ecs --junit-xml=reports/result.xml --import-mode=importlib

python_classes = Test*
python_files = test*
python_functions = test*

# 这里配置一下之后，在allure的界面上，可以看到格式化之后的日志
log_cli = True
log_level = INFO
log_format = %(asctime)s.%(msecs)03d[%(levelname)s]%(pathname)s:%(funcName)s:%(lineno)d: %(message)s
log_date_format = %Y-%m-%d %H:%M:%S
# 虽然pytest自己就可以设置日志相关参数，但是有个大问题，在启用pytest-xdist之后，日志无法输出到控制台
# 这是相关的issue：https://github.com/pytest-dev/pytest-xdist/issues/574
# 因此，我们不得不设置两遍：
# 1. 在pytest侧设置一遍，用于格式化allure中的日志
# 2. 自己用logger.conf来设置一遍logger，

In the last two steps, we install the dependencies and check

Currently there should be these things in the root directory (as shown on the left)

Open the command line tool that comes with pycharm and run pip3 install -r requirements.txt (as shown on the right)

Then enter pytest --version on the command line to confirm whether pytest is installed.

pycharm configuration

Open preferences, set as follows, then restart and open the project

If you already have some running configurations, you need to delete them all first.

first test

We create a test_my_heart.py file and edit the contents

import logging


def my_heart_status():
    return "bad"


def test_my_heart():
    """
    测试一下我的心脏是否在跳动
    """
    logging.info("即将检查我的心脏")
    assert my_heart_status() == "healthy"

Entering pytest on the command line gets these outputs:

Isn’t it amazing and simple? I guess you may have these questions:

How does pytest know which functions are tests?

Here, pytest recursively searches for the test function in the directory and runs it. The search rules are as follows:

Find py files starting or ending with test

Look for a class starting with Test in the file found in the first step and without an __init__ method (can be omitted)

Look for a function starting with test_ in the file found in the first step, or a method starting with test in the class found in the second step.

What is assert?

Assert is an assertion. In short, assert is followed by an expression, and this expression returns a bool value. If the bool value is True, the test passes; if it is False, the test fails.

Notice! When writing tests in pytest, be sure to use assert to assert test results instead of actively throwing errors (such as raise error). pytest mocks the implementation of assert, so it can capture the failure of the use case and the context of the failure. Pay attention to the context of pytest's active output in the figure below:

If you actively raise an error, there will not be such a clear context output.

Use assert elegantly

import logging


def my_heart_status():
    return "bad"


def test_my_heart():
    """
    测试一下我的心脏是否在跳动
    """
    logging.info("即将检查我的心脏")
    assert my_heart_status() == "healthy", f"检查我的心脏失败，实际结果为{my_heart_status()}"

Comparison of error messages

Generate test report

You can notice that after just running, there is an additional reports folder in the root directory. This folder stores the test results, but this test result is not human-readable, so we need to generate a human-readable test report. Although the output from the command line can also be viewed, the generated test report is too convenient, so I strongly recommend using the web version of the test report to view the test results.

Let’s install allure first: brew install allure. Then run allure generate reports/ecs -o reports/html --clean. We generated a human-readable html test report. Then open this html (pictured left) and convert the language to Chinese (pictured right)

If you want to know more details about allure, you can explore and refer to this link yourself~: qualitysphere.gitee.io/ext/allure/…

Category, test suite, function, package, these four titles can view the execution status of all test cases. But I recommend viewing "Function" the most, because this title is displayed most clearly and requires the least number of clicks. The following is the difference between several tab pages when facing dozens of test cases.

Check the reason for failure

Back to our own example, we click on the function, click on the case we just failed (as shown on the left), we can see which case failed. Going further, we click on a few boxes to see specific logs and test details.

Mark

docs.pytest.org/en/7.1.x/ho…

Tag

Using the mark capability of pytest, we can tag the case and then run different cases according to the tag. For example, the most basic one is to distinguish between P0 and P1 cases. Let’s modify the previous code:

import logging
import pytest


def my_heart_status():
    return "stopped"


def my_heartbeat():
    return -1


@pytest.mark.p0
def test_my_heart_status():
    """
    测试一下我的心脏是否在跳动
    """
    logging.info("即将检查我的心脏")
    assert my_heart_status() == "running"


@pytest.mark.p1
def test_my_heartbeat():
    """
    测试一下我的心脏跳动是否正常
    """
    logging.info("即将检查我的心脏")
    assert 50 < my_heartbeat() < 180

Then we run pytest -m p0, and we will get the result as shown on the right:

You can see that pytest identified both use cases, but only the use case marked p0 was run, while the use case marked p1 was not run:

In addition, all marks need to be declared in advance. In this case, we have already declared some marks in advance in the pytest.ini file we created in [Sharp tools first] (pictured left). If not declared in advance, pytest will alarm (pictured right):

Not running at the moment

If there is a temporary problem with a use case, or the use case has been written but the function has not been written, we can mark the use case to skip it first (pictured left)

Furthermore, we can conditionally skip certain cases. For example, we first check the current environment to see if it is in the ICU. If so, we will ignore the heartbeat.

Different parameters for the same use case

For example, there is now a query test interface. We need to test whether the interface meets expectations under different query conditions. Starting with the simplest: different page sizes:

As you can see, although we only wrote one use case, using pytest.mark.parametrize, we achieved an effect similar to table-driven testing.

The pytest -k parameter is used below. For usage details, please refer to FQA-How to run only one test?

We can also set multiple parameters within a pytest.mark.parametrize:

We can also arrange and combine multiple parameters by adding multiple pytest.mark.parametrizes:

Fixture

pre-dependency

Fixture (Chinese name is fixture) is the most important function of pytest. pytest can specify pre-dependencies through fixture, and pytest will parse the dependency order and then execute functions one by one in order. If the execution of the pre-dependency fails, subsequent operations will automatically not be executed. The above are the characteristics of fixture. When we use fixtures, we mainly focus on its three functions:

Declare pre-dependencies (can be seen in the test report instead of looking at the code).

Cache pre-dependencies in the same scope and allow multiple use cases to share a pre-dependency (by reusing resources, it improves running speed and reduces resource usage).

Clean up resources after the case runs. (convenient~)

Let's take a simple example first. We change the previously actively called my_heart_status and my_heartbeat into pre-dependencies. code show as below:

import logging
import pytest


@pytest.fixture()
def my_heart_status():
    return "running"


@pytest.fixture()
def my_heartbeat():
    return 80


@pytest.mark.p0
def test_my_heart_status(my_heart_status):
    """
    测试一下我的心脏是否在跳动
    """
    logging.info("即将检查我的心脏")
    assert my_heart_status == "running"


@pytest.mark.p1
def test_my_heartbeat(my_heartbeat):
    """
    测试一下我的心脏跳动是否正常
    """
    logging.info("即将检查我的心脏")
    assert 50 < my_heartbeat < 180

To explain, in the above code, by adding the @pytest.fixture decorator to the function, pytest can collect all fixtures. Then pytest will analyze the input parameters of each test case, find the corresponding fixture by name, execute the fixture, and pass the fixture results to the test case.

At the same time, after we use the fixture, we can also see the pre-dependency of the test case in the test report. If the pre-dependency fails and the use case fails, it can also be clearly seen in the test:

conftest

Just now we wrote the test cases and fixtures in the same py file, then these fixtures can only be used in this py file. If you want the fixture to be used by multiple py files, you need to write the fixture to the conftest.py file. pytest will recursively search for a file named conftest.py in the directory and make these fixtures available in its subdirectories.

For example: there are three levels of directories, two conftest.py files and three test.py files.

- conftest.py # 有fixtureA
- test_x.py
- TestOtherDir  # 这是一个文件夹
  - conftest.py. # 有fixtureB
  - test_y.py
  - TestZDir  # 这是一个文件夹
    - test_z.py

In the above example, fixtureA can be used by three test.py files. fixtureB can only be used by test_y.py and test_z.py.

Let's create a new conftest.py file and put both fixtures in it.

Scope & caching and shared fixtures

A great use of fixtures is that multiple cases in the same scope share the same pre-dependency. Let's change the above example and assume that all the information about heart can be obtained through one method.

# 在 conftest.py中
import logging
import pytest
class Heart:
    def __init__(self, status, beat):
        self.status = status
        self.beat = beat


@pytest.fixture()
def my_heart():
    logging.info("获取heart信息")
    return Heart("running", 80)


# 在 test_heart.py中
@pytest.mark.p0
def test_my_heart_status(my_heart):
    """
    测试一下我的心脏是否在跳动
    """
    logging.info("检查心脏状态")
    assert my_heart.status == "running"


@pytest.mark.p1
def test_my_heartbeat(my_heart):
    """
    测试一下我的心脏跳动是否正常
    """
    logging.info("检查心跳")
    assert 50 < my_heart.beat < 180

Then as shown below, the my_heart function is called twice. This is because the default scope of each fixture is function level, that is, each test case will execute the fixture again.

There are four types of scopes:

function: Run this fixture once for each test case (default)

class: All methods in the class only run the fixture once (there can be multiple test methods in a class)

module: A .py file only executes the fixture once

session: Each time the pytest command is called, it is only executed once (across multiple py files and multiple folders)

Let's change its scope to module and see the execution results. You can see that this fixture will only be executed once.

If we put two test cases under two py files, they will be executed twice.

If we put two test cases under two py files, but the scope is set to session, they will only be executed once.

For more examples, please refer to this document: blog.csdn.net/Tangerine02…

Nested

In addition to the case that can set pre-dependencies through fixtures, the fixture itself can also set pre-dependencies~

# 在 conftest.py中
import logging
import pytest

class Heart:
    def __init__(self, status, beat):
        self.status = status
        self.beat = beat

@pytest.fixture()
def prepare():
    return "abc"

@pytest.fixture()
def my_heart(prepare):
    logging.info("获取heart信息" + prepare)
    return Heart("running", 80)

Parameters can be passed to Fixture

We can obtain more intuitive test reports and pre-dependency cache by converting function calls into fixture dependencies. But parameters can be passed when calling the function, so how do we pass parameters to the fixture?

Continuing with the above example, we need to examine the hearts of different ages and genders:

In the fixture, you need to set the first parameter to request, and then you can get all the parameters passed by the test case through request.param.

In the test case, the usage is basically the same as pytest.mark.parametrize, just add indirect=True.

# 在 conftest.py 中
import pytest

class Heart:
    def __init__(self, status, sex, beat):
        self.status = status
        self.sex = sex
        self.beat = beat

@pytest.fixture(scope="module")
def my_heart(request):
    logging.info("获取heart信息")
    return Heart("running", request.param["sex"], request.param["age"] * 5)

# 在 test_my_heart.py 中
import logging
import pytest

@pytest.mark.parametrize("my_heart", [{
    
    "sex": "female", "age": 13}, {
    
    "sex": "male", "age": 15}], indirect=True) # 注意这里！
def test_my_heartbeat(my_heart):
    """
    测试一下我的心脏跳动是否正常
    """
    logging.info("检查心跳")
    assert 50 < my_heart.beat < 180

The scope of the passable Fixture

As mentioned earlier, fixtures in the same scope will only be executed once. So if we pass different parameters to the fixture, what will happen to the scope?

Following the above example, the code of conftest.py remains unchanged, and we modify the code of test_my_heart.py:

import logging
import pytest

user1 = {
    
    "sex": "male", "age": 15}
user2 = {
    
    "sex": "female", "age": 13}

@pytest.mark.parametrize("my_heart", [user1], indirect=True)
def test_my_heart_status(my_heart):
    """
    测试一下我的心脏是否在跳动
    """
    logging.info("检查心脏状态")
    assert my_heart.status == "running"


@pytest.mark.parametrize("my_heart", [user2], indirect=True)
def test_my_heartbeat(my_heart):
    """
    测试一下我的心脏跳动是否正常
    """
    logging.info("检查心跳")
    assert 50 < my_heart.beat < 180

If we run this again, we will find that the fixture will be executed twice. And if we change user2 to user1, it will only be run once . Therefore, we can conclude that for a parameter-passable fixture, the parameter + fixture together constitute the only pre-dependency.

There is a small question, how does Python determine whether two values ​​are equal? For Python, there are two operators to determine whether two values ​​are equal, == and is. == is used to determine whether the values ​​are equal; is is used to determine whether the addresses are equal. For Pytest fixtures, it uses is to determine whether they are equal. Therefore, two identical dicts will also cause the fixture to be executed twice. Only by declaring a global variable and then both using this global variable will the fixture be executed only once.

For more information about the difference between Python == and is, you can refer to this discussion: www.zhihu.com/question/20…

Although @pytest.mark.parametrize can set the scope (scope) and can override the scope of the original fixture. However, this is not recommended and will not be discussed here.

Resource cleanup

For example, if we want to observe Heart, we must first connect a listener to Heart, and then uninstall the listener after our test is completed. The first recommended method is to use yield, which is simpler and clearer.

@pytest.fixture(scope="module")
def my_heart(request):
    logging.info("连接监听器")
    logging.info("获取heart信息")

    yield Heart("running", request.param["sex"], request.param["age"] * 5)

    logging.info("卸载监听器")

The second recommendation is to use the fixture's request parameter, request.addfinalizer(). Let's modify the previous my_heart fixture.

@pytest.fixture(scope="module")
def my_heart(request):
    logging.info("连接监听器")

    def teardown():  # 这个函数的名字是随意的，也可以叫别的
        logging.info("卸载监听器")

    request.addfinalizer(teardown) # 这一行，注册一个fixture生命周期结束后运行的函数

    logging.info("获取heart信息")
    return Heart("running", request.param["sex"], request.param["age"] * 5)

There are two points to note:

The first is that there are subtle logical differences between the two methods. When the code segment in the red box in the picture below reports an error (raise error), using addfinalizer will execute the logic of teardown (provided that addfinalizer runs before the error reporting code segment), while using yield No.

The second thing to note is that when writing teardown, you must consider the failure situation in the red box. For example, there is a fixture that creates resources and has built-in teardown logic for deleting resources. When the resource creation fails, teardown may report a NotFound error when deleting the resource. At this time, the error should not be raised.

Execution order and cleanup order

The first thing to note is that due to concurrent testing and different test sets each time, the order in which pytest runs test cases is always irregular. Therefore, it is best not to have any order dependency between cases. If cases depend on the same fixture, it is best to restore the fixture to its original state after each execution.

Secondly, the execution order and cleaning order of the fixture itself are traceable, and they are executed in the following order.

First execute the fixture with autouse=True

Secondly, the fixtures that a certain case depends on are executed. If a case depends on multiple fixtures, they are executed in order, from left to right.

Secondly, the fixtures that the fixture depends on are executed. If a fixture depends on multiple fixtures, they are executed in order from left to right, and the dependencies are parsed recursively according to DFS (depth first).

As for the cleaning operation, the above order will be reversed, that is, the fixture that is executed first will be cleaned last.

Let's take a look at a new py file. The left side shows the rules of DFS, and the right side shows the impact of autouse. For this case, you can modify the code and execute it yourself to experience the dependencies.

# test_order.py
import pytest

@pytest.fixture()
def a():
    print("准备AAA")
    yield "a"
    print("清理AAA")

@pytest.fixture()
def b():
    print("准备BBB")
    yield "b"
    print("清理BBB")

@pytest.fixture()
def c(a, b):
    print("准备CCC")
    yield "c"
    print("清理CCC")

def test_order_1(c):
    pass

准备AAA
准备BBB
准备CCC
PASSED
清理CCC
清理BBB
清理AAA
# test_order.py
import pytest

@pytest.fixture()
def a():
    print("准备AAA")
    yield "a"
    print("清理AAA")

@pytest.fixture(autouse=True)
def b():
    print("准备BBB")
    yield "b"
    print("清理BBB")

@pytest.fixture()
def c(a, b):
    print("准备CCC")
    yield "c"
    print("清理CCC")

def test_order_1(c):
    pass

准备BBB
准备AAA
准备CCC
PASSED
清理CCC
清理AAA
清理BBB

FAQ

How to run only one test?

There are three ways

One is to directly click on the arrow of pycharm. If there are any problems after clicking, please refer to the chapter [Sharpen your tools first] to set up pycharm.

One is to call pytest -k XXX directly on the command line, where XXX is your test function or test class. If your test function name overlaps with the test function name elsewhere, you can add the path, such as pytest FFF -k XXX. Where FFF is the path of the use case you want to run, it can be precise to a folder or file.

One is to directly pytest XXX without using -k. For the format of XXX, please refer to the figure below.

A certain fixture cannot be found

Python functions can easily be overwritten by variables with the same name or other functions. There is a very weird example:

# 在 conftest.py 中
from a_fixture import *
from a import *

# 在 a_fixture.py 中
@pytest.fixture
def aaa():
    pass

# 在 a.py 中
aaa = None

In the above example, running pytest --fixtures cannot find the fixture aaa. But if you replace the order of the two imports, you can see it again.

Only run the case that failed last time

After running, the running results will be saved in the reports directory. We can run pytest --lf to only run the last failed case.

allure's report is confusing

Because the results of each run will be saved in the reports directory, there will be more and more reports, making the reports generated by allure more and more complex. Therefore, after a while, you can delete all the reports directory so that the reports will be clear.

What rules are used to collect log files in a single case?

Generally speaking, what a case executes (including pre-dependencies, its own logic, and resource cleanup logs) will be in the log of a case.

However, if multiple cases rely on the same fixture, the fixture's creation log will only appear in the log of the first executed case, and subsequent cases will not have this part of the log.

What do the yellow items in the allure report mean?

It means that the code panics (raise error). This is usually caused by the code not handling errors.

The pytest multi-threaded plug-in pytest-parallel is not compatible with the test report plug-in all-pytest

Version pytest-multithreading-allure-1.0.5, use: requirement.txt

Solution: testerhome.com/topics/3274…

Related questions: github.com/allure-fram…

Reference documentation

docs.pytest.org/en/7.1.x/in…

blog.csdn.net/tangerine02…

blog.csdn.net/totorobig/a…

qualitysphere.gitee.io/ext/allure/

Finally: The complete software testing video tutorial below has been compiled and uploaded. Friends who need it can get it by themselves.【保证100%免费】

Software Testing Interview Document

We must study to find a high-paying job. The following interview questions are the latest interview materials from first-tier Internet companies such as Alibaba, Tencent, Byte, etc., and some Byte bosses have given authoritative answers. After finishing this set I believe everyone can find a satisfactory job based on the interview information.