2023Python's hottest third-party open source testing framework - pytest

1. Introduction

This article is about the hottest third-party unit testing framework in the Python world: pytest.
It has the following main features:

• assert Output detailed information when the assertion fails (no need to remember self.assert*the name )
• Automatic discovery of test modules and functions
• Modular fixtures to manage various test resources
• unittestFully compatible with , nosebasically compatible with
• Very rich plug-in system, with more than 315 third-party plug-ins, the community is prosperous

Like the previous introduction unittestand nose, we will introduce the features pytestof .

 

2. Use case writing

noseSimilarly , pytesttest cases in the form of functions and test classes are supported. The biggest difference is that you can use assertthe statement to make assertions without worrying about the lack of detailed context information it will generate in ornose .unittest

For example, in the following test example, the assertion test_upperin failed:

import pytest

def test_upper():
    assert 'foo'.upper() == 'FOO1'

class TestClass:
    def test_one(self):
        x = "this"
        assert "h" in x

    def test_two(self):
        x = "hello"
        with pytest.raises(TypeError):
            x + []

And when using pytestto  execute a test case, it will output detailed (and multi-colored) context information:

=================================== test session starts ===================================
platform darwin -- Python 3.7.1, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /Users/prodesire/projects/tests, inifile:
plugins: cov-2.6.0
collected 3 items

test.py F..                                                                         [100%]

======================================== FAILURES =========================================
_______________________________________ test_upper ________________________________________

    def test_upper():
>       assert 'foo'.upper() == 'FOO1'
E       AssertionError: assert 'FOO' == 'FOO1'
E         - FOO
E         + FOO1
E         ?    +

test.py:4: AssertionError
=========================== 1 failed, 2 passed in 0.08 seconds ============================

It is not difficult to see that pytestboth the test code context and the value of the measured variable are output. Compared with noseand unittest, pytestallow users to write test cases in a simpler way, and get a richer and more friendly test result.

3. Use case discovery and execution

unittestnoseBoth supported use case discovery and execution capabilities pytestare supported. pytestAutomatic (recursive) discovery of supported use cases:

• By default, test_*.pyall *_test.pytest case files that match or in the current directory, test functions teststarting or test methods starting with Testin test classes starting testwith
• use pytestcommand
• nose2Same idea as , by specifying specific parameters in the configuration file, the name patterns of use case files, classes and functions can be configured (fuzzy matching)

pytestExecuting specific use cases is also supported:

• Specify test file path
  • pytest /path/to/test/file.py
• Specify test class
  • pytest /path/to/test/file.py:TestCase
• specify test method
  • pytest another.test::TestClass::test_method
• specify test function
  • pytest /path/to/test/file.py:test_function

Fourth, the test fixture (Fixtures)

pytestThe style of the test fixtureunittest is very different from that of , , noseand . nose2It can not only setUprealize tearDownthe test pre-test and cleanup logic of and , but also has many other powerful functions.

4.1 Declaration and use

pytestA test fixture in is more like a test resource, you just define a fixture and then use it directly in a use case. Thanks pytestto the dependency injection mechanism of , you don't need from xx import xxto display the import through the form, you only need to specify the parameters with the same name in the parameters of the test function, for example:

import pytest

@pytest.fixture
def smtp_connection():
    import smtplib

    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)

def test_ehlo(smtp_connection):
    response, msg = smtp_connection.ehlo()
    assert response == 250

In the above example, a test fixture is defined smtp_connection, and a parameter with the same name is defined in the test function test_ehlosignature , then pytestthe framework will automatically inject the variable.

4.2 Sharing

pytestIn , the same test fixture can be shared by multiple test cases in multiple test files. Just define conftest.pythe file , and write the definition of test fixtures in this file, then all test cases of all modules (Module) in the package can use conftest.pythe test fixtures defined in .

For example, if a test fixture is test_1/conftest.pydefined , then test_a.pyand test_b.pycan use the test fixture; but test_c.pycannot.

`-- test_1
|   |-- conftest.py
|   `-- test_a.py
|   `-- test_b.py
`-- test_2
    `-- test_c.py

4.3 Validity levels

unittestnoseBoth support test pre- and cleanup levels: test method, test class, and test module.

pytestThe test fixtures also support various validation levels and are more abundant. Set by specifying scopethe parameter :

• function —— function level, that is, before calling each test function, the fixture will be regenerated
• class —— class level, before calling each test class, the fixture will be regenerated
• module —— module level, before loading each test module, the fixture will be regenerated
• package —— package level, before loading each package, the fixture will be regenerated
• session —— session level, before running all use cases, only generate fixture once

When we specify the effective level as the module level, the example is as follows:

import pytest
import smtplib

@pytest.fixture(scope="module")
def smtp_connection():
    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)

4.4 Test Prep and Cleanup

pytestThe test fixture can also implement test pre-test and clean-up. The two logics are split by yieldthe statement , and the writing method becomes very simple, such as:

import smtplib
import pytest

@pytest.fixture(scope="module")
def smtp_connection():
    smtp_connection = smtplib.SMTP("smtp.gmail.com", 587, timeout=5)
    yield smtp_connection  # provide the fixture value
    print("teardown smtp")
    smtp_connection.close()

In the above example, yield smtp_connectionthe previous statement is equivalent to the test pre-test, and the prepared test resource yieldis returned by smtp_connection; while the latter statement will be after the end of the use case execution (to be exact, the end of the statement cycle of the effective level of the test fixture) Execution, equivalent to test cleanup.

If the procedure for generating test resources (as in the example smtp_connection) supports withstatements , it can also be written in a simpler form:

@pytest.fixture(scope="module")
def smtp_connection():
    with smtplib.SMTP("smtp.gmail.com", 587, timeout=5) as smtp_connection:
        yield smtp_connection  # provide the fixture value

pytestIn addition to the functions introduced in this article, the test fixture also has more advanced gameplay such as parameterized fixtures, factory fixtures, and using fixtures in fixtures.

5. Skipped tests and expected failures

pytestIn addition to supporting unittestand nosetestmethods of skipping tests and predicting failures, it also provides corresponding methods pytest.markin :

• Skip tests directly via skip decorator or pytest.skip function
• Conditionally skip tests via skipif
• Expected test failure via xfail

Examples are as follows:

@pytest.mark.skip(reason="no way of currently testing this")
def test_mark_skip():
    ...

def test_skip():
    if not valid_config():
        pytest.skip("unsupported configuration")

@pytest.mark.skipif(sys.version_info < (3, 6), reason="requires python3.6 or higher")
def test_mark_skip_if():
    ...

@pytest.mark.xfail
def test_mark_xfail():
    ...

6. Sub-test/parameterized test

pytestIn addition to the support unittestin TestCase.subTest, it also supports a more flexible way of writing subtests, that is 参数化测试, through pytest.mark.parametrizedecorators .

In the following example, define a test_evaltest function and pytest.mark.parametrizespecify 3 sets of parameters through the decorator, then 3 subtests will be generated:

@pytest.mark.parametrize("test_input,expected", [("3+5", 8), ("2+4", 6), ("6*9", 42)])
def test_eval(test_input, expected):
    assert eval(test_input) == expected

In the example, the last set of parameters is deliberately allowed to cause failure, and you can see rich test result output when you run the test case: 

========================================= test session starts =========================================
platform darwin -- Python 3.7.1, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /Users/prodesire/projects/tests, inifile:
plugins: cov-2.6.0
collected 3 items

test.py ..F                                                                                     [100%]

============================================== FAILURES ===============================================
__________________________________________ test_eval[6*9-42] __________________________________________

test_input = '6*9', expected = 42

    @pytest.mark.parametrize("test_input,expected", [("3+5", 8), ("2+4", 6), ("6*9", 42)])
    def test_eval(test_input, expected):
>       assert eval(test_input) == expected
E       AssertionError: assert 54 == 42
E        +  where 54 = eval('6*9')

test.py:6: AssertionError
================================= 1 failed, 2 passed in 0.09 seconds ==================================

If the parameters are replaced pytest.param, we can also have higher-level gameplay, such as knowing that the last set of parameters failed, so mark it as xfail:

@pytest.mark.parametrize(
    "test_input,expected",
    [("3+5", 8), ("2+4", 6), pytest.param("6*9", 42, marks=pytest.mark.xfail)],
)
def test_eval(test_input, expected):
    assert eval(test_input) == expected

If the values ​​of multiple parameters of the test function want to be arranged and combined with each other, we can write like this:

@pytest.mark.parametrize("x", [0, 1])
@pytest.mark.parametrize("y", [2, 3])
def test_foo(x, y):
    pass

In the above example x=0/y=2, , x=1/y=2, x=0/y=3and x=1/y=3are brought into the test function respectively, and are executed as four test cases.

7. Test result output

pytestThe test result output unittestof noseis more abundant than that of and , and its advantages are:

• Highlight output, pass or fail will be distinguished by different colors
• Richer context information, automatically output code context and variable information
• Test progress display
• Test result output layout is more friendly and easy to read

If you exchange experience in software testing, interface testing, automated testing, and interviews. If you are interested, you can add software testing communication: 1085991341, and there will be technical exchanges with colleagues.

Eight, plug-in system

pytestThe plug-ins are very rich, and plug-and-play, as users do not need to write additional code.

In addition, thanks to the pytestgood architecture design and hook mechanism, its plug-in writing has also become easy to use.

Nine. Summary

This concludes three introductions to the Python testing framework. After writing so much, the judges are probably tired of reading. We might as well list a horizontal comparison table to summarize the similarities and differences of these unit test frameworks:

If you don't want to install or disallow third-party libraries, then unittestis the best and only option. On the contrary, pytestit is undoubtedly the best choice, and many Python open source projects use it pytestas a unit testing framework.

The above content is the entire content of this article. I hope the above content is helpful to you. Friends who have been helped are welcome to like and comment.

END meager strength

Finally, I would like to thank everyone who has read my article carefully. Looking at the fans’ growth and attention all the way, there is always a need for reciprocity. Although it is not a very valuable thing, you can take it away if you need it:

These materials should be the most comprehensive and complete preparation warehouse for [software testing] friends. This warehouse has also accompanied tens of thousands of test engineers through the most difficult journey, and I hope it can help you too!

加入我的软件测试交流群：110685036免费获取~（同行大佬一起学术交流，每晚都有大佬直播分享技术知识点）

Software testing interview applet

The software test question bank maxed out by millions of people! ! ! Who is who knows! ! ! The most comprehensive quiz mini program on the whole network, you can use your mobile phone to do the quizzes, on the subway or on the bus, roll it up!

The following interview question sections are covered:

1. Basic theory of software testing, 2. web, app, interface function testing, 3. network, 4. database, 5. linux

6. web, app, interface automation, 7. performance testing, 8. programming basics, 9. hr interview questions, 10. open test questions, 11. security testing, 12. computer basics

method of obtaining: