pytest learning summary 2.13 - skip, xfail use the complete set

This article has participated in the "Newcomer Creation Ceremony" event to start the road of gold creation together.

2.13 skip and xfail handle unsuccessful test cases

  Mark test functions that don't work on certain platforms or that you want to fail, so that pytest can handle them accordingly, and render a summary of the test session, while keeping the test suite green, skipping means you only do it if certain conditions are met only expect the tests to pass, otherwise pytest should skip running the tests entirely.

  Common example is skipping windows-only tests on non-windows platforms xfail means you expect the test to fail for some reason. Another common example is testing for features that have not yet been implemented or bugs that have not yet been fixed.

  When a test passes, despite an expected failure (marked with pytest.mark.xfail ), it is an xpass and will be reported in the test summary.

2.13.1 Skipping test cases

1. The decorator marks the skip and passes the skip reason:

import pytest
@pytest.mark.skip(reason="no way of currently testing this")
def test_the_unknown():
    ...
复制代码

2. pytest.skip mandatory skip

def test_function():
    if not valid_config():
        pytest.skip("unsupported configuration")
复制代码

3. Skip use cases in parameterization:

import pytest
@pytest.mark.parametrize("i", range(50))
def test_num(i):
    if i in (17, 25):
        pytest.skip("跳过用例")
复制代码

4. Skip platforms are all use cases for win

import sys
import pytest
    if not sys.platform.startswith("win"):
    pytest.skip("skipping windows-only tests", allow_module_level=True)
复制代码

5. Conditional skip use cases

If you wish to conditionally skip something, here is an example of a test function marked to be skipped when running on pre-Python 3.10 interpreters:

import sys
@pytest.mark.skipif(sys.version_info < (3, 10), reason="requires python3.10 or higher˓→")
def test_function():
    ...
复制代码

6. Skip certain test cases based on certain conditions

Share skipif flags between modules (py files) to skip certain tests in a module based on certain conditions:

# content of test_mymodule.py
import mymodule
minversion = pytest.mark.skipif(mymodule.__versioninfo__ < (1, 1), reason="at least mymodule-1.1 required" )

@minversion
def test_function():
    pass
复制代码

# test_myothermodule.py
from test_mymodule import minversion

@minversion
def test_anotherfunction():
    pass
复制代码

7. To skip a class, use the skipif tag on the class

import pytest, sys
@pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
class TestPosixCalls:
def test_function(self):
    "will not be setup or run under 'win32' platform"
复制代码

8. To skip all test functions of a module, you can use the global test tag

import pytest
# test_module.py
pytestmark = pytest.mark.skipif(...)
复制代码

9. Skip files or directories: just use pytest followed by the use case of the corresponding directory

2.13.2 Marking a test case as failed

1. Use the xfail marker to indicate that you want the test to fail:

@pytest.mark.xfail
def test_function():
    ...
复制代码

2. Setup failed during test run

No other code is executed after the pytest.xfail() call. This is because it does this internally by throwing a known exception.

import pytest
@pytest.mark.parametrize("i", range(50))
def test_num(i):
    if i in (17, 25):
        pytest.xfail("预期失败")
复制代码

3. Failure under a certain condition, cause

@pytest.mark.xfail(sys.platform == "win32", reason="bug in a 3rd party library")
def test_function():
    ...
复制代码

4. Prohibit running marked use cases

@pytest.mark.xfail(run=False)
def test_function():
    ...
复制代码

5. Set strict to make xfail's test case run result as failure

import pytest
@pytest.mark.xfail(strict=True)
def test_function():
    pass
复制代码

[pytest]
xfail_strict=true
复制代码

6. Related code examples

import pytest
xfail = pytest.mark.xfail
@xfail
def test_hello():
    assert 0
@xfail(run=False)
def test_hello2():
    assert 0
@xfail("hasattr(os, 'sep')")
def test_hello3():
    assert 0
@xfail(reason="bug 110")
def test_hello4():
    assert 0
@xfail('pytest.__version__[0] != "17"')
def test_hello5():
    assert 0
def test_hello6():
    pytest.xfail("reason")
@xfail(raises=IndexError)
def test_hello7():
    x = []
    x[1] = 1
复制代码

C:\Users\mc\Desktop\python基础>pytest -rx test_module.py
======================================================================= test session starts ========================================================================
platform win32 -- Python 3.9.6, pytest-7.1.1, pluggy-0.13.1
rootdir: C:\Users\mc\Desktop\python基础
collected 7 items                                               
test_module.py xxxxxxx                               [100%]

===================================================================== short test summary info ======================================================================
XFAIL test_module.py::test_hello
XFAIL test_module.py::test_hello2
  reason: [NOTRUN]
XFAIL test_module.py::test_hello3
  condition: hasattr(os, 'sep')
XFAIL test_module.py::test_hello4
  bug 110
XFAIL test_module.py::test_hello5
  condition: pytest.__version__[0] != "17"
XFAIL test_module.py::test_hello6
  reason: reason
XFAIL test_module.py::test_hello7
======================================================================== 7 xfailed in 0.19s ========================================================================
复制代码

2.13.3 Using parameterized skip/exit

import sys
import pytest
@pytest.mark.parametrize(
    ("n", "expected"),
    [
        (1, 2),
        pytest.param(1, 0, marks=pytest.mark.xfail),
        pytest.param(1, 3, marks=pytest.mark.xfail(reason="some bug")),
        (2, 3),
        (3, 4),
        (4, 5),
        pytest.param(10, 11, marks=pytest.mark.skipif(sys.version_info >= (3, 0), reason="py2k˓→")),
    ],
)
def test_increment(n, expected):
    assert n + 1 == expected
复制代码

C:\Users\mc\Desktop\python>pytest -vs test_module.py
======================================================================= test session starts ========================================================================
platform win32 -- Python 3.9.6, pytest-7.1.1, pluggy-0.13.1 -- c:\python39\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\mc\Desktop\python
collected 7 items

test_module.py::test_increment[1-2] PASSED
test_module.py::test_increment[1-0] XFAIL
test_module.py::test_increment[1-3] XFAIL (some bug)
test_module.py::test_increment[2-3] PASSED
test_module.py::test_increment[3-4] PASSED
test_module.py::test_increment[4-5] PASSED
test_module.py::test_increment[10-11] SKIPPED (py2k˓→)
============================================================= 4 passed, 1 skipped, 2 xfailed in 0.12s ==============================================================
复制代码