pytest framework advanced self-study series | Customized test report

Book source: Fang Lizhi Liang Lili "pytest framework and automated testing application"

Organize the teacher's course content and experimental notes while studying, and share them with everyone. Any infringement will be deleted. Thank you for your support!

Attach a summary post: pytest framework advanced self-study series | summary

Allure currently supports the vast majority of features available, except for environments with pytest. You can add the information needed for the report in the code to make the report richer and easier to understand.

Customize detailed step-by-step instructions

The most notable feature of Allure reports is that it enables each test to be described in great detail. This can be achieved with the @allure.step decorator, which adds an invocation of the annotated method or function to the report with the provided parameters.

Steps can be called externally or nested

Methods annotated with @step can be stored outside the test and imported when needed. Step methods can be nested to any depth. Create a steps.py file in the root directory of this chapter.

code show as below:

def imported_step():
    print("非常重要的步骤！")

Create the test_step_nested.py file at the same level, and the steps can be directly added to the test method, or other files can be called. For example: call the imported_step method in the steps file in the test_with_imported_step method.

It can also be used nestedly, and the method parameters of the call are arbitrary. The step_with_nested_steps method nestedly calls the nested_step method and then nests the nested_step_with_arguments method.

code show as below:

import allure
from steps import imported_step

@allure.step
def passing_step():
    print("通过的步骤")
    pass

@allure.step
def step_with_nested_steps():
    print("带有嵌套的步骤nested")
    nested_step()

@allure.step
def nested_step():
    print("调用带有参数arg的步骤")
    nested_step_with_arguments(1, 'abc')

@allure.step
def nested_step_with_arguments(arg1, arg2):
    print("带有两个参数：", arg1, arg2)
    pass

def test_with_imported_step():
    passing_step()
    print("外部导入")
    imported_step()

def test_with_nested_steps():
    passing_step()
    step_with_nested_steps()

When generating an Allure report, the status of each step is shown in "Execution" to the right of the name. Nested steps organized in a tree-like collapsible structure

D:\SynologyDrive\CodeLearning\WIN\pytest-book\venv\Scripts\python.exe "C:/Program Files/JetBrains/PyCharm Community Edition 2022.3.2/plugins/python-ce/helpers/pycharm/_jb_pytest_runner.py" --path D:\SynologyDrive\CodeLearning\WIN\pytest-book\src\chapter-6\test_step_nested.py 
Testing started at 14:39 ...
Launching pytest with arguments D:\SynologyDrive\CodeLearning\WIN\pytest-book\src\chapter-6\test_step_nested.py --no-header --no-summary -q in D:\SynologyDrive\CodeLearning\WIN\pytest-book\src\chapter-6

============================= test session starts =============================
collecting ... collected 2 items

test_step_nested.py::test_with_imported_step PASSED                      [ 50%]通过的步骤
外部导入
非常重要的步骤！

test_step_nested.py::test_with_nested_steps PASSED                       [100%]通过的步骤
带有嵌套的步骤nested
调用带有参数arg的步骤
带有两个参数： 1 abc


============================== 2 passed in 0.03s ==============================

Process finished with exit code 0

Steps can have description lines with parameters

Steps can have a description line that supports passing placeholders for positional and keyword arguments. Default arguments for keyword arguments will also be captured so that descriptions and specific arguments can be seen in the description of the Allure report.

code show as below:

import allure

@allure.step('步骤可以有描述行，位置参数显示输入参数："{0}"，关键字显示输入参数："{key}"')
def step_with_title_placeholders(arg1, key=None):
    pass

def test_steps_with_placeholders():
    step_with_title_placeholders(1, key='这是关键字参数')
    step_with_title_placeholders(2)
    step_with_title_placeholders(3, '这是位置参数')

fixture and conftest also support step

Fixtures also support steps, below is an example of testing using fixtures defined in the conftest.py module (these fixtures will be parsed by pytest even if not imported directly).

The steps initialized in Allure's report (in conftest.py) are shown in separate "front" trees for configuration initialization and destruction.

code show as below:

conftest.py
import allure
import pytest

@allure.step('step in conftest.py')
def conftest_step():
    pass

@pytest.fixture
def fixture_with_conftest_step():
    conftest_step()
    
test_step_conftest_fixture.py
import allure

@allure.step
def passing_step():
    pass
def test_with_step_in_fixture_from_conftest(fixture_with_conftest_step):
    passing_step()

Different Types of Accessories Supplemental Test Instructions

Reports can display many different types of attachments that supplement the results of a test, step, or fixture. Attachments can be created in the following ways.

allure.attach(body，name，attachment_type，extension)：

(1) body: the original content to be written into the file.

(2) name: A string with a file name.

(3) attachment_type: one of allure.attachment_type values.

(4) extension: used as an extension to create a file.

allure.attach.file(source, name, attachment_type, extension): source represents a string containing the file path, and other parameters are the same as those of allure.attach.

code show as below:

import allure
import pytest

@pytest.fixture
def attach_file_in_module_scope_fixture_with_finalizer(request):
    allure.attach('这个文本的附件是文件级的', '这可以是文件名字', allure.attachment_type.TEXT)
    def finalizer_module_scope_fixture():
        allure.attach('这个文本的附件是文件级的结束部分', '下面调用finalizer', allure.attachment_type.TEXT)
    request.addfinalizer(finalizer_module_scope_fixture)
    
def test_with_attacments_in_fixture_and_finalizer(attach_file_in_module_scope_fixture_with_finalizer):
    print("这是通过fixture调用上面的步骤，步骤中包含结束finalizer调用")
    pass

Attachments can be added to the Allure report, and there are various forms of attachments, which can be pictures, text, or HTML web pages.

code show as below:

import allure

def test_multiple_attachments():
    allure.attach.file('1.jpg', attachment_type=allure.attachment_type.JPG)
    allure.attach.file('2.yaml', attachment_type=allure.attachment_type.YAML)
    allure.attach('<head></head><body>这是个网页说明：下面可有多个</body>', '附加一个网页的具体说明', allure.attachment_type.HTML)

Attachments are shown in the context of the test entity they belong to. Attachments of type HTML will be rendered and displayed on the report page. This is a convenient way to provide some customization to your own test results.

All types listed in the code comments are acceptable

Customize various types of content descriptions

A detailed description of the test can be added, as well as giving the report as much context as needed. These can be done in a few ways: you can add the @allure.description decorator providing a description string, you can provide some HTML with @allure.description_html to be rendered in the "description" section of your test case, or just from the test Get the description in the method's docstring.

Support for unicode strings

Languages of different countries can be used so that the different languages can be displayed correctly.

code show as below:

import allure

def test_unicode_in_docstring_description():
    """unicode描述使用不同国家的语言
    hello my friend
    你好伙计 
    """
    assert 42 == int(6*7)

Rendered description_html from HTML

You can add a description in html format and display it in the report without any sense of disobedience.

code show as below:

import allure

def test_unicode_in_docstring_description():
    """unicode描述使用不同国家的语言
    hello my friend
    你好伙计
    """
    assert 42 == int(6*7)

@allure.description_html("""
<h1>测试带有一些复杂的描述</h1>
<table style="width:100%">
  <tr>
    <th>名</th>
    <th>姓</th>
    <th>年龄</th>
  </tr>
  <tr align="center">
    <td>小明</td>
    <td>李</td>
    <td>50</td>
  </tr>
  <tr align="center">
    <td>小三</td>
    <td>唐</td>
    <td>94</td>
  </tr>
</table>
""")
def test_html_description():
    print("测试通过网页描述")
    assert True

@allure.description("""
这是一个多功能描述，计算
""")
def test_description_from_decorator():
    assert 42 == int(6*7)

Dynamic description, which can replace the description at the beginning

It is also possible to dynamically update the original description allure.dynamic.description from within the test, which is a very flexible way. The code is as follows:

import allure

def test_unicode_in_docstring_description():
    """unicode描述使用不同国家的语言
    hello my friend
    你好伙计
    """
    assert 42 == int(6*7)

@allure.description_html("""
<h1>测试带有一些复杂的描述</h1>
<table style="width:100%">
  <tr>
    <th>名</th>
    <th>姓</th>
    <th>年龄</th>
  </tr>
  <tr align="center">
    <td>小明</td>
    <td>李</td>
    <td>50</td>
  </tr>
  <tr align="center">
    <td>小三</td>
    <td>唐</td>
    <td>94</td>
  </tr>
</table>
""")
def test_html_description():
    print("测试通过网页描述")
    assert True

@allure.description("""
这是一个多功能描述，计算
""")
def test_description_from_decorator():
    assert 42 == int(6*7)
    
@allure.description("""
这段描述将在测试结束后被替换
""")
def test_dynamic_description():
    assert 42 == int(6*7)
    allure.dynamic.description('这是最后的描述，用于替换前面的描述！')

Custom Test Title

Test titles can be made more readable using the special @allure.title decorator. The title supports placeholders for parameters and supports dynamic replacement, which can also be viewed in "Features".

code show as below:

import allure
import pytest

@allure.title("这是一个正常的标题")
def test_with_a_title():
    assert 2 + 2 == 4

@allure.title("这个标题支持其他国家文字：xxx")
def test_with_unicode_title():
    assert 3 + 3 == 6
    
@allure.title("参数化标题：adding {param1} with {param2}")
@pytest.mark.parametrize('param1, param2, expected', [
    (2,2,4),
    (1,2,5)
])
def test_with_parameterized_title(param1, param2, expected):
    assert param1 + param2 == expected
    
@allure.title("动态标题")
def test_with_dynamic_title():
    assert 2 + 2 == 4
    allure.dynamic.title("替换动态标题为最终标题")

various links

Reports can show links to defect systems or test management systems, and Allure has the ability to integrate @allure.link, @allure.issue, and @allure.testcase descriptions. These features allow for direct links to test cases and defect management systems and elsewhere.

code show as below:

import allure

TEST_CASE_LINK = 'http://81.70.24.116:8088/zentao/bug-view-26.html'

@allure.link('http://81.70.24.116:8088/zentao/testcase-browse-1.html')
def test_with_link():
    pass

@allure.link('http://81.70.24.116:8088/zentao/product-index-no.html', name='单击进入项目')
def test_with_named_link():
    pass

@allure.issue('140', '测试出现问题的结果')
def test_with_issue_link():
    pass

@allure.testcase(TEST_CASE_LINK, '测试登录健壮性')
def test_with_testcase_link():
    pass

Customize various labels

Sometimes, there is a need for flexibility in the tests to be performed. pytest allows the use of the mark decorator @pytest.mark (pytest docs).

Allure allows to mark tests in a similar way using 3 types of markup modifiers that can construct the structure of the report: BDD style markup, this markup can represent modules, functions, stories, severity labels, custom labels.

code show as below:

BDD style markup has two decorators: @allure.feature and @allure.story. Used to label tests according to the features and stories of a particular project (see the BDD article on Wikipedia for background). To mark certain features or stories as belonging to subsystems, use names starting with the epic_ prefix.

code show as below:

import allure

def test_without_any_annotations_that_wont_be_executed():
    pass

@allure.story('epic_1')
def test_with_epic_1():
    pass

@allure.feature('feature_1')
def test_with_feature_1():
    pass

@allure.story('story_1')
def test_with_story_1():
    pass

@allure.feature('feature_2')
@allure.story('story_1')
def test_with_story_1_and_feature_2():
    pass

@allure.feature('feature_2')
@allure.story('story_2')
def test_with_story_2_and_feature_2():
    pass

A different set of tests can be specified using the following command-line options to perform operations on a comma-separated list of values:

Entering different commands and parameters in the console will have different execution results. pytest test_feature_story_epic.py --allure-stories story_1,story_2 Execute the test methods of @allure.story(story_1) and @allure.story(story_2) in this code. pytest test_feature_story_epic.py --allure-features feature_2 --allure-stories The two parameters in story_2 are in the relationship of OR, that is, as long as one condition is met, it will be executed, so two tests are executed here.

The execution results are as follows:

PS D:\SynologyDrive\CodeLearning\WIN\pytest-book\src\chapter-6> pytest .\test_feature_story_epic.py --allure-stories story_1,story_2 --alluredir=./result 
c:\users\guoliang\appdata\local\programs\python\python37\lib\site-packages\pep8.py:110: FutureWarning: Possible nested set at position 1
  EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
Test session starts (platform: win32, Python 3.7.7, pytest 7.4.0, pytest-sugar 0.9.7)
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
sensitiveurl: .*
rootdir: D:\SynologyDrive\CodeLearning\WIN\pytest-book
plugins: allure-pytest-2.13.2, assume-2.4.3, base-url-2.0.0, cov-4.1.0, flakes-4.0.5, freezegun-0.4.2, html-3.2.0, httpserver-1.0.6, instafail-0.5.0, metadata-3.0.0, mock-3.11.1, ordering-0.6, pep8-1.0.1, picked-0.4.6, random-order-1.1.0, repeat-0.9.1, rerunfailures-12.0, selenium-4.0.1, sugar-0.9.7, timeout-2.1.0, variables-3.0.0
collected 6 items / 3 deselected / 3 selected                                                                                                                                                                                          

 src/chapter-6/test_feature_story_epic.py ✓✓✓                                                                                                                                                                            100% ██████████

Results (0.09s):
       3 passed
       3 deselected
PS D:\SynologyDrive\CodeLearning\WIN\pytest-book\src\chapter-6>

severity flag

To mark test levels by severity, the @allure.severity decorator can be used. It takes allure.severity_level enumeration value as parameter. The severity levels of bugs in this enumeration usually include levels 4 to 5. Explained in the code comments below.

code show as below:

import allure

def test_with_no_severity_label():
    pass

@allure.severity(allure.severity_level.TRIVIAL)
def test_with_trivial_severity():
    pass

@allure.severity(allure.severity_level.NORMAL)
def test_with_normal_severity():
    pass

@allure.severity(allure.severity_level.NORMAL)
class TestClassWithNormalSeverity(object):
    def test_inside_the_normal_severity_test_class(self):
        pass
    
    @allure.severity(allure.severity_level.CRITICAL)
    def test_inside_the_normal_severity_test_class_with_overriding_critical_severity(self):
        pass

Severity modifiers can be applied to functions, methods, or entire classes.

By using the --allure-severities command line option with comma-separated severity levels, only tests with the corresponding severity are run. The relationship at this level is also an OR relationship, as long as one of the conditions is met, it will be executed.

The execution results are as follows:

PS D:\SynologyDrive\CodeLearning\WIN\pytest-book\src\chapter-6> pytest -s -v .\test_severity.py --allure-severities normal,critical --alluredir=/tmp/my_allure_results
c:\users\guoliang\appdata\local\programs\python\python37\lib\site-packages\pep8.py:110: FutureWarning: Possible nested set at position 1
  EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
Test session starts (platform: win32, Python 3.7.7, pytest 7.4.0, pytest-sugar 0.9.7)
cachedir: .pytest_cache
metadata: {'Python': '3.7.7', 'Platform': 'Windows-10-10.0.22621-SP0', 'Packages': {'pytest': '7.4.0', 'pluggy': '0.13.1'}, 'Plugins': {'allure-pytest': '2.13.2', 'assume': '2.4.3', 'base-url': '2.0.0', 'cov': '4.1.0', 'flakes': '4.
0.5', 'freezegun': '0.4.2', 'html': '3.2.0', 'httpserver': '1.0.6', 'instafail': '0.5.0', 'metadata': '3.0.0', 'mock': '3.11.1', 'ordering': '0.6', 'pep8': '1.0.1', 'picked': '0.4.6', 'random-order': '1.1.0', 'repeat': '0.9.1', 'rerunfailures': '12.0', 'selenium': '4.0.1', 'sugar': '0.9.7', 'timeout': '2.1.0', 'variables': '3.0.0'}, 'Driver': None, 'Capabilities': {}}
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
sensitiveurl: .*
rootdir: D:\SynologyDrive\CodeLearning\WIN\pytest-book
plugins: allure-pytest-2.13.2, assume-2.4.3, base-url-2.0.0, cov-4.1.0, flakes-4.0.5, freezegun-0.4.2, html-3.2.0, httpserver-1.0.6, instafail-0.5.0, metadata-3.0.0, mock-3.11.1, ordering-0.6, pep8-1.0.1, picked-0.4.6, random-order-1.1.0, repeat-0.9.1, rerunfailures-12.0, selenium-4.0.1, sugar-0.9.7, timeout-2.1.0, variables-3.0.0
collected 5 items / 1 deselected / 4 selected                                                                                                                                                                                          

 src\chapter-6\test_severity.py::test_with_no_severity_label ✓                                                                                                                                                            25% ██▌       
 src\chapter-6\test_severity.py::test_with_normal_severity ✓                                                                                                                                                              50% █████     
 src\chapter-6\test_severity.py::TestClassWithNormalSeverity.test_inside_the_normal_severity_test_class ✓                                                                                                                 75% ███████▌  
 src\chapter-6\test_severity.py::TestClassWithNormalSeverity.test_inside_the_normal_severity_test_class_with_overriding_critical_severity ✓                                                                              100% ██████████

Results (0.09s):
       4 passed
       1 deselected
PS D:\SynologyDrive\CodeLearning\WIN\pytest-book\src\chapter-6>

Retry information display

Allure allows aggregating information about tests re-executed within a test run, as well as the history of test execution over time. For retries, the pytest rerun plugin can be used on failures.

For example, if we have a very unreliable step method that often fails, we can add the parameter --reruns=5, after specifying the parameter in the pytest startup options, we will see all unsuccessful runs of this on the Retries tab A test attempt.

code show as below:

import allure
import random
import time

@allure.step
def passing_step():
    pass

@allure.step
def flaky_broken_step():
    if random.randint(1,5) != 1:
        raise Exception('Broken!')
    
def test_broken_with_randomized_time():
    passing_step()
    time.sleep(random.randint(1,3))
    flaky_broken_step()