allure and pytest

Introduction

• A lightweight, flexible, multi-language test reporting tool.
• Supports multiple platforms and luxurious report framework.
• It can provide detailed test reports, test steps, Log and other information.
• Java language development, but supports pytest, JavaScript, PHP, ruby ​​and other languages ​​or frameworks.
• Can be integrated into Jenkins.

Install

To use Allure, you need to install Java and Allure.

• Java: Since Allure is developed in Java language, Java needs to be installed; there are many Java installation tutorials on the Internet, so I won’t go into details here.
• Allure: I only have win10 system, so this chapter is mainly operated under win10.
  • Download address: https://github.com/allure-framework/allure2/releases
    If github download is not smooth, you can consider other websites: https://repo.maven.apache.org/maven2/io/qameta/allure /allure-commandline/
  • Configure environment variables: After decompression, add the bin directory to the PATH environment variable
    
  • Environment verification: Enter the command line allure --versionand the corresponding version can be printed normally.
    
  • To use allure with pytest, you need to install the corresponding plug-in:pip install allure-pytest

use

Official documentation: https://docs.qameta.io/allure-report/

getting Started

If you want to see the allure report, you need to do 2 steps:
1. When executing pytest, associate allure: pytest command with --alluredir 结果存放目录or --alluredir=结果存放目录
2. Open the execution report: allure serve 结果存放目录
The content of test_demo.py is as follows:

def test_add_positive():
    assert 1 + 3 == 4

def test_add_negative():
    assert -1 + -3 == -4

Execution use case:

After the command is successfully executed, a result directory will be generated in the current directory, and the execution results will be stored in the result directory.

By allure serve ./resultopening the test report.

allure decorator

allure provides some decorators, the more commonly used ones are as follows:

method	Parameter value	Parameter Description
@allure.epic(*epics)	epic description	Define a project, used when there are multiple projects. Generally, a string is passed as a parameter to represent the project name. Below is the feature.
@allure.feature(*features)	branch	Use cases are divided according to modules (similar to a business divided into several steps/functions). Generally, a string is passed as a parameter to represent the branch name. Below is the story.
@allure.story(*stories)	story	Similar to a scenario (that is, a function is divided into several aspects for verification), a string is generally passed as a parameter to represent the scenario name.
@allure.title(test_title)	Use case title	Set a name for the specified use case. In the test report, the use case name displayed is not the function name, but the set test_title.
@allure.description(test_description)	Use case description	The test_description parameter describes the details of the current use case, such as preconditions, verification scenarios, expected results, etc. that need attention.
@allure.severity(allure.severity_level.xxx)	Use case priority	Supports setting priorities for use cases. Once a use case fails to execute, the defect level can be determined. Currently allure can set 5 levels: BLOCKER, CRITICAL, NORMAL, MINOR, TRIVIAL.
@allure.step(test_step)	Use case steps	Similar to executing the test process in steps, allure will count the execution of the steps to facilitate problem location and debugging. If the code in allure.step() is executed successfully, the allure report will show that the step was successfully executed. If the execution fails, the corresponding step will be displayed.
@allure.testcase(url, name=None)	Use case related links	Set the address where the use case is stored to facilitate access to the details of the application case. When name is not None, the text is displayed as name, otherwise the URL where the use case is located is displayed.
@allure.issue(url, name=None)	Defect address	Corresponds to the defect address in the defect management system, and the usage is the same allure.testcase().
@allure.link(url, link_type='link', name=None)	Define link	Used to define a link that needs to be displayed in the test report. The default link_type is link, which can be set to tms or issue. When it is tms, the usage is allure.testcase()the same; when it is issue, the usage is allure.issue()the same. The usage of url and name parameters is allure.testcase()consistent with.
allure.attach(body, name=None, attachment_type=None, extension=None)	Add display as attachment	Display the python object in the report according to the specified attachment type. The supported attachment_type (attachment type) is in allure.attachment_type.
allure.attach.file(source, name=None, attachment_type=None, extension=None)	Add attachments	Read the file according to the specified attachment type and display it in the report. The supported attachment_type (attachment type) is in allure.attachment_type.

@allure.epic(*epics)、@allure.feature(*features)、@allure.story(*stories)

Epic, feature, and story have a hierarchical relationship and are displayed hierarchically in the report.
An example is as follows:
test_demo.py content is as follows:

import allure

@allure.epic("epic的使用1")
@allure.feature("测试模块1")
def test_case1():
    assert 1 + 3 == 4


@allure.feature("测试模块1")
def test_case2():
    assert -1 + -3 == -4

@allure.feature("测试模块1")
@allure.story("某部分验证1")
def test_case3():
    assert True

@allure.feature("测试模块1")
@allure.story("某部分验证1")
def test_case4():
    assert True

@allure.story("某部分验证1")
def test_case5():
    assert True

@allure.story("某部分验证2")
def test_case6():
    assert True

Execute pytest test_demo.py --alluredir ./resultthe command and open the allure report as follows:

Click the Show all button, and the hierarchical display instructions are as follows:

@allure.title(test_title)

@allure.title(test_title): is to set the name for the specified use case. In the test report, the use case name displayed is not the function name, but the set test_title.
The content of test_demo.py is as follows:

import allure

def test_case1():
    assert 1 + 3 == 4

@allure.title("测试xxxx点")
def test_case2():
    assert True

The allure test report displays as follows:

@allure.description(test_description)

@allure.description(test_description): Describe the detailed information of the current use case, such as preconditions, verification scenarios, expected results, etc. that need attention. After setting, in the allure report, the details page of the use case will display the Description field, and the test_descriptioncontent of the field is.
Example:
test_demo.py content is as follows:

import allure

def test_case1():
    print("这里是test_case1的测试内容")
    assert True

@allure.description("这条用例是为了验证...巴拉巴拉...的场景")
def test_case2():
    print("这里是test_case2的测试内容")
    assert True

In the allure report, the execution details of test_case2() are shown as follows:

@allure.severity(allure.severity_level.xxx)

@allure.severity(allure.severity_level.xxx): Supports setting priorities for use cases. Once a use case fails to execute, the defect level can be determined. Currently allure can be set to 5 levels:

• BLOCKER: Blocking defect, which is equivalent to the main function not being implemented, very serious
• CRITICAL: Serious defects, missing function points, or although the main process is implemented, there are major problems
• NORMAL: General defects (edge ​​cases, format errors, etc.)
• MINOR: Minor defects (interface does not meet UI requirements, etc.)
• TRIVIAL: minor defects (such as no prompts, poor user experience, etc.)

Example:
test_demo.py content is as follows:

import allure

def test_case1():
    print("这里是test_case1的测试内容，未设置优先级")
    assert True

@allure.severity(allure.severity_level.BLOCKER)
def test_case2():
    print("这里是test_case2的测试内容，优先级为BLOCKER，阻塞缺陷")
    assert True

@allure.severity(allure.severity_level.CRITICAL)
def test_case3():
    print("这里是test_case3的测试内容，优先级为CRITICAL，严重缺陷")
    assert True

@allure.severity(allure.severity_level.NORMAL)
def test_case4_success():
    print("这里是test_case4的测试内容，优先级为NORMAL，一般缺陷")
    assert True

@allure.severity(allure.severity_level.NORMAL)
def test_case4_fail():
    print("这里是test_case的测试内容，优先级为")
    assert False

@allure.severity(allure.severity_level.MINOR)
def test_case5():
    print("这里是test_case5的测试内容，优先级为MINOR，次要的、较小的缺陷")
    assert True

@allure.severity(allure.severity_level.TRIVIAL)
def test_case6():
    print("这里是test_case6的测试内容，优先级为TRIVIAL，轻微缺陷（比如无提示、用户体验不好等）")
    assert True

After execution, the allure report displays as follows:

allure.step(test_step)

allure.step(test_step): Similar to executing the test process in steps, allure will count the execution of the steps to facilitate problem location and debugging. If the code in allure.step() is executed successfully, the allure report will show that the step was successfully executed. If the execution fails, the corresponding step will be displayed.
allure.step(test_step)It is for the steps of the test process. It is not recommended to decorate the entire use case. It is recommended to put it in the test case method or function. The usage method is as follows:

with allure.step("步骤说明"):
    # 代码实现对应步骤

Example:
test_demo.py content is as follows:

import allure

def test_send_mail_without_step():
    print("步骤1：输入用户名admin，密码password")
    print("登录成功")
    print("步骤2：新建邮件")
    print("步骤3：编辑邮件")
    print("步骤4：发送邮件")
    print("发送成功消息提示")
    print("步骤5：退出确认")
    print("退出成功页面")


def test_send_mail_with_step_success():
    with allure.step("步骤1：用户登录"):
        print("输入用户名admin，密码password")
        print("登录成功")

    with allure.step("步骤2：新建并编辑邮件"):
        print("新建邮件")
        print("编辑邮件")

    with allure.step("步骤3：发送邮件"):
        print("发送邮件")
        print("发送成功消息提示")

    with allure.step("步骤4：退出登录"):
        print("退出确认")
        print("退出成功页面")

def test_send_mail_with_step_fail():
    with allure.step("步骤1：用户登录"):
        print("输入用户名admin，密码password")
        print("登录成功")

    with allure.step("步骤2：新建并编辑邮件"):
        print("新建邮件")
        assert False
        print("前面报错了，接下来的不会被执行了")
        print("编辑邮件")

    with allure.step("步骤3：发送邮件"):
        print("发送邮件")
        print("发送成功消息提示")

    with allure.step("步骤4：退出登录"):
        print("退出确认")
        print("退出成功页面")

allure report shows the following:

@allure.testcase(url, name=None)

@allure.testcase(url, name=None): Sets the jump link URL for the specified use case. If name is None, the link will be displayed directly. If name is not None, name will be displayed in the report. Click to jump to the specified link. It is generally used to set the accessible link of the use case and jump directly to the use case details.
Example:
test_demo.py content is as follows:

import allure

def test_case1():
    print("这里是test_case1的测试内容")
    assert True

@allure.testcase("https://www.baidu.com/")
def test_case2():
    print("这里是test_case2的测试内容")
    assert True

@allure.testcase("https://www.baidu.com/", "用例所在位置")
def test_case3():
    print("这里是test_case3的测试内容")
    assert True

allure report shows the following:

@allure.issue(url, name=None)

@allure.issue(url, name=None): Sets the defect link URL for the specified use case. If name is None, the link will be displayed directly. If name is not None, name will be displayed in the report. Click to jump to the specified link. It is generally used to set the corresponding defect link, which can jump directly to the specified defect details page.
Example:
test_demo.py content is as follows:

import allure

def test_case1():
    print("这里是test_case1的测试内容")
    assert True

@allure.issue("https://www.baidu.com/")
def test_case2():
    print("这里是test_case2的测试内容")
    assert True

@allure.issue("https://www.baidu.com/", "缺陷访问地址")
def test_case3():
    print("这里是test_case3的测试内容")
    assert True

allure report shows the following:

@allure.link(url, link_type='link', name=None)

@allure.link(url, link_type='link', name=None): Used to define a link that needs to be displayed in the test report. The default link_type is link, which can be set to tms or issue. When it is tms, the usage is allure.testcase()the same; when it is issue, the usage is allure.issue()the same. The usage of url and name parameters is allure.testcase()consistent with.
Example:
test_demo.py content is as follows:

import allure

def test_case1():
    print("这里是test_case1的测试内容")
    assert True

@allure.link("https://www.baidu.com/")
def test_case2():
    print("这里是test_case2的测试内容")
    assert True

@allure.link("https://www.baidu.com/", name="相关业务文档")
def test_case3():
    print("这里是test_case3的测试内容")
    assert True

allure report shows the following:

Add attachments

allure provides attach, which supports adding attachments to test reports. There are two ways to attach. Attach can add data, text, pictures, videos, web pages and other data.
Supported attachment types allure.attachment_typeare defined in:

method	illustrate
allure.attach(body, name=None, attachment_type=None, extension=None)	Display the content of the body in the form of attachment, name is the attachment name, attachment_type is the attachment type, and extension is generally the attachment extension.
allure.attach.file(source, name=None, attachment_type=None, extension=None)	Display the file (source) in the report as an attachment. name is the attachment name, attachment_type is the attachment type, and extension is generally the attachment extension. source is the local path of the attachment.

attach() displays the Python object as the specified attachment type, attach.file() displays the local file in the report as the specified attachment type, attach() is for strings and other contents, and attach.file() is for files. attach.file() will read the contents of the file according to the specified attachment type and display it in the report, while attach() will display the python string object according to the specified attachment type.
Example:
test_demo.py content is as follows:

import allure

@allure.title("attach.file()针对于文件来源的附件")
def test_attach_file():
    with allure.step("attach.file()添加图片"):
        allure.attach.file("D:\\test_code\\allure_demo\\csdn.png", name="csdn",
                           attachment_type=allure.attachment_type.PNG, extension=".png")

    # 没有指定name，系统会默认分配附件名称，CSV格式显示正常
    with allure.step("attach.file()添加CSV文件"):
        allure.attach.file("D:\\test_code\\allure_demo\\成绩表.csv",
                           attachment_type=allure.attachment_type.CSV)

@allure.title("attach()和attach.file()的区别")
def test_attach_or_file():
    with allure.step("attach.file()添加文本txt内容"):
        allure.attach.file("D:\\test_code\\allure_demo\\操作步骤.txt", name="文本标题：操作步骤",
                           attachment_type=allure.attachment_type.TEXT)

    # 参数一致，attach.file()会将文件中的内容按照指定附件类型读取并展示到报告中，attach()则是将python的字符串对象按照指定附件类型展示
    with allure.step("attach添加文本txt内容"):
        allure.attach("D:\\test_code\\allure_demo\\操作步骤.txt", name="文本标题：操作步骤",
                           attachment_type=allure.attachment_type.TEXT)

@allure.title("attach()的使用")
@allure.description("attach()将python对象按照指定的附件类型展示，attachment_type有/未指定类型时的区别")
def test_attach():
    # attach()的attachment_type参数，用来确定以什么类型展示，以下是attachment_type参数有无指定的情况
    with allure.step("attach添加HTML内容(指定了attachment_type为HTML格式)"):
        allure.attach('<a href="http://www.eastmoney.com">'
                      '<img class="ts-logo" src="https://g1.dfcfw.com/g3/201909/20190912110958.jpg" alt="东方财富网——财经资讯门户"></a>',
                      name="东方财富网首页", attachment_type=allure.attachment_type.HTML)

    with allure.step("attach添加HTML内容(未指定html格式，默认是文本形式)"):
        allure.attach('<a href="http://www.eastmoney.com">'
                      '<img class="ts-logo" src="https://g1.dfcfw.com/g3/201909/20190912110958.jpg" alt="东方财富网——财经资讯门户"></a>',
                      name="东方财富网首页")

The attachments used are as follows:

Execute the pytest command: pytest test_demo.py --clean-alluredir --alluredir ./result, open the allure report:allure serve ./result

allure report shows the following:

The allure parameter of pytest

After python installs the allure-pytest library, the pytest command can use allure parameters.

Commonly used allure parameters are as follows:

parameter	describe
--alluredir=DIR	Store the pytest execution results in the specified directory DIR.
--clean-alluredir	The execution results of pytest, allure, are cumulative by default, that is, the previous execution statistical results will be added. After adding this parameter, only the current execution results will be counted. For example, if you open the allure report after the second execution, the contents of the first execution will be displayed. If you only want the results of this execution, you can add this parameter.
--allure-severities=SEVERITIES_SET	Specify the use case to execute the corresponding priority. In SEVERITIES_SET, if there are multiple priorities, use commas ,to connect them.
--allure-epics=EPICS_SET	Specify the use case to execute the corresponding epic. If there are multiple epics, use commas ,to connect them.
--allure-features=FEATURES_SET	指定执行对应 feature 的用例，如果有多个 feature，使用英文逗号,连接。
--allure-stories=STORIES_SET	指定执行对应 story 的用例，如果有多个story，使用英文逗号,连接。

示例（参数内容的示例都是用当前的代码）：
test_demo.py 内容如下：

import allure

@allure.epic("epic1的用例")
@allure.feature("feature2的用例")
def test_case1():
    print("这里是test_case1的测试内容，未设置优先级")
    assert True

@allure.story("story2的用例")
@allure.severity(allure.severity_level.BLOCKER)
def test_case2():
    print("这里是test_case2的测试内容，优先级为BLOCKER，阻塞缺陷")
    assert True

@allure.epic("epic1的用例")
@allure.story("story1的用例")
@allure.testcase("https://wwww.baidu.com")
@allure.severity(allure.severity_level.CRITICAL)
def test_case3():
    print("这里是test_case3的测试内容，优先级为CRITICAL，严重缺陷")
    assert True

@allure.epic("epic3的用例")
@allure.feature("feature2的用例")
@allure.severity(allure.severity_level.NORMAL)
def test_case4_success():
    print("这里是test_case4的测试内容，优先级为NORMAL，一般缺陷")
    assert True

@allure.feature("feature1的用例")
@allure.story("story3的用例")
@allure.severity(allure.severity_level.NORMAL)
def test_case4_fail():
    print("这里是test_case的测试内容，优先级为NORMAL")
    assert False

@allure.epic("epic2的用例")
@allure.severity(allure.severity_level.MINOR)
def test_case5():
    print("这里是test_case5的测试内容，优先级为MINOR，次要的、较小的缺陷")
    assert True

@allure.feature("feature3的用例")
@allure.severity(allure.severity_level.TRIVIAL)
def test_case6():
    print("这里是test_case6的测试内容，优先级为TRIVIAL，轻微缺陷（比如无提示、用户体验不好等）")
    assert True

统计当前执行结果

--clean-alluredir：pytest 的执行结果，allure 默认是累计的，即会加上之前的执行统计结果，加上这个参数后会只统计当前执行结果。比如第二次执行后打开 allure 报告，会显示有第一次执行的内容，只想要本次执行结果的，可以加上该参数。
示例：
执行了pytest test_demo.py --alluredir ./result命令后，再执行pytest test_demo.py::test_case4_fail --alluredir ./result命令。

allure 报告显示如下：

加了--clean-alluredir参数后（即执行命令：pytest test_demo.py::test_case4_fail --alluredir ./result --clean-alluredir）

allure 报告显示如下：

allure 过滤用例

按照优先级

allure 报告显示如下：

按照 epic

allure 报告显示如下：

按照 feature

allure 报告显示如下：

按照 story

allure 报告显示如下：

生成报告

我们在进行测试用例执行时，经常会有失败的用例需要调试，所以中间可能会有多个版本的测试报告，最后生成最终版本的测试报告。
报告的数据肯定是从 pytest 执行用例统计得来的。pytest 执行用例时，需要加上--alluredir 存储路径参数，将 pytest 的执行统计结果存放到指定的存储路径，有2种方式查看测试报告：

• 方式 1：测试完成后在线查看报告，命令为allure server 存储路径
• 方式 2：测试完成后从测试结果生成静态报告，再打开查看静态报告：
  • 生成静态报告：allure generate 存储路径 -o 新路径 --clean（注意：覆盖路径加-c或--clean）
  • 打开查看静态报告：allure open -h ip地址 -p 端口 新路径

2 种方式都是统计最近一次 pytest 命令执行后的结果，pytest 中有些参数也会影响报告的统计结果，比如加了--clean-alluredir就只会显示当前 pytest 执行用例的统计结果，而不会将之前的统计结果包含进来。所以从统计执行结果来说，方式 1 和方式 2 对 pytest 统计执行结果是一样的，只是渲染成 html 的方式不一样，方式 1 用于本地渲染和查看，方式 2 用于本地渲染和对外展示（即给其他设备也能查看）。

方式 1 前面的示例用过很多次了，这里主要讲下方式 2：方式 2 是对已统计的 pytest 执行结果生成新的测试报告：
test_demo.py 内容如下：

import pytest

def test_passed():
    assert True

def test_fail():
    assert False

def test_skip():
    pytest.skip("测试跳过这条用例")

def test_broken():
    raise ValueError("测试出现异常")

执行 pytest 命令：

从上面截图可以看出，当前目录自动生成了个 result 目录，是本次 pytest 执行结果。接下来根据 result 生成静态报告：

allure generate ./result -o ./report --clean执行成功后，当前目录自动生成 report 目录，存放了生成的 html 文件及其组件、数据内容。

可直接用浏览器打开文件，如下：

也可以使用allure open -h ip地址 -p 端口 新路径命令打开报告：

Allure 介绍

• 一个轻量级、灵活的、支持多语言的测试报告工具。
• 支持多平台，奢华的 report 框架。
• 能提供详尽的测试报告、测试步骤、Log 等信息。
• Java 语言开发，但支持 pytest、JavaScript、PHP、ruby 等语言或框架。
• 可以集成到 Jenkins。

安装

Allure 的使用，需要安装 Java 和 Allure。

• Java：由于 Allure 是 Java 语言开发的，所以需要安装 Java；网上很多 Java 的安装教程，这里就不详细说明了。
• Allure：本人只有 win10 系统的，所以本章主要是在 win10 下操作的。
  • Download address: https://github.com/allure-framework/allure2/releases
    If github download is not smooth, you can consider other websites: https://repo.maven.apache.org/maven2/io/qameta/allure /allure-commandline/
  • Configure environment variables: After decompression, add the bin directory to the PATH environment variable
    
  • Environment verification: Enter the command line allure --versionand the corresponding version can be printed normally.
    
  • To use allure with pytest, you need to install the corresponding plug-in:pip install allure-pytest

use

Official documentation: https://docs.qameta.io/allure-report/

getting Started

If you want to see the allure report, you need to do 2 steps:
1. When executing pytest, associate allure: pytest command with --alluredir 结果存放目录or --alluredir=结果存放目录
2. Open the execution report: allure serve 结果存放目录
The content of test_demo.py is as follows: