The unit test framework of the Python standard library - unittest

To build an automated testing framework with Python, we need to organize use cases and test execution. Here, the blogger recommends Python's standard library-unittest.

Unittest is a member of the xUnit series of frameworks. If you know the other members of xUnit, it should be easy for you to use unittest. They all work in the same way.

The core working principle of unittest

The four core concepts in unittest are: test case, test suite, test runner, test fixture .

Let's explain the meaning of these four concepts separately, first look at a static class diagram of unittest (the following class diagram and explanation are from the Internet, the original link ):

unittest class diagram

An instance of TestCase is a test case. What is a test case? It is a complete test process, including setting up the environment before the test (setUp), executing the test code (run), and restoring the environment after the test (tearDown). The essence of unit test is here. A test case is a complete test unit. By running this test unit, a certain problem can be verified.

And multiple test cases are collected together, it is TestSuite, and TestSuite can also nest TestSuite.

TestLoader is used to load TestCase into TestSuite. There are several loadTestsFrom__() methods, which are to find TestCase from various places, create instances of them, add to TestSuite, and return a TestSuite instance.

TextTestRunner is to execute test cases, where run(test) will execute the run(result) method in TestSuite/TestCase.
The result of the test will be saved to the TextTestResult instance, including information such as how many test cases were run, how many succeeded, and how many failed.

The construction and destruction of a test case environment is a fixture.

A class inherits unittest.TestCase, which is a test case, but if there are multiple test methods that start with, then each such method will generate a TestCase instance when it is loaded, such as: there are in a class Four test_xxx methods, and finally four test cases when loaded into the suite.

The whole process is clear here:

Write the TestCase, and then load the TestCase to TestSuite by TestLoader, and then run the TestSuite by TextTestRunner. The result of the run is saved in TextTestResult. When we execute it through the command line or unittest.main(), main will call run in TextTestRunner to execute it. Or we can execute the use case directly through TextTestRunner. Here is an explanation. When the Runner is executed, the execution result is output to the console by default. We can set it to output to a file and view the result in the file (you may have heard of HTMLTestRunner, yes, through it, the result can be output to In HTML, a beautiful report is generated, which is the same as TextTestRunner, which can be seen from the name, we will talk about this later).

unittest instance

Let's get a better understanding of unittest through some examples.

Let's first prepare some methods to be tested:

mathfunc.py

def add(a, b):
    return a+b

def minus(a, b):
    return a-b

def multi(a, b):
    return a*b

def divide(a, b):
    return a/b

Simple example

Next we write a test for these methods:

test_mathfunc.py

# -*- coding: utf-8 -*-

import unittest
from mathfunc import *


class TestMathFunc(unittest.TestCase):
    """Test mathfuc.py"""

    def test_add(self):
        """Test method add(a, b)"""
        self.assertEqual(3, add(1, 2))
        self.assertNotEqual(3, add(2, 2))

    def test_minus(self):
        """Test method minus(a, b)"""
        self.assertEqual(1, minus(3, 2))

    def test_multi(self):
        """Test method multi(a, b)"""
        self.assertEqual(6, multi(2, 3))

    def test_divide(self):
        """Test method divide(a, b)"""
        self.assertEqual(2, divide(6, 3))
        self.assertEqual(2.5, divide(5, 2))

if __name__ == '__main__':
    unittest.main()

Results of the:

.F..
======================================================================
FAIL: test_divide (__main__.TestMathFunc)
Test method divide(a, b)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:/py/test_mathfunc.py", line 26, in test_divide
    self.assertEqual(2.5, divide(5, 2))
AssertionError: 2.5 != 2

----------------------------------------------------------------------
Ran 4 tests in 0.000s

FAILED (failures=1)

It can be seen that a total of 4 tests were run, 1 failed, and the reason for the failure was given, which 2.5 != 2 means that our divide method is problematic.

This is a simple test, there are a few points that need to be explained:

In the first line, the identification of the result of each use case execution is given, success is ., failure is F, error is E, and skip is S. It can also be seen from the above that the execution of the test has nothing to do with the order of the methods. Test_divide is written in the fourth, but it is executed in the second.
Each test method test starts with, otherwise it will not be recognized by unittest.
Adding verbosity parameters to unittest.main() can control the level of detail of the error report output. The default is 1, if set 0, the execution results of each use case will not be output, that is, there is no line 1 in the above results; if set 2, The detailed execution results are output as follows:

test_add (__main__.TestMathFunc)
Test method add(a, b) ... ok
test_divide (__main__.TestMathFunc)
Test method divide(a, b) ... FAIL
test_minus (__main__.TestMathFunc)
Test method minus(a, b) ... ok
test_multi (__main__.TestMathFunc)
Test method multi(a, b) ... ok

======================================================================
FAIL: test_divide (__main__.TestMathFunc)
Test method divide(a, b)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:/py/test_mathfunc.py", line 26, in test_divide
    self.assertEqual(2.5, divide(5, 2))
AssertionError: 2.5 != 2

----------------------------------------------------------------------
Ran 4 tests in 0.002s

FAILED (failures=1)

It can be seen that the detailed execution status of each use case, the name of the use case, and the description of the use case are output (add """Doc String"" in the code example under the test method, and the character will be used when the use case is executed. As the description of this use case, adding appropriate comments can make the output test report easier to read )

Organize TestSuite

The above code example shows how to write a simple test, but there are two problems, how do we control the order of execution of the use cases? (The test methods in the example here are not necessarily related, but the use cases you write later may have a priority, you need to execute method A first, and then execute method B), we are going to use TestSuite. The cases we add to TestSuite will be executed in the order in which they were added .

The second question is that we only have one test file now, we can directly execute the file, but if there are multiple test files, how to organize them, you can't execute them one by one, the answer is also in TestSuite.

Here is an example:

In the folder we create a new file, test_suite.py :

# -*- coding: utf-8 -*-

import unittest
from test_mathfunc import TestMathFunc

if __name__ == '__main__':
    suite = unittest.TestSuite()

    tests = [TestMathFunc("test_add"), TestMathFunc("test_minus"), TestMathFunc("test_divide")]
    suite.addTests(tests)

    runner = unittest.TextTestRunner(verbosity=2)
    runner.run(suite)

Results of the:

test_add (test_mathfunc.TestMathFunc)
Test method add(a, b) ... ok
test_minus (test_mathfunc.TestMathFunc)
Test method minus(a, b) ... ok
test_divide (test_mathfunc.TestMathFunc)
Test method divide(a, b) ... FAIL

======================================================================
FAIL: test_divide (test_mathfunc.TestMathFunc)
Test method divide(a, b)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\py\test_mathfunc.py", line 26, in test_divide
    self.assertEqual(2.5, divide(5, 2))
AssertionError: 2.5 != 2

----------------------------------------------------------------------
Ran 3 tests in 0.001s

FAILED (failures=1)

As you can see, the execution is the same as we expected: three cases were executed, and the order was executed in the order in which we added to the suite.

The TestSuite addTests() method is used above , and the TestCase list is directly passed in. We can also:

# 直接用addTest方法添加单个TestCase
suite.addTest(TestMathFunc("test_multi"))

# 用addTests + TestLoader
# loadTestsFromName()，传入'模块名.TestCase名'
suite.addTests(unittest.TestLoader().loadTestsFromName('test_mathfunc.TestMathFunc'))
suite.addTests(unittest.TestLoader().loadTestsFromNames(['test_mathfunc.TestMathFunc']))  # loadTestsFromNames()，类似，传入列表

# loadTestsFromTestCase()，传入TestCase
suite.addTests(unittest.TestLoader().loadTestsFromTestCase(TestMathFunc))

Note that it is not possible to sort the cases with the TestLoader method, and at the same time, the suite can also be set in the suite.

Output the results to a file

The use case is organized, but the results can only be output to the console, so there is no way to view the previous execution records. We want to output the results to a file. Very simple, look at the example:

Modify test_suite.py :

# -*- coding: utf-8 -*-

import unittest
from test_mathfunc import TestMathFunc

if __name__ == '__main__':
    suite = unittest.TestSuite()
    suite.addTests(unittest.TestLoader().loadTestsFromTestCase(TestMathFunc))

    with open('UnittestTextReport.txt', 'a') as f:
        runner = unittest.TextTestRunner(stream=f, verbosity=2)
        runner.run(suite)

Executing this file, you can see that UnittestTextReport.txt is generated in the same directory , and all execution reports are output to this file. Now we have a test report in txt format.

test fixture之setUp() tearDown()

The entire test above basically ran down, but there may be some special situations: what if my test needs to prepare the environment before each execution, or need to clean up after each execution? For example, you need to connect to the database before execution, and you need to restore data and disconnect after execution. You can't add code to prepare and clean up the environment in every test method.

This will involve the test fixture we mentioned before, modify test_mathfunc.py :

# -*- coding: utf-8 -*-

import unittest
from mathfunc import *


class TestMathFunc(unittest.TestCase):
    """Test mathfuc.py"""

    def setUp(self):
        print "do something before test.Prepare environment."

    def tearDown(self):
        print "do something after test.Clean up."

    def test_add(self):
        """Test method add(a, b)"""
        print "add"
        self.assertEqual(3, add(1, 2))
        self.assertNotEqual(3, add(2, 2))

    def test_minus(self):
        """Test method minus(a, b)"""
        print "minus"
        self.assertEqual(1, minus(3, 2))

    def test_multi(self):
        """Test method multi(a, b)"""
        print "multi"
        self.assertEqual(6, multi(2, 3))

    def test_divide(self):
        """Test method divide(a, b)"""
        print "divide"
        self.assertEqual(2, divide(6, 3))
        self.assertEqual(2.5, divide(5, 2))

We added setUp() and tearDown() two methods (in fact, these two methods of TestCase were rewritten). These two methods are executed once before and after the execution of each test method. SetUp is used to prepare the environment for the test, and tearDown is used to clean up The environment is ready for future testing.

Let's execute it again:

test_add (test_mathfunc.TestMathFunc)
Test method add(a, b) ... ok
test_divide (test_mathfunc.TestMathFunc)
Test method divide(a, b) ... FAIL
test_minus (test_mathfunc.TestMathFunc)
Test method minus(a, b) ... ok
test_multi (test_mathfunc.TestMathFunc)
Test method multi(a, b) ... ok

======================================================================
FAIL: test_divide (test_mathfunc.TestMathFunc)
Test method divide(a, b)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\py\test_mathfunc.py", line 36, in test_divide
    self.assertEqual(2.5, divide(5, 2))
AssertionError: 2.5 != 2

----------------------------------------------------------------------
Ran 4 tests in 0.000s

FAILED (failures=1)
do something before test.Prepare environment.
add
do something after test.Clean up.
do something before test.Prepare environment.
divide
do something after test.Clean up.
do something before test.Prepare environment.
minus
do something after test.Clean up.
do something before test.Prepare environment.
multi
do something after test.Clean up.

You can see that setUp and tearDown are executed once before and after each case is executed.

If you want to prepare an environmental case before any execution, and then clean up the environment after the implementation of all the case, we can use setUpClass() with tearDownClass():

...

class TestMathFunc(unittest.TestCase):
    """Test mathfuc.py"""

    @classmethod
    def setUpClass(cls):
        print "This setUpClass() method only called once."

    @classmethod
    def tearDownClass(cls):
        print "This tearDownClass() method only called once too."

...

The execution results are as follows:

...
This setUpClass() method only called once.
do something before test.Prepare environment.
add
do something after test.Clean up.
...
do something before test.Prepare environment.
multi
do something after test.Clean up.
This tearDownClass() method only called once too.

You can see that both setUpClass and tearDownClass are executed only once.

Skip a case

What if we temporarily want to skip a case and not execute it? Unittest also provides several methods:

skip decorator

...

class TestMathFunc(unittest.TestCase):
    """Test mathfuc.py"""

    ...

    @unittest.skip("I don't want to run this case.")
    def test_divide(self):
        """Test method divide(a, b)"""
        print "divide"
        self.assertEqual(2, divide(6, 3))
        self.assertEqual(2.5, divide(5, 2))

carried out:

...
test_add (test_mathfunc.TestMathFunc)
Test method add(a, b) ... ok
test_divide (test_mathfunc.TestMathFunc)
Test method divide(a, b) ... skipped "I don't want to run this case."
test_minus (test_mathfunc.TestMathFunc)
Test method minus(a, b) ... ok
test_multi (test_mathfunc.TestMathFunc)
Test method multi(a, b) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK (skipped=1)

You can see that the total number of tests is still 4, but the divide() method is skipped.

decorator skip a total of three unittest.skip(reason), , unittest.skipIf(condition, reason), unittest.skipUnless(condition, reason)unconditional skip skip, skipIf skipped when the condition is True, skipUnless skipped when the condition is False.

TestCase.skipTest() method

...

class TestMathFunc(unittest.TestCase):
    """Test mathfuc.py"""

    ...

    def test_divide(self):
        """Test method divide(a, b)"""
        self.skipTest('Do not run this.')
        print "divide"
        self.assertEqual(2, divide(6, 3))
        self.assertEqual(2.5, divide(5, 2))

Output:

...
test_add (test_mathfunc.TestMathFunc)
Test method add(a, b) ... ok
test_divide (test_mathfunc.TestMathFunc)
Test method divide(a, b) ... skipped 'Do not run this.'
test_minus (test_mathfunc.TestMathFunc)
Test method minus(a, b) ... ok
test_multi (test_mathfunc.TestMathFunc)
Test method multi(a, b) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.001s

OK (skipped=1)

The effect is the same as the decorator above, skipping the divide method.

Advanced-use HTMLTestRunner to output beautiful HTML reports

We can output a text execution report in txt format, but the text report is too simple, do you want a more tall HTML report? But unittest itself does not have HTML reports, so we can only turn to external libraries.

HTMLTestRunner is a third-party unittest HTML report library. First, we download HTMLTestRunner.py and put it in the current directory, or under your'C:\Python27\Lib', then it can be imported and run.

download link:

Official original version: http://tungwaiyip.info/software/HTMLTestRunner.html

Gray and blue modified version: HTMLTestRunner.py (format has been adjusted, Chinese display)

Modify our test_suite.py:

# -*- coding: utf-8 -*-

import unittest
from test_mathfunc import TestMathFunc
from HTMLTestRunner import HTMLTestRunner

if __name__ == '__main__':
    suite = unittest.TestSuite()
    suite.addTests(unittest.TestLoader().loadTestsFromTestCase(TestMathFunc))

    with open('HTMLReport.html', 'w') as f:
        runner = HTMLTestRunner(stream=f,
                                title='MathFunc Test Report',
                                description='generated by HTMLTestRunner.',
                                verbosity=2
                                )
        runner.run(suite)

In this way, during execution, we can see the execution in the console, as follows:

ok test_add (test_mathfunc.TestMathFunc)
F  test_divide (test_mathfunc.TestMathFunc)
ok test_minus (test_mathfunc.TestMathFunc)
ok test_multi (test_mathfunc.TestMathFunc)

Time Elapsed: 0:00:00.001000

And output the HTML test report HTMLReport.html, as shown in the figure:

html report

Now there is also a beautiful HTML report. In fact, you can find that the execution method of HTMLTestRunner is very similar to TextTestRunner. You can compare it with my example above, which is to replace the runner in the class diagram with HTMLTestRunner, and display the TestResult in HTML. If you study enough Deep, you can write your own runner to generate more complex and beautiful reports.

To summarize :

Unittest is a unit testing framework that comes with Python, and we can use it as a use case organization and execution framework for our automated testing framework.
The process of unittest: Write TestCase, then Load TestCase to TestSuite by TestLoader, and then Run TestSuite by TextTestRunner. The result of the operation is saved in TextTestResult. When we execute it through the command line or unittest.main(), main will call in TextTestRunner. run to execute, or we can execute use cases directly through TextTestRunner.
A class that inherits unittest.TestCase is a TestCase, and test the method starting with it is loaded as a real TestCase when it is loaded.
The verbosity parameter can control the output of the execution result. It 0 is a simple report, 1 a general report, or 2 a detailed report.
You can add a case or suite to the suite through addTest and addTests, and you can use the loadTestsFrom__() method of TestLoader.
With setUp(), tearDown(), setUpClass()and tearDownClass()may be disposed before the execution environment embodiment, and in the embodiment with environmental cleanup
We can skip a case through skip, skipIf, skipUnless decorators, or use the TestCase.skipTest method.
Add stream to the parameter, you can output the report to a file: you can use TextTestRunner to output txt reports, and you can use HTMLTestRunner to output html reports.