Since it was selected as a programming language in 2018, Python has always been at the top of the major rankings. It currently ranks third in the Tiobe Index, behind Java and C. With the widespread use of this programming language, a Python-based automated testing framework has also emerged, and continues to develop and enrich.
Therefore, developers and testers need to consider many factors when choosing a test framework for the project at hand, including: the script quality of the framework, the simplicity of test cases, and the possible technical weaknesses of running modules. In order to avoid "choice difficulty", I have prepared five types of Python-type automated testing frameworks for comparison and discussion.
1.Robot Framework
As one of the most important Python testing frameworks, RobotFramework is mainly used in the development and acceptance of test-driven types. Although developed in Python, it also runs on .Net-based IronPython and Java-based Jython. At the same time, as a Python framework, Robot is also compatible with platforms such as Windows, MacOS, and Linux.
Use prerequisites
Before using Robot Framework(RF), you need to install Python 2.7.14 and above. I recommend you use Python 3.6.4 to ensure proper comments are added to code segments and to track program changes. At the same time, you also need to install the Python package manager – pip.
Of course, you must also download the corresponding development framework, for example: PyCharm Community Edition, which is more popular in the developer circle. In addition, since the code snippet itself does not depend on any IDE, you can completely use the existing IDE tools at hand.
advantage
- By using the keyword-driven-test method, it can help testers easily create readable test cases, thus simplifying the entire automation process.
- Users can easily test the syntax in the data.
- Because it is composed of various common tools and testing libraries, it not only has a large ecosystem, but also its individual elements can be used in separate projects.
- The framework is highly extensible due to having many types of APIs.
- Although not a built-in function, Robot can help users run various parallel tests through Selenium Grid
shortcoming
Despite all the above conveniences, Robot Framework is cumbersome in creating custom HTML reports. At most you can use it to generate short reports in xUnit format.
In addition, Robot Framework is not good at parallel testing.
Robot's features compared with competing products
Since Robot has a rich built-in library and can use a simpler test-oriented DSL (Domain Specific Language), if you are new to the field of automated testing and lack development experience, it is better to use it as a Python testing framework than Pytest or Pyunit, which will be mentioned below, is easier to get started. Of course, if you need to develop a complex automation framework, please use Pytest, or other types of Python frameworks.
2.Pytest
Pytest, which is suitable for a variety of software testing, is another Python-type automated testing framework. With its open source and easy-to-learn features, this tool is often used by QA (Quality Analysis) teams, development teams, personal teams, and various open source projects. Given that Pytest has practical functions such as "assert rewriting (assertrewriting)", many large Internet applications, such as Dropbox and Mozilla, have switched from the unittest (Pyunit) mentioned below to Pytest.
Use prerequisites
Apart from basic Python knowledge, users do not need more technical reserves. In addition, users only need to have a test device with a command line interface, and have installed the Python package manager and IDE tools for development.
advantage
- In the past, developers had to wrap their tests in various large classes. Today, Pytest allows users to write more compact test suites.
- Other testing tools require the developer or tester to use a debugger, or log inspection to detect the source of a particular value. When users use Pytest to write test cases, they can store all the values in the test case until they learn which values failed and which values were asserted.
- Since there is not much boilerplate code involved, users can easily write and understand various tests.
- Test fixture (Fixture) function is often used to add a parameter to the test function, and return a different value. In pytest, you can modularize one fixture by using another. At the same time, users can use multiple fixtures to cover all combinations of parameters without rewriting test cases.
- The pytest developers keep the framework extensible by releasing various utility plugins. For example: pytest-xdist can be used to execute parallel tests without using other testers. At the same time, unit tests can also be parameterized without duplicating any code.
- By providing developers with various special routines, it can make the writing of test cases easier and less error-prone, and the code will become shorter and easier to understand.
shortcoming
The special routines mentioned above also mean that users must give up certain compatibility. Although it is convenient for users to write test cases, these use cases cannot be used with any other testing framework.
The characteristics of pytest are compared with competing products
In terms of writing functional test cases and developing complex frameworks, Pytest outperforms UnitTest, which will be mentioned below. However, it is similar to Robot Framework and is only suitable for developing simple test frameworks.
3.UnitTest/PyUnit
UnitTest/PyUnit, inspired by JUnit, is also a standardized Python-like automated testing framework for unit testing. Its base class TestCase provides various assertion methods, and all cleanup and setup routines. Therefore, every method in the TestCase subclass is prefixed with "test" to indicate that they can be run as test cases. Users can use the load method and the TestSuite class to group and load various tests.
Of course, you can also build a custom test runner by combining them. Just as we use Junit to test Selenium, UnitTest will also use UnitTest-sml-reporting and can generate reports of various XML types.
Use prerequisites
Since UnitTest uses Python by default, we don't need any prerequisites. In addition to the basic knowledge of the Python framework, you can also additionally install pip and IDE tools for development.
advantage
- Developers do not need to install any other modules.
- UnitTest is a derivative of xUnit, and its working principle is very similar to other xUnit frameworks. So for those who don't have a strong Python background, they can get started quickly.
- Users can run individual test cases in a much simpler manner. You only need to pre-order the name on the terminal, and the framework flexibly executes tests for various use cases and produces refined output.
- It is able to generate various test reports within milliseconds.
shortcoming
Although the framework commonly uses snake_case to name various Python codes, since it is derived from Junit, it still retains some traditional camelCase naming methods. This tends to be confusing.
Because it supports too many abstract methods, the purpose of the test code is sometimes unclear. Requires a lot of boilerplate code.
The characteristics of PyUnit compared with competing products
Similar to the views of some Python developers, I think: Pytest can cultivate the programming habit of testers to write better automation code in a very compact way. Although UnitTest is the default automated testing framework for Python, its working principle and naming rules are still slightly different from standard Python code. At the same time, it also requires too much boilerplate code. So the framework is not very popular.
4.Behave
We all know: behavior-driven development (behavior-driven
development, BDD), is a method based on agile software development. It encourages collaboration among developers, business actors, and QA personnel. As another Python testing framework, Behave allows teams to avoid various complex situations to perform BDD testing. Essentially, the framework is very similar to SpecFlow and Cucumber, and is often used to perform automated testing. Users can write test cases in an easy-to-read language that can be pasted into the code during its execution. Moreover, those behavioral norms and steps that are set can also be reused in other test solutions.
Use prerequisites
Anyone with basic knowledge of Python can use Behave. Other prerequisites include:
You must first install Python 2.7.14 and above.
Requires a Python package manager or pip to work with Behave.
Most developers will choose Pycharm as the development environment, of course you can also choose other IDE tools.
advantage
- Because the system behavior uses semi-formal (semi-formal) language, and domain vocabulary (domain
- vocabulary), so Behave helps maintain consistent behavior across organizations.
- Let those development teams working on different modules have similar characteristics through coordinated processing.
- The individual functional blocks it builds are capable of executing various test cases.
- Since there are more details worthy of user reasoning and thinking, the target product will have better specification.
- Based on a similar specification format, it enables project managers and stakeholders to have a clearer understanding of the output of the development and QA teams.
shortcoming
Only suitable for black box testing.
Behave's features compared with competing products
Behave is best suited for black box testing like simple web testing. However, Behave is not a good choice for integration tests and unit tests with complex scenarios. As an alternative to Behave, some developers and testers will recommend pytest-bdd. This tool combines all the advantages of Pytest and implements various behavior-driven testing scenarios.
5.Lettuce
Lettuce is another behavior-driven automation tool based on Cucumber and Python. Lettuce mainly focuses on those mundane tasks that are characteristic of behavior-driven development. Not only is it easy to use, but it also makes the entire testing process smoother and even more fun.
Use prerequisites
You need to install Python 2.7.14 and above with IDE. Of course, you can also use Pycharm or any other IDE tool. At the same time, you also need to install the Python package manager.
advantage
- Similar to other BDD testing frameworks, Lettuce enables developers to create multiple scenarios and describe different functions in simple natural language.
- Based on a similar specification format, development and QA teams can coordinate and work together.
- For black box testing, Lettuce is very suitable for those behavior driven test cases
shortcoming
To successfully implement behavior-driven testing, there needs to be ongoing communication between the development team, QA staff, and stakeholders. Lack of such communication can create process ambiguity and lead to problems for teams. Competitive comparison
In the opinion of average developers and automation testers, Cucumber is more practical in performing BDD tests. In the eyes of Python developers and QA personnel, Pytest-bdd is the best. The power of pytest lies in: its framework enables compact and easy-to-understand code, and can be tightly integrated into behavior-driven tests.
epilogue
To sum up, among the above five automated testing Python frameworks, Pytest, Robot Framework and UnitTest can be mainly used for functional and unit testing, while Lettuce and Behave are only suitable for behavior-driven testing. Through further comparison, we think: For functional testing, Pytest is the best. If you are new to Python-based automated testing, RobotFramework is the best tool to get you started. Although its functionality is limited, it is very easy to use. For Python-based BDD testing, Lettuce and Behave are equally good. However, if you already have some experience with Pytest, please use Pytest-bdd. I hope this article can help you choose a suitable Python testing framework and carry out testing work smoothly.
Finally, I would like to thank everyone who has read my article carefully. Reciprocity is always necessary. Although it is not a very valuable thing, you can take it away if you need it:
These materials should be the most comprehensive and complete preparation warehouse for [software testing] friends. This warehouse has also accompanied tens of thousands of test engineers through the most difficult journey, and I hope it can help you! Partners can click the small card below to receive