foreword
This article introduces and compares five Python frameworks for automated testing to help you make a reasonable choice of tools, and to smoothly carry out and accelerate application testing. Public Zhonghao [Programmer Ashin] Reply 66, you can get learning materials for Python automated testing!
Therefore, developers and testers need to consider many factors when choosing the best testing framework for the project at hand, including: the scripting quality of the framework, the simplicity of the test cases, and the possible technical weaknesses of the running module. In order to avoid the "difficulty of choosing", I have prepared five types of automated testing frameworks in Python for you to compare and discuss.
1.Robot Framework
As one of the most important Python testing frameworks, Robot Framework is mainly used in test-driven development and acceptance. Although developed in Python, it also runs on .Net-based IronPython and Java-based Jython. At the same time, as a Python framework, Robot is also compatible with platforms such as Windows, MacOS, and Linux.
Use prerequisites
Before using Robot Framework(RF), you need to install Python 2.7.14 and above. I recommend that you use Python 3.6.4 to ensure that appropriate comments are added to code snippets and changes to the program can be tracked. At the same time, you will also need to install the Python package manager – pip.
Of course, you must also download the corresponding development framework, such as the PyCharm Community Edition, which is popular among developers. Also, since the code snippets themselves are not dependent on any IDE, you can stick with the IDE tools you have at hand.
advantage
By using a keyword-driven-test approach, it can help testers easily create readable test cases, which in turn simplifies the entire automation process.
Users can easily test the syntax in the data.
Because it is composed of various common tools and test libraries, it has a large ecosystem and can use its various elements in separate projects.
The framework is highly extensible due to having many types of APIs.
Although not a built-in feature, Robot can pass the Selenium Grid (see https://www.lambdatest.com/blog/lambdatest-now-live-with-an-online-selenium-grid-for-automated-cross-browser- testing/) to help users run various parallel tests.
shortcoming
Despite all the above mentioned conveniences, Robot Framework is more cumbersome in creating custom HTML reports. At most you can use it to generate short reports in xUnit format.
In addition, Robot Framework is not very good at parallel testing.
Comparison of Robot's Features and Competitive Products
Since Robot has a wealth of built-in libraries and can use a simpler test-oriented DSL (Domain Specific Language), if you are new to the field of automated testing and lack development experience, it is better to use it as a Python testing framework than Pytest or Pyunit, which will be mentioned below, is easier to get started with. Of course, if you need to develop complex automation frameworks, then please also use Pytest, or other types of Python frameworks.
2.Pytest
Pytest, suitable for many kinds of software testing, is another Python-type automation testing framework. With its open source and easy-to-learn features, the tool is often used by QA (Quality Analysis) teams, development teams, individual teams, and various open source projects. Given that Pytest has useful features like "assert rewriting", many large Internet applications, such as Dropbox and Mozilla, have switched from unittest (Pyunit) to Pytest, which will be mentioned below.
Use prerequisites
In addition to basic Python knowledge, users do not need more technical reserves. In addition, users only need to have a test device with a command line interface, and have installed the Python package manager and IDE tools for development.
advantage
In the past, developers had to wrap their tests in various large classes. Today, Pytest allows users to write more compact test suites.
Other testing tools require developers or testers to use a debugger, or to detect the source of a particular value by means of log inspection. When users write test cases with Pytest, they can store all the values in the test cases until they know which values fail and which values are asserted.
Since there is not much boilerplate code involved, users can easily write and understand various tests.
Test fixture functions are often used to add a parameter to a test function and return a different value. In Pytest, you can modularize one fixture by using another. At the same time, users can use multiple fixtures to cover tests to all combinations of parameters without rewriting test cases.
Pytest developers keep the framework extensible by publishing various useful plugins. For example: pytest-xdist can be used to execute parallel tests without using other testers. At the same time, unit tests can also be parameterized without duplicating any code.
By providing developers with a variety of special routines, it can make the writing of test cases easier and less error-prone, and the code will be shorter and easier to understand.
shortcoming
The special routines mentioned above also mean that the user must give up certain compatibility. Although it is convenient for users to write test cases, these test cases cannot be used with any other test framework.
Comparison of Pytest Features and Competitive Products
In terms of writing functional test cases and developing complex frameworks, Pytest outperforms UnitTest which will be mentioned below. However, it is similar to Robot Framework and is only suitable for developing simple test frameworks.
If you are considering using Pytest, please refer further to Test Automation with Pytest and Selenium WebDriver.
3.UnitTest/PyUnit
Inspired by JUnit, UnitTest/PyUnit is also a standardized Python-like automation testing framework for unit testing. Its base class TestCase provides various assertion methods, as well as all cleanup and setup routines. Therefore, each method in a TestCase subclass is prefixed with "test" to identify that they can be run as test cases. Users can use the load method and the TestSuite class to group and load various tests. Of course, you can also build custom test runners by combining them. Just as we use Junit to test Selenium, UnitTest also uses UnitTest-sml-reporting and can generate various XML-type reports.
Use prerequisites
Since UnitTest uses Python by default, we don't need any prerequisites. In addition to basic knowledge of the Python framework, you can additionally install pip, as well as IDE tools for development.
advantage
Developers do not need to install any other modules.
UnitTest is a derivative of xUnit and works very similarly to other xUnit frameworks. Therefore, for those who do not have a strong Python background, they can also get started quickly.
Users can run individual test cases in an easier way. All you need to do is pre-order a name on the terminal, and the framework flexibly executes tests for various use cases and produces refined output.
It is able to generate various test reports within milliseconds.
shortcoming
Although snake_case is commonly used by the framework to name various Python code, since it is derived from Junit, it still retains some traditional camelCase naming methods. This is often confusing.
Because it supports too many abstract methods, the purpose of the test code is sometimes unclear.
Requires a lot of boilerplate code.
Comparison of PyUnit's Features and Competitive Products
Similar to the opinion of some Python developers, I think: Pytest can cultivate testers to write better automation code in a very compact way, this programming habit. Although UnitTest is the default automated testing framework for Python, its working principles and naming conventions are still slightly different from standard Python code. At the same time, it also requires too much boilerplate code. So the framework is not very popular.
4.Behave
We all know that behavior-driven development (BDD) is an innovative method based on agile software development. It encourages collaboration among developers, business participants, and QA staff. As another Python testing framework, Behave allows teams to avoid complex situations and perform BDD testing. Essentially, this framework is very similar to SpecFlow and Cucumber, and is often used to perform automated tests. Users can write test cases in an easy-to-read language that can be pasted into the code during their execution. Moreover, those set behavioral norms and procedures can also be reused in other test solutions.
Use prerequisites
Anyone with basic knowledge of Python can use Behave. Other prerequisites include:
You must first install Python 2.7.14 and above.
Requires the Python package manager or pip to work with Behave.
Most developers will choose Pycharm as the development environment, of course, you can also choose other IDE tools.
advantage
Behave helps maintain consistent behavior across an organization because the system behavior uses a semi-formal language, as well as a domain vocabulary.
Let those development teams working on different modules have similar characteristics by coordinating processing.
Each function block it builds can execute various test cases.
Because there are more details worth reasoning and thinking about, the target product will have better specification.
Based on a similar specification format, it enables project managers and stakeholders to have a clearer view of the output of development and QA teams.
shortcoming
For black box testing only.
Behave's Features Compared with Competitors
Behave is best for black box testing like simple web testing. However, Behave is not a good choice for integration and unit tests with complex scenarios. As an alternative to Behave, some developers and testers recommend pytest-bdd. The tool incorporates all the advantages of Pytest and implements various behavior-driven testing scenarios.
If you are considering using Behave, then you can borrow Behave with Selenium (see https://www.lambdatest.com/support/docs/display/TD/Behave+with+Selenium±+Running+Behave+Automation+ Automation scripts mentioned in Scripts+on+LambdaTest+Selenium+Grid).
5.Lettuce
Lettuce is another behavior-driven automation tool based on Cucumber and Python. Lettuce mainly focuses on those mundane tasks that are characteristic of behavior-driven development. Not only is it easy to use, but it can make the entire testing process smoother and even more fun.
Use prerequisites
You need to install Python 2.7.14 with IDE, and above. Of course, you can also use Pycharm or any other IDE tool. At the same time, you also need to install the Python package manager.
advantage
Similar to other BDD testing frameworks, Lettuce enables developers to create multiple scenarios and describe different functions in simple natural language.
According to a similar specification format, development and QA teams can work together in coordination.
For black box testing, Lettuce is well suited for those behavior-driven test cases.
shortcoming
To successfully implement behavior-driven testing, continuous communication is required between the development team, QA staff, and stakeholders. The absence of such communication can lead to blurred processes and problems for the team.
Competitive comparison
In the eyes of regular developers and automation testers, Cucumber is more practical for performing BDD tests. In the eyes of Python development and QA personnel, Pytest-bdd is the best. The power of Pytest is that its framework enables compact and easy-to-understand code that is tightly integrated into behavior-driven testing.
Epilogue
To sum up, among the above five Python frameworks for automated testing, Pytest, Robot Framework, and UnitTest can be mainly used for functional and unit testing, while Lettuce and Behave are only suitable for behavior-driven testing. By further comparison, we believe: For functional testing, Pytest is the best. If you are new to Python-based automation testing, Robot Framework is the best tool for getting started. Although its functionality is somewhat limited, it is very easy to use. For Python-based BDD testing, Lettuce and Behave are equally good. However, if you already have some experience with Pytest, then ***please use Pytest-bdd. ***, I hope this article can help you choose a suitable Python testing framework and carry out your testing work smoothly.