Overview
We mainly do some exchanges on automated testing of interface functions from two aspects, one is what is automated testing, the other is why automated testing, and also include some of our own understanding of the following three issues:
Clarify why
automated testing is carried out Clarify how automated testing is carried out Clarify the
evaluation
of automated testing implementation Cognition on the basis of automation:
common misconceptions
Use automation to completely replace manual testing.
Discover more new bugs with automated tests.
What kind of cognition should be formed
The purpose of automated testing is not simply to reduce or replace manual testing, but to enable testers to do more and more meaningful testing (including manual testing).
Automated tests are used to verify that functionality that previously worked correctly still works.
Why automate testing
Not to automate for the sake of automation, but to carry out some kind of automation in order to implement a set of solutions to solve the problem, certainly to solve some problems in the testing process and introduce automated testing.
Pros and Cons
Now that you choose to automate some problems, you must first understand the pros and cons of automated testing itself.
1. About the
cost
The cost of machine resources replaces the cost of labor, which solves the problem of repetitive test execution costs to a certain extent.
2. About efficiency
Improve the efficiency of test execution and shorten the test cycle, to a certain extent, solve the problem that the test cycle increases with the increase of version iterations (increase of function points).
3. About test coverage
Testing functions through the recording and playback of automated testing tools and data-driven testing can improve test coverage and solve the problem of low test coverage in regression testing to a certain extent.
4. About finding problems
Automated testing has good consistency and repeatability, and to a certain extent solves the problem of consistency in the process of manual repeated execution.
5. About the process
Automated testing tools are introduced into the entire testing process as a role to improve the fluency of test execution.
cons
1. About personnel
The testers are additionally required to have certain test development capabilities, which introduces the problem of higher requirements for the testers' capabilities.
2. About the cost
The cost of automation test development varies depending on the choice of automation framework (or tool), but all have high development costs, which introduces the issue of development costs.
3. About maintenance
With version iterations and functional changes, the problem of automated code development and maintenance is introduced.
4. About finding problems
Due to its own limitations (mostly used in regression testing and stable version scenarios), automated testing finds fewer problems.
When we recognize the advantages of automation, have we considered the impact of its disadvantages, can we deal with it, and how?
If you want to do good work, you must first sharpen your tools
Through the above, it is not difficult to find that automated testing is a double-edged sword. Therefore, we need to consider the following aspects as to whether automation is carried out:
do and don't
Products in the IT industry and even other industries can be automated, so whether automation is not a question of whether or not it can be done, but whether there is a suitable time or stage and a suitable way to do it. Before implementing automated testing, product development needs to be done. Process analysis, usually need to meet the following conditions at the same time:
The software requirements change infrequently (more than 10% of the changes are frequent changes, of course, 10% is not a fixed value)
The project cycle is long enough The
automated test cases can be reused
The scenarios that are generally suitable for software test automation are as follows:
Regression testing, repeating a single test operation causes unnecessary waste
. It is also mentioned above that automated testing must be introduced to solve some problems in the testing process. If it is automated for the sake of automation, it will undoubtedly fail.
Airplane and Cannon
The choice of framework and tools is the first problem we face after confirming the automation. There was a stalk on the Internet before that instant noodles have no soul when they are cooked. Of course, this is a joke. The development of automated testing must be selected based on the characteristics of the system under test. Blindly choosing an automated testing framework (or tool) regardless of the characteristics and scenarios of the system under test (system framework) is soulless, and the probability of automation failure will be relatively high.
Let's briefly introduce several automated test-driven frameworks for automated development.
First of all, we need to understand that the automated testing framework is more inclined to a design idea. This idea guides the use of tools or self-developed development, and not only one kind of framework can be used. Combined with the characteristics of the system under test, it is generally to choose a variety of test frameworks combination of testing and design requirements (development, maintenance perspective).
Record playback test framework
The principle adopted by the recording and playback testing framework is to achieve the purpose of automated testing by recording the linear scripts generated by the application and playing them back.
Advantages: The tester's test development ability is the least required, and the required script can be obtained by recording.
Disadvantages: generally do not have the ability to make logical judgments, poor maintainability, and low efficiency.
Adapt to the scene: Not recommended, the traditional UI automation test is gradually weakened. Regarding U automation, it is necessary to know whether the system under test meets the conditions for automation. In a project where the system under test changes frequently, the implementation of UI automation is undoubtedly a big hole, and the later maintenance work is enough to make the heart tired. Forced to abandon automated testing.
The Test Library Architecture Framework
The core idea of the test library framework can be summarized as the decoupling of system function operation and business logic. All functional operations supported by the test system are encapsulated in the test library. The test script calls the test library and transmits external test data. The test library is written by the automated test development project (you may not understand the business) and is responsible for the change of the control. And maintenance, the writing of test scripts can be written by the automated test development project that has a good grasp of the business, and is responsible for the change and maintenance of business logic and test data.
Advantages: No matter which layer of the system under test changes (code layer or business layer, etc.), only the corresponding personnel are required to change and maintain.
Disadvantages: The maintenance work caused by the change is attached to the automated test development engineers and business testers at the same time, and the maintenance code is built up.
Adaptation scenarios: This framework is generally applied based on various automated development methods (tool-based or non-tool-based self-developed R&D + continuous integration).
The Data-Driven Testing Framework
The core idea of data-driven can be summarized as data (test data, configuration data) and code decoupling. The principle of this framework is to use data-driven scripts for testing. Data-driven scripts store data input in a separate data file. The script only stores the code. The input of the script is read directly from the file at runtime, so the same script (Code template) can be run in different test cases, realizing the separation of code and data.
Advantages: For business personnel to convert from code-oriented development to configuration-oriented design (parameter combination design), the development difficulty and development cost are reduced, and the scalability of test cases is improved, similar tests can be quickly expanded, and automated code is realized Does not increase with the growth of use cases.
Disadvantages: The maintenance of test scripts is the responsibility of automated test development engineers, requiring knowledge of automated programming and business logic. The initial test script design cost is high and has certain limitations (for the same test content and with the same test Logic).
Applicable scenarios: It is more suitable for scenarios where the test content and test logic are highly repetitive, and the tested object has high requirements on the scalability and reusability of test cases.
The Keyword-Driven or Table-Driven Testing Framework
Keyword-driven is an extension of data-driven logic. Its core idea can be summarized as data code flow (logic) decoupling, and at the same time, it completes the mapping between code and test description (test description for the object under test). The principle of the framework is based on the data-driven basis, which completes the splitting, abstracting, and encapsulation of the tested objects to map them into "keywords" (test descriptions). When writing test cases, only the keywords are required. By combining, the test case development of different scenarios can be completed.
Advantages: For business manual testers, the development from code or configuration-oriented development to natural language (test description)-oriented development reduces development difficulty and maintenance cost to the greatest extent, and improves the scalability and organization of test cases. This enables automation code not to grow with the growth of use cases.
Disadvantages: It requires a high degree of test development ability and business understanding of testers.
Applicable scenarios: The object under test contains complex business processes (logic), of course, complex ones can be made simpler and more ok.
What is automated testing
Knowledge about automation
Automated testing is the process of converting human-driven testing behavior into machine-executed code.
Usually, after designing the test case and completing the review, the tester performs the side test step by step according to the test case, and obtains the comparison between the actual result and the expected result.
In this process, in order to reduce the input of manpower, time or hardware resources and improve the efficiency of test execution, the concept of automated testing was introduced.
Processes that are automated
Automated testing is not only about the use of tools and code development, it is all about ensuring product quality and services. Like manual testing, the core of automated testing is test analysis and design. The performance of automated testing itself depends on whether the appropriate method is selected and tested. Adequate test analysis and coverage is done.
Test Analysis and Test Case Design
Regarding how to conduct test analysis and test design, because there are too many introductions, special exchanges will be held in the future. I will not explain them here. Remember to pay attention.
Automated test pre-research
Two clear:
Clarify the background of the system under test (the development or testing stage of the system under test, changes in requirements, project cycle, etc.)
Clarify the automation testing goals (specific modules, automation conversion rate, efficiency improvement, etc.)
Test framework selection
Only in terms of implementation, many automated testing frameworks (or tools) can be automated, or any one of them is not very reluctant, so the choice of automation frameworks (or tools) is not artificial (what would I do? , or which framework is easier to master), but choose an automation framework (or tool) based on the object under test.
Two scenarios:
Scenario 1: The multi-dimensional query function is similar to the filtering query of a certain treasure product.
Scenario 2: A relatively complex business process, similar to a level-by-level approval process system.
Thinking: Regarding these two scenarios, how do we choose an automated testing framework (or tool)?
Scenario 1 Features: Similar to the WEB terminal of a certain treasure, the multi-dimensional (brand, origin) combined query of different keywords (clothes, football) has a single query logic and strong reusability. Recommended use: test library architecture framework + data-driven framework.
Scenario 2 Features: Similar to the approval application process, it contains complex business logic and multi-user roles. Recommended use: test library framework + data-driven framework (configuration decoupling) + keyword-driven framework
test case development
two notes
Normative and fit: The development norm and the development process must be in line with the idea of the automated testing framework. If you choose the test library framework, then when you use reverse programming, you find that there are still functional operations that need to be encapsulated. Develop in the test library, call in the use case, instead of developing in the use case.
Development cost and maintenance cost: The development cost and maintenance cost must be considered in the development and design. The development cost determines the efficiency, and the maintenance cost determines whether the automation can continue to operate effectively. At the same time, pay attention to the solution to the cost problem. On the premise of effective coverage of the object, the cost is reduced through the framework design and optimization scheme, rather than by doing less and more coarsely to reduce the cost.
For example, when selecting the test library framework to realize the automation of the query business in the above scenario:
If we write an automated use case for "keyword (shoes) + brand (Nike)", and write another automated use case for "keyword (shoes) + brand (Nike) + shipping location (Nanjing)", is there any way to do this? Where to optimize?
Combined with the business characteristics, we use the data-driven framework on the basis of the test library framework to separate its single query logic from the test data, so that when we develop other use cases, we only need to add test data (input, Corresponding expectations), avoiding the development and maintenance costs caused by repeated development.
test case execution
Jenkins is generally used for continuous integration.
Test case result feedback
Two requirements:
The result information is clear: what is the specific reason for the failure
The result information is clearly stated: the specific reason for the failure
Test case maintenance
a police station
Be vigilant that when the maintenance cost of use cases increases with the number of use cases, it is necessary to optimize the use case design framework in time to avoid excessive use case maintenance costs and lead to use cases being abandoned or abandoned.
Thinking: How to evaluate how well our automated testing is carried out?
The following dimensions, let's see how to choose? (multiple choice questions)
About automated testing motivation
What is the background of our automated testing requirements?
The project cycle is long, the old functions tend to be stable, the project manpower is tight, more things need to be done, and the investment in repeated testing needs to be reduced?
Other teams are doing automation, and we are also following suit.
For automated testing purposes
For what purpose do we carry out automated testing? What is the purpose we hope to achieve, which needs to be clarified in the pre-research stage of automation.
Improve test efficiency, reduce repetitive investment, and free up time to do more things.
The purpose of automation is to complete automation.
About automation itself
Does the chosen automated testing framework (or tool) fit the system under test, evaluating development costs, maintenance costs, and solutions provided?
Combined with business feature scenarios, select the appropriate time and method to carry out automation
Before automation development, conduct sufficient test analysis and design to ensure appropriate test coverage (test point test results).
Other teams are using XX tools, and we can learn to use them to automate our team.
Just create one or two test data to cover an interface, and expect the result to be verified.
About the effect of automated testing
whether we have achieved our set goals.
Have we done more valuable things, such as using the time saved by automation to do more in-depth analysis of the core business, dig out some problems, etc. Get moving, you don't need manpower to do it.
This guy is really badly maintained, and the maintenance is not good. Forget about manual testing.
Encouragement: [Tutorials that may help you]
These materials should be the most comprehensive and complete preparation warehouse for friends who do [software testing]. This warehouse has also accompanied me through the most difficult journey. I hope it can also help you! Everything should be done as early as possible, especially in the technology industry, and the technical foundation must be improved.
Follow my WeChat public account [Program Yuanmuzi] to get it for free~