Automated testing is a major advancement in software testing technology. We all know that automated testing can improve work efficiency and reduce repetitive work, but in practice, we always encounter various problems, leading to the blind zone of automated testing . How to do automated testing is a problem that many companies urgently want to solve. Alibaba's one-stop R&D efficiency enhancement platform-Yunxiao, will open the live broadcast of "Alibaba Continuous Integration and Continuous Delivery Layered Automation" at 16:00 on December 8 to provide enterprises with automated testing solutions. Today we first look at some interpretations of Alibaba guests.
Guest introduction: Jin Tong: Senior Product Manager of Alibaba B2B Business Group. Engaged in the research and development and testing of Internet systems for many years. Now he is mainly responsible for the product design of cloud efficiency layered automated testing.
automated test
The so-called automated test is a test method that simulates the manual operation of the user with the program, which can help the tester to liberate from the repetitive and boring manual test. But in the process of automation practice, testers often find that there is a big gap between ideal and reality.
The above picture is not difficult to see that the number of automation failures in this week of Ali’s department not only is not proportional to the number of bugs found, but also wasted the time for the testers to investigate 41 automation failures, and this time is for the original intention of automated bug checking. Meaningless, unnecessary, and non-technical.
Why do external environments, business changes, application environment problems, execution machine problems, data problems, and framework problems all cause so many failures? But is the probability of finding bugs so low?
Combined with our many years of automation practice and summary, automation has these shortcomings as shown below:
Cost:
High personnel cost: Basically understand the code language of a certain automation framework, have a certain coding ability, and the code logic must be clear, otherwise how to ensure reasonableness, logic, business and robustness, which greatly affect the success rate of automation factor? How can we ensure that the automated test script itself has no bugs?
Environmental costs are high: development environment, operating environment, scheduling environment, etc., students who have been exposed to the code know that an environment installation cannot be completed in less than half a day or even a day. At the same time, if automation is connected to the project automation process, or scheduled monitoring, it is necessary to develop a scheduling platform. How much will these costs be for a test group from zero to a certain company, or even a company? How many man-days of work are required to complete these?
effect:
From the analysis of Figure 1, we can know the effect. However, Figure 1 is only a sample of a certain department of Ali in a single week, and 41 investigations have been wasted. If such an automated test runs for a year, what will the effect be? Can you ensure that there are no such interference failures? Can the number of failures be proportional to the number of bugs?
Coverage rate:
There are often students complaining that the coverage of automation is low, and many branches and logic cannot be covered. This is mostly due to the misunderstanding of these students. Many people regard UI automation as the entire automation test. However, there is no automated testing framework that can cover the testing of all the function points of a system, so there is a view that the coverage of "automation" is low. How to increase the coverage of automation?
Timeliness:
In fact, the failure of automation caused by the 10 business changes in Figure 1 is evidence of this shortcoming. The so-called business change refers to the normal project change, but the automation failure caused by the script not being updated in time. This failure just proves that automated testing is useful, as long as the content covered by the test changes, the automation can be tested. How to improve the timeliness of our scripts?
Faced with these problems, we can't help asking ourselves: Is it really necessary to do automated testing? If necessary, how to reduce these costs and how to improve test results? After continuous practice, we have introduced a layered automated testing strategy.
What is the concept of hierarchical automation?
Before understanding hierarchical automation, let's look at the automation test pyramid. The automated testing pyramid is familiar in the testing field, where ui represents page-level system testing, service represents service business testing (interface testing), and unit represents unit testing. The higher the pyramid, the greater the effort and workload required.
Unit test (unit):
It can simulate various abnormal scenarios through the mock framework, with the least external dependence, and a test method that can achieve the smallest test granularity. Because of the low dependency, it can be easily executed anytime and anywhere, and troubleshooting is simple. This is the foundation of all tests.
Interface test (service):
The tester is required to be very clear about the structure of the system and the scheduling between systems, and at the same time understand the logical relationship of the interface, otherwise it is easy to miss some abnormal scenarios in the interface test code. Because this layer contains some business logic and an integration of multiple interfaces, compared to unit testing, there are more external dependencies, resulting in problem positioning not as accurate as the unit testing layer. Therefore, the investment will be more than unit testing.
Page test (ui):
It is a common black box automated test scenario. It is the closest to the user's real scene, and it is easy to find problems, but its implementation cost is the highest and it is too easy to be externally dependent, which affects the success rate of the script, so it is at the top of the pyramid, but it is not the whole of the pyramid. 80% of the disadvantages of automated testing are due to ui automation.
The above is the main three layers of hierarchical automation. It can be seen that hierarchical automated testing advocates a test strategy that divides the system into layers and uses appropriate automated methods to test at different levels. Whether a project can be covered by automation depends on whether the test leader’s layering strategy is reasonable and comprehensive.
Through practice, it is found that if you simply use layered automated testing, the R&D test ratio can be increased to 4:1 at most, then the average Alibaba R&D test ratio is 8:1. How is this achieved?
Alibaba's layered automation practice
In the long-term practice of Alibaba’s hierarchical automation, the two important shortcomings of automation cost and effect
To make breakthroughs, carry out the dual revolution of layered automation tools and project process, and finally achieve the industry-leading R&D test ratio.
First, the layered automation tool revolution
Automated testing frameworks, whether ui, interfaces or units, external open source frameworks, fee-based software, etc., have many advantages and disadvantages. Alibaba Test integrates the practice of multiple frameworks, improves and innovates them, breaks through the many problems of traditional automation frameworks, greatly reduces the cost of automation, and improves the effect of automation. As shown in the figure below, the four important tools, AUI focuses on UI automation, SAT focuses on interface automation, Amon focuses on unit testing, and Perf focuses on performance. The weaknesses of the traditional testing framework are fully overcome and modified, and finally shotguns are replaced. Comprehensively improve test efficiency.
Not only that, but Alibaba Cloud also implements tool-based and platform-based manual work in the entire project process from demand-development-testing-release, as shown in the figure.
The development link starts from branching, to the deployment environment and unit test of self-test, all platform tooling. One-click branching, one-click deployment, and one-click single test integration. You can view the results of environment deployment and code scan results such as findbugs, PMD, Sonar, etc. in less than a cup of coffee.
In the testing process, there are two main products, useful cases and defects in manual testing. The platform is precipitated, and there is no need to do some file transfer. In addition to the layered automation related test platform and tools introduced above, the efficiency of automated testing is improved and finally realized Platform and tooling of the overall test work.
Second, the project process revolution
In addition to the cost reduction and effectiveness improvement of a single tool, cloud efficiency also optimizes the project process.
The following figure shows our common project process, in which automated testing work often only has a single automated testing framework for testing.
Through the previous analysis, hierarchical automation can fully cover our entire project functions. Therefore, we changed the process to the process shown in the figure below.
Such a process, after long-term practice, has found that the R&D test ratio has increased to 3:1 at most. Is there room for improvement?
When we look at these processes again, we can see that testing, especially automated testing, is independent of the development project process. The most direct problem brought about by this process is that the automatic discovery of problems is not timely, and there is no good intervention guarantee for the development of self-test projects. At the same time, it is triggered manually, and the influence of human factors is very large. This is important for limiting the development and testing ratio. the reason.
Assuming that our project uses a hierarchical automated testing strategy, and integrates its triggering-troubleshooting-result feedback into the entire demand-development-test-release project process, what effect will it produce? What?
The automated result feedback has become a necessary part of the release, and every automated test result has become an important development part of the release. What effect will it produce?
Can our automated testing only be used for test-involved project testing, or can it only be used for test-involved releases? Can we still use it for daily business protection? Is it possible to discover problems in the early stages of bugs in the project?
Whether it is the test tool revolution or the test process improvement, we have many questions. It is these questions that make automated testing a double-edged sword. If used well, it can greatly improve efficiency and reduce cost. If not used well, it may become a double-edged sword. Chicken ribs hinder efficiency improvement. Therefore, learning how to conduct automated testing, how to grasp the hierarchical strategy of hierarchical automation, and how to integrate hierarchical automation into the project process are the testing skills that companies must master.
Recommend good articles:
How do automated testing practitioners choose jobs? What is the future career development?
The basics of automated testing, everything you know and not know is here!
A pack of sad spicy strips to share-top ten best automated testing tools
About software testing! Everything you want to know is here, Xiaobai must see!
What you need to know before doing automated testing
10 years of software testing engineers' perceptions-to friends who are still confused
What kind of person is suitable for software testing?
Knowledge to understand python automated testing(3)
Which is more suitable for automated testing, Python or Java?
The daily work of software testers
Play with Python+Selenium automated testing in 10 minutes, and teach you a quick start!
Finally: Welcome to follow the editor to receive a summary of the core knowledge of Python automated test engineers with a 300-page pdf document! Software testing technology exchange group: (313782132) The content of these materials are all the knowledge points that the interviewer must ask during the interview. The chapter includes many knowledge points, including basic knowledge, Linux essentials, Shell, Internet program principles, Mysql Database, package capture tool topics, interface testing tools, advanced testing-Python programming, Web automation testing, APP automation testing, interface automation testing, advanced continuous integration testing, testing architecture development testing framework, performance testing, security testing, etc.