In software testing, we often encounter two basic problems (difficulties):
- It is difficult to guarantee that there will be no missed tests : we have done a lot of tests, but we don’t know how the tests will go, and we have no confidence that there will be problems after the software goes online;
- Select test cases to be executed : When faced with a large number of regression test cases, we do not have enough time to complete the test. How to select effective test cases? Although we will have some strategies, such as risk-based testing strategies, operating profile-based testing strategies or combined testing strategies, they mainly rely on the experience of testers and are relatively subjective.
People tried to solve such basic problems, which gave rise to "precision testing". Under the agile development model, the development pace is accelerated and testing resources are less than before. This problem is more prominent, so we pay more attention to "accurate testing".
1. What is precision testing? Accurate testing is to answer two basic questions through data: what is measured and what is to be measured? That is, accurate testing uses specific algorithms, technical means and tools to analyze code, program running processes, test cases, etc. and the relationships between them. Thereby obtaining relevant information and knowledge, accurately positioning and optimizing the test scope (such as streamlining test cases), and using accurate data to evaluate test results and product quality, making the entire testing process more efficient, accurate and credible, while effectively reducing missed tests. risks and minimize testing costs. Accurate testing is an important trend in the intelligent construction of quality engineering and is also the embodiment of digital software testing . It allows us to clearly understand the testing process and achieve the quantitative testing goals we need (such as test coverage).
2. Accurate test implementation method
Achieving accurate testing is relatively simple in principle. The key is to achieve two basic tasks:
- It can complete effective code dependency analysis, and even extend to business dependency analysis, so as to correctly and accurately identify the code scope affected by each code modification. The code impact scope can be accurate to class methods, function levels or code blocks;
- Establish a mapping relationship between code and test cases, so that test cases that need to be executed can be recommended based on the identified impact scope.
If you want to establish a mapping relationship between code and test cases, or evaluate the effects of accurate testing, you will generally use code coverage analysis to better understand which codes are executed by the test cases, and to further clarify which codes are covered during test execution. , which codes are not covered in test execution, etc.
3. Accurate testing practice
In the practice of accurate testing implementation, you need to use some open source tools or develop some platforms yourself to achieve the above two basic tasks. For example, you can use the code coverage monitoring platform to collect dynamic code coverage data when the program is running, and build a use case knowledge base based on this; use the open source JVM-sandbox (https://github.com/alibaba/jvm- sandbox) can record real system operation conditions (i.e. traffic recording).
There will also be different treatments for manually executed test cases and automated test scripts. For example, manually executed test case recording will use the SDK embedded in the client to provide a UI interface for users to perform recording operations and complete data cleaning, collection, and reporting, and then analyze it in real time on the server. Automated test scripts are relatively simple. They can be executed one by one to collect coverage data, and it is easy to establish the relationship between code and use cases. You can refer to: ByteDance’s accurate testing practice and the design logic behind smart eyes.
Dependency analysis at the code level can also be extended to the analysis of the call chain, because the information collected when the program is running can more truly reflect the calling relationship (dependency) of the code, just like when we do code dependency analysis , rising from source code to compiled Binary bytecode, can more truly reflect the method/code block calling relationship.
Also, we need to create an index for the use case ID to improve the efficiency of use case recommendation. Based on the use case's associated method (or code block), upstream and downstream call links, and corresponding coverage, we build different versions of call information with different granularities, and provide test case index services, use case recall services, etc. In today's era of prevalent AI, we can naturally introduce knowledge graphs and machine learning algorithms to further optimize the effect of accurate testing. For example:
- The code dependency structure can be stored through a graph, such as a structure such as "(code belongs to package)-[include]->(file)-[include]->(function)-[call]->(function)". After obtaining the original data of the project call chain, we then deeply traverse each call link to collect the corresponding relationship of each package, file, function, as well as the path, location, input and output parameters, comments, code lines and other information;
- You can further collect the weight of the "use case-function call chain" to recommend use cases based on the weight;
- According to the similarity of use cases, some use cases with high similarity can be excluded, such as segmenting all use cases, establishing a vocabulary, using tf-idf method to calculate the text similarity between use cases, and using GCN (graph convolutional neural network) to calculate Use case similarity.
4. Frequently Asked Questions
Q1: How to build an accurate testing system from 0 to 1?
A1: You can start based on the Java technology stack and corresponding tools and defeat them one by one. Start with coverage analysis and development to understand the effectiveness of test cases and improve the quality and testing efficiency of test cases;
Then do code dependency analysis, use Code diff to understand the scope of the code, slowly establish the dependency relationship between the code and test cases, and achieve more accurate and effective testing; finally, move towards a fully automated approach to build an efficient The precise testing system means completing the construction of code knowledge base and use case knowledge base, and completing the construction of tool platforms such as traffic recording, automatic analysis of call chains, automatic recommendation and recall of use cases.
Q2 : Can you provide a complete, detailed, and reusable accurate test plan? Want to know more about which open source tools can be used to build this accurate testing platform? How to form a tool chain that can support the rapid implementation of accurate testing?
A2: : ByteDance and Youkudu introduced earlier have implemented complete implementation solutions, which are due to the two basic capabilities of traffic collection and code analysis. Traffic collection can be done based on the open source JVM-sandbox, although it requires secondary development. .
For code analysis, on the one hand, you can use code diff tools to understand code changes; on the other hand, you can use AST tools (Babel, jscodeshift, esprima, recast, acorn, estraverse, etc.), coverage analysis tools (such as JaCoCo), and Java Dependence Analysis ( JDA) + Java's own jdeps and other solutions for code dependency analysis.
Q3 : How to achieve precise testing from 1 to N, and achieve large-scale replication from one team?
A3 : Once an accurate testing system (platform) is built, it is relatively easy to promote based on the benefits of using the team, because the benefits are obvious, especially when running accurate tests in a fully automated way, it can also be coordinated with some unified rules and processes , and more importantly, the precision testing platform must be seamlessly connected with the company's R&D platform. Ideally, it should be flexibly integrated with the CI/CD pipeline, allowing the team to use it without any fuss.
Q4 : Accurate testing can only be used for regression. How to enable new function testing?
A4: Because regression test cases are constantly increasing and will reach a huge level, the cost of full regression is very high. At the same time, the amount of new/modified code is relatively small and the scope of impact is limited. There is no need to run all regression test cases. , selecting use cases based on experience will lead to missed tests, so it is very necessary to do accurate testing. New functions are relatively limited, and test cases written for new functions need to be executed, so there is generally no need for an "accurate testing" strategy.
However, with the help of an accurate testing platform, test coverage can be better achieved, improving the quality of test cases and the adequacy of test results. Moreover, the new function will be the old function in the next iteration, and the test cases written for it will also become regression test cases. Therefore, the new function also needs to be run on the accurate testing platform to obtain information about the code and test cases, and improve the code knowledge base and use case knowledge. library.
The following are relatively good learning tutorial resources that I have collected. Although they are not very valuable, if you happen to need them, you can leave a message in the comment area [777] and just take them away.
Friends who want to get information, please like + comment + collect , three times in a row!
After three consecutive rounds , I will send you private messages one by one in the comment area~