[Live broadcast preview] Shangyun vs. Xiayun: Reduce costs and increase laughter? Cut leeks?

1. Background introduction

Since there are many cases of co-construction of multiple domains, on the one hand, applications continue to expand with business development, and the complexity of each application code will continue to increase. How to accurately and comprehensively determine the scope of impact of code modifications will become increasingly important. On the other hand, the co-construction process If the impact of common changes in each domain cannot be accurately estimated, there will be test omissions; if the information in each domain is asymmetric, one party may change and the other party will not be aware of it, resulting in some impacts caused by inadequate assessment. Based on the above background, the introduction of accurate testing platform practices in the merchant domain can help QA scan the interface range of development changes for each version, and can effectively improve the coverage and reliability of testing.

Based on the exploration and practice of precise testing on merchant addresses in the second quarter and achieved certain benefits; in the third quarter, large-scale practice was expanded. Therefore, based on the core business needs of merchants, four core applications were selected and accumulated several continuous Iterative process and outcome data. The following is some practical data and experience using the precision testing platform after several iterations.

2. Accurate testing practice in merchant domain

Test flow chart

Accurate test implementation plan

Accurately test application nodes

After the test, before the smoke:

According to the service that changes the score document, pull and confirm the scope of the change;
Confirm with the developer whether the interface changes are reasonable, and clarify the test scope for the test;
Confirm the accuracy of the platform;
Confirm the list of items that need to be supplemented with automation;

Before a round of testing is completed:

Automation of interface modifications is carried out, with a pass rate of 100%;
Confirm that the change service coverage is 75%.

The contribution of accurate testing and the difference before and after the introduction

Confirm the scope of the modified interface and clarify the scope of testing;
Added a new unautomated change interface to accurately obtain the automated execution results of the changed interface;
Code that improves service and demand dimensions.

3. Merchant iteration data

positive return

Rich use cases

Help us supplement test scenarios and use cases, and avoid missing tests due to inaccuracies in code or manual evaluation.

Example 1: During the iteration, it was found through the recommended interface that an interface was affected. The technical solution did not reflect the change. Change a line of code. The changed interface has filtering review order logic. It is necessary to add a regression scenario. One interface has not been evaluated and the rest are recommended. It is recommended to add and modify interfaces normally.

Impact area: To filter risk-controlled audit orders, you need to return to the scene and confirm whether the audit orders are filtered normally. If the jump is normal and there is no audit order, the technical solution has no record of changes to this interface.

Benefits: Confirming the impact area ensures that some unassessed scenarios are covered to avoid possible online problems.

The platform recommends an interface outside the scope of the technical solution:

It was later confirmed that there had indeed been changes:

Platform accuracy

Accuracy rate: a/(b+c) *100% (a: the number of changed interfaces recommended by the platform; b: the number of new interfaces, c: the number of old interfaces changed);

The current accuracy rate of the platform: Taking the latest iteration 528 data as an example, the number of interface changes is 18+, the recommendation is 15+, the average accuracy rate is: about 80%;

The following is the 526 version of the Groot service, with 15+ interface changes and 15+ platform recommendations. All were successfully recommended, with an accuracy of 100%;

The platform recommends a list of interfaces, a total of 10+, including 5+ backdoor interfaces for processing stock data.

To sum up, from the perspective of platform recommendation and actual interface changes developed, the current recommendation accuracy rate of a certain service platform is 100%, which helps QA accurately confirm the number of interface changes and accurately locate the test scope.

Rich automation cases

Including the new Dubbo/Http interface and the old Dubbo/Http interface are recommended. The automated Case has been completed for the changed interface, and the execution of automation is more accurate; the unfinished automation can be targeted to the left, and the technical solutions can be compared to check for deficiencies. ;The number of automated Cases moved left: 50+.

problem found

Interception of 2 valid questions: belong to other departments respectively. Multiple domains participate in the code development of the warehouse. When multiple domains are co-constructed, it is impossible to accurately estimate the scope of impact caused by changes in each domain. Accurate recommendations can cover the scope of the impact and focus on automated result analysis of the changed interface to save money. The environment and other cases affect time, triggering the automation project to return to old functions, exposing problems in advance.

Awareness in advance: After testing, relevant automation is triggered through the change interface recommended by the platform, and a problem introduced by the migrated code is intercepted in advance to reduce subsequent risks.

Assist in testing: After version 526 was tested, the platform recommended interface changes and automated cases. It was discovered that a new requirement code would affect existing functions after being merged, and repairs were made in a timely manner. Avoid problems being discovered in the post-processing stage and reduce risks in advance.

Improve efficiency

Automatically create plan execution - efficiency improvement points: efficiency improvement of 0.5-1h/per person per iteration, incremental code preview and analysis more convenient, saving 0.1-0.5h/per person per iteration.

Incremental preview

Through incremental comparison, there is no need to re-pull the old and new codes to compare and confirm the changes. You can directly pull the analysis and comparison to more intuitively confirm the code change points, confirm whether the scope of influence has returned, and improve human efficiency.

Practice process data

Version 527 iteration: number of interface changes: 15+; number of test left-shift interfaces: 6+; number of platform recommendation results: 11+; accuracy ratio: about 73%.

526 version iteration: number of interface changes: 90+; number of test left-shift interfaces: 10+; number of platform recommendation results: 73+; accuracy ratio: about 81%.

Version 525 iteration: number of interface changes: 40+; number of test left-shift interfaces: 30+; number of platform recommendation results: 25+; accuracy ratio: about 63%.

524 version iterations: number of interface changes: 22+; number of test left-shift interfaces: 8+; number of platform recommendation results: 11+; accuracy ratio: about 72%.

4. Introduction to precision testing platform

Introduction to implementation plan

The accurate testing platform is mainly based on abstract syntax tree (AST) to generate method call chain and make accurate recommendations. During version iteration, "method call chain of each application" and "global interface call chain" are generated. ("Difference analyzer" + "recommendation engine") extracts "change interface" and "influence interface" based on "change code", and then recommends Relevant use cases (automation + functional use cases + capital loss use cases), combined with accurate measurement, present the test quality of iterative versions, as follows:

Code Analyzer - Link Analyzer: Generate a method call chain based on the latest submission, mark the specific methods of the entry implementation classes of interfaces such as Http, Dubbo, Grpc, etc., and record the interface-related attribute information and store the knowledge;
Interface call chain extractor: Open up Trace2.0 to extract the interface call chain of the previous iteration and store it in the knowledge base;
Code Analyzer - Difference Analyzer: Based on Code Diff (latest submission - online submission), extract the changed method from Code Diff, combine it with the knowledge base to launch the change interface, and combine the change interface with the method call chain to locate the affected interface;
Recommendation engine: (change interface + impact interface) + automation use case + functional use case interface knowledge base => automation use case + functional use case;
Accurate measurement: Combine code coverage platform, automation platform, use case platform, etc. to measure test quality.

5. Summary of practical experience

Summarize

In the third quarter, we worked together through virtual teams to promote precise testing. In each version, we followed up on several core service applications of familiar modules. We also had some practice in some independent projects. In independent projects, each project was tested based on the demand dimension. All modified applications are accurately tested and recommended. This period mainly includes testing left-shift automation cases and following up on the reasons for stock automation failures. We also supplement the application stock interface automation cases to improve the coverage of application automation, and try to ensure that the old interfaces for each change are recommended. All interfaces can be automatically covered, helping to save the time and cost of testing and regression when old interfaces are returned;

When multiple domains are co-constructed, this can also help to accurately recommend the test range. Accurate testing can effectively improve test coverage and reliability, help testers discover potential problems, avoid information deviation leading to missed interfaces in multi-domain co-construction, and recommend interfaces that have not been evaluated for development changes;

Throughout the third quarter, each version has recommended 100% data through the recommendation of multiple applications. There are also some platform issues that need to be optimized. The data is being collected continuously. Taking the latest iteration data as an example, the average accuracy The rate is 80+%. So far in the third quarter: multiple applications, several iterations, and multiple students have invested a total of 6d+ in the form of virtual groups. The preliminary results are as follows: output accurate test process interaction diagram, intercept defects, and recommend unassessed Change interface, left-shift automated Case, automatically create a left-shift plan and execute it automatically: efficiency improvement 0.5-1h (per person per iteration); there are some special cases in the process, such as an independent project adding and modifying interfaces. If it is not recommended normally, the platform will also need to adjust the strategy for accurate recommendation later.

Follow-up planning and outlook

Through some problems discovered in the practice in the third quarter, the platform also actively cooperates with the improvement. I believe that based on the platform support, we will make accurate test recommendations based on the demand dimension in the future. More applications will use the precise testing platform, and each iteration will advance the meeting based on the demand dimension. More efficient and accurate analysis to gradually achieve the best testing results and maintenance costs; through each version iteration, continuously use collected data to discover some problems that affect accuracy, and optimize and improve the accuracy platform system again to improve accuracy Test the accuracy of the platform and repair the problems and defects that have been discovered; when the accuracy increases to a certain level, it can provide basis and reference for future testing work.

* Text/Jeremy

This article is original to Dewu Technology. For more exciting articles, please see: Dewu Technology official website

Reprinting without the permission of Dewu Technology is strictly prohibited, otherwise legal liability will be pursued according to law!

Dewu business domain accurate testing practice