About the Author
Haibing, Ctrip R&D Energy Efficiency Manager and SRE, focuses on automated testing and tool technology for energy efficiency improvement.
1. Background
As the scale of Ctrip's air ticket BU business continues to increase, the business system has become increasingly complex, and various problems and challenges have also arisen. For R&D testing teams, they face various performance dilemmas, including high business complexity, heavy data construction workload, full regression testing, high communication costs, large number of test cases that are difficult to reuse, large amount of test data maintenance, and automated use cases. Management and other issues. Each will affect the efficiency and quality of the testing team and bring challenges to the software development process.
To sum up, there are two core difficulties: cost and complexity.
In terms of cost , we usually need to make a trade-off between cost and quality. We need to ensure quality while iterating quickly, and we need to ensure quality under limited investment.
In terms of complexity , when business rules are accumulated for a period of time, the complexity of business processes, rules, scenarios and data processing increases quadratically or exponentially after superposition, which brings great challenges to test quality work.
2. Exploration: Exploration and practice of automated regression testing
To address these challenges, we conduct some ongoing explorations in our quality and testing efforts:
1) AUTO test platform: a conventional programmable use case design management and execution platform.
2) Recording and playback testing: Introducing TCP Replay for playback testing has the advantage of simplifying the test design. The disadvantage is that there are no expected test results and manpower is required to pay attention to the analysis results.
3) Test data construction: Use scripts combined with interface access to generate data to solve the problem of daily data dependence. However, the labor cost of construction is high, and it requires continuous investment and maintenance.
4) Data MOCK platform: Build multiple MOCK platforms for different scenarios to solve test data problems.
5) Various coverage platforms: measure the scope and workload of testing.
6) Optimize the test environment: Build a benchmark test environment and sub-environments to ensure the needs of continuous debugging and testing.
In addition, there are many other attempts at testing frameworks, SQL log analysis, unit test case generation, Chaos fault drill platform, etc.
Problems encountered during exploration
The above various explorations have achieved certain results, but there are still two problems:
1) Automated testing focuses on automated execution, maintenance work is still "manual", and the output ratio is not very optimistic.
2) In scenarios where it is necessary to construct a large amount of test data, write scenarios, have a large scope of regression testing, and release frequently, whether it is manual testing or automated testing, development testing still faces a huge workload, including the maintenance of use cases and data. The pain points of testing were not effectively addressed.
new construction goals
Therefore, the R&D team gave us a new construction goal: "The quality must be improved and guaranteed. It is okay to waste electricity, but we cannot waste people."
Based on the above exploration experiences and new goals, we came up with the implementation idea of "using real production traffic and data for regression testing". In the end, we transformed this idea into concrete ideas and implemented it, building an automated testing platform AREX that combines recording + playback + comparison, which is now open source (open source address https://github.com/arextest ) .
Recording: Not only records production requests, but also records the data involved in request processing.
Playback: Not only playback requests, but also MOCK the involved data into the application.
Comparison: Replace test assertions with difference comparisons of recorded playback.
3. Introduction to AREX platform
3.1 What is AREX
AREX includes Java Agent and service components such as front-end, storage, scheduling, reporting, and database. The overall architecture is shown in the figure below.
AREX regression testing logic
The implementation logic of AREX regression testing is to conduct comprehensive and rapid regression testing of new versions of applications by recording nearly full business scenario requests and data in the production environment, and then playing back these real requests and data in the test environment.
target scenario
Test scenarios that require frequent and large amounts of data creation;
Scenarios that require a large number of business regression tests;
Test development scenarios where human resources are scarce;
Frequent releases and frequent regression testing scenarios;
Next, let’s explain how AREX works from the perspective of workflow.
The figure below is the call chain of an application that is not connected to AREX.
When connected to AREX, as shown in the figure below, the top is the traffic recording process. During the normal call processing process, the AREX Java Agent intercepts the data on the call link and saves it to the AREX database.
Below is the playback process. During the playback request, the third-party dependencies are not actually accessed. AREX cuts off the call chain, and the Agent reads the previously collected data from the database and returns it to the caller.
After the AREX playback test case is executed, AREX calls the comparison SDK for comparison, outputs the difference results, and generates a test report.
4. AREX technology implementation and optimization
Although the concept of traffic recording and playback is already a commonplace, it is not that easy to put it into practice. There are many problems that need to be solved. The following will introduce in detail the technical difficulties our team encountered when implementing the AREX platform and how we solved these problems.
4.1 AREX gent recording and playback principle
The following uses a simple function to illustrate the recording and playback principle of AREX Agent:
The left side is the function before conversion, and the right side is the function after conversion.
In the function entry part, playback judgment is made. If playback is required, use the collected data as the return result, which is Mock.
The function exit part makes a recording judgment. If recording is required, the intermediate data that the application needs to save is saved to the AREX database.
The principle of dependency package injection is that simple.
4.2 AREX technical challenges
AREX AGENT technology stack
Because of the good performance and the code is easier to read and understand, we use the ByteBuddy library to implement bytecode modification.
In addition, SPI (Service Provider Interface) is used, which is a set of APIs provided by Java for expansion. It can be used to enable framework extensions and component replacement. Our injection component is implemented through this plug-in mode.
Implement TRACING
The recording function of AREX needs to record requests, responses, and third-party request responses, etc., and requires a unique Key to connect these data in series so that it can be completely used as a test case. This Key is the Record ID of AREX.
The following classes implement the AREX call chain. In short, at the entry of the call chain, the Agent will generate a unique Record ID and save it to the Thread Local variable. Inject Agent code into the function head and tail of the application. This code will read the value in the Thread Local variable and store it together with the intercepted data.
Solve the problem of missing call chain
The above are the common basic scenarios in AREX TRACING delivery implementation. In addition, there are a lot of scenarios where multi-threading, asynchronous and other technologies are used in our applications. In this scenario, the call chain will be lost, which brings great difficulties to the concatenation of data.
In order to solve the problem of missing call chain, we implemented the encapsulation of Runnable, Callable, ForkJoinTask, and Async Client classes.
The basic principle is to replace the original code with our encapsulated class where the multi-threaded code is called, while retaining the original functions in the encapsulated class and realizing the data reading and writing functions of AREX at the same time.
Tips : Wrapper (packaging class) is a keyword. Search for Wrapper in the code base to see the implementation of all AREX packaging classes.
Implement record and playback injection
The following is a code example to implement recording and playback.
@Override
public List<MethodInstrumentation> methodAdvices() {
ElementMatcher<MethodDescription> matcher = named("doFilter")
.and(takesArgument(0, named("javax.servlet.ServletRequest")))
.and(takesArgument(1, named("javax.servlet.ServletResponse")));
return Collections.singletonList(new MethodInstrumentation(matcher, FilterAdvice.class.getName()));
}
When we need to implement the injection of a new dependent library, how do we implement this plug-in?
First, we use three elements to locate the injected function:
Module (ModuleInstrumentation: FilterModuleInstrumentationV3): The concept of logic management is to put multiple injection classes or encapsulation classes into one module.
Type (TypeInstrumentation: FilterInstrumentationV3): This is the class where we want to locate the injected object, that is, the injected application class.
Function (MethodInstrumentation): This function should be located at the function where the modification is injected.
As in the above example, the class we want to inject is the CacheAspectSupport class, the module it is in is the CacheModuleInstrumention module, and the function we want to inject and modify is the doFilter function.
Next, implement the function of function injection code through three steps. The following is a more intuitive look at how AREX Agent implements code injection through a Mybatis Query() function:
Step 1: Associate the function named METHOD_NAME_Query (string "Query") with the class QueryAdvice, which is the class that implements the injection function.
Step 2: The QueryAdvice class implements the function OnMethodEnter() and annotates ByteBuddy's Annotation.
Step 3: The QueryAdvice class will be injected into the head and tail of the Query function.
If playback is required, the queried data is stored in a local variable; if playback is not required, execution continues.
The following is the injection code of the Query function before exiting:
If the MOCK result meets the conditions, the MOCK data is returned;
If the current status is recording, the original data of the SQL+ query results will be saved to the AREX database.
Implement version management
Popular components often have multiple versions used in different systems at the same time, and the implementation methods of different versions may be very different or even incompatible.
To address this problem, AREX is compatible with multiple versions. When the application starts, the AREX Agent will capture the information of all dependent packages, such as the Manifest.MF file of the JAR package, obtain the version information of the class library from the Manifest, and then start the corresponding AREX injection code based on the version information. This implementation is compatible with multiple versions.
As shown in the figure below, the version range that the current injection script adapts to is set, so that AREX can identify the component versions that the application depends on before loading these classes, and then match the versions when the classes are loaded to ensure correct code injection.
Implement code isolation
Since most AREX usage scenarios are recorded in a production environment and played back in a test environment, stability is crucial. For the stability of the system and to prevent the Agent's code from affecting the code execution of the application under test, AREX implements code isolation and interoperability.
The AREX core JAR is loaded in an independent ClassLoader and is not interoperable with the user's application code. In order to ensure that the injected code can be accessed correctly at runtime, the ClassLoader is simply modified, as shown in the figure below.
Solve the time problem
Many of Ctrip's business scenarios are time-sensitive. We often encounter situations where the recording time has expired during playback, and the business logic cannot proceed, resulting in playback failure.
We use our own implementation of currentTimeMillis() to proxy Java's original currentTimeMillis() call. The recording and playback of time will be performed according to the scene at the time of recording, thus realizing the mock of time.
This scenario is described in detail in AREX Agent #182:
Fix caching issues
In actual applications, various caches are used to improve runtime performance. Inconsistent execution results due to differences in cached data are a big problem in recording and playback.
AREX provides the function of dynamic class Mock. The implementation method is to configure the method of accessing the local cache into a dynamic class. It is equivalent to you customizing this method for Mock. The data of the dynamic class method you configured will be recorded and played back in the production environment. The corresponding matching data is returned.
Of course, this approach also has shortcomings:
Cache configuration is easy to ignore and has a great impact on the playback pass rate;
Each application has its own cache implementation, which cannot be processed in advance and requires manual participation and configuration costs.
4.3 Other advantages of AREX
Support writing interface testing
To verify the business correctness of the modified system, it is not enough to just verify the returned results. It is usually also necessary to verify the correctness of the intermediate process data, such as whether the data content written by the business system to the database is correct, etc.
In response to this, AREX also provides perfect support for writing interface tests.
During the recording and playback process, AREX will record the external database requests made by the new and old versions of the system, and compare the two requests. If there are differences, they will be displayed in the playback report.
Since AREX MOCK handles all requests for third-party dependencies, supports verification of databases, message queues, Redis data, and even supports verification of runtime memory data, and no calls to the database will actually occur during playback, so it does not Dirty data will be generated.
Quickly locate production problems
In actual use, AREX can also be used to quickly locate production problems.
After production problems occur, it is difficult for developers to reproduce them locally due to version differences, data differences and other issues. Debugging is very costly and labor-intensive.
Using AREX, you can force the problematic Case to be recorded in the production environment (a unique Record ID will be generated in the response message). Then start the local development environment, add this Record ID to the header of the request message, and then you can restore the recording locally. requests and data, and then use local code to directly debug production problems.
5. Implementation and prospects of AREX automated testing
5.1 The implementation effect of AREX in Ctrip
After each BU is connected to AREX, in addition to the initial learning costs of familiar tools and configurations, it significantly reduces the workload of test developers in automated use case development, data MOCK, and data construction, forming a virtuous cycle and reducing leakage. Tested and increased coverage, effectively improving product quality.
Among them, the effect is most significant in two scenes:
Technical reconstruction projects, especially scenarios where the request and response are not modified. In this usage scenario, there is almost no need for testers to participate, and developers themselves can quickly conduct self-tests through AREX to ensure quality.
Developers improve the quality of self-tests.
5.2 AREX optimization
During the initial trial period of AREX within Ctrip, everyone’s evaluation of the tool was still very good. However, various problems arise when expanding the scope of use, especially when other teams do not actively access it:
The false positive rate is high (time, uuid, serial number, etc.), and there are many comparison and filtering configurations in the early stage.
The expected confirmation of code changes is cumbersome and requires a lot of manual intervention.
We are currently focusing on optimizing AREX configuration and comparison capabilities.
Configuration enhancement
At this stage, ensuring a high comparison pass rate requires a lot of manual intervention (comparison ignores configuration, etc.), so the first thing to do is to reduce user configuration costs:
Visually demonstrate differences
Manual marking operations improve ease of use
Configuration updates can be recalculated and reexecuted
homogeneous aggregation
Through aggregation, similar errors are aggregated in multiple dimensions to facilitate developers to observe differences. In the end, in most cases, developers only need to confirm one difference to remove most of the comparison differences.
Algorithmic noise reduction
1) Pre-analysis noise reduction
Pre-analysis noise reduction is to release the production version of the recorded traffic to the test environment, play back the version and compare the differences, and identify "noise" points similar to tokens, serial numbers, etc. in advance.
The noise is then tagged into the rule base as a knowledge base.
Finally, Schema changes of messages and data are identified and active noise reduction is performed to reduce the cost of manual configuration by users.
2) Compare knowledge base
Comparison is the core capability of AREX, but the current comparison is relatively rough and has a high false positive rate. We hope to increase the accumulation of comparison knowledge (without manual intervention) in iterative testing and form comparison knowledge. library to help users accurately identify valid differences. For example, convert Schema definition into effective comparison rules.
Accurate testing
Precision testing is to narrow the scope of testing. Precision testing will also be introduced in AREX in the future. The main purpose is to clarify the source of comparison differences.
We plan to associate code changes, code execution links and AREX playback, allowing users to "observably" confirm problems through two-way tracing of code changes and difference results. First, identify whether the differences are expected differences caused by code updates, and further proactively filter and identify differences that identify unexpected problems.
After comparing and optimizing the difference results, the cost of R&D configuration can be effectively reduced and the accuracy of the difference results can be improved. This is the real value of AREX's automated regression testing.
6. Write at the end
Through continuous optimization, AREX has gradually achieved the goal of using real traffic and data for regression testing, reduced costs, improved quality, and achieved the goals set in the early stages of construction.
Of course, there are still many areas that need to be optimized and improved, including algorithms, performance, support range, etc., which require further optimization and development. We also hope that all interested people can join in the co-construction of our AREX open source project.
[Recommended reading]
Application SQL performance risk identification and early warning, Ctrip Financial Payment AppTrace implementation practice
Application of UI automated testing based on BDD concept in portability simulation
“Ctrip Technology” public account
Share, communicate, grow