Brief description of test data
What is the test data of the soul torture? Maybe you would say without hesitation, isn’t the test data the data we use for testing every day? This statement is too general. In fact, test data refers to data related to the test, which can be divided into the following categories.
1. Test request data
Test request data is the test data we often understand. This part of the data is the necessary input for the execution of the test case (the test case here refers to the automated test case, usually in the form of a test script).
It can be directly coupled in the test case, or it can be placed in an external file. In the case of external files, as mentioned in our previous tweets, test data can be stored in JSON, YAML, TXT, Excel File (xlsx), CSV, SQL files, and even .py files.
The use of this part of the test data, through the previous sharing, I think everyone should not be unfamiliar anymore. For test request data, it is also divided into the following two cases.
Mandatory data
The data that must be carried when sending the request is the mandatory data. For example,
in the UI automation test, the data that you cannot submit the form without filling in;
in the API automation test, the parameters and data that you must carry in the request. Otherwise, an error will be reported when sending the request.
Non-mandatory data
The data that is not mandatory when sending the request is the non-mandatory data. For example:
in the UI automation test, the data that can be submitted without filling in the form;
in the API automation test, the data that you do not carry in the request will also be sent to the request. Parameters and data that will not report errors.
2. Test expected data
Test expected data, usually used as data for comparison with the result data generated after the test. This part of the data is often accompanied by the existence of the assertion function.
Used to determine whether the test result data generated based on the test request data is the same as the expected test data. If they are the same, it means that the business behavior is in line with expectations; if they are not the same, it means that the business behavior is inconsistent with the demand, and there may be a bug.
3. Test result data
That is, after the input of the test request data, the result data generated by the system, this part of the data is also divided into two cases:
Pure result data
Refers to unanalyzed, aggregated data. For example: the result data of a certain test case. Their function is often used to compare with the test expectation data provided by the user to verify the correctness of the business.
Aggregated result data
Aggregated result data, usually referred to as test reports. By aggregating pure result data, the aggregated result data can tell us more about the quality of the system.
For example: after a test, the test report can tell us how many test cases succeeded, how many test cases failed, and which module the failed test cases belonged to.
Through the comparison of multiple test reports, we can see which test modules often have problems, which modules are basically stable, and which modules have decreased performance. The analysis of aggregated data helps to improve our test strategy.
How to prepare test request data
Test request data preparation often consumes more time in automated testing. How to effectively prepare test data is actually a proposition without a standard answer. Based on my previous project experience, several commonly used data preparation methods are listed here for reference only.
1. Manually create according to business rules
This is currently the easiest way, the tester directly provides the test request data, including mandatory data and non-mandatory data. Save the test request data by directly providing it to the test method or using an external file. The test request data saved in the external file is read and applied to the test case (test script) one by one through the data drive when the test is in progress.
The test request data created by hand has a disadvantage, that is, the test request data will never change, which is not in line with normal user usage.
2. Use third-party fake data library to automatically generate
In order to better simulate the use of normal users, you can use third-party fake data, such as the faker library commonly used in python. By calling these fake data libraries, you can generate test data closer to normal users.
But this kind of data is generally only used when creating data. For example, registration and filling in the feedback form. For query data, it is not applicable, because query data usually requires that the data already exist in the system database.
3. Data file
obtained through SQL query The method of obtaining test request data through SQL query is a common way. Generally, this method is suitable for the situation where a request for data is from the output of different services.
For example: test the product deduction interface, then the input of this interface must have parameters such as user id, product id, product price, user balance, etc., and these parameters are provided by one or more services. So using SQL statement combination query is a quicker way.
The test data is obtained through SQL statement query. If there are too many tables to join, there will be data generation efficiency problems.
4. Automatically generate data according to the test platform. The
data platform (Data platform) is a popular data generation solution in recent years. It integrates the above several data production methods and provides a unified interface so that users can easily Generate test data without worrying about how the data is generated, but the construction of the data construction platform requires the test team to have a certain amount of development architecture capabilities.
5. Copy from the production environment
By copying the traffic of the production environment, it is used for test environment testing. This method is commonly used in performance testing. Test data is constructed by copying online traffic to the test environment. In the process of big data testing, some companies will copy directly from the production environment to the test environment in order to save costs or make the test data closer to the actual user usage scenario data.
Common solutions include TcpCopy, goreplay, etc. This method has relatively high requirements for the test team's architectural capabilities and code development capabilities, and often requires the cooperation or even leadership of the development team. It is generally implemented through the establishment of special projects within the company.
Timing of preparation of test request data
Regarding which stage of the test to create test request data, there are usually the following two ways in the industry.
1. Preparation before the test run
Prepare before the test runs, that is, the test data exists in the form of hard code. It can be hard coded directly in the test method or written in data files of various formats. The manual creation of test data based on business rules as mentioned above is the best example of preparation before the test runs.
2. Prepare when the test is run
Preparation during test operation means that the test data is not specified in advance, that is, there is no test data file in the test code. The test data is to directly generate the test data required by the test case by invoking the data construction platform or by combining the query DB when the test is running, and then start the test.
As to when the test data should be prepared, there is currently no unified conclusion, and it does not matter whether it is good or bad. You can choose freely according to your own situation.
Problems and countermeasures of test request data
Currently, no matter what method is used to prepare the data, no matter what timing is used to generate the test request data, the test request data may have the following problems:
1. Test data is out of date
This situation is common when the test request data is prepared in advance. For example: a set of data is used for the deduction of coupons, but usually coupons have an expiration date. After the coupon expires, using this set of data for testing will inevitably cause the test to fail. Therefore, the test request data prepared in advance must be maintained regularly.
2. Multiple runs lead to different test results.
This situation often occurs because the data is prepared in advance. For example: a set of test data is provided for user registration. When the first test runs, the test will pass normally, but the second test will fail because the user already exists.
For the situation where DB write operation is required during the test, it is best to perform a tear down operation after the test is over to restore the system to the state before the test.
3. Test data is unavailable due to environment switching.
Generally, the release of a product must be tested in several test environments. For example, development environment, test environment, integrated test environment, pre-production environment, production environment, etc. The test data of each environment may be different. Switching environments must ensure that the test data is available.
The problem of unavailability of test data due to environment switching can be solved in the following two ways:
Ensure that each test environment uses the same set of data.
This method is cumbersome and suitable for new projects. Create the same test data for each test environment to avoid test errors caused by test environment switching.
The test framework has the ability to switch test environments and automatically find corresponding environmental data.
This method is relatively common. Different test environments can have different test data. The test framework has the ability to automatically mount the test data of the corresponding test environment after switching the test environment.
4. The test data is changed during the test run The
test data may be changed dynamically during the test. For example, the user's balance is stored in the database, and the test data is generated when the test is run. That is, when the test is running, it is found that the user balance is insufficient when the user balance is checked and obtained.
In this case, it is usually necessary to change the conditions of the test data generation, that is, write the query statement more robustly to ensure that the acquired user must have a balance. Or add a conditional judgment, if it is found that there is no balance, another service is called to recharge the user.
5. Concurrent running makes test data unavailable.
Running test cases concurrently, or multiple people running the same test case at the same time, may cause multiple test cases to operate on the same set of data. This may cause the test to fail (for example: different people take the same piece of test data for registration operations).
In this case, you can code to allow the test framework to support concurrent runtimes using the same data file, but this usually involves a lot of investment. In order to avoid investing too much development effort, in most cases, multiple classes are used to support concurrency, and the test cases under one class are executed sequentially to avoid test cases under the same test class and access the same test data at the same time.
to sum up
What I shared today is the common problems of test request data in test operation. Careful you may find that although these problems have their own solutions, they all have a commonality, that is, the emergence of these problems is due to the lack of test data management.
Welcome to pay attention to [The Way of Infinite Testing] public account , reply [Receive Resources]
Python programming learning resources dry goods,
Python+Appium framework APP UI automation,
Python+Selenium framework Web UI automation,
Python+Unittest framework API automation,
Resources and codes are sent for free~
There is a QR code of the official account at the bottom of the article, you can just scan it on WeChat and follow it.
Remarks: My personal public account has been officially opened, dedicated to the sharing of test technology, including: big data testing, functional testing, test development, API interface automation, test operation and maintenance, UI automation testing, etc., WeChat search public account: "Infinite The Way of Testing", or scan the QR code below:
Add attention and let us grow together!