background
Quote : According to a 2008 Aberdeen Group research report, for Web sites, a 1-second page load delay is equivalent to reducing PV (page view) by 11%, which is equivalent to reducing customer satisfaction by 16%. From a monetary point of view, it means that if a website earns 100,000 yuan a day, it may lose a total of 250,000 yuan in sales in a year because the page load speed is 1 second slower than the competition.
Compuware analyzed more than 150 websites and 1.5 million viewed pages and found that an increase in page response time from 2 seconds to 10 seconds leads to a 38% abandonment rate of page views. It can be seen that website performance has a direct relationship with business goals, and it is very important to perform performance testing on websites.
Students who do performance testing know that there are many scenarios in performance execution, and each company will have its own defined scenario nouns that are not listed here. I generally use four scenarios for performance (baseline scenario, capacity scenario, stability scenario, exception scenario) to cover a performance requirement. Let's talk about how these four scenarios are implemented and how to cover performance requirements.
actual landing
Before talking about the four scenarios, let's talk about the general process of performance testing. The process is the rule. In Mencius - "Li Lou Chapters and Sentences": " Li Lou Zhiming, the coincidence of the public loses the son, and if you don't follow the rules, you can't make a square ." After having this process, you will not mess around when encountering performance testing, and you can work according to this process.
Regarding what needs to be done for each action, the person doing the performance test will know what needs to be done at a glance, so I will not explain the specific content here. After we have completed the preconditions, we can carry out the scene design and execution. The performance analysis will not be explained in this lesson. The Jmeter pressure tool is used here. TPS refers to the Throughput value in the Summary Report component in Jmeter, and the back-end interface is not considered. For the loading time of page elements such as js and css on the front-end page, other stress tools can also be used in the specific performance test. There is no limit to what stress tool to use. Here we mainly explain how to use the scene.
现在我也找了很多测试的朋友，做了一个分享技术的交流群，共享了很多我们收集的技术文档和视频教程。
如果你不想再体验自学时找不到资源，没人解答问题，坚持几天便放弃的感受
可以加入我们一起交流。而且还有很多在自动化，性能，安全，测试开发等等方面有一定建树的技术大牛
分享他们的经验，还会分享很多直播讲座和技术沙龙
可以免费学习！划重点！开源的！！！
qq群号：110685036
Baseline Scenario
The benchmark scenario is a single interface for stress testing. If everyone knows this, how should we do the benchmark scenario?
- Environmental preparation
Internet companies generally choose online stress testing for performance testing. We all know that the cost of building a stress testing environment that is consistent with the online environment is relatively high, such as hardware costs, software costs, labor costs, maintenance costs, etc., so Internet companies choose online stress testing. Do a stress test on it. For non-first-line Internet companies or traditional industries, stress testing is generally done in a test environment. The banking and insurance companies I have worked in have their own stress testing environments. Their environments are relatively independent, and it is relatively easy to do stress testing and tuning. It is more convenient.
In the stress test environment, it is recommended that the machine configuration is consistent with the online machine configuration, and the application container configuration must also be consistent with the online environment. Only in this way can the stress test results be true. To put it bluntly, the stress test environment needs to be consistent with the online environment. If it is not consistent, it needs to be scaled down according to a certain ratio.
- data preparation
For data preparation, we need to consider the amount of bedding data in the stress test environment. Generally, the bedding data source is the real survival database data after desensitization. Before the stress test starts, the database needs to be backed up to facilitate troubleshooting and problem location. If the amount of data is inconsistent with the amount of online data, the stress test results will be unreliable. You can think about the performance of a few hundred databases and an online million or tens of millions of databases. During the test, the amount of bedding data in the stress test environment needs to be considered.
- Parameter preparation
When doing performance stress test scripts, we need to prepare parameterized data. Many people think that a few pieces of data are enough for performance testing, and a large amount of data is not needed for performance testing. This will cause the pressure to be inconsistent with the real living environment. For example, dozens of pieces of data Data operations generate tens of millions or even hundreds of millions of businesses, so it is hard to say the credibility of the scenarios obtained from the stress test. So how many parameters are reasonable?
To answer this question, it is necessary to determine what scenario the parameterization is used in, because the scenario is different, and the parameterized data is also different. If the scenario is not considered, 100 threads, TPS is 800, the system runs for 1 hour, data type, and restrictions and quantities are calculated as follows:
If the scenario is taken into consideration, we will analyze what kind of data is used according to the actual business scenario in order to calculate the amount of parameterized data. The data here includes repeated and non-repeated data. Here we use our unified login as an example. For login The business requires two parameters, one is the account number and the other is the password (this does not consider logging in through the verification code). The account number and password must be able to log in to the system, otherwise the subsequent business operations cannot be completed. Obviously, different people must log in with different accounts.
➢ Scene 1
When we are writing scripts, we configure as many users as there are threads, and let each thread use the same user to perform business operations in a loop. However, such parameter configuration can only meet specific business scenarios. Use the token to operate the business, and exit the system after work. In such a scenario, we need to prepare as much user data as there are threads.
➢ Scenario 2
For an e-commerce system, it is not advisable to follow the above parameters, because the behavior of continuously purchasing goods with an account is completely inconsistent with the real scene. What should we do at this time? In such a scenario, we need to consider TPS and duration, user data calculation method reference:
➢ Scene Three
If a thread can recycle fixed parameters, in this case we need to judge according to the actual business, for example, in the case of 100 pressure threads, we prepare 1000 pieces of data, and we can let each thread use 10 different pieces of data. There is no fixed limit on the number of such scenarios, and it can only be judged based on actual business. However, before configuring this parameter, you need to consider the type of the parameter. If it is recyclable data, it is very rare in real performance scenarios, that is to say, only There are basically no real scenarios using one or several test data.
- parametric data
After knowing how many parameters are used, it is necessary to solve where to obtain the parameterized data. The purpose of this step is to ensure that the parameter data is valid. Generally, we have two sources of parameterized data, one is the existence of the background database, and the other is the absence of the database Data created by pressure tools.
In short, parameterized data sources need to ensure the validity of the data, and these data need to meet the following two conditions:
✓ Satisfy the data distribution of the production environment
✓ Meet the data volume requirements in performance scenarios
For a system, whether the parameterized data is reasonable will directly affect whether the stress test results are meaningful. If the amount of data used by the pressure tool is small, then the application server, cache server, database server, etc. will use a small amount of cache for processing, resulting in the inability to occupy the cache of various back-end servers to the size that should be in the real scene, so In this state, the pressure in the real scene cannot be tested.
There are also implications for database-attached storage devices. If the amount of data is small, the corresponding storage I/O usage is low. If too many parameters are obtained, the pressure on the system will be great; if too few parameters are obtained, which does not conform to the data volume in the real scene, the real pressure of the system cannot be tested.
If the parameters come from a database, we usually want to check the data histogram in the database. For data taken directly from production, the distribution of data is more accurate. But for some data created in the test environment, it is necessary to check whether the data distribution is consistent with the production after the data is created. After the above preparations, you can start benchmarking. We need to pay attention to two points when doing benchmarking:
When do we end up with the baseline scenario in our implementation? Or under what circumstances can the benchmark scenario be ended? Generally, during the stress test, the system resource utilization rate of interface calls reaches 90% or the system resources are completely used up, then the benchmark scenario can be stopped. Note that performance problems need to be tuned to ensure that resources are fully used, so as to achieve TPS and response time is the most reasonable .
Capacity scenario
For the capacity scenario, it is to combine all the interfaces of the previous benchmark scenario in a certain proportion and execute them together. The purpose of the capacity scenario is to answer the question "what is the maximum online capacity?". There is a problem here is how to determine the ratio between the interface and the interface? How to configure it in the pressure tool? If these problems cannot be solved, we will not be able to design capacity scenarios. Some students may say that it is enough to put all the interfaces in one scene and run them directly in multiples of 1, 2, and 3 times. There is no need to consider the ratio of each interface. , Just run the maximum value directly, but there is a gap between the scene and the actual scene, and the scene that comes out is out of reality, so how should we design it? Another question is when will the capacity scenario stop?
First of all, we can extract production logs. I believe that every company has a log system. If there is no log system, you can use the awk command to extract logs, and you can also use ELFK to extract logs. After having the above data, you can sort out the business logic.
Our company can obtain the stress-tested logs through the lambda platform, and obtain the interface data we want for a period of time through the search conditions:
If there is no such system, we can obtain relevant data in Nginx logs, which can be processed by writing python scripts or using shell commands to obtain the data we want. We need the data format as:
In this way, we can know the number of each request in each time period, and we can get the corresponding business ratio.
If it is ELFK extracting survival logs, you can install such an environment configuration by yourself, set relevant conditions and extract the data types of the corresponding relationship of the above data.
Note: If the high concurrency time point of a certain request is not at the same time point as other requests, multiple scenarios must be simulated, because the business model in the scenario will change.
Assuming that we can get the ratio of calls to requests of the stress-tested interface through the above steps, we can also divide the number of certain requests by the total number of requests to get the ratio of each interface. Here we use a table to illustrate (note that the Data is hypothetical):
Here is a summary of the steps to obtain the production data ratio:
Please note that there can be multiple business scenarios in the actual project business statistics process. This idea can solve any inconsistency between performance scenarios and production scenarios. After the business ratio is obtained, we need to sort out the business scenarios and use the scenarios to cover the business ratio. How should the scenario ratio be designed? You can think about it and design several scenarios that can cover the above interface business ratio. I won’t explain it here, and you can think about it yourself. .
In JMeter, the Throughput Timer will be used to control TPS.
Regarding the scene ratio of the Throughput Controller controller, you can check the information and solve it.
With the above business ratios, we can think about when the capacity scenario will end? We need to determine a goal when doing capacity scenarios, otherwise there will be no end time point. Do you still remember that when we allocate business proportions, we allocate a total amount of data to each interface as a proportion, and this total amount is the goal of our capacity test.
Stability Scenario
We mentioned earlier that the capacity scenario is to see the maximum capacity that the system can withstand, while the stability scenario mainly looks at the performance stability of the system when it provides long-term services, and observes the cumulative effect of the system during long-term operation . Therefore, the running time is a very important indicator in the stability scenario.
Here we need to understand that the stability running time is not fixed, it depends on the specific application scenarios of the business system, so how should the stability time be calculated? A calculation method for calculating execution time is provided here.
Assume that according to the statistics of the production system, in the previous operation and maintenance cycle, there was a business capacity of 60 million, and the maximum TPS obtained in the capacity scenario was 1000. Then, we can calculate by the following formula:
Stable running time = 60 million (cumulative business volume) ÷ 1000 (TPS) ÷ 3600 (seconds) ≈ 16 (hours).
No matter what kind of system it is, if you want to run a stable scenario, you must determine a cumulative business volume (the total business volume at a time). When we execute it, we directly use the maximum TPS to run it. If a system is normal under the maximum TPS state Only by running can we verify whether the system can stand the test.
Abnormal scene
In the design of abnormal scenarios, the system architecture is firstly analyzed, and the demand points in the abnormal scenarios are analyzed first, and then corresponding cases are designed to cover them. Why analyze system architecture? Because in an application, after we have tested the function, there are usually two types of abnormal problems: one is the abnormality of the architecture level; the other is the performance abnormality caused by the capacity. For architectural-level exceptions, we can only analyze them from the perspective of architecture.
From the perspective of performance technology, common testing methods such as host failure, network failure, and application failure are basically used. Now there are chaos tools on the market, and you can also use chaos tools to do abnormal testing, which provides some common methods to help Everyone do the operation:
✓ Host: power off, reboot, shutdown, etc.;
✓ Network: shut down the network card with the ifdown command, simulate jitter packet loss, delay retransmission, etc.;
✓ Apply: kill, stop, etc.
Let’s talk about the logic of abnormal scenario design here. The first step is to list all the components in the technical architecture and analyze the points that may cause abnormalities. The second step is to design corresponding scenarios based on the analysis of abnormal points. In many cases, it is based on feelings, which will lead to incomplete coverage of abnormal scenarios. Here, it is recommended to refer to the FMEA failure model to design abnormal scenarios, because FMEA allows us to have rules to follow. It should be noted here that there is logic for any methodology Don't overuse it when it comes to practice, but pay attention to the rationality of the content.
This is just to provide a design idea. When you land, you can use the ones that don’t fit your company or practice. I hope you can design a set of abnormal scene models that fit your own system.
The following are supporting learning materials. For friends who do [software testing], it should be the most comprehensive and complete preparation warehouse. This warehouse also accompanied me through the most difficult journey. I hope it can help you too!
Software testing interview applet
The software test question bank maxed out by millions of people! ! ! Who is who knows! ! ! The most comprehensive quiz mini program on the whole network, you can use your mobile phone to do the quizzes, on the subway or on the bus, roll it up!
The following interview question sections are covered:
1. Basic theory of software testing, 2. web, app, interface function testing, 3. network, 4. database, 5. linux
6. web, app, interface automation, 7. performance testing, 8. programming basics, 9. hr interview questions, 10. open test questions, 11. security testing, 12. computer basics
Information acquisition method:
