1. Background
When doing performance tests, many people use the number of concurrent users to measure the performance of the system. They feel that the more concurrent users the system can support, the better the performance of the system. The relationship between them is therefore very necessary to explain.
2. Definition of terms
ØNumber of concurrent users: refers to the users who operate the business in the real system. In performance testing tools, it is generally called the number of virtual users. Note that the number of concurrent users is very different from the number of registered users and the number of online users. , the number of concurrent users will definitely put pressure on the server, and the number of online users is only "hanging" on the system, which does not put pressure on the server. The number of registered users generally refers to the number of users existing in the database.
Ø TPS : Transaction Per Second, the number of transactions per second, is a very important indicator to measure system performance.
3. Vu and TPS conversion
Ø Simple example: TPS is explained in terms of transactions per second, but transactions are made by virtual users. If a virtual user completes a transaction within 1 second, then TPS is obviously 1; if a certain transaction If the service response time is 1ms, then a user can complete 1000 transactions in 1 second, and the TPS is 1000; if the service response time is 1s, then a user can only complete 1 transaction in 1 second. To reach 1000TPS, at least 1000 users are required; therefore, it can be said that 1 user can generate 1000TPS, and 1000 users can also generate 1000TPS, it is nothing more than a response time.
Ø Complex formula:
Imagine a complex scenario with multiple scripts, each of which defines multiple transactions (for example, there are 100 requests in a script, we call these 100 consecutive requests Action, only the 10th request and the 20th request are defined separately transaction 10 and transaction 20) The specific formula is as follows:
Symbols represent meanings:
Vui represents the number of concurrent users used by the i-th script
Rtj represents the time spent in the jth transaction of the i-th script, which will affect the entire Action time
Rti represents the time for the i-th script to complete all operations at one time, that is, the Action time
n means the nth script
m represents m transactions in each script
Then the TPS of the jth transaction = Vui/Rti
Total TPS=
4. How to get Vu and TPS
ØAcquisition of the number of concurrent users (Vu)
New system: There is no historical data for reference and can only be evaluated by business units.
Old system: For systems that have already been online, you can select the number of people who use the system at peak times and within a certain period of time. These people are considered to be the number of online users, and 10% of the number of concurrent users can be used. For example, within half an hour, the system will be used. The number of users is 10,000, so taking 10% as the number of concurrent users is basically enough.
Ø TPS acquisition
New system: There is no historical data for reference and can only be evaluated by business units.
Old system: For the system that has been launched, you can select the peak time, obtain the business volume and total business volume of each transaction in the system within 5 minutes or 10 minutes, and calculate the TPS according to the number of completed transactions per unit time, that is, the business transaction volume. Number/unit time (5*60 or 10*60)
5. How to evaluate the performance of the system
For server-side performance, TPS is used as the main measure to measure the performance of the system, and the number of concurrent users is supplemented to measure the performance of the system. If the number of concurrent users must be used to measure the performance of the system, a premise is required, that is, how long does it take for the transaction to be completed? , because when the system load is not high, adding the thinking time (the value of thinking time is equal to the transaction response time) to the script can basically double the number of concurrent users, so it is not too much to use the number of concurrent users to measure the performance of the system big meaning.
6. Related cases
Through a large number of performance tests, we found that there is no need to use tens of thousands of users to conduct concurrent tests. As long as the system processes business time fast enough, hundreds of users or even dozens of users can achieve the goal. In addition, I consulted many performance testing projects done by experts, and basically there were no more than 5000 users concurrently.
Therefore, for large-scale systems with very high business volume and sufficient hardware configuration, 5,000 concurrent users are sufficient; for small and medium-sized systems, 1,000 concurrent users are sufficient.
7. Performance Testing Strategy
Doing performance testing requires a set of standardized processes and testing strategies. The number of concurrent users is only one of the indicators to consider. When doing load testing, the number of users is generally increased according to gradient pressure, not without estimation. Next, if you add tens of thousands of users at a time, the transaction failure rate is very high, and the response time is very long, which has exceeded the user's tolerance range. It doesn't make much sense to do so. It's like "how much money can do as many things" , you need to choose the relevant strategy.
8. Loadrunner VS PTS
As can be seen from the comparison items in the figure below, PTS is more acceptable to customers than Loadrunner (LR).
| direction |
Contrast |
Loadrunner |
PTS |
Remark |
| infrastructure |
Does the hardware and software environment of the system under test need to be purchased additionally? |
need |
unnecessary |
Infrastructure software and hardware are provided by Alibaba Cloud, only need to purchase services |
| Does the press environment require additional purchases? |
need |
unnecessary |
Infrastructure software and hardware are provided by PTS, only need to purchase services |
|
| cost |
cost |
Very expensive |
Inexpensive, pay-as-you-go |
Commercial tool licenses are very expensive |
| Function |
Function |
powerful |
stronger |
Many functions of LR are basically useless, there is no need for a large horse-drawn trolley |
| Ease of use |
operation, learning, etc. |
difficulty |
easy |
LR is not easy to get started |
| stability |
system stability |
more stable |
very stable |
Inexplicable errors often occur during LR stress testing |
| scene simulation |
scene simulation condition |
more real |
very real |
The distributed clusters of PTS distributed all over the country can realistically simulate real-life scenarios, while LR is not easy to simulate, and even if it can, the communication between the control machine and the press is often dropped. |
9. Summary
Ø The performance of the system is determined by TPS and has little to do with the number of concurrent users. Under the same TPS, it can be decompressed by different numbers of users (set by adding thinking time).
Ø The maximum TPS of the system is certain (within a range), but the number of concurrent users is not necessarily and can be adjusted.
Ø It is recommended not to set too long a thinking time during performance testing to put pressure on the server in the worst case.
Ø Under normal circumstances, for large-scale systems (with large business volume and many machines), 5,000 concurrent users are sufficient for stress testing, and for small and medium-sized systems, 1,000 concurrent users are sufficient for stress testing.
The article is reproduced from: Open Source China Community [ http://www.oschina.net ]