Write custom directory title here
1. What is performance testing
Performance testing is to test the quality of software performance, it is a non-functional test,It tests various performance indicators of the system by simulating various normal, peak and abnormal load conditions through automated testing tools.
2. Why learn performance
Perform a performance test:It is to meet the business scenarios in our real products, support the simultaneous access of a large number of users, and the business needs of high concurrency.
For example, the number of people online on WeChat every day:
It improves the performance of our products mainly in the following aspects:
Provides a measure of the system
jmeter memory CPU; basic unit of measure 50% 30%
These tests are helpful for benchmarking parameters such as response time to measure application speed, we all know power is critical. This is critical for performance comparisons of applications.
It is helpful to evaluate the scalability of the application.
The daily visits of Ma Bingbing's official website are 1W, and the subsequent visits reach 2W and 10W. The
performance test is helpful to check whether the application has the ability to expand more users.
Helps to check the robustness of the application
With the time of production passing the stress test, we can check the robustness of the application when the workload is higher than expected or higher than the application threshold.
Helps to check the reliability of the application
Different types of performance testing are done to check whether the application is reliable and provides correct and consistent output. Performance testing, such as load testing and endurance testing, helps to assess the correctness of the system under the expected workload.
3. The difference between performance testing and functional testing
Functional testing: The main concern is whether our product functions meet the product functional requirements specifications, mainly refers to the normal use cases and abnormal use cases of our product functions, and special cases (mainly focus on our product manuals, Normal and abnormal situations, as well as our special situation)
Performance testing: After the function is completed, we mainly focus on whether the various indicators of the product (such as response time, cpu, memory, throughput, etc.) meet the requirements of our product business requirement scenarios. (time, resource)
1. What is the difference between performance testing and functional testing?
The testing focus is different.
Functional testing: mainly to see whether the implemented functions of our products conform to our product documentation and the forward and reverse use cases listed in the product documentation. For example: we log in to our Ma Bingbing official website
.
Enter a normal user name and password, and the login is successful. Wrong
use case: Enter a normal user name and wrong password, and the login is unsuccessful.
Special case: Enter a user name with spaces and a normal password, whether the login is successful.
Performance test:The main thing to look at is whether some of our performance indicators meet our product documents
For example: we log in to our Ma Bingbing official website
1W users use the correct user name and password to log in at the same time, and the login success time is within 2S.
At the same time, the cpu usage rate of our server is lower than 40%, and the memory usage rate is lower than 50%.
2. When is the performance test done?
Performance testing must be done after our functional testing is completed
3. When we are making our products, can we only do functions but not performance?
If it is your company’s own system, if the number of people using it is small, you don’t need to do performance.
If it is a commercial product, when the number of users is relatively large, this performance must be done.
4. The performance test mainly considers these aspects:
time characteristic: The period from when the user sends the request to when the response is received
resource utilization: Including: cpu, memory, hard disk
server reliability: refers to the continuous operation of the server under relatively high load conditions
Configurable optimization: Refers to server configuration optimization, business logic optimization, code optimization, etc.
Fourth, the classification of performance testing (emphasis)
1. General performance test (baseline test)
When applying lower pressure to the product system, check the operating status of the system and record the relevant data as a basic reference
Baseline: For a computer, 90% of the CPU and 70% of the memory will affect the performance.
For example: Ma Bingguan V2.0 version, simulated 1W users place orders on our server, and its response time is 2S -- -- This is our general performance test. It is the presence of a reference to do.
V3.0 simulates 10,000 users placing orders on our server. Let’s take a look again. Its response time is 8 seconds. Does its response time have corresponding performance problems?
2. Stability test
Check whether the system is stable by loading the system with a certain business pressure (such as CPU resource usage at 70% to 90%) and running it for a period of time.
monkey testMa Soldier Yanxuan APP run our monkey after work
1000000 9 minutes
10000000 90 minutes
Mobile phone --" after-sales, performance (stability)
a. Why do stability performance tests?
The official website of Ma Bingbing can resist our 1W users placing orders at the same time, so can we guarantee to place orders for a long time?
3. Load test
It is a test that gradually increases the load of the system , tests the changes in system performance, and finally determines the maximum load that the system can withstand under the condition of full system performance indicators. In short, the load test is to determine the processing capacity of the system and the various thresholds it can withstand by gradually increasing the pressure
Ma Bingbing’s official website requires no more than 3s to place an order.
100 people place an order for 1s.
1000 people place an order for 2S.
2000 people place an order for 2.5s.
3000 people place
an order for 3S.
a. Why do load tests?
For example: we buy tickets for 12306. Is it okay for us to buy tickets normally, but can you guarantee that 12306 can also operate normally during holidays or Spring Festival?
4. Stress test
It is a test to gradually increase the load of the system, test the change of system performance, and finally determine under what load conditions, it is in a failure state, and obtain the maximum service level that the system can provide. Stress testing is to gradually increase the load to saturate some system resources or even fail them.
3000 people place orders—>4000–>5000
a. Why do stress tests?
In the process of using the software, the number of users is beyond our control. Is it possible to exceed our maximum usage? How should we deal with it?
What is the biggest difference between load testing and our stress testing?
The load test is gradually increased from the very beginning.
The stress test is gradually increased from
the load. The load test test is only to measure the maximum load of our products, and our stress test is to gradually increase the load. , pushing our system across
5. Concurrency testing
Test multiple users accessing the same application at the same time, and send multiple requests in a very short period of time to verify the server's ability to handle concurrency
a. Why do concurrency testing?
The official website of Horse Soldiers can withstand the visits of a large number of users, but can you guarantee that there will be no problems when a certain commodity is sold on our official website of Horse Soldiers ?
Five, performance test indicators (emphasis)
RT -Response time (response time)
system processing capability is also called throughput (HPS, TPS, QPS)
number of concurrent users
error rate (for example, 3 per thousand, 1000 out of 3 errors)
CPU
Memory (memory)
disk throughput
network throughput
1. Response time
The entire time from the client sending a request to getting a response generally includes network response time + server response time.
Note that we don't care about the time the client clicks and the time it takes for the client to process the page display.
User Acceptance Criteria:
For example, the 2-5-10 principle, that is, according to the normal user experience, if the user can get a response within 2 seconds, the user will feel that the speed is very fast; When a response is received within 10 seconds, the user will feel that the response speed of the system is slow, but it is acceptable. If there is no response after more than 10 seconds, the user will feel unacceptable.
Industry reference standard:
The acceptable response time for different businesses in different industries is different. In general, for online real-time transactions:
Internet companies: less than 500 milliseconds, such as about 10 milliseconds for Taobao business.
Financial enterprises: less than 1 second is preferred, and less than 3 seconds for some complex businesses.
Insurance companies: less than 3 seconds is better.
Manufacturing: preferably under 5 seconds.
2. System processing capacity (HPS, TPS, QPS) throughput
System processing capability is also called our throughput. What is throughput?
Throughput refers to the number of requests processed by the system in unit time (calculated in seconds) .
For single-user systems, response time can be a good measure of system performance, but for concurrent systems, throughput is usually required as a performance indicator.
In general, the following indicators are used to measure:
HPS (Hits Per Second): the number of hits per second, and the unit is hits per second.
TPS (Transaction per Second): The number of transactions processed by the system per second, the unit is pen/second.
QPS (Query per Second): The number of queries processed by the system per second, and the unit is times/second.
For Internet services, if some services have one and only one request connection, then TPS=QPS=HPS,
If it is a pressure test on an interface (single scene), and this interface will not request other interfaces internally, then tps=qps, otherwise, tps≠qps
single-service multi-interface request
The number of transactions processed per second, generally TPS is for the entire system. How much transaction processing can an application system complete in 1 second? In distributed processing, a transaction may correspond to multiple requests. To measure the processing capacity of a single interface service, QPS is more used.
standard
Regardless of TPS, QPS, HPS, this indicator is a very important indicator to measure the processing capacity of the system, the bigger the betterAccording to experience, in general:
Financial industry: 1000TPS~9000TPS, excluding Internet-based activities
Insurance industry: 100TPS~1000TPS, excluding Internet-based activities
Manufacturing industry: 10TPS~50TPS
Internet e-commerce: 10000TPS~1000000TPS
Internet medium-sized websites : 1000TPS~50000TPS
Internet small website: 500TPS~10000TPS
3. Number of concurrent users
Number of concurrent users: the number of users sending requests to the server at the same time
Common misunderstanding:
the total number of users using the system (only registered users, data stored in the database) the
total number of online users using the system (after successful login, operating other tasks, only online users)
if 100w registered users The number of online users is 10w, of which 3w users are viewing videos at the same time, then 3w is the number of concurrent users
A correct understanding
of the number of concurrent users refers to the number of users who open the system and perform business operations at the same time. ) system, the maximum number of concurrent users is the concurrent access capability of the system.
For short connection (after successful login, no longer care about this system) system, the maximum number of concurrent users is not equal to the concurrent access capability of the system, but is related to various conditions such as system architecture and system processing capability.
http: The request can only be sent by the client, and the server responds passively.
4. Error rate
The error rate refers to the probability of a failed transaction when the system is under load . Error rate = (number of failed transactions / total number of transactions) * 100%. For a system with better stability, its error rate should be caused by timeout, which is the timeout rate.
eg: Ma Soldier’s official website ordering operation: 100 orders were placed, 1 of which was wrong, and the error rate was 1%, 100 of which must be under load, that is, the TPS is 100 times/s eg: the maximum processing request of
TPS The capacity is 10,000 times/s, but 100 order requests are sent, 99 of them are successful, and 1 is wrong. This is not called an error rate, but a bug.
The timeout rate will not affect the amount, and the effect is not large.
Internet industry reference standard:
Different systems have different requirements for the error rate, but generally it does not exceed 3 per thousand, that is, the success rate is not less than 99.7%
5、CPU
The central processing unit is a very large-scale integrated circuit, which is the computing core (Core) and control core (Control Unit) of a computer. Its function is mainly to interpret computer instructions and process data in computer software.
1. CPU main frequency: That is, the CPU clock frequency, the frequency directly determines the performance of the CPU. In GHz, the higher the value, the higher the frequency, the stronger the performance, and the greater the power consumption. The main frequency of the CPU can be increased by overclocking, that is, increasing the voltage, but it will increase heat generation and instability.
2. CPU digits: Refers to the number of floating-point numbers that the processor can calculate at one time. Generally, the higher the CPU digits, the faster the CPU will perform operations. 64-bit chips are common now.
3. CPU cache: It is a temporary memory located between the CPU and the memory. Its capacity is much smaller than that of the memory, but the exchange speed is much faster than that of the memory. Generally speaking, the cache of the CPU can be divided into a first-level cache and a second-level cache. Cache and L3 cache, cache performance directly affects CPU processing performance.
4. Number of CPU cores: Refers to the number of CPU cores that process data intensively on the CPU. The number of cores directly affects the speed of CPU processing.
5. Number of CPU threads: The number of CPU threads refers to the number of threads on the same processor that multiple threads execute synchronously and share the execution resources of the processor, which can significantly improve performance with almost no extra cost. Multi-threading technology can prepare more data to be processed for the high-speed computing core and reduce the idle time of the computing core. In general, the more threads, the more powerful the performance.
6. CPU process: Refers to the CPU process technology, generally in nanometers. The lower the value, the more transistors can be integrated, that is, the more powerful the performance, so the improvement of the process technology provides the basis for the performance improvement. And a more advanced process will also reduce heat and reduce power consumption.
The CPU indicator mainly refers to the CPU utilization rate, including user state (user), system state (sys), waiting state (wait), and idle state (idle). CPU utilization should be lower than the industry warning value range, that is, less than or equal to 75%;CPU sys% is less than or equal to 30%, CPU wait% is less than or equal to 5%. Single-core CPUs also need to comply with the above indicators.
User indicates: what percentage of CPU time is spent running in user mode space or user processes (running user space processes). Typical user space programs include: Shells, databases, web servers...
The meaning of System is similar to that of User. System indicates: what percentage of time the CPU spends running in kernel space. Allocating memory, IO operations, creating child processes... are all kernel operations. This also shows that when the IO operation is frequent, the System parameter will be high.
wait In the computer, the operation of reading and writing the disk is much slower than the running speed of the CPU. The CPU load processes the data, and the data generally needs to be read from the disk to the memory to be processed. When the CPU initiates a read and write operation, it needs to wait for the disk drive to read the data into the memory (refer to: JAVA IO and NIO understanding), which causes the CPU to have nothing to do during the waiting period. The time the CPU is in this wait state is measured by the Wait parameter.
Idel said: CPU is idle time ratio
6. Memory
Memory is one of the important components in the computer, it is a bridge to communicate with the CPU. All programs in the computer run in the memory, so the performance of the memory has a great impact on the computer.
In order to maximize memory utilization, modern operating systems store caches in memory. Therefore, 100% memory utilization does not mean that there is a bottleneck in memory. The measurement of bottlenecks in the system mainly depends on the utilization of swap space in SWAP (swapping with virtual memory). Under normal circumstances, the utilization rate of SWAP swap space should be lower than 70%. Too many swaps will cause low system performance.
7. Disk throughput
Disk throughput refers to the amount of data passing through a disk per unit of time without disk failure.
The disk indicators mainly include the number of megabytes read and written per second, the disk busy rate, the number of disk queues, the average service time, the average waiting time, and the space utilization rate. Among them, the disk busy rate is an important basis to directly reflect whether the disk has a bottleneck. Generally, the disk busy rate should be lower than 70%.
8. Network throughput
Network throughput refers to the amount of data that passes through the network per unit of time without network failure. The unit is Byte/s . The network throughput index is used to measure the system's demand for network equipment or link transmission capabilities. When the network throughput index is close to the maximum transmission capacity of the network device or link, it is necessary to consider upgrading the network device. 10M/s
The network throughput index mainly includes how many megabytes of traffic enters and exits per second, and generally cannot exceed 70% of the maximum transmission capacity of the device or link.