1 Introduction
For operation and maintenance engineers, they need to know the performance bottlenecks of the servers they maintain. For example, what is the concurrency per second of my current architecture, what is the maximum concurrency that my server can accept, and what causes my performance problems; if the current architecture The performance bottleneck is almost reached, whether the horizontal expansion performance improves greatly, or the vertical expansion performance improves greatly.
If you need to understand this information, you need to work hard in two aspects, one is to perform performance testing on the server, and the other is to perform performance monitoring on the server.
Through the performance test of the server : we can understand the performance bottleneck of the current architecture, and can also test the horizontal expansion and vertical expansion of the architecture, and provide data reference for the subsequent expansion of the architecture.
Through the performance monitoring of the server : we can understand whether the current server's CPU, memory, IO and other resources are exhausted, we can add triggers to the monitoring system, once the server resources are about to reach the bottleneck, we can trigger an alarm to let the operation Maintenance personnel can handle it, and can also trigger an automated expansion of the architecture (if it is a cloud platform, directly call the api to create the host, and ansible to deploy applications and programs)
This article will introduce some practices that I use jmeter to test performance bottlenecks in my work. The performance test in this article is applicable to mobile Internet architecture, and there are other better test methods for non-mobile Internet architecture.
2. Introduction to Jmeter distributed pressure measurement
In the scenario of using jmeter to do a large concurrent stress test at work, there will be a single machine with limited memory, CPU, and network IO. The server pressure has not yet risen, but the stress test server has crashed due to too much pressure from the simulation. In order to allow the jmeter tool to provide more powerful load capacity, jmeter provides a mechanism for multiple machines to generate load at the same time. The following is the architecture diagram.
Principle: For example, if I configure the number of threads in the jmeter server to be 10 and the number of cycles to be 100, that is, I will initiate 1000 requests to the test server. I have 3 agent servers. If I choose to start the stress test remotely on the server side, each The agent will initiate 10 100 requests to the test server , so the requests generated by this stress test are 10 100*3=3000 times.
If you don't understand the principle very well, you will understand it after reading the following operations.
3. Construction of Jmeter distributed pressure test environment
3.1. Instructions before building
Server environment description: For performance testing, you can directly purchase the press on the cloud platform on demand, and release the press once the test is over.
Distributed environment stress server requirements:
-
Need server (control machine) and agent (press machine), the agent is built in the linux (centos 6.5) server environment, and the server is built in the windows (server 2012) environment.
-
Most of the bottlenecks in stress testing are bandwidth. It is necessary to ensure that the bandwidth of the press is higher than that of the server, otherwise the pressure will not rise.
-
It is necessary to ensure that both the agent and server are in the same network, and in a multi-NIC environment, it is necessary to ensure that the started network cards are in the same network segment.
-
Need to ensure the time synchronization between the server and the agent.
- Turn off the firewall.
3.2. Windows deployment jmeter
(1) Deploy the jdk environment, configure the path variable, and the installation effect is as follows
(2) Go directly to the official website to download the latest binary source code package.
(3) Unzip jmeter to the specified directory and set the path variable. After the installation is complete, run the jmeter command on the command line. If jmeter can be started normally, the environment configuration is ok.
3.3.Linux deployment jmeter
(1) Download and install
wget http://mirrors.tuna.tsinghua.edu.cn/apache//jmeter/binaries/apache-jmeter-3.1.zip
unzip apache-jmeter-3.1.zip -d /usr/local/
cd /usr/local/
ln -s apache-jmeter-3.1/ jmeter(2) Configure the startup script
#!/bin/bash
# chkconfig: 345 26 74
# description: jmeter agent
myip=`ifconfig eth0 |awk '/inet addr/{gsub(/addr:/,"");print $2}'`
cmd="/usr/local/jmeter/bin/jmeter-server -Djava.rmi.server.hostname=$myip"
start(){
$cmd &
}
stop(){
jmeter_pid=`ps aux | grep jmeter-server | grep -v grep | awk '{print $2}'`
for pid in $jmeter_pid;do
kill -9 $pid
done
}
act=$1
case $act in
'start')
start;;
'stop')
stop;;
'restart')
stop
sleep 2
start;;
*)
echo '[start|stop|restart]';;
esac
(3) Start the jmeter agent service and verify whether it is listening on port 1099
[root@jmeter-agent-01 ~]# /etc/init.d/jmeter-agent start
[root@jmeter-agent-01 ~]# netstat -lntp | grep 1099
tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN 414/java3.4. Distributed environment configuration
(1) Ensure that the server and agnet are installed correctly.
(2) The Agent starts and monitors port 1099.
(3) In the bin directory under the jmeter installation directory of the server machine, find the properties file, modify the remote host option, and add the addresses of three agent servers.
(4) Start jmeter server, multi-network card mode needs to specify the IP address to start
jmeter -Djava.rmi.server.hostname=192.168.10.61(5) Verify whether the distributed environment is successfully built
1. After jmeter is started, the list of remote hosts you added will appear in the following options
2. Create a request test: Create a request to visit Baidu, the number of visits is once, the configuration is as follows:
Click Start directly, the jmeter server machine initiates a request, and the result is as follows
After requesting all the previous request data, after selecting the remote start all, check the initiated request is three times, that is, each agent server requests once according to the server configuration.
If your environment does not report an error after selecting all startups, and the number of initiated requests is the same as the number of agent servers, it means that the jmeter distributed stress test environment has been successfully built and can be tested.
4. Jmeter affirmation
4.1. Introduction to Assertions
There are two commonly used jmeter assertions, one is response assertion, and the other is response time assertion. If the response content does not meet the configuration of the assertion, the request is considered to be a failure.
Response assertion: Determine whether the response content contains the specified character information, which is used to determine whether the content returned by the api interface is correct.
Response time assertion: Determine whether the response time exceeds the expected time, and is used to determine whether the return time of the api interface exceeds the expected time.
4.2. Assertion configuration
(1) Modify http to the actual api test request.
(2) Assertion adding method: Right-click the http request of the test plan, select Add→Assertion→Add response assertion and assertion duration.
(3) Configuration response assertion: Our interface normally returns a code value of 2000. If the interface returns a code value of not 2000, it means that the interface is abnormal. For testing purposes, here is modified to the interface that returns a code value of not 2222, which means that the access failed.
(4) Configure assertion response time: If the request interface time is set to exceed 1 millisecond, the request is considered to have failed.
(5) Verify the assertion configuration: Initiate an http request, because the returned content code value is not 2222, and the access time exceeds 1 millisecond, the access is considered to be failed.
5. Jmeter variable configuration
Examples of scenarios where variables are used: we need to test the performance curve model, that is, slowly changing from light pressure to heavy pressure to test our performance inflection point. At this time, jmeter needs to configure multiple thread groups, and each thread group needs to be set http request, such as the figure below; because the curve model of each test performance is the same interface, every time you modify the interface, you need to modify the http request. If you use variables at this time, it means that you only need to modify the api every time you modify the api The variables can be.
Method of setting variables: in the test plan
Reference variable:
6. Analysis of Jmeter performance test results
The following is an aggregate report of a performance curve model test that I performed (requests increased from 3,000 to 30,000 per second): From a simple look, you can see that the inflection point of performance is 27,000 requests per second, and the TPS processing capacity can reach 6,000. Per second, 99% of the user response time is 60 milliseconds, and the maximum response time is 71 milliseconds. The performance is still good.
Concurrency bottleneck : When requests increased from 27,000 to 30,000 requests per second, our TPS dropped from 6000 to 4500. It can be seen that the concurrency bottleneck is processing up to 6000 requests per second.
Response time: We can see that TPS stays at or below 3500, and 99% of users have a response time of 11 milliseconds. As TPS increases, our response time also increases, and we can see our TPS At 3,500 responses per second, it has no effect on the response time.
Note that this is only one of the interfaces of my business. We have hundreds of interfaces in production. Different interfaces return data, code logic, and executed SQL. If you need to do performance testing, you should choose one of the hotspot interfaces. Perform a performance test on each interface, and after the results are obtained, perform a specific analysis of the performance bottleneck to the lowest level?
Aggregate report parameter description: The unit is milliseconds
-
Label: Define the name of the HTTP request
-
Samples: Indicates how many requests were issued in this test
-
Average: Average response time-by default it is the average response time of a single request
-
Median: The median, which is the response time of 50% of users
-
90% Line: 90% of users’ response time
-
Min: The minimum response time to visit the page
-
Max: The maximum response time for visiting the page
-
Error%: the number of error requests/total number of requests
-
Throughput: By default, the number of completed requests per second (request per second)
- KB/Sec: The amount of data received from the server per second
7. Monitoring during testing
7.1. Concurrent test monitoring
Concurrent testing directly initiates a specified number of requests, such as initiating 100,000 requests together to look at the processing capacity of the system. If you need the resource usage information of the server at this time, you cannot use the zabbix monitoring system, because generally 100,000 requests are processed. It can be processed in 20 seconds, but zabbix data collection is once every minute, so the collected data is obviously inaccurate, so you need to query the performance of the server in real time through the monitoring command that comes with the system, for example, you can use dstat Or dynamic monitoring commands such as glances to analyze the performance of the system.
Supplement: It is not necessary to perform such real-time monitoring to test every interface. For example, after testing most of my interfaces TPS can reach 5000, but one of the interfaces can only reach 2000. At this time, you need to monitor in real time during the test. Check it out. What causes the performance to fail. Is it because the returned data is too large that the network bandwidth is full; or the SQL execution time is too long and the database load is high, or the code has a problem that causes the web service cpu to take up high.
7.2. Stability test monitoring
The stability test is to continuously simulate a specified number of requests to access the server. For example, I initiate 4000 requests to the test server every second for 12 hours to see what happens to the server. At this time, I need to use zabbix to monitor. , The following is part of the monitoring interface I did performance testing, including tomcat requests per second, server entrance traffic, statistics of http status codes requested per minute for the entire cluster, and server resource usage information.
Imprint: Author: Simon flying ice, after a 90 it man, has been working in Beijing, love sports, love of adventure, love of travel. Original: http://www.fblinux.com/?p=951, submitted by the author, the copyright belongs to the original creator. Unless it is impossible to confirm, we will mark the author and source. If there is any infringement, please let us know, we will delete it immediately and apologize, thank you .