[TOC]

Nginx some experience in pressure measurement and performance analysis

Nginx some experience in pressure measurement

Watch bandwidth of the NIC, NIC queue

Gigabit Ethernet single machine to measure the pressure nginx performance, there will be the bandwidth bottleneck, this time to observe the CPU and bandwidth, CPU did not run full discovery, si is not full, but the NIC bandwidth traffic has reached a bottleneck for Gigabit NIC said that this small b, rather than a large B; then the theoretical upper limit of Gigabit Ethernet is 128MB / s, but due to various factors, often difficult to reach 128, but in general to 110MB / s or more is already a bottleneck .
- Gigabit Ethernet, based on the card will be a bottleneck, on this basis, then Nginx as a Web Server for http:
  - 0KB，QPS = 48w (return 200)
  - 0KB，QPS = 29w (return 302)
  - 1B， QPS = 23w
  - 1KB，QPS = 9w
- Nginx direct return 200 and return 302, QPS lot of difference (Gigabit Ethernet basis)
  - In the Gigabit Ethernet, return 200, QPS 48w; return 302, QPS 29w. The reason is because the amount of data returned Response to return 302 the result to be bigger, resulting in resulting in a second processing request not so much because of the presence of traffic bottlenecks NIC
When the measured pressure side has four Gigabit Ethernet NIC after, put pressure on the flow side of the card went to the bottleneck, and this time you need to adjust the pressure on the end.
4 NIC binding, short connection when the measured pressure, was found to have 100% si, and is a fixed number of the CPU, the card will be described binding repeat.
- After the card rebind did not repeat, si have improved, overall performance improved slightly
24 Nucleation single, single Gigabit Ethernet at the scene, the first NIC bottlenecks; the next scene four gigabit Ethernet card, CPU si soft-break bottlenecks.
Pressure measurement time, pay attention to the big bag, small packets of different treatments for the packet, CPU consumption will be more, because the soft interruption, Baotou solution of the problem, check the packet data level. sar observations when, in general, rxpck / s and txpck / s only need to focus on data without bottlenecks, the bottleneck will be built on the CPU; and the need to focus on rxkB / s and txkB / s, this NIC determines the traffic bandwidth limit. Not only to observe the server, the client depends on whether there is a bottleneck (CPU, network card bandwidth), error conditions, etc.

Watch Nginx CPU consumption is uniform

For normal, CPU consumption of each worker processes nginx should be uniform, if a difference of more than 10%, even 20% or more, then there must exist the problem of uneven CPU consumption. nginx will make the current version of the CPU uneven, due to the closure of the accept_mutex, normal, each worker process should all be about the same CPU consumption, if open accept_mutex on; evenly to the front of several nginx worker processes.
The best posture is to open reuseport, but this needs to be used with attention and dynamic upstream, or if frequent reload will result in a large number of RST.

Hyper-Threading CPU attention

Disabled, enable Hyper-Threading CPU, in the case of nginx single process, and there is no significant difference
- Pressure measurement time, whether hyperthreading turned off, there is no significant difference

Concerned more with a single instance

After a stand-alone single-instance performance bottlenecks after reaching the limit or there, or as stand-alone multi-instance, did not improve; there is no stand-alone single-instance performance bottleneck, you can improve the performance of multi-instance

End pressing concern

When the pressure test, to find a performance inflection point; if up against a bottleneck, you also need to callbacks, until you find a best performance inflection point.
- When long connected to wrk if the number of threads, the number of concurrent too great, it will make Nginx only time a worker process can degrade performance; short time processes, system parameters net.ipv4.tcp_tw_recycle wrk end to be set to 1 let port multiplexing, otherwise there will be a number of connection error
- Thus a process is the pressure will be the pressure side (the number of threads, number of concurrent) will decrease, increase, to observe Nginx data server, and then to obtain the best performance data
wr pressure go up? To Why not go? CPU bottleneck, or a memory bottleneck? If the cpu, then how to give more CPU? Of course, the number of threads. But also the number of threads and cores CPU pressure side match.
- top -H see threads, each thread did not want to run over, so as to maximize the performance, if each thread ran full, then you can not wrk maximize performance, which is unable to provide the highest pressure. If each thread did not run over, but the QPS or do not increase, then Nginx is a performance issue here.

Indicators must be observed when measuring pressure

CPU【top】

top, observe CPU consumption, but also to observe the number of each CPU core, and soft interrupt data concerned si, si to 100% have a problem

LAN Bandwidth [sar -n DEV 1 100 | grep em1]

sar -n DEV 1100 | grep em1

Observe the NIC bandwidth, the bandwidth of the network card to see whether the bottleneck

Ifconfig can then see if there is packet loss and the like.

Even built per second [netstat -s | grep active]

netstat -s |grep active
    6262441249 active connections openings
复制代码

By netstat -s |grep activeactive connections to get the current, and then do the difference.

5W QPS short connection, if there is a short upstream connection, then built per second even number should be around 10W

[] Ss -lnt connection queue

ss -lnt

ss -lnt |grep -E ":6001|:6002"

State       Recv-Q Send-Q                                   Local Address:Port                                                  Peer Address:Port
复制代码

When the socket in a listening state (Listening),
- Recv-Q represents the current value of the syn backlog.
- Send-Q syn backlog means the maximum value.
If the connection queue is too small, and the need to adjust the system configurations nginx

Disk IO

Nginx tuning some experience

Nginx performance indicators

Relevant basic indicators:

Nginx requests processed per second (ie:: QPS) Requests per second (RPS)
- RPS for HTTP Requests: http nginx number of requests processed per second
- RPS for HTTPS Requests (SSL / TLS transactions per second) (TPS): nginx treated https requests per second
- Follow in response to data 0 - 100 KB in size exhibit different situations --1--10
  - 0KB indicates an empty http response, such as the return error code 302
Connections per Second (CPS): nginx new connection requests processed per second
- Including http and https
Throughput: the size of the volume capacity of data throughput, the reaction can handle nginx
latency: delay and delay distribution

Other concerns:

Watch CPU core number from 1 - In n, nginx performance performance
- Use taskset , a can wrk process is bound to a CPU core; performance situation can accurately test different CPU Nucleation
Nginx is concerned about the role of different manifestations webserver or reverse proxy
Connections, sometimes referred to as the number of concurrent, it refers to the number of simultaneous service requests. That is, those who have already sent a request (Request), but has not yet received complete response (Response) The number of requests.

Nginx is necessary to adjust the parameters

Nginx is necessary to adjust the parameters:

worker_processes     auto;
worker_rlimit_nofile 10240;

worker_connections 10240;

tcp_nopush  on;
tcp_nodelay on;


keepalive_timeout  300s;     
keepalive_requests 1000000;
复制代码

Recommended parameter adjustment:

proxy_connect_timeout 60;
proxy_send_timeout 60;
proxy_read_timeout 60;
复制代码

Linux system must adjust the parameters

conntrack parameters

Generally, set nf_conntrack_max to 200w, nf_conntrack_buckets 1/4 or 1/2 nf_conntrack_max, to prevent the barrel is too large to affect performance.

[[email protected] ingress]$ cat /proc/sys/net/netfilter/nf_conntrack_buckets
524288

[[email protected] ingress]$ cat /proc/sys/net/netfilter/nf_conntrack_max
2097152
复制代码

backlog queue

net.core.somaxconn
- Maximum number of connections can be queued Nginx accept. If resulted in generally too small Nginx performance issues can view the kernel log you found the state
- Together with instructions for adjusting NGINX listen.
net.core.netdev_max_backlog
- Packet sent to the rate buffer before the CPU card; this value may be increased to improve the performance of the machine have a high bandwidth

echo 32768 > /proc/sys/net/core/somaxconn
echo 819200 > /proc/sys/net/ipv4/tcp_max_syn_backlog
复制代码

File descriptor

sys.fs.file-max
- Linux file system allows a maximum number of description
nofile
- The maximum number of file descriptors allowed the application level, the general settings /etc/security/limits.conffile

Port Modify /etc/sysctl.conf, then sysctl -p Entry Into Force

net.ipv4.ip_local_port_range
- port range of ports
For pressure measurement end, if a short link
- net.ipv4.tcp_tw_reuse = 1
  - It represents the open port multiplexing. TIME-WAIT sockets allow re-used for new TCP connection, the default is 0, indicating off;
- net.ipv4.tcp_tw_recycle = 1
  - Represents a rapid recovery of open TCP connections TIME-WAIT sockets, the default is 0, it off.

NIC queue, CPU soft interrupt si

20-24 Nucleation stand-alone, single Gigabit Ethernet at the scene, the first NIC bottlenecks; the next scene four gigabit Ethernet card, CPU si soft-break bottlenecks.
4 NIC binding, short connection when the measured pressure, was found to have 100% si, and is a fixed number of the CPU, the card will be described binding repeat. After the card rebind did not repeat, si have improved, overall performance improved slightly
CPU idle is zero there will be no problem, only si 100% will have a problem
- si to 100%, the CPU in a large number of soft interrupt handling, this time, note cards and CPU-bound soft interrupt this somewhat bottleneck. NIC queue or is not enough, or is too CPU core, or is cpuset CPU card queues and bindings for the process repeated.

[ "I welcome the attention of the public micro-channel number: Linux server system development, quality article send back will vigorously by the public micro-channel number"]