linux - server performance evaluation

Factors affecting the performance of Linux servers

1. cpu
2. RAM
3. Disk IO
4. Network IO

System Performance Evaluation Criteria

Factors affecting performance	it is good	Bad	bad
CPU	user% + sys%< 70%	user% + sys%= 85%	user% + sys% >=90%
RAM	Swap In(si)=0 Swap Out(so)=0	Per CPU with 10 page/s	More Swap In & Swap Out
Disk	iowait % < 20%	iowait % =35%	iowait % >= 50%

Wherein:
% User: indicates the percentage of time the CPU is in user mode.
% sys: indicates the percentage of time the CPU is in the system mode.
% iowait: indicates CPU waits for input and output completion percentage.
swap in: ie si, represents the virtual memory page import, that exchange from SWAP DISK into RAM
swap OUT: That is so, represents the virtual memory page is derived, namely switching from RAM to SWAP DISK.

System performance analysis tools

1. Common system commands
   Vmstat, sar, iostat, netstat, free, ps, top , etc.
2. Common combinations:
   with vmstat, sar, iostat detect whether a CPU bottleneck.
   With free, vmstat detect whether the memory bottleneck.
   With iostat detect whether the disk I / O bottlenecks.
   Netstat detecting whether the network bandwidth bottleneck.

The overall system load query (uptime)

uptime
  08:21:34 up 36 min,  2 users,  load average: 0.00, 0.00, 0.00
 
#当前服务器时间：    08:21:34
#当前服务器运行时长  36 min
#当前用户数          2 users
#当前的负载均衡      load average  0.00, 0.00, 0.00，分别取1min,5min,15min的均值

This query command the CPU load, the number of CPU cores to obtain N should be observed, observation behind two digital, digital / N, if the value is less than 0.7 can be obtained worry.

CPU performance evaluation

vmstat command

Monitoring system CPU utilization vmstat command, the command can display summary information about the relative performance between the various resources of the system, here we mainly use it to watch a CPU load.

vmstat 5 5: monitoring every 5 seconds for a total of 5 times

• r: represents the run queue (that is really how many processes assigned to CPU), how much the general run queue, indicates that the CPU is busy, resulting in high CPU usage (number of processes running and waiting for CPU time slice, in principle, a core CPU run queue no more than 2, the whole system queue can not exceed two times the total number of cores, or on behalf of the system pressure is too large)
• b: that the process blocked (number of processes waiting for resources, such as disk is waiting for I / O, network I / O, etc.)
• swpd: virtual memory size has been used, if greater than 0, indicating a lack of physical memory to your machine, if not the cause of memory leaks program, then you should upgrade the memory or memory-intensive tasks to migrate to other machines.
• cs: context switching times per second
• us: user CPU time
• sy: system CPU time, if too high, the system is calling for a long time, for example, is frequently IO operations.
• id: idle CPU time
• wt: IO CPU wait time
• free: free physical memory size
• buff: a buffer between the device and the device. Linux / Unix systems is used to store, catalog there what the cache, permissions, etc.
• cache: a buffer between the memory and cpu
• si: read per second from disk into virtual memory size, if this value is greater than 0, indicating that not enough physical memory or a memory leak, and memory-intensive process're looking to get rid of.
• so: virtual memory size of the disk writes per second, if this value is greater than 0, above.
• bi: the number of blocks per second received block device, where the block device refers to all disk blocks and other devices on the system, the default block size is 1024byte
• bo: number of blocks sent by the second block, for example, we read the file, must be greater than 0 BO. bi bo and generally have close to zero, or is IO too often, need to be adjusted.
• wa: IO CPU wait time.
• us: user CPU time
• sy: system CPU time, if too high, the system is calling for a long time, for example, is frequently IO operations.
• id: idle CPU time, in general, id + us + sy = 100, generally I think id be idle CPU usage, us is user CPU usage, sy is the system CPU usage.

Emphasis participation number r, b, swpd, free, buff, cache, si, so, bi, bo

sar command

sar analysis including: reading and writing of documentation, use of system calls, disk I / O, CPU efficiency, memory usage, process activities and related activities such as IPC

Format: sar [options] [-A] [-o file] t [n]

Wherein:
T: sampling separation, n is the sample number, the default value is. 1
-o File: shows a command results in a file stored in a binary format, file is the file name.

options for the command-line options, sar command common options are as follows:

• The sum of all reports: -A
• Statistical information output CPU usage: -u
• -v: Statistical information output inode, documents and other kernel tables
• -d: each block activity information output device
• -r: Statistical information output memory and swap space
• -b: Displays statistics I / O transfer rate, and
• -a: file reading and writing case
• The number of output process statistics, create a second process: -c
• Statistical information output memory pages: -R
• -y: terminal equipment activities
• -w: output system information exchange activities

sar -u 2 5: Output cpu usage and statistics

Output Description:

CPU: all statistics are expressed as the mean of all the CPU.
% user: display the percentage of total CPU time used at the user level (file application) operation.
% nice: displaying at a user level, a nice operation, the total percentage of time occupied by CPU.
% system: the percentage of the total CPU time in the kernel level (Kernel) operation.
% iowait: displaying a percentage of waiting I / O operation takes a total time of the CPU.
% steal: manager (hypervisor) to provide services to another virtual process while waiting for the percentage of virtual CPU.
% idle: displays CPU idle time occupancy percentage of the total time of CPU.

1. If the value of% iowait too high, indicates the presence of the hard disk I / O bottlenecks
2. If the value of% idle high but the system slow to respond, it is possible to wait for the CPU to allocate memory, should increase the memory capacity
3. If the value is consistently lower than 1% idle, the system is relatively low CPU processing power, indicating that the system resources need to be resolved is the most CPU

mpstat command

View all CPU core information: mpstat -P ALL 2

pidstat command

Each decomposition process uses the cpu usage information: pidstat -u 1 -p process ID

Memory performance evaluation

free command

• free
• free -m units of M
• G units in free -g

pidstat command

pidstat -p process ID -r sampling interval in seconds

vmstat command

vmstat 2 3

[root@node1 ~]# vmstat 2 3
procs ———–memory———- —swap– —–io—- –system– —–cpu——
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 162240 8304 67032 0 0 13 21 1007 23 0 1 98 0 0
0 0 0 162240 8304 67032 0 0 1 0 1010 20 0 1 100 0 0
0 0 0 162240 8304 67032 0 0 1 1 1009 18 0 1 99 0 0

Description:

• Memory
  swpd column indicates the amount of memory to the memory switching exchange zone (in units of k). Swpd If the value is not 0, or larger, as long as the value si, so long as 0, in which case generally do not worry, do not affect system performance.
  column indicates the number of free physical memory currently free (in units of k)
  BUFF column indicates the number of memory buffers cache, generally read and write device only needs to block buffer.
  column indicates the number of cache memory page cached, usually as a file system cached, frequently accessed files are cached, if the cache value is large, indicating that more number of files cached, if this time the bi IO is relatively small, indicating that the file system efficiency Better.
• swap
  Si column is represented by a disk into memory, which is the amount of RAM memory into the swap area.
  so the column is represented by a memory disk transferred, that is, memory swap enter the amount of memory.
  Under normal circumstances, the value of si, so are zero, if the value si, so the long-term is not zero, it indicates a lack of system memory. The need to increase system memory.

Disk I / O performance evaluation

1. Disk storage foundation
   knowledge of RAID storage, depending on the application, choose a different RAID mode.
   Possible instead of direct disk I / O with the memory read and write the files or frequently accessed data processing operation into the memory, because the memory read and write operations to a thousand times higher than the efficiency of disk reads and writes directly.
   Frequently read and write files with long-term change file independent, are placed on different disk devices.
   For a write operation of data frequently, consider using raw device instead of the file system.
2. The advantage of using raw device are:
   data can be read directly without going through the operating system level cache, saving memory resources, avoiding the memory resource contention.
   Avoid the file system-level maintenance overhead, such as file systems need to maintain the super block, I-node and so on.
   Avoiding the operating system cache read-ahead function, reducing the I / O request.
3. The disadvantage of using raw device are:
   data management, space management is not flexible, need very professional people to operate.

iostat command

Evaluate the use of iostat disk

[root@webserver ~]# iostat -d 2 3
Linux 2.6.9-42.ELsmp (webserver) 12/01/2008 _i686_ (8 CPU)
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 1.87 2.58 114.12 6479462 286537372
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 0.00 0.00 0.00 0 0
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 1.00 0.00 12.00 0 24

The output of each of the above is explained as follows:

• Blk_read / s represents the number of data blocks read per second.
• Blk_wrtn / s represents the number of data blocks written per second.
• All Blk_read represents the number of blocks read.
• Blk_wrtn represents all number of blocks written.

1. By value Blk_read / s and Blk_wrtn / s have a basic understanding of the read and write performance of the disk, if Blk_wrtn / s value is large, the disk write operations are frequent, consider optimizing disk optimizer or, if Blk_read / s value is large, the disk directly read many operation, data may be read into memory to operate.
2. For values ​​of these two options do not have a fixed size, depending on the system application, have different values, but there is still a rule can be followed: a long-term, large data read and write is certainly not normal, it case will affect system performance.

sar command

Use sar assess disk performance
by "sar -d" combination, you can do a basic statistical system's disk IO, please see the following output:

[root@webserver ~]# sar -d 2 3
Linux 2.6.9-42.ELsmp (webserver) 11/30/2008 _i686_ (8 CPU)
11:09:33 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
11:09:35 PM dev8-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:09:35 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
11:09:37 PM dev8-0 1.00 0.00 12.00 12.00 0.00 0.00 0.00 0.00
11:09:37 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
11:09:39 PM dev8-0 1.99 0.00 47.76 24.00 0.00 0.50 0.25 0.05
Average: DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
Average: dev8-0 1.00 0.00 19.97 20.00 0.00 0.33 0.17 0.02

Need to focus on the meaning of several parameters:

1. It represents the average latency per await device I / O operations (in milliseconds).
2. svctm represents the average service time per device I / O operations (in milliseconds).
3. % Util represents one second in a few percent of the time for I / O operations.

On to the disk IO performance, the following general criteria:

1. Normally svctm await should be less than the value, and the size and svctm properties associated disk, CPU, memory load will also affect the value svctm, too many requests will indirectly leads to an increase svctm value.
2. Magnitude values ​​generally depends await svctm values ​​and I / O queue length and the I / O request mode, if the value of svctm await very close, showing almost no I / O wait, disk performance very good, if the value is far await higher than the svctm, said I / O queue waiting too long, the application running on the system will slow down, then the problem can be solved by replacing the hard disk faster.
3. % Util term value is also an important indicator to measure the disk I / O, if% util close to 100%, the resulting disk I / O request is too much, I / O system has been working at full capacity, the disk may be a bottleneck . Over time, it is bound to affect system performance by optimizing program or to resolve this problem by replacing the higher, faster disk.

Network Performance Evaluation

(1) ping command detecting connectivity of the network
(2) via a network interface netstat -i combination detection condition
(3) through a combination of routing table information detection system netstat -r
display network operating system (4) by a combination of sar -n state
(5) by command ifstat