If a certain Linux has a performance problem, and you check it after logging in, which commands will you use to check those aspects in the first minute?
At Netflix, we have a large number of EC2 instances in the cloud and several performance analysis tools to monitor and troubleshoot performance issues. Including the full cloud monitoring product Atlas and the instance granular Vector. Although these tools can help us solve most problems, sometimes we still need to log in to the instance and run some standard Linux commands to assist in troubleshooting.
Minute 1: Summary
In this article, Netflix's performance engineering team will show you how to do performance troubleshooting from the command line in the first 60 seconds, using some standard Linux tools along the way.
In 60 seconds, you can get an overview of system resource usage and running processes by running the following ten commands. Look for error and saturation metrics, then resource utilization related metrics.
The word saturation may be difficult to understand. You can Google the USE method. The so-called saturation means that the resource load exceeds its processing capacity, which can be expressed as the length of the request queue or waiting time.
uptimedmesg | tailvmstat 1mpstat -P ALL 1pidstat 1iostat -xz 1free -msar -n DEV 1sar -n TCP,ETCP 1top
This is the 10 commands, some commands need to install the sysstat package in advance, please pay attention. The indicator data exposed by these commands will help you complete the USE Method (this is a methodology for locating performance bottlenecks) analysis. Including checking various aspects such as CPU, memory, and hard disk, and checking indicators related to utilization, saturation, and errors. In addition, pay attention to the application of exclusion method. When we exclude a certain type of resource problem, it means that the scope of investigation is narrowed, which is conducive to subsequent troubleshooting.
The following sections summarize the commands, with examples from a production system. For more information on these tools, see their manuals.
1、uptime
$ uptime23:51:26 up 21:31, 1 user, load average: 30.02, 26.43, 19.02
This is a quick way to see the system load average, which indicates how many tasks (processes) are running on the system. On Linux systems, these numbers include processes that need to be running on the CPU as well as processes that are waiting for I/O (usually disk I/O). It's just a rough indication of system load, just look at it a little bit. You'll also need other tools to learn more about the specifics.
These three figures show the results obtained by exponentially compressing the total load average of the system in one minute, five minutes, and fifteen minutes. From this we can see how the load on the system changes over time. For example, if you are checking a problem and see that the value corresponding to 1 minute is much smaller than the value of 15 minutes, then it may indicate that the problem has passed and you did not observe it in time.
In the example above, the system load is increasing over time because the most recent minute load value was over 30, while the 15-minute load average was only 19. Such a significant difference contains many meanings, such as CPU load. For further confirmation, you need to run the vmstat or mpstat command. Please refer to the following chapters for these two commands.
2、dmesg | tail
$ dmesg | tail[1880957.563150] perl invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0[...][1880957.563400] Out of memory: Kill process 18694 (perl) score 246 or sacrifice child[1880957.563408] Killed process 18694 (perl) total-vm:1972392kB, anon-rss:1953348kB, file-rss:0kB[2320864.954447] TCP: Possible SYN flooding on port 7001. Dropping request. Check SNMP counters.
If there are system messages in the system, this command will display the last 10 messages. Find out if there are any Errors from these messages. These Errors may help you locate system performance problems. The above example includes oom-killer, and TCP drops a request.
Don't miss this step! The dmesg command is always worth a try.
3、vmstat 1
$ vmstat 1procs ---------memory---------- ---swap-- -----io---- -system-- ------cpu-----r b swpd free buff cache si so bi bo in cs us sy id wa st34 0 0 200889792 73708 591828 0 0 0 5 6 10 96 1 3 0 032 0 0 200889920 73708 591860 0 0 0 592 13284 4282 98 1 1 0 032 0 0 200890112 73708 591860 0 0 0 0 9501 2154 99 1 0 0 032 0 0 200889568 73712 591856 0 0 0 48 11900 2459 99 0 0 0 032 0 0 200890208 73712 591860 0 0 0 0 15898 4840 98 1 1 0 0^C
vmstat(8), short for virtual memory stat, is a commonly used tool (first created for BSD decades ago). It prints a summary of key server statistics on each line.
The operating parameter of vmstat is 1, print once per second, and count the situation in the last second, but please ignore the first line. The statistics in the first line are not the situation in the previous second, but the average value since the OS startup. As far as we troubleshoot the problem, it doesn't help much.
Columns to focus on
-
r : Total number of processes running or waiting to run. Compared with the average load data in uptime, the r value does not include I/O, which can better reflect the CPU saturation. If this value is greater than the number of cores of the CPU, it means that it is too saturated.
-
free : free memory in KB units. If you find a long string of numbers here, it means you still have a lot of memory available :) We will introduce
free -mcommands later, which can better explain the situation of free memory. -
si,so : The amount of Swap in and out. If these values are non-zero, it means that your memory is not enough.
-
us, sy, id, wa, st : These are CPU decomposition values, which are an average value for all CPUs, not for a certain core. us is user time, sy is system time (kernel), id is idle, wa is wait I/O, st is stolen time (only virtual machines need to pay attention to st)
Add the user mode time and the kernel mode time to know whether the CPU is busy. A constant level of wa indicates a performance bottleneck in terms of I/O. If wa is high, idle will be high, because the CPU has been waiting for I/O to obtain data, and there is no way to continue computing.
System time system time is necessary for I/O processing. A high system time average, say over 20%, can be further probed, perhaps the kernel is handling I/O inefficiently.
In the above example, the CPU time is mainly spent on the user, which means that the application program in the user mode is occupying the CPU time. The average CPU utilization is also well over 90%. But this is not necessarily a problem, r check the saturation by the value of the column.
4、mpstat -P ALL 1
$ mpstat -P ALL 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)07:38:49 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle07:38:50 PM all 98.47 0.00 0.75 0.00 0.00 0.00 0.00 0.00 0.00 0.7807:38:50 PM 0 96.04 0.00 2.97 0.00 0.00 0.00 0.00 0.00 0.00 0.9907:38:50 PM 1 97.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 2.0007:38:50 PM 2 98.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 1.0007:38:50 PM 3 96.97 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.03[...]
This command prints the time spent per CPU core, which can be used to check for imbalances. A high CPU usage, possibly caused by a single-threaded application.
5、pidstat 1
$ pidstat 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)07:41:02 PM UID PID %usr %system %guest %CPU CPU Command07:41:03 PM 0 9 0.00 0.94 0.00 0.94 1 rcuos/007:41:03 PM 0 4214 5.66 5.66 0.00 11.32 15 mesos-slave07:41:03 PM 0 4354 0.94 0.94 0.00 1.89 8 java07:41:03 PM 0 6521 1596.23 1.89 0.00 1598.11 27 java07:41:03 PM 0 6564 1571.70 7.55 0.00 1579.25 28 java07:41:03 PM 60004 60154 0.94 4.72 0.00 5.66 9 pidstat07:41:03 PM UID PID %usr %system %guest %CPU CPU Command07:41:04 PM 0 4214 6.00 2.00 0.00 8.00 15 mesos-slave07:41:04 PM 0 6521 1590.00 1.00 0.00 1591.00 27 java07:41:04 PM 0 6564 1573.00 10.00 0.00 1583.00 28 java07:41:04 PM 108 6718 1.00 0.00 0.00 1.00 0 snmp-pass07:41:04 PM 60004 60154 1.00 4.00 0.00 5.00 9 pidstat^C
pidstat is a bit like top's statistical view for a specific process. Unlike top, pidstat scrolls the print instead of clearing the screen. This is useful for observing patterns over time, and for recording (copy-pasting) what you see into your investigative notes.
The above example identifies two java processes responsible for consuming CPU. %CPUOne column is the sum of all CPUs; 1591%showing that this java process is consuming almost 16 CPUs.
6、iostat -xz 1
$ iostat -xz 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)avg-cpu: %user %nice %system %iowait %steal %idle73.96 0.00 3.73 0.03 0.06 22.21Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %utilxvda 0.00 0.23 0.21 0.18 4.52 2.08 34.37 0.00 9.98 13.80 5.42 2.44 0.09xvdb 0.01 0.00 1.02 8.94 127.97 598.53 145.79 0.00 0.43 1.78 0.28 0.25 0.25xvdc 0.01 0.00 1.02 8.86 127.79 595.94 146.50 0.00 0.45 1.82 0.30 0.27 0.26dm-0 0.00 0.00 0.69 2.32 10.47 31.69 28.01 0.01 3.23 0.71 3.98 0.13 0.04dm-1 0.00 0.00 0.00 0.94 0.01 3.78 8.00 0.33 345.84 0.04 346.81 0.01 0.00dm-2 0.00 0.00 0.09 0.07 1.35 0.36 22.50 0.00 2.55 0.23 5.62 1.78 0.03[...]^C
This is a great tool for understanding block devices (disks), including the applied workload and resulting performance metrics.
-
r/s, w/s, rkB/s, wkB/s : These are reads, writes, read KB, and write KB per second for the device. Use these to describe workloads. A performance problem may simply be due to excessive load.
-
await : The average time of I/O, in milliseconds. This is the time spent by the application's I/O requests, including time queued and serviced. An average time that is greater than expected may be an indicator of device saturation, or a device problem.
-
avgqu-sz : The average number of requests sent to the device. Values greater than 1 may be evidence of saturation (although devices can often process requests in parallel, especially virtual devices in front of multiple backend disks).
-
%util : Device utilization. This is actually a busy percentage, showing how much time the device is doing work per second. Values greater than 60% generally result in poor performance (should be seen in wait), though it depends on the device. Values close to 100% usually indicate saturation.
If the storage device is a logical disk device facing many backend disks, then 100% utilization may simply mean that some I/O is being processed 100% of the time, however, the backend disks may be far from being saturated, possibly able to handle more work.
Remember that poor disk I/O performance is not necessarily an application problem. A number of techniques are commonly used to perform I/O asynchronously so that applications do not block and are directly affected by latency (for example, read ahead on reads and buffering on writes).
7、free -m
$ free -mtotal used free shared buffers cachedMem: 245998 24545 221453 83 59 541-/+ buffers/cache: 23944 222053Swap: 0 0 0
The two columns on the right:
-
buffers : buffer cache for block device I/O
-
cached : page cache for the file system
Usually, we need to check whether these values are close to 0, and if they are close to 0, it may lead to higher disk I/O (use iostat to confirm further) and worse performance. The values in the example above look ok, each with a lot of MB left.
-/+ buffers/cache Provides more intuitive values for the amount of used and free memory. Linux uses free memory for caching, but can be reclaimed quickly if an application needs it. So in a way, cached memory should be included in the free memory column, and this line does just that. There's even a website, linuxatemyram, that explains this confusing situation.
This can create additional confusion if using ZFS on Linux, as we do in some services, because ZFS has its own filesystem cache, which is not properly reflected by the listing free -m. It looks like the system is low on free memory, when in fact this memory is available from the ZFS cache when needed.
8、sar -n DEV 1
$ sar -n DEV 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)12:16:48 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil12:16:49 AM eth0 18763.00 5032.00 20686.42 478.30 0.00 0.00 0.00 0.0012:16:49 AM lo 14.00 14.00 1.36 1.36 0.00 0.00 0.00 0.0012:16:49 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0012:16:49 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil12:16:50 AM eth0 19763.00 5101.00 21999.10 482.56 0.00 0.00 0.00 0.0012:16:50 AM lo 20.00 20.00 3.25 3.25 0.00 0.00 0.00 0.0012:16:50 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00^C
Use this tool to check the throughput of the network interface: rxkB/s and txkB/s, as a measure of the workload, and of course, check if some throttling is triggered. In the example above, eth0 receives 22Mbytes/s, which is 176Mbits/sec (much lower than the limit of e.g. 1Gbit/sec).
This version is also %ifutil used for device utilization (full duplex maximum in both directions), which is something we measure using Brendan's nicstat tool. As with nicstat, this is hard to get right, and doesn't seem to work (0.00) in this example.
9、sar -n TCP,ETCP 1
$ sar -n TCP,ETCP 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)12:17:19 AM active/s passive/s iseg/s oseg/s12:17:20 AM 1.00 0.00 10233.00 18846.0012:17:19 AM atmptf/s estres/s retrans/s isegerr/s orsts/s12:17:20 AM 0.00 0.00 0.00 0.00 0.0012:17:20 AM active/s passive/s iseg/s oseg/s12:17:21 AM 1.00 0.00 8359.00 6039.0012:17:20 AM atmptf/s estres/s retrans/s isegerr/s orsts/s12:17:21 AM 0.00 0.00 0.00 0.00 0.00^C
This is a summary view of some key TCP metrics. These indicators include:
-
active/s : the number of locally initiated TCP connections per second (e.g. via connect())
-
passive/s : The number of remotely initiated TCP connections per second (e.g. via accept())
-
retrans/s : the number of TCP retransmissions per second
Active and passive counts are often used as a rough measure of server load: the number of newly accepted connections (passive), and the number of downstream connections (active). For the time being, you can think of active as outbound and passive as inbound, but this is not strictly correct (for example, sometimes localhost to localhost).
Retransmissions are a sign of a network or server problem; it could be an unreliable network (such as the public internet), or it could be an overloaded server dropping packets. The above example shows only one new TCP connection per second.
10、top
$ toptop - 00:15:40 up 21:56, 1 user, load average: 31.09, 29.87, 29.92Tasks: 871 total, 1 running, 868 sleeping, 0 stopped, 2 zombie%Cpu(s): 96.8 us, 0.4 sy, 0.0 ni, 2.7 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 stKiB Mem: 25190241+total, 24921688 used, 22698073+free, 60448 buffersKiB Swap: 0 total, 0 used, 0 free. 554208 cached MemPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND20248 root 20 0 0.227t 0.012t 18748 S 3090 5.2 29812:58 java4213 root 20 0 2722544 64640 44232 S 23.5 0.0 233:35.37 mesos-slave66128 titancl+ 20 0 24344 2332 1172 R 1.0 0.0 0:00.07 top5235 root 20 0 38.227g 547004 49996 S 0.7 0.2 2:02.74 java4299 root 20 0 20.015g 2.682g 16836 S 0.3 1.1 33:14.42 java1 root 20 0 33620 2920 1496 S 0.0 0.0 0:03.82 init2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd3 root 20 0 0 0 0 S 0.0 0.0 0:05.35 ksoftirqd/05 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H6 root 20 0 0 0 0 S 0.0 0.0 0:06.94 kworker/u256:08 root 20 0 0 0 0 S 0.0 0.0 2:38.05 rcu_sched
The top command includes many of the metrics we examined earlier. It is good for seeing how the workload has changed compared to the output of previous commands.
One downside of the top command is that it can be difficult to see patterns over time, whereas tools like vmstat and pidstat that provide scrolling output may be clearer. If you don't pause the output quickly enough (Ctrl-S to pause, Ctrl-Q to continue), the screen will clear and evidence of intermittent problems will be lost.
follow-up analysis
There are many more commands and methods that you can apply to dig deeper. Check out Brendan's Linux performance tools tutorial at Velocity 2015, which covers observability, benchmarking, tuning, static performance tuning, profiling, and tracing through more than 40 commands.
Source: This article is transferred from the official account SRETalk.