In the previous article on common Java troubleshooting methods , the troubleshooting method for cpu iowait was not written. The main reason is that I have not encountered any high cpu iowait cases before. Unfortunately, I have encountered two consecutive cases in the past week. Starting the case of cpu iowait, I have learned a lot of system-level knowledge through these two cases. Maybe this knowledge is nothing to those who are familiar with the system, but I think it is worth sharing for the students who write Java (because Java basically does not use it). For storage type scenarios, generally speaking, the chance of encountering high iowait will be much lower than other types of problems).
When iowait is high, the most important thing is to find out which process is consuming io first, and solve the problem at the fastest speed, but some default tools of linux such as top, iostat, etc. can only see the consumption of io , but it doesn't correspond to which process is consuming. The better tool for positioning is iotop, but it may not be easy to install in some environments. After installing it, execute iotop directly, and you can see which process it is. It consumes a lot of io. In terms of solving the problem, it is usually solved by killing the process after finding the process (there is another way to see the process that consumes io for a period of time by opening syslog and blk_dump, but in my two I have tried it in one case and the effect is not ideal).
But generally speaking, the above method can only barely solve the problem, but it still does not locate the problem in the program. It may even be restarted and the iowait is still high, so other tools are needed for further investigation. Fortunately, the system level is With such a tool, blktrace/debugfs can be used to locate which file (or which) files are read or written causing high iowait (this method is mainly learned from a blog by Bo Yu of Alibaba Group Kernel Group ).
After installing blktrace, first mount -t debugfs none /sys/kernel/debug, and then you can use iostat to see which device is consuming io. For example, if you see that sda is consuming, then you can execute blktrace /dev /sda, some sda.blktrace.* files will be automatically generated in the executed directory during execution. When you feel that the collection is almost the same, you can stop it with ctrl+c.
Then execute blkparse sda.blktrace.* > result.log, regenerate result.log and execute grep 'A' result.log | head -n 5 to see that in the process of collection, the place that consumes more io is in Which, for example, in the case I encountered after execution, I saw:
8,0 11 1 0.000071219 248366 AR 218990140 + 16 <- (8, 9) 148994872 8,0 11 1 0.000071219 248366 Ar 218990140 + 16 <- (8, 9) 148994872 8,011 0.000071219 248366 AR 218990140 + 16 < - (8,9) 148994872 8,0 11 1 0.000071219 248366 AR 218990140 + 16 <- (8,9) 148994872 8,0 11 1 0.000071219 248366 AR 218990140 + 16 <- (94 behind 48994 here) In the end, it means reading (WS if it is writing), which is the same as what iostat saw before. io is mainly caused by a large number of reads. Through the above information 8,0 and (8,9), it can be seen that the main consumption is in sda9 (this can also be seen through iostat), and 148994872 after (8,9) represents the read sector number. Then you can find the block size of sda9 through debugfs -R 'stats' /dev/sda9 | grep 'Block size'. In the case I encountered, the block size is 4096, which is usually the default for ext2/ext3 file systems. The size of each sector is 512, 148994872/(4096/512) = 18624359 to find the corresponding block number on the file system, and then through debugfs -R 'icheck 18624359' /dev/sda9 to find the corresponding inode, in my In the case encountered, the result after execution is: Block Inode number 18624359 18284971 and debugfs also provides the function of finding specific files through inode number, so it is easy to do, continue to execute debugfs -R ' ncheck 18284971', after execution, you will see information similar to the following: Inode Pathname 18284971 [corresponding file name] After finding the file name, it is easy to do, you can combine the previous iotop or lsof to find the corresponding process id, and then You can see how to avoid a lot of reading this file from the code level. In addition to the above case, iowait in some cases is actually relatively simple, such as reading and writing huge files (usually may occur when a large number of exceptions occur)... In solving the two cases with high cpu iowait encountered last week, one of them was caused by the above business code, but the other was related to the raid card configuration, because from the perspective of iostat, the amount of writing at that time was not very large. It's big, but the iowait is relatively high. After asking the system people for help, they told me that it was because of the problem of the RAID card writing strategy configuration. I didn't understand the configuration of the RAID card before. There will be a raid card on the server, and the current raid card basically has a cache. In order to ensure the security of the data in the cache, the raid card usually has a battery or a capacitor, and the failure rate of the capacitor is relatively low. , raid card will provide the configuration of write strategy. The write strategy is usually Write Back and Write Through. Wirte Back means that the write is successful after writing to the cache, and Write Through means that writing to the disk is successful. Usually, the raid card The write strategy will be divided into two configurations: normal, and when there is a problem with the battery or capacitor. In the case encountered, it is configured to use Write Through when there is a problem with the battery/capacitor. At that time, the battery of the Raid card of the machine was configured. There is a fault, so the strategy is switched to Write Through, and the iops that can be supported is naturally greatly reduced. In our scenario, it does not matter if the local data is lost, so the Raid card strategy can be configured even if the battery fails Also still uses Write Back. Usually various Raid cards will provide tools to configure the write strategy, such as HP card hpacucli, you can view the information of the hard disk and the Raid card through cat /proc/scsi/scsi (you can use cat /proc/mdstat to view the raid information first), It is helpful to confirm the cache/cache capacity/battery of the raid card and the iops that the hard disk itself can support. Because it is recommended that when encountering a scene with high iowait, you can first look at the write strategy of the raid card. If there is no problem, then locate the specific root cause through iotop, blktrace, debugfs, lsof, etc. It can be seen that even for Java students, when troubleshooting problems, they still need to have some knowledge of various system tools and hardware levels. Because it is recommended that when encountering a scene with high iowait, you can first look at the write strategy of the raid card. If there is no problem, then locate the specific root cause through iotop, blktrace, debugfs, lsof, etc. It can be seen that even for Java students, when troubleshooting problems, they still need to have some knowledge of various system tools and hardware levels. Because it is recommended that when encountering a scene with high iowait, you can first look at the write strategy of the raid card. If there is no problem, then locate the specific root cause through iotop, blktrace, debugfs, lsof, etc. It can be seen that even for Java students, when troubleshooting problems, they still need to have some knowledge of various system tools and hardware levels.