Have you ever experienced: Early in the morning it was crazy bomb alarm wake up, due to high online application CPU usage ......
Have you ever experienced: arrived in the company, the bench has not warm it up, but inbox bunch of customer service complaints e-mail, you need to troubleshoot problems logging locate ......
Have you ever experienced: from work bell will soon ring, but you have to work overtime, statistics per application, peak various indicators such as per minute, due to the BOSS bring these indexes, foreign bragging Technology Conference on tomorrow . .....
Have you ever experienced: the night is refreshing sleep, operation and maintenance on duty called you crazy CALL, because the online application memory there is a problem ......
I'm sure the above scenario, a high probability that you have experienced. Speak truth, in fact, whether you are experienced, you have to count them today with friends. Because I will be combined with previous experience summary, open the altar managed to ape the door, pulled out a holy canon cast or two.
Audio Bible in a day, Rome was not built in a day, Rome was not built in a day, experience is not a pit and contributed to the anti-wolf have surgery, we start with the overall situation, take a look at this technology Collection ( not see it does not matter, feel very cattle breaking the line).
This book is really hard-core technology, is divided into four major strokes, adopted piecemeal, let's dismantling one by one.
The first one trick: online application CPU occupancy too high.
~ Demolition recruit:
采用 top 命令，找出 CPU 占用最高的进程 PID；
通过 ps -ef | grep PID 查看对应的应用，看看是谁在作祟；
采用 jstack -l PID >> PID.log 获取进程的堆栈信息；
采用 ps -mp PID -o THREAD,tid,time 拿到占用 CPU 最高的线程 tid；
采用 printf "%x\n" tid 获取 16 进制的线程 TID；
采用 grep TID -A20 PID.log 确定是线程哪儿出了问题。Finally ~: leg pain Medical legs Bianzheng differentiation, remedy. Identify the position of the code, the code is adjusted.
The first 2 strokes: online application memory overflow.
~ Demolition recruit:
采用 top 命令，找出应用对应的 PID；
采用 jmap -heap PID 确认一下分配的内存少不少；
采用 jmap -histo:live PID | more 找出分析最耗内存的对象【留意占用多少G的对象】；
采用 ps -efL | grep PID | wc -l 查看进程创建的线程数；
采用 ll /proc/PID/task | wc -l 也可以查看进程创建的线程数；
采用 netstat -apn | grep PID | wc -l 查看进程网络连接数。Finally ~: leg pain Medical legs Bianzheng differentiation, remedy.
a. 如果内存分配确实小，适当调整内存；
b. 对象被频繁创建，且不释放，优化代码；
c. 不断创建线程或者不断进行网络连接，优化代码。
The first 3 strokes: troubleshooting business problems.
~ Demolition recruit:
采用 tail -fn 200 log_file 实时查询线上日志；
找准日志搜所关键字keyWord，例如 orderId、mobileId、reqId 等；
采用 grep keyWord log_file 查询关键字所在的行的日志；
采用 grep -C n keyWord log_file 匹配关键字所在行的上下 n 行；
采用 grep keyWord log_file | wc -l 匹配关键字的的行数有多少。Finally ~: be the largest log search tail, grep with the actual scene investigation log.
The first 4 strokes: statistical problems of BOSS.
~ Demolition recruit:
Using cat log_file read log files; using cut command log timestamp taken out; if taken according to the second statistics seconds; if taken according to the statistics minutes min; using statistical weight to be uniq -c; employed in accordance with a first sort -nr the numerical size of the column is reverse; using head -1 displays only the first line.
Finally ~: statistical problems solved, then the statistical peak per second command how to write it?
Such as log:
1118 115856 066 - REQID0000000000188 ... ...
Command combinations:
SecondPeak=`cat log_file|cut -d, -f1| cut -c 1-11|sort|uniq -c |sort -nr|head -1`
Well, today's share coming to an end, do not know how much you get, understand and do not understand, recommend your collection for a rainy day.
