Security failure causes high CPU processing problems

Unbridled spread of the epidemic in the world, but in my country has been effectively controlled Greater China, which not only reflects the comprehensive strength of a country is also reflected in our Greater China hundreds of millions of popular unity can overcome again and again to live in the outside world can not seem to fight all kinds of the power of the country lies in the people of God of War, the great unity of power, and we as the IT industry, operation and maintenance technicians, not only to have their own real good technology, but also the quality of the project team members have a sense of operation and maintenance, in order to do a good job with the line operation and maintenance and technical support operations under the on-line system, good non-functional, such as the prevention and control of many functional services, ensure service high availability, high reliability, high maintenance, high stability, high security and other business enterprises to ensure operating normally.
Security failure causes high CPU processing problems
Otherwise, when members uploaded to the server code works or installation package to deploy a service control will have inadvertently uploaded a *** of the code, invasive procedures, the system will lead to paralysis, similar to us during this epidemic, we own wearing masks , strong body, and out of the cell to do a variety of security, passes, body temperature, spray anti-virus community and other, similar to our server set up a firewall, configure the access ports, good security code for each scan a document, regularly check performance testing and other types of non-functional indicators such as inspection and testing to ensure proper server operation, but sometimes overlooked one important, cause problems, for example, this time we have a test server application stage appeared high CPU, and can not log normal status quo, the specific reasons as follows:

Problem Cause:
February 28 afternoon near the six, the developers made a sudden I said, 187 shots to the server can not log in and asked if I changed the password, as Figure 1:
Security failure causes high CPU processing problems
(Figure I)
think this time in addition to the installation experience test operation and maintenance monitoring tool is useful to 187 services, but also 10 days before the thing, but not to modify your password, so I'm also curious to see the next login, found that indeed there is a problem, re-enter the password does not work, the following Figure 2:
Security failure causes high CPU processing problems
( Figure II)

We found 187 does not log, but the message indicates that the server is not shut down, but ssh link was changed string, then the first reaction forehead, *** were using an executable SSH back door, and these components to to provide installation services in the form of malicious software resides.
Just out of curiosity and availability deploy monitoring tools, operation and maintenance services I log monitoring program, found also collects information resources such as CPU 187 service, as Figure III, but CPU utilization is high, you should be using what malware provide for his own service, but also shows that 187 service is still available, but the new ssh connection can not be linked,

![](https://s1.51cto.com/images/blog/202003/01/ad704e19df4c78c0dfb31ff67afbb7b4.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=)

, (Figure 3)
Fortunately, there is a lazy before sexual habits, have opened a CRT in another computer, run out rarely go off, just have to open 187 such as the server did not pass, it can also directly access, service discovery is leading to tsm high CPU, cron memory usage is high, the next question that developers do not have service, found no use to kill first.
Security failure causes high CPU processing problems

于是就直接先kill掉,然后修改了下系统登录密码,但是还是要把问题追究到底,发现kill该进程后发现CPU立马掉下来,如下图四
Security failure causes high CPU processing problems
(图四)
通过查证:tsm64是负责通过SSH暴力破解传播挖矿机和后门的扫描器,可以发送远程命令来下载和执行恶意软件。
看了下该进程对应的服务,安装路径配置路径如下:
root 31803 31798 84 07:44 ? 08:36:57 /tmp/.X19-unix/.rsync/c/lib/64/tsm --library-path /tmp/.X19-unix/.rsync/c/lib/64/ /usr/sbin/httpd rsync/c/tsm64 -t 505 -f 1 -s 12 -S 8 -p 0 -d 1 p ip
发现该服务应该只是一个shell服务,而且看了下远程监控收集的记录,发现是2月27日凌晨四点多的时候被侵入,植入病毒,导致CPU使用率高,也导致我们CRT无法正常登录,如下图五和图六:
Security failure causes high CPU processing problems
(图五)
Security failure causes high CPU processing problems
(图六)
分析下应该是有开启服务进程,才会导致CPU和内存偏高,而引起内存偏高的是cron进程,于是通过crontab -e发现确实被开启了进程服务,如下图七
Security failure causes high CPU processing problems
(图七)
接下来直截了当,停止服务,然后删除对应路径下文件和定时作业,继续观察两天,如下图8发现确实没有在复现问题。
(图八)
Security failure causes high CPU processing problems
图九
Security failure causes high CPU processing problems

Summary:
Although this problem ten minutes total time from discovery to solve, but it is purely down to luck, quick fix, also illustrates the multidimensional non-functional service troubleshooting, operation and maintenance technicians divergent thinking technology and comprehensive knowledge, mainly still have to combat is king, this problem is mainly due to:
a: server password is set too simple lead to take advantage of, the main reason.
Two: server security settings is not perfect
three: the project team to upload the document does not conduct rigorous audits lead to upload files with a virus which leads to problems caused by this.
Four: server system user login privileges is not perfect;
five: do not panic encounter problems, to meditate, to be calm;

Guess you like

Origin blog.51cto.com/372550/2474677