Linux intrusion troubleshooting
Preface
When an enterprise has a hacking, system crash or other security incident that affects the normal operation of the business, it is urgent to deal with it as soon as possible, so that the enterprise’s network information system can resume normal work in the shortest possible time, further find the source of the intrusion, and restore the process of the intrusion accident. At the same time, solutions and preventive measures are given to recover or reduce economic losses for the enterprise.
In view of common attack events, combined with the analysis and resolution of emergency response incidents in the work, some ideas for the investigation of Linux server intrusion are summarized.
1. Intrusion investigation ideas
Account security
Basic use:
1、用户信息文件 /etc/passwd
root:x:0:0:root:/root:/bin/bash
account:password:UID:GID:GECOS:directory:shell
用户名：密码：用户ID：组ID：用户说明：家目录：登陆之后的 shell
注意：无密码只允许本机登陆，远程不允许登陆
2、影子文件 /etc/shadow
root:$6$oGs1PqhL2p3ZetrE$X7o7bzoouHQVSEmSgsYN5UD4.kMHx6qgbTqwNVC5oOAouXvcjQSt.Ft7ql1WpkopY0UV9ajBwUt1DpYxTCVvI/:16809:0:99999:7:::
用户名：加密密码：密码最后一次修改日期：两次密码的修改时间间隔：密码有效期：密码修改到期到的警告天数：密码过期之后的宽限天数：账号失效时间：保留
who 查看当前登录用户（tty 本地登陆 pts 远程登录）
w 查看系统信息，想知道某一时刻用户的行为
uptime 查看登陆多久、多少用户，负载状态
Intrusion investigation:
1、查询特权用户特权用户(uid 为0)
[root@localhost ~]# awk -F: '$3==0{print $1}' /etc/passwd
2、查询可以远程登录的帐号信息
[root@localhost ~]# awk '/\$1|\$6/{print $1}' /etc/shadow
3、除root帐号外，其他帐号是否存在sudo权限。如非管理需要，普通帐号应删除sudo权限
[root@localhost ~]# more /etc/sudoers | grep -v "^#\|^$" | grep "ALL=(ALL)"
4、禁用或删除多余及可疑的帐号
usermod -L user 禁用帐号，帐号无法登录，/etc/shadow 第二栏为 ! 开头
userdel user 删除 user 用户
userdel -r user 将删除 user 用户，并且将 /home 目录下的 user 目录一并删除
History command
Basic use:
View the system commands executed by the account through the .bash_history file
1、root 用户的历史命令
histroy
2、打开 /home 各帐号目录下的 .bash_history，查看普通帐号执行的历史命令。
为历史的命令增加登录的 IP 地址、执行命令时间等信息：
1）保存1万条命令
sed -i 's/^HISTSIZE=1000/HISTSIZE=10000/g' /etc/profile
2）在/etc/profile的文件尾部添加如下行数配置信息：
######jiagu history xianshi#########
USER_IP=`who -u am i 2>/dev/null | awk '{print $NF}' | sed -e 's/[()]//g'`
if [ "$USER_IP" = "" ]
then
USER_IP=`hostname`
fi
export HISTTIMEFORMAT="%F %T $USER_IP `whoami` "
shopt -s histappend
export PROMPT_COMMAND="history -a"
######### jiagu history xianshi ##########
3）source /etc/profile 让配置生效
生成效果： 1 2018-07-10 19:45:39 192.168.204.1 root source /etc/profile
3、历史操作命令的清除：history -c
但此命令并不会清除保存在文件中的记录，因此需要手动删除 .bash_profile 文件中的记录。
Intrusion investigation:
进入用户目录下，导出历史命令。
cat .bash_history >> history.txt
Check abnormal port
Use netstat network connection command to analyze suspicious port, IP, PID
netstat -antlp | more
查看下 pid 所对应的进程文件路径，
运行 ls -l /proc/$PID/exe 或 file /proc/$PID/exe（$PID 为对应的 pid 号）
Check for abnormal processes
Use the ps command to analyze the process
ps aux | grep pid
Check startup items
Basic use:
Schematic diagram of system operation level:
| Run level | meaning |
|---|---|
| 0 | Shutdown |
| 1 | Single user mode can be imagined as the safe mode of windows, mainly used for system repair |
| 2 | Incomplete command line mode, without NFS service |
| 3 | Complete command line mode, is the standard character interface |
| 4 | System reservation |
| 5 | Graphics mode |
| 6 | Restart |
View run level commands runlevel
System default permission level
vi /etc/inittab
id=3：initdefault #系统开机后直接进入哪个运行级别
Boot configuration file
/etc/rc.local
/etc/rc.d/rc[0~6].d
Example: When we need to boot our own script, we only need to drop the executable script in the /etc/init.d directory, and then create a soft link in the /etc/rc.d/rc*.d file.
Note: The * here represents the seven levels of 0,1,2,3,4,5,6
root@localhost ~]# ln -s /etc/init.d/sshd /etc/rc.d/rc3.d/S100ssh
Here sshd is the script file of a specific service, S100ssh is its soft link, and the beginning of S represents auto-start when loading; if it is a script file starting with K, it represents the need to be closed when the run level is loaded.
Intrusion investigation:
Startup file:
more /etc/rc.local
/etc/rc.d/rc[0~6].d
ls -l /etc/rc.d/rc3.d/
Check scheduled tasks
Basic use
1. Use crontab to create scheduled tasks
-
Basic command
crontab -l 列出某个用户cron服务的详细内容 Tips：默认编写的crontab文件会保存在 (/var/spool/cron/用户名 例如: /var/spool/cron/root crontab -r 删除每个用户cront任务(谨慎：删除所有的计划任务) crontab -e 使用编辑器编辑当前的crontab文件 如：*/1 * * * * echo "hello world" >> /tmp/test.txt 每分钟写入文件
2. Use the anacron command to achieve asynchronous timing task scheduling
-
Use Cases
每天运行 /home/backup.sh 脚本： vi /etc/anacrontab @daily 10 example.daily /bin/bash /home/backup.sh 当机器在 backup.sh 期望被运行时是关机的，anacron会在机器开机十分钟之后运行它，而不用再等待 7天。
Intrusion investigation
Focus on whether there are malicious scripts in the following directories
/var/spool/cron/*
/etc/crontab
/etc/cron.d/*
/etc/cron.daily/*
/etc/cron.hourly/*
/etc/cron.monthly/*
/etc/cron.weekly/
/etc/anacrontab
/var/spool/anacron/*
Tips:
more /etc/cron.daily/* 查看目录下所有文件
Check service
Service starts automatically
The first modification method:
chkconfig [--level 运行级别] [独立服务名] [on|off]
chkconfig –level 2345 httpd on 开启自启动
chkconfig httpd on （默认level是2345）
The second modification method:
修改 /etc/re.d/rc.local 文件
加入 /etc/init.d/httpd start
The third modification method:
Use the ntsysv command to manage auto-start, you can manage independent services and xinetd services.
Intrusion investigation
1. Query installed services:
RPM package installation service
chkconfig --list 查看服务自启动状态，可以看到所有的RPM包安装的服务
ps aux | grep crond 查看当前服务
系统在3与5级别下的启动项
中文环境
chkconfig --list | grep "3:启用\|5:启用"
英文环境
chkconfig --list | grep "3:on\|5:on"
Source code package installation service
查看服务安装位置 ，一般是在/user/local/
service httpd start
搜索/etc/rc.d/init.d/ 查看是否存在
Check abnormal files
1. View sensitive directories, such as files in the /tmp directory, and pay attention to hidden folders. Folders named "..." have hidden attributes
2. Get the creation time of the discovery of WEBSHELL and remote control Trojans, how to find out the files created in the same time range?
You can use the find command to search, such as find /opt -iname “*” -atime 1 -type f to find out the files visited by /opt the day before
3. For suspicious files, you can use stat to create and modify the time.
Check the system log
The default storage location of logs: /var/log/
View log configuration: more /etc/rsyslog.conf
| Log file | Description |
|---|---|
| / var / log / cron | Logs related to system timing tasks are recorded |
| / var / log / cups | Log of printing information |
| / var / log / dmesg | Record the kernel self-check information when the system is turned on, you can also use the dmesg command to directly view the kernel self-check information |
| / var / log / mailog | Record mail information |
| / var / log / message | A log that records important information about the system. This log file will record most of the important information of the Linux system. If there is a problem with the system, the first thing to check should be this log file. |
| / var / log / btmp | Record the error login log. This file is a binary file and cannot be viewed directly with vi. Instead, use the lastb command to view |
| / var / log / lastlog | Record the log of the last login time of all users in the system. This file is a binary file, not directly vi, but use the lastlog command to view |
| / var / log / wtmp | Permanently record the login and logout information of all users, as well as the system startup, restart, and shutdown events. The same file is also a binary file, you cannot directly vi, but you need to use the last command to view |
| / var / log / utmp | Record the information of the currently logged-in user. This file will continue to change with the user's login and logout, and only records the information of the currently logged-in user. Similarly, this file cannot be directly vi, but use w, who, users and other commands to query |
| / var / log / secure | Record authentication and authorization information, as long as the program involving account and password will be recorded, such as SSH login, su switch user, sudo authorization, even adding users and modifying user passwords will be recorded in this log file |
Log analysis skills:
1、定位有多少IP在爆破主机的root帐号：
grep "Failed password for root" /var/log/secure | awk '{print $11}' | sort | uniq -c | sort -nr | more
定位有哪些IP在爆破：
grep "Failed password" /var/log/secure|grep -E -o "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"|uniq -c
爆破用户名字典是什么？
grep "Failed password" /var/log/secure|perl -e 'while($_=<>){ /for(.*?) from/; print "$1\n";}'|uniq -c|sort -nr
2、登录成功的IP有哪些：
grep "Accepted " /var/log/secure | awk '{print $11}' | sort | uniq -c | sort -nr | more
登录成功的日期、用户名、IP：
grep "Accepted " /var/log/secure | awk '{print $1,$2,$3,$9,$11}'
3、增加一个用户kali日志：
Jul 10 00:12:15 localhost useradd[2382]: new group: name=kali, GID=1001
Jul 10 00:12:15 localhost useradd[2382]: new user: name=kali, UID=1001, GID=1001, home=/home/kali
, shell=/bin/bash
Jul 10 00:12:58 localhost passwd: pam_unix(passwd:chauthtok): password changed for kali
#grep "useradd" /var/log/secure
4、删除用户kali日志：
Jul 10 00:14:17 localhost userdel[2393]: delete user 'kali'
Jul 10 00:14:17 localhost userdel[2393]: removed group 'kali' owned by 'kali'
Jul 10 00:14:17 localhost userdel[2393]: removed shadow group 'kali' owned by 'kali'
# grep "userdel" /var/log/secure
5、su切换用户：
Jul 10 00:38:13 localhost su: pam_unix(su-l:session): session opened for user good by root(uid=0)
sudo授权执行:
sudo -l
Jul 10 00:43:09 localhost sudo: good : TTY=pts/4 ; PWD=/home/good ; USER=root ; COMMAND=/sbin/shutdown -r now
2. Tools
Rootkit killing
-
chkrootkit
Website: http://www.chkrootkit.org
使用方法： wget ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz tar zxvf chkrootkit.tar.gz cd chkrootkit-0.52 make sense #编译完成没有报错的话执行检查 ./chkrootkit -
rkhunter
Website: http://rkhunter.sourceforge.net
使用方法： Wget https://nchc.dl.sourceforge.net/project/rkhunter/rkhunter/1.4.4/rkhunter-1.4.4.tar.gz tar -zxvf rkhunter-1.4.4.tar.gz cd rkhunter-1.4.4 ./installer.sh --install rkhunter -c
Virus killing
-
Clamav
Website: http://www.clamav.net/download.html
Installation method one:
1、安装 zlib： wget http://nchc.dl.sourceforge.net/project/libpng/zlib/1.2.7/zlib-1.2.7.tar.gz tar -zxvf zlib-1.2.7.tar.gz cd zlib-1.2.7 #安装一下gcc编译环境： yum install gcc CFLAGS="-O3 -fPIC" ./configure --prefix= /usr/local/zlib/ make && make install 2、添加用户组 clamav 和组成员 clamav： groupadd clamav useradd -g clamav -s /bin/false -c "Clam AntiVirus" clamav 3、安装 Clamav tar –zxvf clamav-0.97.6.tar.gz cd clamav-0.97.6 ./configure --prefix=/opt/clamav --disable-clamav -with-zlib=/usr/local/zlib make make install 4、配置 Clamav mkdir /opt/clamav/logs mkdir /opt/clamav/updata touch /opt/clamav/logs/freshclam.log touch /opt/clamav/logs/clamd.log cd /opt/clamav/logs chown clamav:clamav clamd.log chown clamav:clamav freshclam.log 5、ClamAV 使用： /opt/clamav/bin/freshclam 升级病毒库 ./clamscan –h 查看相应的帮助信息 ./clamscan -r /home 扫描所有用户的主目录就使用 ./clamscan -r --bell -i /bin 扫描bin目录并且显示有问题的文件的扫描结果Installation method two:
#安装 yum install -y clamav #更新病毒库 freshclam #扫描方法 clamscan -r /etc --max-dir-recursion=5 -l /root/etcclamav.log clamscan -r /bin --max-dir-recursion=5 -l /root/binclamav.log clamscan -r /usr --max-dir-recursion=5 -l /root/usrclamav.log #扫描并杀毒 clamscan -r --remove /usr/bin/bsd-port clamscan -r --remove /usr/bin/ clamscan -r --remove /usr/local/zabbix/sbin #查看日志发现 cat /root/usrclamav.log |grep FOUND
webshell killing
Linux version:
河马 WebShell 查杀：http://www.shellpub.com
RPM check
System integrity can be verified through the -Va that comes with rpm. Check all rpm software packages to see which commands have been replaced:
./rpm -Va > rpm.log
If everything is checked properly, no output will be generated. If there are inconsistencies, they will be displayed. The output format is an 8-bit long string, and each character is used to represent a comparison between the file and an attribute in the RPM database. The result, if it is. (dot), it means the test passed.
验证内容中的8个信息的具体内容如下：
S 文件大小是否改变
M 文件的类型或文件的权限（rwx）是否被改变
5 文件MD5校验是否改变（可以看成文件内容是否改变）
D 设备中，从代码是否改变
L 文件路径是否改变
U 文件的属主（所有者）是否改变
G 文件的属组是否改变
T 文件的修改时间是否改变
If the command is replaced, if you restore it back:
文件提取还原案例：
rpm -qf /bin/ls 查询 ls 命令属于哪个软件包
mv /bin/ls /tmp 先把 ls 转移到 tmp 目录下，造成 ls 命令丢失的假象
rpm2cpio /mnt/cdrom/Packages/coreutils-8.4-19.el6.i686.rpm | cpio -idv ./bin/ls 提取 rpm 包中 ls 命令到当前目录的 /bin/ls 下
cp /root/bin/ls /bin/ 把 ls 命令复制到 /bin/ 目录 修复文件丢失
Linux security check script
Github project address:
https://github.com/grayddq/GScan
https://github.com/ppabc/security_check
https://github.com/T0xst/linux
