0x00 Preface
The Linux system has a very flexible and powerful log function, which can save almost all operation records and retrieve the information we need from it. This article briefly introduces Linux system logs and log analysis techniques.
0x01 Log Introduction
Log default storage location: /var/log/
Check the log configuration: more /etc/rsyslog.conf
| log file | illustrate |
|---|---|
| /var/log/cron | Logs related to system timing tasks are recorded |
| /var/log/cups | Logs that record print information |
| /var/log/dmesg | It records the kernel self-test information when the system is turned on, and you can also use the dmesg command to directly view the kernel self-test information |
| /var/log/maillog | Record email information |
| /var/log/message | A log that records important system information. This log file will record most of the important information of the Linux system. If there is a problem with the system, the first thing to check should be this log file |
| /var/log/btmp | Record the error login log. This file is a binary file and cannot be viewed directly with vi, but you need to use the lastb command to view it |
| /var/log/loadlog | Record the log of the last login time of all users in the system. This file is a binary file and cannot be directly vi. Instead, use the lastlog command to view it |
| /var/log/wtmp | Permanently record the login and logout information of all users, and record the system startup, restart and shutdown events at the same time. Also this file is also a binary file, you cannot directly vi, but you need to use the last command to view |
| /var/log/utmp | Record the information of the currently logged-in user. This file will change continuously with the user's login and logout, and only record the information of the currently logged-in user. Similarly, this file cannot be directly vi, but use w, who, users and other commands to query |
| /var/log/secure | Record verification and authorization information, as long as the account and password are involved in the program will be recorded, such as SSH login, su switch user, sudo authorization, even adding users and modifying user passwords will be recorded in this log file |
Several important logs: Login failure record: /var/log/btmp //lastb Last login: /var/log/lastlog //lastlog Successful login record: /var/log/wtmp //last login log record: /var/log/secure
Current login user information: /var/run/utmp //w, who, users
History command record: history only cleans up the current user: history -c
0x02 Log analysis skills
A. Commonly used shell commands
Commonly used shell commands under Linux such as: find, grep, egrep, awk, sed
Tips:
1. grep displays several lines of information before and after:
标准unix/linux下的grep通过下面參数控制上下文：
grep -C 5 foo file 显示file文件里匹配foo字串那行以及上下5行
grep -B 5 foo file 显示foo及前5行
grep -A 5 foo file 显示foo及后5行
查看grep版本号的方法是
grep -V
2. grep finds all files containing a certain string
grep -rn "hello,world!"
* : 表示当前目录所有文件，也可以是某个文件名
-r 是递归查找
-n 是显示行号
-R 查找所有文件包含子目录
-i 忽略大小写
3. How to display certain lines of a file:
cat input_file | tail -n +1000 | head -n 2000
#从第1000行开始，显示2000行。即显示1000~2999行
4、find /etc -name init
//在目录/etc中查找文件init
5. Just display the account of /etc/passwd
`cat /etc/passwd |awk -F ':' '{print $1}'`
//awk -F指定域分隔符为':'，将记录按指定的域分隔符划分域，填充域，$0则表示所有域,$1表示第一个域,$n表示第n个域。
6、sed -i ‘153,$d’ .bash_history
删除历史操作记录，只保留前153行
B. Log analysis skills
/var/log/secure
1、定位有多少IP在爆破主机的root帐号：
grep "Failed password for root" /var/log/secure | awk '{print $11}' | sort | uniq -c | sort -nr | more
定位有哪些IP在爆破：
grep "Failed password" /var/log/secure|grep -E -o "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"|uniq -c
爆破用户名字典是什么？
grep "Failed password" /var/log/secure|perl -e 'while($_=<>){ /for(.*?) from/; print "$1\n";}'|uniq -c|sort -nr
2、登录成功的IP有哪些：
grep "Accepted " /var/log/secure | awk '{print $11}' | sort | uniq -c | sort -nr | more
登录成功的日期、用户名、IP：
grep "Accepted " /var/log/secure | awk '{print $1,$2,$3,$9,$11}'
3、增加一个用户kali日志：
Jul 10 00:12:15 localhost useradd[2382]: new group: name=kali, GID=1001
Jul 10 00:12:15 localhost useradd[2382]: new user: name=kali, UID=1001, GID=1001, home=/home/kali
, shell=/bin/bash
Jul 10 00:12:58 localhost passwd: pam_unix(passwd:chauthtok): password changed for kali
#grep "useradd" /var/log/secure
4、删除用户kali日志：
Jul 10 00:14:17 localhost userdel[2393]: delete user 'kali'
Jul 10 00:14:17 localhost userdel[2393]: removed group 'kali' owned by 'kali'
Jul 10 00:14:17 localhost userdel[2393]: removed shadow group 'kali' owned by 'kali'
# grep "userdel" /var/log/secure
5、su切换用户：
Jul 10 00:38:13 localhost su: pam_unix(su-l:session): session opened for user good by root(uid=0)
sudo授权执行:
sudo -l
Jul 10 00:43:09 localhost sudo: good : TTY=pts/4 ; PWD=/home/good ; USER=root ; COMMAND=/sbin/shutdown -r now
/var/log/yum.log
Software installation, upgrade and uninstallation logs:
\~~~yum install gcc yum install gcc
[root@bogon ~]# more /var/log/yum.log
Jul 10 00:18:23 Updated: cpp-4.8.5-28.el7_5.1.x86_64 Jul 10 00:18:24 Updated: libgcc-4.8.5-28.el7_5.1.x86_64 Jul 10 00:18:24 Updated: libgomp-4.8.5-28.el7_5.1.x86_64 Jul 10 00:18:28 Updated: gcc-4.8.5-28.el7_5.1.x86_64 Jul 10 00:18:28 Updated: libgcc-4.8.5-28.el7_5.1.i686 ~~~
Check how many IP accesses:
awk '{print $1}' log_file|sort|uniq|wc -l
To see how many times a page has been visited:
grep "/index.php" log_file | wc -l
Check how many pages are visited by each IP:
awk '{++S[$1]} END {for (a in S) print a,S[a]}' log_file > log.txt
sort -n -t ' ' -k 2 log.txt 配合sort进一步排序
Sort the number of pages accessed by each IP from small to large:
awk '{++S[$1]} END {for (a in S) print S[a],a}' log_file | sort -n
Check which pages are visited by a certain IP:
grep ^111.111.111.111 log_file| awk '{print $1,$7}'
Pages that remove search engine statistics:
awk '{print $12,$1}' log_file | grep ^\"Mozilla | awk '{print $2}' |sort | uniq | wc -l
Check how many IP visits there were during the hour at 14:00 on August 16, 2015:
awk '{print $4,$1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}'| sort | uniq | wc -l
View the top ten ip addresses accessed
awk '{print $1}' |sort|uniq -c|sort -nr |head -10 access_log
uniq -c is equivalent to group statistics and put the statistics at the top
cat access.log|awk '{print $1}'|sort|uniq -c|sort -nr|head -10
cat access.log|awk '{counts[$(11)]+=1}; END {for(url in counts) print counts[url], url}
Top 10 most visited files or pages
cat log_file|awk '{print $11}'|sort|uniq -c|sort -nr | head -10
cat log_file|awk '{print $11}'|sort|uniq -c|sort -nr|head -20
awk '{print $1}' log_file |sort -n -r |uniq -c | sort -n -r | head -20
The top 20 IPs with the most visits
The number of visits through subdomains is calculated according to the referer, which is slightly inaccurate
cat access.log | awk '{print $11}' | sed -e ' s/http:\/\///' -e ' s/\/.*//' | sort | uniq -c | sort -rn | head -20
List the files with the largest transfer size
cat www.access.log |awk '($7~/\.php/){print $10 " " $1 " " $4 " " $7}'|sort -nr|head -100
List the pages whose output is larger than 200000byte (about 200kb) and the occurrence times of the corresponding pages
cat www.access.log |awk '($10 > 200000 && $7~/\.php/){print $7}'|sort -n|uniq -c|sort -nr|head -100
If the last column of the log records the page file transfer time, it will list the most time-consuming pages to the client
cat www.access.log |awk '($7~/\.php/){print $NF " " $1 " " $4 " " $7}'|sort -nr|head -100
List the most time-consuming pages (more than 60 seconds) and the number of occurrences of the corresponding pages
cat www.access.log |awk '($NF > 60 && $7~/\.php/){print $7}'|sort -n|uniq -c|sort -nr|head -100
List files with transfer time longer than 30 seconds
cat www.access.log |awk '($NF > 30){print $7}'|sort -n|uniq -c|sort -nr|head -20
List the number of each process running on the current server, sorted in reverse order
ps -ef | awk -F ' ' '{print $8 " " $9}' |sort | uniq -c |sort -nr |head -20
View the current number of concurrent accesses of apache
What is the difference between the number of MaxClients in httpd.conf
netstat -an | grep ESTABLISHED | wc -l
You can use the following parameters to view the data
ps -ef|grep httpd|wc -l1388
Count the number of httpd processes, and even a request will start a process for the Apache server.
Indicates that Apache can handle 1388 concurrent requests, and this value can be automatically adjusted by Apache according to the load
netstat -nat|grep -i "80"|wc -l
4341
netstat -an will print the current network link status of the system, while grep -i "80" is used to extract connections related to port 80, and wc -l performs statistics on the number of connections.
The final number returned is the total number of requests for all current port 80
netstat -na|grep ESTABLISHED|wc -l
376
netstat -an will print the current network link status of the system, and grep ESTABLISHED will extract the information of the established connection. Then wc -l statistics
The number finally returned is the total number of established connections of all current port 80.
netstat -nat||grep ESTABLISHED|wc
A detailed record of all established connections can be viewed
Output the number of connections for each ip, and the total number of connections in each state
netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":");if(n<=2)++S[array[(1)]];else++S[array[(4)]];++s[$NF];++N} END {for(a in S){printf("%-20s %s\n", a, S[a]);++I}printf("%-20s %s\n","TOTAL_IP",I);for(a in s) printf("%-20s %s\n",a, s[a]);printf("%-20s %s\n","TOTAL_LINK",N);}'
other collection
Analyze and sort the top 20 URLs with the highest page visits on 2012-05-04 under the log file
cat access.log |grep '04/May/2012'| awk '{print $11}'|sort|uniq -c|sort -nr|head -20
Query the IP address of www.abc.com in the URL address of the visited page
cat access_log | awk '($11~/\www.abc.com/){print $1}'|sort|uniq -c|sort -nr
Get the top 10 IP addresses with the highest access, and you can also query by time
cat linewow-access.log|awk '{print $1}'|sort|uniq -c|sort -nr|head -10
Time period query log time period
cat log_file | egrep '15/Aug/2015|16/Aug/2015' |awk '{print $1}'|sort|uniq -c|sort -nr|head -10
Analyze the reverse order of IPs accessed from 2015/8/15 to 2015/8/16 "/index.php?g=Member&m=Public&a=sendValidCode"
cat log_file | egrep '15/Aug/2015|16/Aug/2015' | awk '{if($7 == "/index.php?g=Member&m=Public&a=sendValidCode") print $1,$7}'|sort|uniq -c|sort -nr
($7~/.php/) If $7 contains .php, it will be output. This sentence means the most time-consuming one hundred PHP pages
cat log_file |awk '($7~/\.php/){print $NF " " $1 " " $4 " " $7}'|sort -nr|head -100
List the most time-consuming pages (more than 60 seconds) and the number of occurrences of the corresponding pages
cat access.log |awk '($NF > 60 && $7~/\.php/){print $7}'|sort -n|uniq -c|sort -nr|head -100
Statistical website traffic (G)
cat access.log |awk '{sum+=$10} END {print sum/1024/1024/1024}'
Count 404 connections
awk '($9 ~/404/)' access.log | awk '{print $9,$7}' | sort
Statistics http status
cat access.log |awk '{counts[$(9)]+=1}; END {for(code in counts) print code, counts[code]}'cat access.log |awk '{print $9}'|sort|uniq -c|sort -rn
concurrent per second
watch "awk '{if($9~/200|30|404/)COUNT[$4]++}END{for( a in COUNT) print a,COUNT[a]}' log_file|sort -k 2 -nr|head -n10"
Bandwidth Statistics
cat apache.log |awk '{if($7~/GET/) count++}END{print "client_request="count}'
Find out the 10 IPs with the most visits in a certain day
cat /tmp/access.log | grep "20/Mar/2011" |awk '{print $3}'|sort |uniq -c|sort -nr|head
What are the IPs with the highest number of IP connections doing that day?
cat access.log | grep "10.0.21.17" | awk '{print $8}' | sort | uniq -c | sort -nr | head -n 10
The 10 periods with the largest number of ip connections in hourly units
awk -vFS="[:]" '{gsub("-.*","",$1);num[$2" "$1]++}END{for(i in num)print i,num[i]}' log_file | sort -n -k 3 -r | head -10
Find the most visited minutes
awk '{print $1}' access.log | grep "20/Mar/2011" |cut -c 14-18|sort|uniq -c|sort -nr|head
Take 5 minutes log
if [ $DATE_MINUTE != $DATE_END_MINUTE ] ;then #
Then judge whether the start timestamp and the end timestamp are equal
START_LINE=sed -n "/$DATE_MINUTE/=" $APACHE_LOG|head -n1#If not equal, take out the line number of the start timestamp and the line number of the end timestamp
View tcp link status
netstat -nat |awk '{print $6}'|sort|uniq -c|sort -rn
netstat -n | awk '/^tcp/ {++S[$NF]};END {for(a in S) print a, S[a]}'
netstat -n | awk '/^tcp/ {++state[$NF]}; END {for(key in state) print key,"\t",state[key]}'
netstat -n | awk '/^tcp/ {++arr[$NF]};END {for(k in arr) print k,"\t",arr[k]}'
netstat -n |awk '/^tcp/ {print $NF}'|sort|uniq -c|sort -rn
netstat -ant | awk '{print $NF}' | grep -v '[a-z]' | sort | uniq -cnetstat -ant|awk '/ip:80/{split($5,ip,":");++S[ip[1]]}END{for (a in S) print S[a],a}' |sort -n
netstat -ant|awk '/:80/{split($5,ip,":");++S[ip[1]]}END{for (a in S) print S[a],a}' |sort -rn|head -n 10
awk 'BEGIN{printf ("http_code\tcount_num\n")}{COUNT[$10]++}END{for (a in COUNT) printf a"\t\t"COUNT[a]"\n"}'
Find the top 20 IPs with the number of requests (often used to find attack sources):
netstat -anlp|grep 80|grep tcp|awk '{print $5}'|awk -F: '{print $1}'|sort|uniq -c|sort -nr|head -n20 netstat -ant |awk '/:80/{split($5,ip,":");++A[ip[1]]}END{for(i in A) print A[i],i}' |sort -rn|head -n20
Use tcpdump to sniff access to port 80 to see who is the highest
tcpdump -i eth0 -tnn dst port 80 -c 1000 | awk -F"." '{print $1"."$2"."$3"."$4}' | sort | uniq -c | sort -nr |head -20
Find more time_wait connections
netstat -n|grep TIME_WAIT|awk '{print $5}'|sort|uniq -c|sort -rn|head -n20
Look for more SYN connections
netstat -an | grep SYN | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | sort -nr | more
List processes by port
netstat -ntlp | grep 80 | awk '{print $7}' | cut -d/ -f1
Viewed the number of connections and the current number of connections
netstat -ant | grep $ip:80 | wc -l netstat -ant | grep $ip:80 | grep EST | wc -l
View the number of IP visits
netstat -nat|grep ":80"|awk '{print $5}' |awk -F: '{print $1}' | sort| uniq -c|sort -n
Linux command to analyze the current link status
netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
watch "netstat -n | awk '/^tcp/ {++S[\$NF]} END {for(a in S) print a, S[a]}'"# You can always monitor through watch
LAST_ACK 5 #Close a TCP connection needs to be closed in two directions, both parties send FIN to indicate the closure of one-way data, when the two parties send the last FIN, the sender is in the LAST_ACK state at this time, When the sender receives the other party's confirmation (Fin's Ack confirmation), the entire TCP connection is actually closed;
SYN_RECV 30 # Indicates the number of requests waiting to be processed;
ESTABLISHED 1597 # Indicates the normal data transmission status;
FIN_WAIT1 51 # Indicates that the server actively requests to close the tcp connection;
FIN_WAIT2 504 # means the client disconnected;
TIME_WAIT 1057 # Indicates the number of requests that have been processed and are waiting for the timeout to end;