nginx日志分析 Nginx访问日志分析

Nginx访问日志分析

 

nginx默认的日志格式

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';

字段说明

127.0.0.1 - - [14/May/2017:12:51:13 +0800] "GET /index.html HTTP/1.1" 200 4286 "http://127.0.0.1/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36" 远程主机IP 请求时间 时区 方法 资源 协议 状态码 发送字节 referer 浏览器信息 

统计访问IP前十

# awk '{print $1}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10
   6958 123.174.51.164
   2307 111.85.34.165 1617 118.112.143.148 1489 117.63.146.40 1404 118.182.116.39 1352 1.48.219.30 1132 60.222.231.46 1129 10.35.1.82 943 27.227.163.200 880 58.253.6.133

统计指定某一天的访问IP

# grep "17/May/2017" /usr/local/nginx/logs/access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -10 # awk '/17\/May\/2017/{print $1}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10 6958 123.174.51.164 2307 111.85.34.165 1617 118.112.143.148 1489 117.63.146.40 1404 118.182.116.39 1352 1.48.219.30 1132 60.222.231.46 1129 10.35.1.82 943 27.227.163.200 880 58.253.6.133 经过测试，在文件较大的时候，先grep再awk速度快很多。

过滤URL

# awk '{print $11}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10 20737 "http://www.adreambox.net/index.php?app=home&mod=User&act=index" 4155 "-" 3981 "http://www.adreambox.net/" 1921 "http://www.adreambox.net/index.php?app=adreambox&mod=Class&act=prensent&id=5&type=2" 1299 "http://www.adreambox.net/index.php?app=home&mod=Public&act=doLogin" 1191 "http://www.adreambox.net/index.php?app=group&mod=Group&act=index&gid=1413" 718 "http://www.adreambox.net/index.php?app=group&mod=Group&act=index&gid=1403" 657 "http://www.adreambox.net/index.php?app=wap&mod=Index&act=index" 657 "http://www.adreambox.net/index.php?act=index&app=home&mod=User" 639 "http://www.adreambox.net/index.php?app=group&mod=Manage&act=index&gid=1413"

统计指定资源

# awk '($7~/\.html$/){print $1 " " $7 " " $9}' /usr/local/nginx/logs/access.log #处理第7个字段以'.html'结尾的行 11.0.8.5 //ckeditor/notexist_path.html 404 11.0.8.5 //ckeditor/CHANGES.html 404 11.0.8.18 //docs/CHANGELOG.html 404 11.0.8.5 //themes/mall/default/seller_order.confirm.html 404 11.0.8.18 //themes/mall/default/header.html 404 11.0.8.5 //themes/store/default/footer.html 404 11.0.8.5 //templates/admin/index.html 404 11.0.8.5 //system/templates/admin/login.html 404 11.0.8.18 //templates/404.html 404 11.0.8.18 //admin/editor/editor/dialog/fck_about.html 404 11.0.8.5 //fckeditor/_whatsnew.html 404 11.0.8.5 //FCKeditor/_docs/whatsnew.html 404 11.0.8.5 //style/gb/help/index.html 404 10.10.1.11 /Login/login.html 404

过滤指定时间后的日志并打印IP

# awk '($4>"[15/May/2017:21:16:38"){print $1}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr
 291031 11.0.8.5 274174 11.0.8.18 2764 10.10.1.11 1193 11.0.8.6 1 127.0.0.1

统计流量

# grep "17/May/2017" /usr/local/nginx/logs/access.log | awk '{sum+=$10}END{print sum}' 95210093059

统计状态码

# awk '{print $9}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10 1271257 200 957444 503 61875 502 32852 404 19121 302 13356 304 2819 500 2789 400 271 499 203 401
转载于https://www.cnblogs.com/keithtt/p/6867019.html

nginx默认的日志格式

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';

字段说明

127.0.0.1 - - [14/May/2017:12:51:13 +0800] "GET /index.html HTTP/1.1" 200 4286 "http://127.0.0.1/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36" 远程主机IP 请求时间 时区 方法 资源 协议 状态码 发送字节 referer 浏览器信息 

统计访问IP前十

# awk '{print $1}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10
   6958 123.174.51.164
   2307 111.85.34.165 1617 118.112.143.148 1489 117.63.146.40 1404 118.182.116.39 1352 1.48.219.30 1132 60.222.231.46 1129 10.35.1.82 943 27.227.163.200 880 58.253.6.133

统计指定某一天的访问IP

# grep "17/May/2017" /usr/local/nginx/logs/access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -10 # awk '/17\/May\/2017/{print $1}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10 6958 123.174.51.164 2307 111.85.34.165 1617 118.112.143.148 1489 117.63.146.40 1404 118.182.116.39 1352 1.48.219.30 1132 60.222.231.46 1129 10.35.1.82 943 27.227.163.200 880 58.253.6.133 经过测试，在文件较大的时候，先grep再awk速度快很多。

过滤URL

# awk '{print $11}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10 20737 "http://www.adreambox.net/index.php?app=home&mod=User&act=index" 4155 "-" 3981 "http://www.adreambox.net/" 1921 "http://www.adreambox.net/index.php?app=adreambox&mod=Class&act=prensent&id=5&type=2" 1299 "http://www.adreambox.net/index.php?app=home&mod=Public&act=doLogin" 1191 "http://www.adreambox.net/index.php?app=group&mod=Group&act=index&gid=1413" 718 "http://www.adreambox.net/index.php?app=group&mod=Group&act=index&gid=1403" 657 "http://www.adreambox.net/index.php?app=wap&mod=Index&act=index" 657 "http://www.adreambox.net/index.php?act=index&app=home&mod=User" 639 "http://www.adreambox.net/index.php?app=group&mod=Manage&act=index&gid=1413"

统计指定资源

# awk '($7~/\.html$/){print $1 " " $7 " " $9}' /usr/local/nginx/logs/access.log #处理第7个字段以'.html'结尾的行 11.0.8.5 //ckeditor/notexist_path.html 404 11.0.8.5 //ckeditor/CHANGES.html 404 11.0.8.18 //docs/CHANGELOG.html 404 11.0.8.5 //themes/mall/default/seller_order.confirm.html 404 11.0.8.18 //themes/mall/default/header.html 404 11.0.8.5 //themes/store/default/footer.html 404 11.0.8.5 //templates/admin/index.html 404 11.0.8.5 //system/templates/admin/login.html 404 11.0.8.18 //templates/404.html 404 11.0.8.18 //admin/editor/editor/dialog/fck_about.html 404 11.0.8.5 //fckeditor/_whatsnew.html 404 11.0.8.5 //FCKeditor/_docs/whatsnew.html 404 11.0.8.5 //style/gb/help/index.html 404 10.10.1.11 /Login/login.html 404

过滤指定时间后的日志并打印IP

# awk '($4>"[15/May/2017:21:16:38"){print $1}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr
 291031 11.0.8.5 274174 11.0.8.18 2764 10.10.1.11 1193 11.0.8.6 1 127.0.0.1

统计流量

# grep "17/May/2017" /usr/local/nginx/logs/access.log | awk '{sum+=$10}END{print sum}' 95210093059

统计状态码

# awk '{print $9}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10 1271257 200 957444 503 61875 502 32852 404 19121 302 13356 304 2819 500 2789 400 271 499 203 401
转载于https://www.cnblogs.com/keithtt/p/6867019.html