Apache configuration-log segmentation and use AWStats log analysis

Preface

​ With the increase in the number of visits to the website, by default, a single apache log file will become larger and larger. The log file takes up a large amount of disk space and it is inconvenient to view related information. If you do not split the log, you can only delete large logs (such as Apache logs) at one time. This will also lose a lot of valuable information for the website, because these logs can be used for access analysis, network security monitoring, and network Operating status monitoring, etc., so managing these massive logs is of great significance to the website.

One: Apache day separation

Log splitting is of great significance to websites. The following two tools automatically split Apache logs according to the date of the day: Apache's own rotatelogs splitting tool and third-party tool cronolog splitting logs

• Log files will take up a lot of disk space under the accumulation of time
• It would be inconvenient to view related information

1.1: Two tools for splitting log files

• Apache comes with rotatelogs split tool to achieve, we only need to add configuration file to open it
• Third-party tool cronolog segmentation

1.2: rotatelogs segmentation tool

• The log file of the configuration website is forwarded to rotatelogs for split processing
• Configuration format————

ErrorLog "| /usr/sbin/rotatelogs -l logs/error_%Y%m%d.log 86400"
CustomLog "| /usr/sbin/rotatelogs -l logs/access_%Y%m%d.log 86400" combined

In the actual production environment, most of a server corresponds to N subdomain sites. In order to facilitate the consent management, you can configure it in a virtual host mode, and use the site name to identify the log file.
For example: ErrorLog "| rotatelogs (absolute path of the command)- l Log file path/website name-error_%Y%m%d.log 86400"

ErrorLog indicates the name of the error log; Customlog access log; /user/sbin/rotalogs indicates the absolute path of the command; -l specifies the path of the log file; %Y%m%d.log 86400 indicates the division by day, the time of a day is 86400 seconds

Two: rotatelogs separate log practice

1: First install Apache

[root@localhost ~]# yum -y install httpd
[root@localhost ~]# rpm -q httpd
httpd-2.4.6-93.el7.centos.x86_64
#找到rotatelogs的路径   复制一下一会要用
[root@localhost ~]# which rotatelogs
/usr/sbin/rotat:elogs

2: Edit the httpd configuration file

[root@localhost ~]# vim /etc/httpd/conf/httpd.conf

3: Restart the service to view the port

[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# setenforce 0
[root@localhost ~]# systemctl start httpd
[root@localhost ~]# netstat -ntap | grep 80
tcp        0      0 20.0.0.41:80            0.0.0.0:*               LISTEN      86943/httpd         
tcp        0      0 0.0.0.0:6000            0.0.0.0:*               LISTEN      18020/X             
tcp6       0      0 :::6000                 :::*                    LISTEN      18020/X           

4: View log files

• If you find the error log and login log, if you do not start the service, there will be no log file. Please pay attention here.

[root@localhost ~]# ls /var/log/httpd/
access_log  error_log

5: Configure rotatelogs separation

[root@localhost ~]# etc/httpd/conf/httpd.conf

6: Restart the service

[root@localhost ~]# systemctl start httpd
#检查apache配置文件
[root@localhost ~]# httpd -t
Syntax OK
[root@localhost ~]# ls /var/log/httpd/
access_log  error_log

7: Go to the host and refresh the web page

8: Check the log file again and a new log file has been generated

[root@localhost ~]# ls /var/log/httpd/
access_log  error_log  www.kgc.com.access_20200805.log  www.kgc.com.error_20200805.log

Two: Cronolog split log practice

■Source code compile and install cronolog tool
■Configure website log file and transfer it to cronolog for segmentation processing
■Configuration format

ErrorLog "I cronolog命令的绝对路径 日志文件路径/网站名-error_ %Y%m%d.log"
CustomLog "I cronolog命令的绝对路径 日志文件路径/网站名_%Y %m%d.log" combined

2.1: Install Cronolog

[root@localhost ~]# cd /opt
[root@localhost opt]# rz -E
rz waiting to receive.
[root@localhost opt]# ls
cronolog-1.6.2-14.el7.x86_64.rpm  rh
[root@localhost opt]# rpm -ivh cronolog-1.6.2-14.el7.x86_64.rpm
#复制软件路径
[root@localhost opt]# which cronolog 
/usr/sbin/cronolog

2.2: Configure log separation

[root@localhost ~]# etc/httpd/conf/httpd.conf

2.3: Restart the service

[root@localhost opt]# systemctl restart httpd
#里面就一个错误日志跟登录日志
[root@localhost httpd]# ls
access_log  error_log

2.4: Go to the host and click refresh

2.5: Check again the new log file has been generated

[root@localhost httpd]# ls
access_log  error_log  www.kgc.com.access_20200805.log
##更该一下时间
[root@localhost httpd]# date
2020年 08月 05日 星期三 20:11:16 CST
[root@localhost httpd]# date -s 07/02/20
2020年 07月 02日 星期四 00:00:00 CST

#再次查看日志文件   生成7月2号的  日志可以看见任意一天的文件比较方便
[root@localhost httpd]# ls
access_log  www.kgc.com.access_20200702.log  www.kgc.com.error_20200702.log
error_log   www.kgc.com.access_20200805.log

Three: Deploy AWStats log analysis system

AWStats log analysis system introduction

• An open source log analysis system developed by Perl language
• It can be used to analyze the access logs of Apache, Samba, Vsftpd, IIS and other servers
• Information combined with crond and other scheduled task services, the log content can be analyzed regularly

Steps

3.1: Install DNS and Apache service

[root@localhost ~]# yum -y install bind httpd
..省略内容

3.2: Configure DNS

• /etc/named.conf ##Main configuration file

• /etc/named.rfc1912.zones ##Zone configuration file

• /var/named/named.localhost ##Regional data configuration file

• Configure the main configuration file

[root@localhost ~]# vim /etc/named.conf
 11 
 12 options {
    
    
 13         listen-on port 53 {
    
     any; };  ##进来后把监听地址改为any
 14         listen-on-v6 port 53 {
    
     ::1; };
 15         directory       "/var/named";         ##目录
 16         dump-file       "/var/named/data/cache_dump.db";
 17         statistics-file "/var/named/data/named_stats.txt";
 18         memstatistics-file "/var/named/data/named_mem_stats.txt";
 19         recursing-file  "/var/named/data/named.recursing";
 20         secroots-file   "/var/named/data/named.secroots";
 21         allow-query     {
    
     any; };    ##把localhost改为any 任何都可以访问地址

• Configure zone profile

zone "abc.com" IN {
    
    
        type master;
        file "abc.com.zone";
        allow-update {
    
     none; };
};

• Configure area data

[root@tom03 ~]# cd /var/named/
[root@tom03 named]# ls
data  dynamic  named.ca  named.empty  named.localhost  named.loopback  slaves
[root@tom03 named]# cp -p named.localhost abc.com.zone
[root@tom03 ~]# vim /var/named/abc.com.zone

3.3: Restart the service and close the security function

[root@localhost named]# setenforce 0
[root@localhost named]# iptables -F
[root@localhost named]# systemctl start named

3.4: Analysis by the client

nslookup resolved successfully

3.5: Configure httpd

[root@tom03 ~]# vim /etc/httpd/conf/httpd.conf
#打开vipv4监听端口
Listen 20.0.0.43:80
#修改指定域名、S、
ServerName www.abc.com:80

[root@tom03 ~]# cd /var/www/html/
[root@tom03 html]# ls
[root@tom03 html]# vim index.html
#编写主页信息、、

<h1>hello</h1>

#重启服务
[root@tom03 ~]# systemctl restart httpd

3.6: Access test

Four: Deploy AWStats

[root@tom03 html]# rz -E
rz waiting to receive.
[root@tom03 html]# ls
awstats-7.6.tar.gz  index.html
[root@tom03 html]# mv awstats-7.6.tar.gz /opt
[root@tom03 html]# cd /opt
[root@tom03 opt]# ls
awstats-7.6.tar.gz  rh
#解压缩
[root@tom03 opt]# tar zxvf awstats-7.6.tar.gz 
[root@tom03 opt]# ls
awstats-7.6  awstats-7.6.tar.gz  rh
[root@tom03 opt]# mv awstats-7.6 /usr/local/awststs
[root@tom03 opt]# cd /usr/local/
[root@tom03 local]# ls
awststs  etc    include  lib64    sbin   src
bin      games  lib      libexec  share
[root@tom03 local]# cd awststs/
[root@tom03 awststs]# ls
docs  README.md  tools  wwwroot
[root@tom03 awststs]# cd tools/
[root@tom03 tools]# ls
awstats_buildstaticpages.pl  logresolvemerge.pl
awstats_configure.pl         maillogconvert.pl
awstats_exportlib.pl         nginx
awstats_updateall.pl         urlaliasbuilder.pl
dolibarr                     webmin
geoip_generator.pl           xslt
httpd_conf
#pl结尾的是脚本文件
[root@tom03 tools]# ./awstats_configure.pl 

Just keep pressing the Enter key,

[root@tom03 tools]# vim /etc/httpd/conf/httpd.conf

• The following is the data written by awstats

• Modify it

<Directory "/usr/local/awststs/wwwroot">
    Options None
    AllowOverride None
#    Order allow,deny          #添加注释
#    Allow from all            #添加注释
    Require all granted        #允许所有人可以访问
</Directory>

• Modify site statistics configuration file

[root@tom03 tools]# cd /etc/awstats/
[root@tom03 awstats]# ls
awstats.www.abc.com.conf          #生成的新文件
[root@tom03 awstats]# vim awstats.www.abc.com.conf

• dirdata data storage directory, because this directory does not exist, so create this directory

[root@tom03 lib]# mkdir awstats
#重启服务
[root@tom03 lib]# systemctl restart httpd

• Visit next

Modify the localhost in http://localhost/awstats/awstats.pl?config=www.kgc.com to the domain name

• Update statistics of web page visit data

[root@tom03 lib]# cd /usr/local/awststs/
[root@tom03 awststs]# ls
docs  README.md  tools  wwwroot
[root@tom03 awststs]# cd tools/
[root@tom03 tools]# ls
awstats_buildstaticpages.pl  logresolvemerge.pl
awstats_configure.pl         maillogconvert.pl
awstats_exportlib.pl         nginx
awstats_updateall.pl         urlaliasbuilder.pl
dolibarr                     webmin
geoip_generator.pl           xslt
httpd_conf
#awstats_configure.pl 配置脚本    awstats_updateall.pl #更新数据
[root@localhost tools]# ./awstats_updateall.pl now
...省略内容

• Visit the website again, the statistics are successful

• Configure periodic tasks

[root@localhost tools]# crontab -e
'//添加一下内容'
*/4 * * * *     /usr/local/awstats/tools/awstats_updateall.pl now
[root@localhost tools]# systemctl start crond
You have new mail in /var/spool/mail/root
[root@localhost tools]# cd /var/www/html
#配置自动跳转
[root@localhost html]# vim xx.html
<html>
 <head>
   <meta http-equiv=refresh content="0;url=http://www.abc.com/awstats/awstats.pl?config=www.abc.com">
 </head>
 <body></body>
</html>

• Restart service

[root@localhost html]# systemctl restart httpd