Course content: To
create an enterprise-level LNMP WEB architecture actual combat
AWK, Sed, Grep analysis nginx log
SHELL into a script to cut the Nginx date;
1. Build an enterprise-level LNMP WEB architecture combat:
When the make source code compilation method builds the nginx web platform, the default nginx web server only publishes one set of where to go back, and the source code is a test page, as shown in the figure
2. Build the actual combat of enterprise-level LNMP web architecture,
AWK sed grep analyzes nginx log
shell and turns into tutorial cutting nginx log;
1. Build the actual combat of enterprise-level LNMP web architecture:
As an operation and maintenance personnel, the job responsibilities are: guarantee the enterprise server, code release, website Update, database, business system maintenance, etc.;
2. In the enterprise, developers have developed a set of website code, which is written based on the PHP language, and requires operation and maintenance personnel to use the PHP website code to achieve access to external users as operation and maintenance How to operate:
Evaluate the number of users (concurrent users) visiting the PHP website in the later period,
purchase online servers (Tencent Cloud Host, 8C16G8M) 8C core CPU, 16G memory, 8M bandwidth
Tencent Cloud Host Operating System (Centos7.x Linux operating system) )
Register the extranet domain name and file for the record at the same time. Cvc.net will later access the domain name to
construct the LAMP and LNMMP WEB structure for publishing PHP website;
//检查硬件环境：
[root@cdeba90ec46e /]# cat /etc/redhat-release
CentOS Linux release 7.6.2003 (Core)
[root@cdeba90ec46e /]# uname -a
Linux cdeba90ec46e 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@cdeba90ec46e /]# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 100G 4.5G 96G 5% /
tmpfs 910M 0 910M 0% /dev
tmpfs 910M 0 910M 0% /sys/fs/cgroup
/dev/sda5 100G 4.5G 96G 5% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 64M 1.1M 63M 2% /run
tmpfs 64M 0 64M 0% /run/lock
tmpfs 64M 0 64M 0% /var/log/journal
tmpfs 910M 0 910M 0% /tmp3. Based on Centos7 Linux cloud host, build a set of LNMP WEB architecture from 0, publish and process PHP website code
LNMP:
L: Linux operating system (centos7) developed based on the linux kernel
N: Nginx open source, free high-performance WEB server Software;
M: MySQL, Mariadb relational database
P: Perl, Python, PHP environment & PHP compiler
The working principle of the LNMP architecture: In the
LNMP WEB architecture, Nginx is a high-performance WEB server, which itself cannot handle PHP. When we receive the HTTP Regtest request sent by the client browser, the Nginx server responds and processes web requests and static resources. The nginx server can directly process and respond to static file requests such as CSS, pictures, videos, and TXT.
PHP dynamic page request nginx cannot be processed directly, the nginx server will transmit the PHP web script to PHP-FPM (process management program) through the interface transfer protocol (gateway protocol)
PPH-FPM calls the PHP parser (PHP-CGI) PHP parser parsing PHP script information. Finally, the PHP interpreter puts the parsed script back into php-fpm, and php-fpm transmits the script information to the mother in the form of fase-CGI, if shown; 
Note, the basis of deploying LNMP in the docker environment surroundings:
[root@localhost ~]# docker run -tid --net=none --name=centos7-nginx.yum.82qiye --privileged=true centos7-ssh:zabbix-agent /sbin/init
27ba4cba71d5991e7cf814fb268c421474cd4018f0a62c560655d2bbb69c0634
[root@localhost ~]# pipework br0 centos7-nginx.yum.82qiye 192.168.1.82/[email protected]
[root@localhost ~]# docker exec -it centos7-nginx.yum.82qiye /bin/bash3. There are two main ways to build LNMP WEB platform from 0;
YUM binary method
MAKE source code compilation method
Here we build LNMP based on YUM binary method (the form of network source, the server can be connected to the external network, and the LAN YUM source is configured) Operation
1. Deploy nginx:
//1、添加Epel-release扩展源
[root@localhost ~]# yum -y install epel-release
[root@localhost ~]# ll /etc/yum.repos.d/|grep -aw epel
-rw-r--r-- 1 root root 951 Oct 3 2017 epel.repo
-rw-r--r-- 1 root root 1050 Oct 3 2017 epel-testing.repo
//2、安装nginx软件包：
[root@localhost ~]# yum -y install nginx //显示Complete！表示安装成功；
//3、检测nginx软件包是否安装成功
[root@27ba4cba71d5 ~]# rpm -qa |grep nginx
nginx-filesystem-1.14.1-9.module_el8.0.0+184+e34fea82.noarch
nginx-mod-http-perl-1.14.1-9.module_el8.0.0+184+e34fea82.x86_64
nginx-mod-mail-1.14.1-9.module_el8.0.0+184+e34fea82.x86_64
nginx-1.14.1-9.module_el8.0.0+184+e34fea82.x86_64
nginx-all-modules-1.14.1-9.module_el8.0.0+184+e34fea82.noarch
nginx-mod-http-xslt-filter-1.14.1-9.module_el8.0.0+184+e34fea82.x86_64
nginx-mod-stream-1.14.1-9.module_el8.0.0+184+e34fea82.x86_64
nginx-mod-http-image-filter-1.14.1-9.module_el8.0.0+184+e34fea82.x86_64
//4、查看端口和进程
[root@27ba4cba71d5 ~]# systemctl start nginx.service
[root@27ba4cba71d5 ~]# ps -ef |grep nginx
[root@27ba4cba71d5 ~]# netstat -tunlp |grep -aw 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 258/nginx: master p
tcp6 0 0 :::80 :::* LISTEN 258/nginx: master p 2. Deploy Mariadb:
//1、安装Mariadb：
[root@27ba4cba71d5 ~]# yum -y install mariadb-server mariadb mariadb-devel
//2、检测Mariadb安装是否成功：
rpm -qa |grep mariadb
//3、查看Mariadb进程和端口号；
[root@27ba4cba71d5 ~]# systemctl start mariadb.service
[root@27ba4cba71d5 ~]# ps -ef |grep mysql
mysql 452 1 0 08:04 ? 00:00:00 /usr/libexec/mysqld --basedir=/usr
root 518 121 0 08:04 ? 00:00:00 grep --color=auto mysql
[root@27ba4cba71d5 ~]# netstat -tunlp |grep -aw 3306
tcp6 0 0 :::3306 :::* LISTEN 452/mysqld3. Deploy PHP:
//1、安装php
[root@27ba4cba71d5 ~]# yum -y install php php-devel php-fpm php-mysql
//2、检测PHP是否安装成功：
[root@27ba4cba71d5 ~]# rpm -qa |grep php
//3、查看PHP进程及端口号
[root@27ba4cba71d5 ~]# systemctl start php-fpm.service
[root@27ba4cba71d5 ~]# ps -ef |grep php
[root@27ba4cba71d5 ~]# netstat -tunlp|grep -aw 9000 //此时没有查到9000端口4. According to the above LNMP deployment instructions, the deployment of the LNMP platform is completed, check its progress
[root@27ba4cba71d5 ~]# ps -ef |grep -wE "nginx|mysqld|php"
root 258 1 0 07:52 ? 00:00:00 nginx: master process /usr/sbin/nginx
nginx 259 258 0 07:52 ? 00:00:00 nginx: worker process
nginx 260 258 0 07:52 ? 00:00:00 nginx: worker process
mysql 452 1 0 08:04 ? 00:00:03 /usr/libexec/mysqld --basedir=/usr
root 598 1 0 08:14 ? 00:00:00 php-fpm: master process (/etc/php-fpm.conf)
apache 599 598 0 08:14 ? 00:00:00 php-fpm: pool www
apache 600 598 0 08:14 ? 00:00:00 php-fpm: pool www
apache 601 598 0 08:14 ? 00:00:00 php-fpm: pool www
apache 602 598 0 08:14 ? 00:00:00 php-fpm: pool www
apache 603 598 0 08:14 ? 00:00:00 php-fpm: pool www
root 647 121 0 08:23 ? 00:00:00 grep --color=auto -wE nginx|mysqld|php5. It is necessary to talk about the configuration integration of nginx and php-fpm, so that when nginx detects that the user requests a PHP dynamic web page, nginx will send the user's request to the back-end PHP-FPM interpreter through the CGI gateway protocol and take it out, and configure nginx.conf code show as below:
//nginx配置：
[root@localhost nginx]#
location / { //第二步
root html;
index index.php index.html index.htm; //加上nginx.php表示引导页。
location ~ \.php$ {
root /usr/share/nginx/html; //更改发布目录 //第一步
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; //添加$document_root;表示发布目录 //第三步
include fastcgi_params;
}
//查看nginx.conf文件并去掉#号空行。
[root@localhost nginx]# grep -vE "#|^$" nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65; //以上为全局配置
server { //以下为server主机的配置
listen 80;
server_name localhost;
location / { //location /是正常匹配，处于正则匹配后执行。
root html;
index index.php index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
location ~ \.php$ { //location ~是正则匹配，是优先匹配，
root /usr/share/nginx/html;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; //指定发布目录的变量，绝对路径也可以：/usr/share/nginx/html;
include fastcgi_params;
}
}
}After the change is complete:
[root@27ba4cba71d5 nginx]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@27ba4cba71d5 nginx]# nginx -s reload6, nginx, php-fpm release directory; /usr/share/nginx/html, create index.php in this directory, the code content is as follows.
[root@27ba4cba71d5 nginx]# vim /usr/share/nginx/html/index.php
<?php
phpinfo();
?>
7. Publish the website through LNMP:
//我们只需要把我们打开发包解压到发布目录就可以了：
[root@localhost ~]# ls
anaconda-ks.cfg web.html.tar
[root@localhost ~]# tar -zxvf web.html.tar /usr/share/nginx/html
3. AWK, Sed, and Grep analyze Nginx logs;
1 As an operation and maintenance personnel, the log content in the enterprise is mainly used for shooting errors and fixing problems. According to the error prompts of the log content, the problem can be located in the first time, and the wrong shooting problems can be identified. Solve problems quickly and reduce business losses.
In fact, the content of the log is used for operation and maintenance personnel, developers, and DBA troubleshooting. In addition, the log content can be analyzed, counted, and evaluated, so as to grasp the portal IP, PV, UV, access, resource allocation, and usage. Wait.
[root@localhost nginx]# cd /var/log/nginx/ //cd到nginx的日志目录
[root@localhost nginx]# ll
total 1764
-rw-rw-r-- 1 nginx root 1527988 Nov 3 22:13 access.log
-rw-r--r-- 1 root root 34543 Oct 31 11:45 access.log-20201031.gz
-rw-rw-r-- 1 nginx root 47309 Nov 1 10:12 access.log-20201101.gz
-rw-rw-r-- 1 nginx root 87592 Nov 2 13:24 access.log-20201102.gz
-rw-rw-r-- 1 nginx root 46903 Nov 3 12:41 access.log-20201103.gz
-rw-rw-r-- 1 nginx root 0 Nov 2 13:25 error.log
-rw-r--r-- 1 root root 1488 Oct 30 13:20 error.log-20201031.gz
-rw-rw-r-- 1 nginx root 299 Nov 1 18:11 error.log-20201102.gz
[root@localhost nginx]# more access.log //每一条都是一个用户的访问请求；
192.168.1.101 - - [03/Nov/2020:12:41:22 +0800] "POST /zabbix.php?action=notifications.get&sid=a2c13a51c1e3ec42&output=ajax HTTP/1.1" 200 436 "http://192.168.1.59/zabbix.php?action=da
shboard.view&ddreset=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"1. Based on the SHELL programming three swordsmen Awk, Sed, Grep analyze the online Nginx logs, analyze and count the total number of requests (visits) of Nginx throughout the day. The instructions and methods of operation are as follows:
//1、统计当天，总的用户访问量：
[root@localhost nginx]# awk '{print $0}' access.log |wc -l //统计我们当天用户访问量； 单引号表示命令段，大括号把命令括起来，表示一个动作，print打印动作，$0表示文本所有内容，wc 统计，-l打印行号
3884
[root@localhost nginx]# sed = access.log|tail -2 // sed = 是显示行号的，tail -2表示显示最后两行
3884
192.168.1.101 - - [03/Nov/2020:22:22:45 +0800] "POST /zabbix.php?action=notifications.get&sid=a2c13a51c1e3ec42&output=ajax HTTP/1.1" 200 436 "http://192.168.1.59/hosts.php?ddreset=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"
[root@localhost nginx]# sed = access.log|tail -2 |head -1 head -1 表示第一行
3884
[root@localhost nginx]# sed = access.log|tail -2 |head -1
3884
[root@localhost nginx]# wc -l access.log|cut -d "" -f1 //-f1便是fire字段
3884 access.log
[root@localhost nginx]# awk 'END{print NR}' access.log
38842. Based on the SHELL programming three swordsmen Awk, Sed, Grep analyze the online Nginx logs, analyze and count the total number of requests (visits) of Nginx from 09:00 to 11:00 throughout the day. The instructions and methods of operation are as follows:
[root@localhost nginx]# awk '/2020:09:00/' access.log|more
[root@localhost nginx]# sed -n '/2020:09:00/'p access.log|more
[root@localhost nginx]# grep -aiw '2020:12:41' access.log|more
[root@localhost nginx]# sed -n '/2020:12:41/,/2020:12:00/'p access.log|head -5 //范围 head表示前5条，tail表示末尾5条
[root@localhost nginx]# awk '/2020:12:41/,/2020:12:00/' access.log|wc -l
39273. Based on the SHELL programming three swordsmen Awk, Sed, Grep, analyze the online Nginx log, analyze and count the total number of requests (visit volume) of Nginx from 09:00 to 11:00 throughout the day, print out the IP address, and at the same time Print the top 20 IP addresses, and add IPs with more than 500 visits to the Linux blacklist; the instructions and methods of operation are as follows:
//都属于正则表达式：
//将09:00~11:00的IP地址打印出来；
[root@localhost nginx]# sed -n '/2020:12:41/,/2020:12:00/'p access.log|awk '{print$1}'|more
192.168.1.101
192.168.1.59
[root@localhost nginx]# sed -n '/2020:12:41/,/2020:12:00/'p access.log|grep -oE "([0-9]{1,3}\.){3}[0-9]{1,3}"|more
192.168.1.101
192.168.1.59
//将访问次数超过500次的IP加入linux的黑名单：
root@localhost nginx]# sed -n '/2020:12:41/,/2020:12:00/'p access.log|grep -oE "([0-9]{1,3}\.){3}[0-9]{1,3}"|sort -n|uniq -c|sort -nr|head -20 //打印访问量排前20名的IP地址；
3964 192.168.1.101
3961 192.168.1.59
注：sort -n正向排序；uniq -c去重并统计，sort -nr 逆向排序，head -20 打印20名；
//打印访问次数超过500次的用户地址；
[root@localhost nginx]# sed -n '/2020:12:41/,/2020:12:00/'p access.log|grep -oE "([0-9]{1,3}\.){3}[0-9]{1,3}"|sort -nr|uniq -c|sort -nr|awk '{if(($1>500)) print$2}'
192.168.1.101
192.168.1.59
//访问量超过500次的加入IPtables防火墙黑名单：（IPtable）
[root@localhost nginx]# for ip in $(sed -n '/2020:12:41/,/2020:12:00/'p access.log|grep -oE "([0-9]{1,3}\.){3}[0-9]{1,3}"|sort -nr|uniq -c|sort -nr|awk '{if(($1>500)) print$2}');do iptable -t filter -A INPUT -s $ip/32 -m tcp -p tcp --dport 80 -j DROP ;done //$ip/32表示IP地址，IPtable规则