Instalación de ELK y recopilación de registros

Preparación del entorno ELK

Configuración: 2 core 2G de memoria
plan de la siguiente manera:
dirección IP:
192.168.59.130:
JDK
Kibana
elasticsearch

192.168.59.131:
jdk
logstash

1. Apague el firewall

systemctl stop firewalld
setenforce 0

2. Sincronización horaria

yum -y install ntpdate
ntpdate  pool.ntp.org

Enlace del paquete relacionado con ELK: https://pan.baidu.com/s/112s9cetAG0PuCjENr0hT4Q
Código de extracción: avsz
3. Dos conjuntos de JDK instalados

unzip ELK.zip
yum -y install jdk-8u131-linux-x64_.rpm
java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

4: instalar elasticsearch

192.168.59.130 安装elasticsearch
yum -y install elasticsearch-6.6.2.rpm
vim /etc/elasticsearch/elasticsearch.yml

Ejecute el servicio elasticsearch

systemctl enable elasticsearch
systemctl start elasticsearch
验证是否运行
tailf /var/log/elasticsearch/fncxy.log

Mostrar información correctamente

5: instalar logstash

192.168.59.131下安装logstash
yum -y install logstash-6.6.0.rpm
vim /etc/logstash/conf.d/msg.conf

Archivo de configuración

开启服务
systemctl start logstash
稍等一分钟查看
ss -nltp | grep 9600
LISTEN     0      50      [::ffff:127.0.0.1]:9600                  [::]:*                   users:(("java",pid=14233,fd=88))
chmod 777 /var/log -R

6192.168.59.130 Un vestido kibana

yum -y install kibana-6.6.2-x86_64.rpm
vim /etc/kibana/kibana.yml

systemctl start kibana
ss -nltp |grep 5601
LISTEN     0      128    192.168.59.130:5601                     *:*                   users:(("node",pid=14381,fd=18))

7: recopilar registros de nginx

yum -y install epel-release
yum -y install nginx

yum -y install httpd-tools
ab -n 1000 -c 1000 http://192.168.1.8/index.html
-n:请求数
-c:并发数
注意：并发数不能大于请求数！！
input{
    
    
	file{
    
    
		path => "/var/log/nginx/access.log"
		type => 'nginx-log'
		start_position => "beginning"
	}
}
filter {
    
    
	grok{
    
    
		match => {
    
    "message" => "%{NGX}"}
}
}
output{
    
    
	elasticsearch{
    
    
		hosts => "192.168.10.130:9200"
		index => "nginx_log-%{+YYYY.MM.dd}"
	}
}

vim /etc/logstash/pipelines.yml
- pipeline.id: nginx
  path.config: "/etc/logstash/conf.d/nginx.conf"
vim /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/nginx
定义正则
NGX %{
    
    IPORHOST:client_ip} (%{
    
    USER:ident}|- ) (%{
    
    USER:auth}|-) \[%{
    
    HTTPDATE:timestamp}\] "(?:%{WORD:verb} (%{NOTSPACE:request}|-)(?: HTTP/%{NUMBER:http_version})?|-)" %{
    
    NUMBER:status} (?:%{
    
    NUMBER:bytes}|-) "(?:%{URI:referrer}|-)" "%{GREEDYDATA:agent}"
chmod 777 /var/log -R
重启logstash
systemctl restart logstash

Código de estado

Valor pv

Las 10 visitas principales
Visitar el gráfico de tendencias

8 Correo de alarma
Python3.6 y enlace elasticalert: https://pan.baidu.com/s/18IBQejJE7rKr9y9pHMyeiA
código de extracción: y8iu

1.安装Python编译环境
yum -y install gcc gcc-c++ openssl openssl-devel

tar zxvf Python-3.6.2.tgz
cd Python-3.6.2
./configure --prefix=/usr/local/python3 --with-openssl
make && make install

2.设置软链接
rm -rf /usr/bin/python #删除以前版本的py
ln -s /usr/local/python3/bin/python3.6 /usr/bin/python #新的3.6上来顶替
ln -s /usr/local/python3/bin/pip3.6 /usr/bin/pip	#新的上来顶替

3.修复yum
vim /usr/bin/yum 将python 修改为python2
vim /usr/libexec/urlgrabber-ext-down 将python 修改为python2

3 安装alert 插件
tar zxvf v0.2.1_elasticalert.tar.gz
 mv elastalert-0.2.1/ /usr/local/elastalert
安装依赖包：
cd /usr/local/elastalert
pip install "elasticsearch<7,>6"
pip install -r requirements.txt -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
python setup.py  install
会生成以下命令
-rwxr-xr-x. 1 root root      422 8月  19 03:13 elastalert-create-index
-rwxr-xr-x. 1 root root      396 8月  19 03:13 elastalert
-rwxr-xr-x. 1 root root      416 8月  19 03:13 elastalert-test-rule
-rwxr-xr-x. 1 root root      430 8月  19 03:13 elastalert-rule-from-kibana

ln -s /usr/local/python3/bin/elastalert* /usr/bin/

cd /usr/local/elastalert
改名字：
mv config.yaml.example config.yaml

vim config.yaml

cd /usr/local/elastalert/example_rules #进入模板库
mv example_frequency.yaml nginx_frequency.yaml  #为自己的监控模板改个名字
vim nginx_frequency.yaml  #修改模板配置文件

配置邮件相关的配置文件
vim email_auth.yaml

Configurar el correo integrado de Linux

yum -y install mailx
vim /etc/mail.rc

Servicio elastalert abierto

elastalert --config /usr/local/elastalert/config.yaml --rule /usr/local/elastalert/example_rules/nginx_frequency.yaml --verbose