ELK log analysis
1. Why use ELK?
In general we need to log analysis scenarios: grep directly in the log file, awk can get the information they want. But in the larger scene, this method is inefficient, problems including how to log archiving is too big, too slow how to do text searches, how multi-dimensional queries. Require centralized log management, log on all servers collect aggregate. Common Solutions is creating a centralized log collection system, the logs on all nodes in the unified collection, management, access.
General large-scale deployment of the system is a distributed architecture, different service modules deployed on a different server, when problems arise, in most cases need to issue critical information exposure, targeted to specific servers and services module, to build a centralized style log system, can improve the efficiency of fault location.
A complete centralized logging system, need to include the following main characteristics: the collection - can collect log data transmissions from multiple sources - how to store log data - to be able to stabilize the log data to a central storage system
analysis - analysis supports UI
warning - can provide error reporting, monitoring mechanisms ELK provide a complete solution, and is open source software, used in conjunction with each other, the perfect convergence, efficient to meet a number of occasions applications. A log mainstream system.
2. ELK components Introduction
ELK is an abbreviation of three open source software, respectively: Elasticsearch, Logstash, Kibana, they are open source software. Added a FileBeat, it is a lightweight log collection processing tools (Agent), Filebeat small footprint, suitable for transmission on each server logs to collect after Logstash, the government has also recommended this tool.
Elasticsearch is an open source distributed search engine that provides collection, analysis, storage of data three functions. Its features include: distributed, zero-configuration, auto-discovery, auto-slice index, index replication mechanism, restful style interfaces, multiple data sources, such as automatic load search.
Logstash mainly used to log collection, analysis, log filtering tools to support large amounts of data acquisition mode. General work of c / s architecture, client installed on the host side need to collect logs, server side is responsible for each node the received log is filtered, modification and other operations in a concurrent to elasticsearch up.
Kibana is also an open source and free tools, you can analyze Kibana friendly Web interface and log Logstash ElasticSearch provided to help summarize, analyze and search for important data logs.
Filebeat affiliated with Beats. Beats currently contains four tools:
- Packetbeat (collect network traffic data)
- Topbeat (collection systems, processes and file system-level CPU and memory usage data)
- Filebeat (collect data files)
Winlogbeat (collect Windows event log data)
3. Experimental deployment
The deployment is filebeats (client), logstash + elasticsearch + kibana (server) component architecture.
Service request arrives on Nginx nginx-server machine; Nginx response to the request, and increase in the access record access.log file; FileBeat new log collected by uploading logs LogStash port 5044; LogStash log information by the unit 9200 incoming port to ElasticSerach; user search logs Kibana access through the browser, server port is 5601;
Kibana port access by 9200 ElasticSerach;
Experimental environment:
The deployment is a single point of ELK with the two machines (CentOS-7.5) ELK server: 192.168.88.100
Nginx Client: 192.168.88.111. Preparation:
Yum configured network source
# wget http://mirrors.aliyun.com/repo/Centos-7.repo
# wget http://mirrors.aliyun.com/repo/epel-7.repo 关闭防火墙：systemctl stop(disable) firewalld 关闭 SELinux：SELINUX=disabled2. Download and install the package:
# mkdir /elk;cd /elk
# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.tar.gz
# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.tar.gz
# wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-linux-x86_64.tar.gz
全部解压缩，并复制到/usr/local/目录下3. Install JDK (java) Environment Tools:
# yum -y install java-1.8*4. Configuration elasticsearch:
1) 新建 elasticsearch 用户并启动(用 elasticsearch 普通用户启动)
# useradd elasticsearch
# chown -R elasticsearch.elasticsearch /usr/local/elasticsearch-6.2.3/
# su - elasticsearch
# cd /usr/local/elasticsearch-6.2.3/
# ./bin/elasticsearch -d
2) 查看进程是否启动成功（等待一下）
# netstat -antp
3) 若出现错误可以查看日志
# cat /usr/local/elasticsearch-6.2.3/logs/elasticsearch.log
4) 测试是否可以正常访问
# curl localhost:9200
## 5. 配置 logstash
Logstash 收集 nginx 日志之使用 grok 过滤插件解析日志，grok 作为一个 logstash 的过滤插件，支持根 据模式解析文本日志行，拆成字段。
1) logstash 中 grok 的正则匹配
vim vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns
WZ ([^ ]*)
NGINXACCESS %{IP:remote_ip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{WZ:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent}
%{QS:xforward}
2) 创建 logstash 配置文件
vim /usr/local/logstash-6.2.3/default.conf input {
beats {
—————————————————————————————
port => "5044"
}
}
#数据过滤
filter {
grok {
match => { "message" => "%{NGINXACCESS}" }
}
geoip {
# nginx 客户端 ip
source => "192.168.88.110"
}
}
#输出配置为本机的 9200 端口，这是 ElasticSerach 服务的监听端口
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
}
}
3) 进入到/usr/local/logstash-6.2.3 目录下，并执行下列命令 后台启动 logstash：nohup bin/logstash -f default.conf & 查看启动日志：tailf nohup.out
查看端口是否启动：netstat -napt|grep 50446. placement kibana
1) 打开 Kibana 配置文件/usr/local/kibana-6.2.3-linux-x86_64/config/kibana.yml，找到下面这行并修改
# vim /usr/local/kibana-6.2.3-linux-x86_64/config/kibana.yml
#server.host: "localhost"
修改为
server.host: "192.168.88.100"
这样其他电脑就能用浏览器访问 Kibana 的服务了；
2) 进入 Kibana 的目录：cd /usr/local/kibana-6.2.3-linux-x86_64
执行启动命令：nohup bin/kibana & 查看启动日志：tail -f nohup.out 查看端口是否启动：netstat -napt|grep 5601
3) 测试：
在浏览器访问 192.168.88.100:5601
到此。ELK 部署完成7. Nginx client configuration
1) yum 安装二进制 nginx 软件包
# yum -y install nginx
2) 下载 filebeat 并解压到/usr/local/
# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.3-linux-x86_64.tar.gz
# tar -xf ./filebeat-6.2.3-linux-x86_64.tar.gz -C /usr/local/
3) 打开文件/usr/local/filebeat-6.2.3-linux-x86_64/filebeat.yml，找到如下位置：修改三处
enable：false #修改为 true
paths：/var/log/*.log #修改为/var/log/nginx/*.log
#output.elasticsearch: #将此行注释掉
#hosts: ["localhost:9200"] #将此行注释掉
output.logstash: #取消此行注释
hosts: ["192.168.88.100:5044"] #取消此行注释并修改 IP 地址为 ELK 服务器地址
4) 切换到/usr/local/filebeat-6.2.3-linux-x86_64 目录下
# cd /usr/local/filebeat-6.2.3-linux-x86_64
后台启动 filebeat：nohup ./filebeat -e -c filebeat.yml &
查看日志：tailf nohup.out
5) 通过浏览器多访问几次 nginx 服务，这样能多制造一些访问日志，访问地址：https://192.168.137.131
6) 访问 Kibana：https://192.168.88.100:5601，点击左上角的 Discover，就可以看到访问日志已经被ELK collected, then follow these steps to complete the setup
⚫ input logstash- *, click on "the Next the STEP"
⚫ select Time Filter, and then click on the "index the Create pattern"
⚫ can then create their own rules query log content