Article directory
- foreword
- 1. Overview of ELK log analysis system
- 2. Overview of Elasticsearch
- 3. Overview of LogStash
- 4. Overview of Kibana
- 5. Deploy ELK
foreword
The ELK platform is a complete set of log centralized processing solutions, using ElasticSearch, Logstash and Kibana three open source tools together to complete more powerful user query, sorting and statistical requirements for logs
1. Overview of ELK log analysis system
1. Introduction to ELK
ElasticSearch:
It is a distributed storage search engine developed based on Lucene (- a full-text search engine architecture), used to store various logs
-
Elasticsearch is developed in Java, through the RESTful web interface, allowing users to communicate with Elasticsearch through a browser
-
Elasticsearch is a real-time, distributed, scalable search and analysis engine that allows full-text, structured searches. It is typically used to index and search large volumes of log data, and can also be used to search many different types of documents
-
The advantage is that it can perform near real-time storage, search and analysis operations on large-capacity data
Kiabana :
- Kibana is usually deployed together with Elasticsearch. Kibana is a powerful data visualization Dashboard for Elasticsearch. Kibana provides a graphical web interface to browse Elasticsearch log data, which can be used to summarize, analyze and search for important data
Logstash:
- as a data collection engine. It supports dynamic collection of data from various data sources, and performs operations such as filtering, analyzing, enriching, and unifying the format on the data, and then stores it in a location specified by the user, and generally sends it to Elasticsearch
- Logstash is written by the JRuby language and runs on the Java Virtual Machine (JVM). It is a powerful data processing tool that can realize data transmission, format processing, and format output. Logstash has a powerful plug-in function and is often used for log processing
Additional components that can be added to Filebeat :
-
Lightweight open source log file data collector. Usually installed on the client side that needs to collect data
-
Filebeat, and specify the directory and log format, Filebeat can quickly collect data and send it to logstash for analysis, or directly to Elasticsearch storage. Compared with logstash running on the JVM, it has obvious advantages in performance and is a replacement for it.
The benefits of combining filebeat with logstash
-
- Logstash has a disk-based white-adaptive buffering system that will absorb the incoming traffic, thus lightening the pressure on Elast1 csearch to continuously write data
-
- Extract from other data sources such as databases, s3 object stores, or messaging queues
-
- Send data to multiple locations, such as s3, HDFS (Hadoop Distributed File System) or write to a file
-
- Use conditional dataflow logic to compose more complex processing pipelines
The concept of caching and message queue
- Cache/message queue (redis, kafka, RabbitMQ, etc.): It can perform traffic peak clipping and buffering for high-concurrency log data. Such buffering can protect data from loss to a certain extent, and can also decouple the application of the entire architecture
Introduction to Fluentd:
- Fluentd: is a popular open source data collector. Due to the shortcomings of logstash being too heavyweight, Logstash has low performance and high resource consumption, and then there is the emergence of Fluentd
- Compared with logstash, Fluentd is easier to use, consumes less resources, has a high performance history, and is more efficient and reliable in data processing. It is welcomed by enterprises and has become an alternative to logstash. It is often used in the EFK architecture
- EFK is also often used as a solution for log data collection in Kubernetes clusters
- Fluentd is generally run through DaemonSet in a Kubernetes cluster so that it can run a Pod on each Kubernetes worker node
- It works by taking container log files, filtering and transforming the log data, and then passing the data to an Elasticsearch cluster where it is indexed and stored
2. Why use ELK
- Logs mainly include system logs, application logs, and security logs. System operation and maintenance and developers can use logs to understand server software and hardware information, check errors in the configuration process and the reasons for errors. Frequent analysis of logs can understand the server load, performance security, and take timely measures to correct errors
- Often we can use tools such as grep and awk to analyze the logs of a single machine basically, but when the logs are scattered and stored on different devices. If you manage tens or hundreds of servers, you are still viewing logs using the traditional method of logging into each machine in turn. Does this feel cumbersome and inefficient. As a top priority, we use centralized log management, such as open source syslog, to collect and summarize logs on all servers. After centralized management of logs, the statistics and retrieval of logs has become a more troublesome thing. Generally, we can use Linux commands such as grep, awk and wc to achieve retrieval and statistics, but for more demanding queries, sorting and statistics, etc. With a huge number of machines, it is inevitable that this method is still a bit powerless
- Generally, a large-scale system is a distributed deployment architecture, and different service modules are deployed on different servers. When a problem occurs, in most cases, it is necessary to locate the specific server and service module based on the key information exposed by the problem, and build a set of centralized Type log system, which can improve the efficiency of locating problems
3. Basic features of a complete log system
- Collection: Ability to collect log data from multiple sources
- Transmission: It can stably analyze and filter log data and transmit it to the storage system
- storage: store log data
- Analysis: Support UI analysis
- WARNING: Ability to provide error reporting, monitoring mechanisms
4. How ELK works
(1) Deploy Logstash on all servers that need to collect logs; or centralize the management of logs on the log server first, and deploy Logstash on the log server
(2) Logstash collects logs, formats and outputs logs to the Elasticsearch cluster
(3) Elasticsearch indexes and stores the formatted data
(4) Kibana queries data from ES clusters to generate charts and displays front-end data
Summary: Loastash, as a log collector, collects data from data sources, filters and formats the data, and then stores them in Elasticsearch. Kibana visualizes and displays the logs.
5. ELK log processing steps
- 1. Centralized log management (Beats)
Management includes four tools:
Packetbeat (collect network traffic data)
Topbeat (collect system, process and file system level CPU and memory usage data)
Filebeat (collect file data)
Winlogbeat (collect Windows event log data)
- 2. Format the log (Logstash) and output it to Elasticsearch
- 3. Index and store formatted data (Elasticsearch)
- 4. Display of front-end data (Kibana)
2. Overview of Elasticsearch
- Provides a distributed multi-user capable full-text search engine
1. Elasticsearch features
- near real time
- cluster
- node
- index
- index(library) → type(table) → document(record)
- Shards and Replicas
In practice, the data stored in the index may exceed the hardware limit of a single node. For example, a 1 billion document needs 1TB space may not be suitable for storage on the disk of a single node, or the search request from a single node is too slow. In order to solve this problem, Elasticsearch provides the function of dividing the index into multiple shards. When creating an index, you can define the number of shards you want. Each shard is a fully functional independent index that can reside on any node in the cluster
Fragmentation:
-
1. Horizontal split expansion to increase storage capacity
-
2. Distributed parallel cross-shard operations, providing performance and throughput
The mechanism of distributed sharding and how the documents of search requests are summarized are completely controlled by elasticsearch, which is transparent to users
Copy:
- 1. High availability to deal with sharding or node failure, for this reason, shard replicas should be on different nodes
- 2. Enhanced performance, increased throughput, searches can be performed on all copies in parallel
Network problems and other problems can come unexpectedly at any time. For robustness, it is strongly recommended to have a failover mechanism, no matter what kind of failure to prevent fragmentation or node unavailability. For this reason, elasticsearch allows us to divide the index One or more copies of a shard, known as a shard copy or replica
3. Overview of LogStash
- A powerful data processing tool
- It can realize data transmission, format processing and format output
- Data input, data processing (such as filtering, rewriting, etc.) and data output
- Consists of three components: Input, Output, Filter Plugin
Input: Get logs
Output: Output logs
Filter Plugin: Filter logs, format processing
1. Main components of LogStash
Shipper
Indexer
Broker
Search and Storage
Web Interface
4. Overview of Kibana
- An open source analysis and visualization platform for Elasticsearch
- Search and view data stored in Elasticsearch indexes
- Advanced data analysis and display through various charts
1. Main functions of Kibana
- Elasticsearch seamless integration
- Integration of data, complex data analysis
- Benefit more team members
- Flexible interface, easier to share
- Simple configuration, visualized multiple data sources
- Simple Data Export
5. Deploy ELK
"There is no picture version but the whole command can be copied and operated, just lazy to intercept"
Environment configuration
Server Type IP Address Components Hardware Aspects of Installation
Node1 node 192.168.113.129 Elasticsearch, Kibana 4 core 4G
Node2 node 192.168.113.130 Elasticsearch 4 core 4G
Apache node 192.168.113.128 Logstash, Apache 2 core 4G
Configure the basic environment
#所有节点，关闭系统防火墙和安全机制
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
1. ELK Elasticsearch cluster deployment (operate on Node1, Node2 nodes)
The operation is shown here on one machine. Actually, both machines need to be configured. It is not necessary to configure both machines. Pay attention to the text and you will be prompted.
1.1 Environment preparation
#更改主机名
#Node1节点：
hostnamectl set-hostname node1
su -
#Node2节点：
hostnamectl set-hostname node2
su -
#Apache节点
hostnamectl set-hostname Apache
su -
#查看Java环境(三台都要)，桌面系统默认已安装
#建议同步安装下，更新一下版本让每台机器同步
[root@node1 opt]# yum -y install java
[root@node1 opt]# java -version ##查看版本信息
openjdk version "1.8.0_342"
OpenJDK Runtime Environment (build 1.8.0_342-b07)
OpenJDK 64-Bit Server VM (build 25.342-b07, mixed mode)
#配置域名解析“（node1和node2）”
[root@node1 opt]# echo '192.168.113.129 node1' >> /etc/hosts
[root@node1 opt]# echo '192.168.113.130 node2' >> /etc/hosts
[root@node1 opt]# cat /etc/hosts
1.2 Deploy Elasticsearch software (node1 and node2)
192.168.113.129
192.168.113.130
#1.安装elasticsearch—rpm包
#上传elasticsearch-5.5.0.rpm到/opt目录下
cd /opt
[root@node1 opt]# rpm -ivh elasticsearch-5.5.0.rpm
#2.加载系统服务
[root@node1 opt]# systemctl daemon-reload
[root@node1 opt]# systemctl enable elasticsearch.service
#3.修改elasticsearch主配置文件
[root@node1 opt]# cp /etc/elasticsearch/elasticsearch.yml{,.bak} ##备份以免配置失误无法挽回，你要自信就随你咯
或 cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak
[root@node1 opt]# vim /etc/elasticsearch/elasticsearch.yml
--17--取消注释，指定集群名字
cluster.name: my-elk-cluster
--23--取消注释，指定节点名字：Node1节点为node1，Node2节点为node2
node.name: node1
--33--取消注释，指定数据存放路径
path.data: /data/elk_data
--37--取消注释，指定日志存放路径
path.logs: /var/log/elasticsearch/
--43--取消注释，改为在启动的时候不锁定内存
bootstrap.memory_lock: false
--55--取消注释，设置监听地址，0.0.0.0代表所有地址
network.host: 0.0.0.0
--59--取消注释，ES 服务的默认监听端口为9200
http.port: 9200
--68--取消注释，集群发现通过单播实现，指定要发现的节点 node1、node2
discovery.zen.ping.unicast.hosts: ["node1", "node2"]
#查看刚刚编辑好的配皆文件过滤注释项，就是我们刚刚编辑的几行
[root@node1 opt]# grep -v "^#" /etc/elasticsearch/elasticsearch.yml
cluster.name: my-elk-cluster
node.name: node1 #注node2节点要改为node2
path.data: /data/elk_data
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: false
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["node1", "node2"]、
#4.创建数据存放路径并授权
#这个数据存放路径是不存在的，自己创建一个并为其修改属主属组
[root@node1 opt]# mkdir -p /data/elk_data
[root@node1 opt]# chown elasticsearch:elasticsearch /data/elk_data/
#5.启动elasticsearch是否成功开启
#开启服务检测服务是否开启，等10s左右
[root@node1 opt]# systemctl start elasticsearch.service
[root@node1 opt]# netstat -natp | grep 9200
tcp6 0 0 :::9200 :::* LISTEN 17854/java
#6.查看节点信息
#查看节点 Node1、Node2 的信息。
#浏览器访问 http://192.168.113.129:9200 http://192.168.113.130:9200
#检查群集状态信息
浏览器访问
http://192.168.113.129:9200/_cluster/state?pretty
http://192.168.113.130:9200/_cluster/state?pretty
Using the above method to view the status of the cluster is not user-friendly. You can manage the cluster more conveniently by installing the Elasticsearch-head plug-in
2.1 Install the Elasticsearch-head plugin
- After Elasticsearch version 5.0, the Elasticsearch-head plug-in needs to be installed as an independent service, which needs to be installed using the npm tool (a package management tool for NodeJS)
- To install Elasticsearch-head, you need to install the dependent software node and phantomjs in advance
- node: is a JavaScript runtime environment based on the Chrome V8 engine
- phantomjs: It is a JavaScript API based on webkit, which can be understood as an invisible browser. It can do anything based on webkit browsers
192.168.113.129
192.168.113.130
#1.编译安装 node
#上传软件包 node-v8.2.1.tar.gz 到/opt
#安装所需编译环境
[root@node1 opt]# yum install gcc gcc-c++ make -y
cd /opt
tar zxvf node-v8.2.1.tar.gz
[root@node1 opt]# cd node-v8.2.1/
[root@node1 node-v8.2.1]# ./configure
[root@node1 node-v8.2.1]# make -j 4 && make install ##耐心等待需要时间有点久
#2.安装 phantomjs
#上传软件包 phantomjs-2.1.1-linux-x86_64.tar.bz2 到/opt
[root@node1 node-v8.2.1]# cd /opt
[root@node1 opt]# tar jxvf phantomjs-2.1.1-linux-x86_64.tar.bz2 -C /usr/local/src/
[root@node1 opt]# cd /usr/local/src/phantomjs-2.1.1-linux-x86_64/bin/ ##切换至解压目录
[root@node1 bin]# cp phantomjs /usr/local/bin/ #将phantomjs可执行命令复制到路径环境变呈中，便于系统识别，也可以使用软链接
#3.安装 Elasticsearch-head 数据可视化工具
#上传软件包 elasticsearch-head.tar.gz 到/opt
cd /opt
[root@node1 opt]# tar zxvf elasticsearch-head.tar.gz -C /usr/local/src/
[root@node1 opt]# cd /usr/local/src/elasticsearch-head/
[root@node1 elasticsearch-head]# npm install ##npm插件是基于前端开发，安装直接使用npm install
#4.修改 Elasticsearch 主配置文件
[root@node1 elasticsearch-head]# vim/etc/elasticsearch/elasticsearch.yml
......
--末尾添加以下内容--
http.cors.enabled: true #开启跨域访问支持，默认为 false
http.cors.allow-origin: "*" #指定跨域访问,允许的域名地址为所有
[root@node1 elasticsearch-head]# systemctl restart elasticsearch.service ##重启服务
#5.启动 elasticsearch-head 服务
#必须在解压后的 elasticsearch-head 目录下启动服务，进程会读取该目录下的 gruntfile.js 文件，否则可能启动失败
[root@node1 elasticsearch-head]# cd /usr/local/src/elasticsearch-head/
[root@node1 elasticsearch-head]# npm run start &
> [email protected] start /usr/local/src/elasticsearch-head
> grunt server
Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100
#回车就退出页面了
#elasticsearch-head 监听的端口是 9100
[root@node1 elasticsearch-head]# netstat -natp | grep 9100
tcp 0 0 0.0.0.0:9100 0.0.0.0:* LISTEN 64114/grunt
#6.通过 Elasticsearch-head 查看 Elasticsearch 信息
通过浏览器访问
http://192.168.113.129:9100/
http://192.168.113.130:9100/
地址并连接群集。如果看到群集健康值为 green 绿色，代表群集很健康
- Enter and connect in the column behind Elasticsearch
http://192.168.113.129:9200
http://192.168.113.130:9200
- Then point connection will find: cluster health value: green (0 of 0)
#7.插入索引
#通过命令插入一个测试索引，索引为 index-demo，类型为 test
[root@node1 elasticsearch-head]# curl -X PUT 'localhost:9200/index-demo/test/1?pretty&pretty' -H 'content-Type: application/json' -d '{"user":"zhangsan","mesg":"hello world"}'
//输出结果如下：
{
"_index" : "index-demo",
"_type" : "test",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"created" : true
}
浏览器访问 http://192.168.113.129:9100/ 查看索引信息，可以看见索引默认被分片5个，并且有一个副本
点击“数据浏览”，会发现在node1上创建的索引为 index-demo，类型为 test 的相关信息
http://192.168.113.129:9100
http://192.168.113.130:9100
2. ELK Logstash deployment (operated on Apache nodes)
Logstash is generally deployed on servers whose logs need to be monitored. In this case, Logstash is deployed on the Apache server to collect the log information of the Apache server and send it to Elasticsearch
192.168.113.128
#1.更改主机名,防火墙记得关
hostnamectl set-hostname apache
#2.安装Apahce服务（httpd）
[root@apache ~]# yum -y install httpd
[root@apache ~]# systemctl start httpd
#3.安装Java环境
[root@apache ~]# yum -y install java
[root@apache ~]# java -version
#4.安装logstash
#上传软件包 logstash-5.5.1.rpm 到/opt目录下
[root@apache ~]# cd /opt
[root@apache opt]# rpm -ivh logstash-5.5.1.rpm
[root@apache opt]# systemctl start logstash.service
[root@apache opt]# systemctl enable logstash.service
#创建软链接，便于系统识别
[root@apache opt]# ln -s /usr/share/logstash/bin/logstash /usr/local/bin/
2.1 Test Logstash
Common options for Logstash commands:
Option Description
-f This option can be used to specify the configuration file of Logstash, and configure the input and output streams of Logstash according to the configuration file
-e is obtained from the command line, the input and output are followed by a string, which can be used as the configuration of Logstash (if it is empty, stdin is used as input and stdout is used as output by default)
-t test configuration file is correct, then exit
Define input and output streams:
#输入采用标准输入，输出采用标准输出（类似管道）
# -e 是将后而的字符当作配苦文件.写入标准输入，输出标准输出
[root@apache opt]# logstash -e 'input { stdin{} } output { stdout{} }'
......
www.baidu.com
2022-08-29T12:05:13.498Z apache www.baidu.com
www.sina.com.cn
2022-08-29T12:05:27.352Z apache www.sina.com.cn
##执行 ctrl+c 退出
#使用 rubydebug 输出详细格式显示，codec 为一种编解码器
#rubydebug显示详细输出，codec为一种解码器，将输入的内容，解码后进行详细输出
[root@apache opt]# logstash -e 'input { stdin{} } output { stdout{ codec=>rubydebug } }'
......
www.baidu.com
{
"@timestamp" => 2022-08-29T12:10:19.732Z,
"@version" => "1",
"host" => "apache",
"message" => "www.baidu.com"
}
#使用 Logstash 将信息写入 Elasticsearch 中
logstash -e 'input { stdin{} } output { elasticsearch { hosts=>["192.168.113.129:9200"] } }'
输入 输出 对接
......
www.baidu.com #键入内容（标准输入）
www.sina.com.cn #键入内容（标准输入）
www.google.com #键入内容（标准输入）
//结果不在标准输出显示，而是发送至 Elasticsearch 中，可浏览器访问 http://192.168.113.129:9100/ 查看索引信息和数据浏览
#真机访问，查看索引信息
打开浏览器 输入http://192.168.113.129:9100/ 查看索引信息
2.2 Define the logstash configuration file
The Logstash configuration file basically consists of three parts: input, output and filter (optional, choose to use as needed
#格式如下：
input {
...}
filter {
...}
output {
...}
#在每个部分中，也可以指定多个访问方式。例如，若要指定两个日志来源文件，则格式如下：
input {
file {
path =>"/var/log/messages" type =>"syslog"}
file {
path =>"/var/log/httpd/access.log" type =>"apache"}
}
#修改 Logstash 配置文件，让其收集系统日志/var/log/messages，并将其输出到 elasticsearch 中
[root@apache opt]# chmod +r /var/log/messages
[root@apache opt]# ll /var/log/messages
-rw-r--r--. 1 root root 2154749 8月 29 20:21 /var/log/messages
[root@apache opt]# vim /etc/logstash/conf.d/system.conf
input {
file{
path =>"/var/log/messages" #指定要收集的日志的位置
type =>"system" #自定义日志类型标识
start_position =>"beginning" #表示从开始处收集
}
}
output {
elasticsearch {
#输出到 elasticsearch
hosts => ["192.168.113.129:9200"] #指定 elasticsearch 服务器的地址和端口
index =>"system-%{+YYYY.MM.dd}" #指定输出到 elasticsearch 的索引格式
}
}
#####################################################################
input {
file{
path =>"/var/log/messages"
type =>"system"
start_position =>"beginning"
}
}
output {
elasticsearch {
hosts => ["192.168.113.129:9200"]
index =>"system-%{+YYYY.MM.dd}"
}
}
[root@apache opt]# systemctl restart logstash ##重启服务
#浏览器访问 http://192.168.113.129:9100/ 查看索引信息
3. ELK Kibana deployment (operated on Node1 node)
A single machine operates 192.168.113.129 (Node1)
#1.安装 Kiabana
#上传软件包 kibana-5.5.1-x86_64.rpm 到/opt目录
cd /opt
[root@node1 opt]# rpm -ivh kibana-5.5.1-x86_64.rpm
#2.设置 Kibana 的主配置文件
[root@node1 kibana]# cp /etc/kibana/kibana.yml{,.bak} ##备份一下
[root@node1 kibana]# vim /etc/kibana/kibana.yml
--2--取消注释，Kiabana 服务的默认监听端口为5601
server.port: 5601
--7--取消注释，设置 Kiabana 的监听地址，0.0.0.0代表所有地址
server.host: "0.0.0.0"
--21--取消注释，设置和 Elasticsearch 建立连接的地址和端口
elasticsearch.url: "http://192.168.113.129:9200"
--30--取消注释，设置在 elasticsearch 中添加.kibana索引
kibana.index: ".kibana"
#3.启动 Kibana 服务
[root@node1 kibana]# systemctl start kibana.service
[root@node1 kibana]# systemctl enable kibana.service
[root@node1 kibana]# netstat -natp | grep 5601
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 50325/node
#4.验证 Kibana
浏览器访问 http://192.168.113.129:5601
第一次登录需要添加一个 Elasticsearch 索引：
Index name or pattern
//输入：system-* #在索引名中输入之前配置的 Output 前缀“system”
- 单击 “create” 按钮创建，单击 “Discover” 按钮可查看图表信息及日志信息。
- 数据展示可以分类显示，在“Available Fields”中的“host”，然后单击 “add”按钮，可以看到按照“host”筛选后的结果
#5.将 Apache 服务器的日志（访问的、错误的）添加到 Elasticsearch 并通过 Kibana 显示 （192.168.113.128机器操作，也就是在装了apache机器）
[root@node1 opt]# vim /etc/logstash/conf.d/apache_log.conf
input {
file{
path => "/etc/httpd/logs/access_log"
type => "access"
start_position => "beginning"
}
file{
path => "/etc/httpd/logs/error_log"
type => "error"
start_position => "beginning"
}
}
output {
if [type] == "access" {
elasticsearch {
hosts => ["192.168.113.129:9200"]
index => "apache_access-%{+YYYY.MM.dd}"
}
}
if [type] == "error" {
elasticsearch {
hosts => ["192.168.113.129:9200"]
index => "apache_error-%{+YYYY.MM.dd}"
}
}
}
[root@apache opt]# cd /etc/logstash/conf.d/
[root@apache conf.d]# /usr/share/logstash/bin/logstash -f apache_log.conf
Access http://192.168.113.129:5601 with a browser to log in to Kibana, click the "Create Index Pattern" button to add an index, enter the previously configured Output prefix apache_access-* in the index name, and click the "Create" button. Add the apache_error-* index in the same way
Also add apache_error
Select the "Discover" tab, select the apache_access-*, apache_error-* indexes just added in the middle drop-down list, and you can view the corresponding charts and log information
4. Filebeat+ELK deployment
Environment configuration
Host IP address installation package/software/tools
Node1 node (2C/4G) node1/192.168.113.129 Elasticsearch Kibana
Node2 node (2C/4G) node2/192.168.113.130 Elasticsearch
Apache node apache/192.168.113.128 Logstash Apache
Filebeat node filebeat/192.16 .113.126 Filebeat
192.168.113.126
[root@filebeat ~]# systemctl stop firewalld
[root@filebeat ~]# systemctl disable firewalld
[root@filebeat ~]# setenforce 0
#1.安装 Filebeat
#上传软件包 filebeat-6.6.0-linux-x86_64.tar.gz 到/opt目录
[root@filebeat opt]# tar zxvf filebeat-6.6.0-linux-x86_64.tar.gz
[root@filebeat opt]# mv filebeat-6.6.0-linux-x86_64 /usr/local/filebeat
#2.设置 filebeat 的主配置文件
[root@filebeat opt]# cd /usr/local/filebeat/
[root@filebeat filebeat]# cp filebeat.yml{,.bak} ##备份
[root@filebeat filebeat]# vim filebeat.yml
......
filebeat.prospectors:
- type: log #21行，指定 log 类型，从日志文件中读取消息，默认是开启的
enabled: true #24行，fales 改为 ture，开始日志
paths: #27行 下添加
- /var/log/messages #指定监控的日志文件
- /var/log/*.log
fields: #49行添加以下内容，可以使用 fields 配置选项设置一些参数字段添加到 output 中，注每次冒号敲完后，要空格后再写东西不然回报错，因为是默认格式
service_name: filebeat
log_type: log
service_id: 192.168.113.126 ##地址是安装 Filebeat的机器
--------------Elasticsearch output-------------------
(151行到164内没注释的全部注释掉)
----------------Logstash output---------------------
output.logstash: #165行取消注释
hosts: ["192.168.113.128:5044"] #167行取消注释，指定 logstash（安装Apache的机器） 的 IP 和端口
......
#3.启动 filebeat
[root@filebeat filebeat]# pwd #要在绝对路径下启动
/usr/local/filebeat
[root@filebeat filebeat]# ./filebeat -e -c filebeat.yml
#4.在 Logstash 组件所在节点上新建一个 Logstash 配置文件（192.168.113.128）
[root@apache ~]# cd /etc/logstash/conf.d/
[root@apache conf.d]# vim logstash.conf
input {
beats {
port => "5044"
}
}
output {
elasticsearch {
hosts => ["192.168.113.129:9200"] ##指定的es主机ip因为要存储
index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}
#启动 logstash
[root@apache conf.d]# pwd
/etc/logstash/conf.d
[root@apache conf.d]# logstash -f logstash.conf
##浏览器验证测试 192.168.113.129:9100
5. Access http://192.168.113.129:5601 with the browser to log in to Kibana, click the "Create Index Pattern" button to add the index "filebeat-*", click the "create" button to create, click the "Discover" button to view the chart information and log information
