1. understand the role of various components
Filebeat is a log file shipping tool After installing the client on your server, filebeat monitors log directory or specified log file, tracking read these files (track file changes, keep reading)
Kafka used to live is a high throughput distributed publish-subscribe messaging system that can handle all the action streaming data consumer-scale site
Logstash is a pipeline with real-time data transmission capabilities, is responsible for the transmission of data from the input end of the pipeline to pipe output end; this tube at the same time also allows you according to their needs in the middle with strainer, Logstash offers many powerful features inside the filter to meet your various application scenarios
ElasticSearch it provides a distributed multi-user capability the full-text search engine, based on RESTful web interfaces
Kibana is ElasticSearch user interface
In practical application scenario, in order to meet the real-time large data retrieval scenario, using Filebeat to monitor the log file, Kafka as an output terminal Filebeat of, Kafka post received in real time Filebeat to Logstash as an output terminal, to Logstash data may not yet be we want to format or data for a particular business, then you can order ElasticSearch as an output data format output by some Logstash after a plug-in filters the data and finally to want to ElasticSearch data can be distributed wealth searched
2. Install and deploy various components, and the configuration profile associated with the various components of the
Downloading the installation package and extract the individual components (these components are available for download in the official website Apache)
Filebeat installation
Extract the directory after the Filebeat
Filebeat configuration is very simple, and only need to configure the full path to monitor the output of the log file IP, general default (default profile filebeat.yml) elasticsearch to output, as output configuration
An output terminal arranged
这里我们不直接输出到ElasticSearch，而是kafka，所以需要配置 beat-kafka.yml，输入端配置如下
输出端配置
因为是输出到kafka，所以需要制定kafka的topic
Kafka的安装
在Filebeat已经配置输出端为kafka的配置，这里kafaka不需要配置任何东西，解压直接用，以下是kafka初用的几个必须的命令,具体查看kafka的官方API
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning现在启动Filebeat:
./filebeat -e -c beat-kafka.yml -d "publish"这里需要说明一点，不同的输出端可以配置不同的.yml文件，所以这里的 beat-kafka.yml文件是kafka对应的配置
以上命令输出没有报错并有如下信息
且可以看到你的监控文件的log的信息字段证明不存在问题
这是kafka的consumer会有如下信息
message就是文件的具体信息
Logstash的安装
Logstash也不要做太多的配置，从搭建Filebeat和Kafka时，已经实现了数据的流通，现在要完成的事是然流到kafka的数据流到logstash，logtash也需要输出，将ElasticSearch作为输出端
对于kafka不同的数据建议logstash用不同的通道去接
一下是配置的两个不同的通道
启动logstash
bin/logstash -f first-pipeline.conf --config.reload.automatic --path.data=/home/hadoop/logstash--path.data=/home/hadoop/logstash是指存放数据的路径
ElasticSearch配置和安装
如果你不要更改ip的话那就不需要任何配置，直接启动
sh bin/elasticsearch通过elasticsearch查询看整个架构是否完整数据是否正确流通
查看索引
curl '192.168.1.20:9200/_cat/indices?v'
根据索引查询数据
curl -XGET '192.168.1.20:9200/test_index/_search?pretty'
这就证明我们整个架构是正确的且数据流通无误
Kibana的配置和安装
Kibana只是ElasticSearch的图形化界面，启动即用