Filebeat+zookeeper+kafka+ELK

The principle of filebeat

Filebeat works like this: when you start Filebeat, it starts one or more inputs that it looks for in the location specified for log data. For every log that Filebeat finds, Filebeat starts a collector. Each collector reads a single log for new content and sends the new log data to libbeat, which will aggregate the events and send the aggregated data to the output configured for Filebeat

Filebeat structure: Consists of two components, inputs (inputs) and harvesters (collectors), which work together to track files and send event data to the output you specify. The harvester is responsible for reading the contents of a single file. The harvester reads each file line by line and sends the content to the output. Start a harvester for each file. The harvester is responsible for opening and closing files, which means that file descriptors remain open while the harvester is running. If a file is deleted or renamed while it is being collected, Filebeat will continue to read the file. A side effect of this is that the space on the disk is reserved until the harvester shuts down. By default, Filebeat keeps files open until close_inactive is reached

Kafka 简介

Kafka is a message queue, which is mainly used to process message queues in the state of large amounts of data , and is generally used for log processing. Since it is a message queue, it Kafkaalso has the corresponding characteristics of the message queue.

The benefits of kafka message queues

Application decoupling

buffer

Traffic clipping

Asynchronous processing

Introduction to zookeeper

It is a distributed service framework and a sub-project of Apache Hadoop. It is mainly used to solve some data management problems often encountered in distributed applications, such as: unified naming service, state synchronization service, cluster management, distributed Management of application configuration items, etc.

Equivalent to file system + notification mechanism

---

Build Filebeat+zookeeper+kafka+ELK

CPU name	ip	system	components
node1	192.168.100.142	centos7	Elasticsearch,Elasticsearch-head, Kiabana
node2	192.168.100.140	centos7	Elasticsearch,Elasticsearch-head
apache	192.168.100.141	centos7	httpd, logstash,
filebeat1	192.168.100.145	centos7	zookeeper,kafka
filebeat2	192.168.100.134	centos7	zookeeper,kafka
filebeat3	192.168.100.144	centos7	zookeeper,kafka

This experiment was carried out in the last ELK experiment, so I will not demonstrate the deployment of ELK. If you need it, you can go to my last blog, which introduces ELK and deployment steps in detail.

Closer to home, let's start this experiment

1. Install zookeeper kafka (operate in fiebeat1 - 3)

(1) Unzip and install the zookeeper soft armor package

[root@bogon opt]# tar zxvf apache-zookeeper-3.5.7-bin.tar.gz 
[root@bogon opt]# mv apache-zookeeper-3.5.7-bin /usr/local/zookeeper-3.5.7     
[root@bogon opt]# cd /usr/local/zookeeper-3.5.7/
[root@bogon conf]# cp zoo_sample.cfg zoo.cfg
[root@bogon conf]# vim zoo.cfg 

[root@bogon conf]# cd ..
[root@bogon zookeeper-3.5.7]# mkdir data logs
[root@bogon zookeeper-3.5.7]# echo 1 > data/myid
## 给每个机器指定对应的节点号 

(2) Start zookeeper

View the three node zookeeper status

---

(2). Install kafka (operate in fiebeat1 - 3)

[root@bogon bin]# cd /opt/
[root@bogon opt]# tar zxvf kafka_2.13-2.7.1.tgz 
[root@bogon opt]# mv kafka_2.13-2.7.1 /usr/local/kafka

(3) Modify the configuration file

[root@bogon opt]# cd /usr/local/kafka/
[root@bogon kafka]# cd config/
[root@bogon config]# vim server.properties 

(4) Add relevant commands to the system environment

[root@bogon kafka]# vim /etc/profile
[root@bogon kafka]# source /etc/profile

(5) Enable kafka

create topic

（192.168.100.145）

test topic

Release message (192.168.100.145)

Consuming messages (192.168.100.145)