1. ELK application case
ELK deployment case:
This architecture is a bit complicated, so here is an interpretation of the architecture. This architecture diagram is divided into 5 layers from left to right. The functions and meanings of each layer are introduced as follows:
The first layer, data collection layer
The data collection layer is located on the leftmost business server cluster. Filebeat is installed on each business server for log collection, and then the collected original logs are sent to the Kafka+zookeeper cluster.
The second layer, the message queue layer
After the original log is sent to the Kafka+zookeeper cluster, it will be stored centrally. At this time, filbeat is the producer of the message, and the stored message can be consumed at any time.
The third layer, data analysis layer
As a consumer, Logstash will go to the Kafka+zookeeper cluster node to pull the original log in real time, then analyze, clean, and filter the obtained original log according to the rules, and finally forward the cleaned log to the Elasticsearch cluster.
The fourth layer, data persistent storage
After the Elasticsearch cluster receives the data sent by logstash, it performs operations such as writing to disk, building an index library, and finally stores the structured data on the Elasticsearch cluster.
The fifth layer, data query, display layer
Kibana is a visual data display platform. When there is a data retrieval request, it reads data from the Elasticsearch cluster, and then performs visual graphing and multi-dimensional analysis.