(18) Introduction to go-micro microservice ELK

1 What is ELK

ELK is the abbreviation of three [open source software], respectively: Elasticsearch, Logstash, Kibana , they are all open source software, and a new Beats is added .

• Elasticsearch: It is a distributed search engine that provides three functions of collecting, analyzing and storing data

  • Main features Real-time analysis Distributed real-time file storage, and each field is indexed Document-oriented, all objects are documents High availability, easy to expand, support cluster (Cluster), fragmentation and replication (Shards and Replicas).

• Logstash: It is mainly a tool for collecting, analyzing and filtering logs

  • Virtually any data can be accessed
  • Can be combined with various external applications
  • Support elastic expansion

• Kibana: Provides a friendly web interface that can help aggregate analysis and search data

  • KibanaIt is a web platform based on Apachean open source protocol and JavaScriptwritten in a language that Elasticsearchprovides analysis and visualization. It can Elasticsearchlookup in the index, interact with data, and generate table graphs of various dimensions.

• Beats: is a lightweight log collection and processing tool (Agent)

  • ELKA new member of the protocol stack, a lightweight open source log file data collector. serverAfter installing t on the server where log data needs to be collected Filebea, and specifying the log directory or log file, Filebeatthe data can be read, sent to Logstashfor analysis, or directly sent to Elasticsearchfor centralized storage and analysis.

Two Beats' Six Tools

• Packetbeat: network data (collect network traffic data)

• Metricbeat: Metrics (collects system, process and filesystem level data)

• Filebeat: log files (collect file data)

• Winlogbeat : windows event log (collects Windows event log data)

• Auditbeat: audit data (collect audit logs)

• Heartbeat: Runtime monitoring (collecting data when the system is running)

Features of the Three ELK System

• Powerful collection capability: able to collect log data from multiple sources

• Stable transmission capability: able to transmit log data to the central system

• Distributed storage: log data can be stored according to business needs, and can be dynamically expanded

• Analysis & Display & Reminder: With data analysis ability, data display ability and alarm ability

Four ELK+beats system architecture

Logstash obtains data from various data sources (such as log files, standard input Stdin, etc.) through Filebeat, and then processes the data through the filtering plug-in, and then outputs it to Elasticsearch through the Elasticsearch output plug-in, and displays it through Kibana.

Five ELK advantages

• Flexible processing: elasticsearch is a real-time full-text index with powerful search functions

• The configuration is relatively simple: all elasticsearch uses JSON interfaces, logstash uses module configuration, and kibana's configuration file part is simpler.

• Efficient retrieval performance: Based on the excellent design, although each query is real-time, it can also achieve second-level response to tens of billions of data queries.

• Cluster linear expansion: both elasticsearch and logstash can be flexibly expanded linearly

• The front-end operation is good: the front-end design of kibana is more beautiful, and the operation is simple. On the Kibana interface, you only need to click the mouse to complete the search and aggregation functions and generate a dazzling dashboard.

six last

• So far, the introduction of go-micro microservice ELK has been officially completed.

• Next, I started writing the code for filebeat to collect logs. I hope you pay attention to bloggers and columns, and get the latest content as soon as possible. Every blog is full of dry goods.