[Teach you how to understand ELK] Data visualization tools and techniques

Yuxian: CSDN content partner, CSDN rising star mentor, rising star creator in the full stack field, 51CTO (Top celebrity + expert blogger), github open source enthusiast (go-zero source code secondary development, game back-end architecture https: //github.com/Peakchen)

ELK is a data processing and visualization platform composed of three open source components: Elasticsearch, Logstash and Kibana. The following is a detailed explanation of these components and an introduction to the use of related interfaces:

Elasticsearch

Elasticsearch is a distributed search and analysis engine based on the Lucene search engine. It can handle large-scale data, supports real-time search and analysis, and provides a RESTful API interface.

The main features of Elasticsearch include:

Distributed data storage and processing
Real-time search and analysis
Multi-tenant support
High Availability and Fault Tolerance
Geospatial search and analysis

Elasticsearch's RESTful API interface includes:

Index Management API
Document Management API
search API
Aggregation API
Management API

scenes to be used:

Real-time search and analysis
Log Management and Analysis
E-commerce website search and recommendation system

Logstash

Logstash is an open source data collector that collects, processes and transforms data from various sources. It supports multiple input and output formats and provides rich filter plug-ins.

Logstash’s main features include:

Data collection and transfer
Data Filtering and Transformation
Multiple input and output formats supported
Plug-in scalability

Logstash plug-ins include:

input plugin
filter plugin
output plugin

scenes to be used:

Log collection and processing
Data Transfer and Transformation
Event handling and forwarding

Kibana

Kibana is an open source data visualization tool that can be integrated with Elasticsearch to provide real-time data analysis and visualization capabilities. It supports multiple visualization types and provides rich query and analysis capabilities.

Kibana’s main features include:

Real-time data visualization and analysis
Multiple visualization types supported
Query and Analysis Functions
User management and permission control

Kibana visualization types include:

line chart
bar chart
pie chart
heat map
map

scenes to be used:

Real-time data visualization and analysis
Business data monitoring and analysis
Security incident analysis and response

Links to literature materials:

Elasticsearch official documentation: Elasticsearch Guide [8.9] | Elastic
Logstash official documentation: Logstash Reference [8.9] | Elastic
Kibana official documentation: Kibana Guide [8.9] | Elastic
Elastic Stack Chinese Community: Elastic Chinese Community
The Definitive Guide to Elasticsearch (Chinese version): Elasticsearch: The Definitive Guide | Elastic
Logstash plugin list: https://www.elastic.co/guide/en/logstash/current/plugins-list.html
Kibana plug-in center: https://www.elastic.co/guide/en/kibana/current/plugins.html

Here is an example implementation of data visualization using ELK:

data source

Use Logstash to collect access logs of a web application and send the log data to Elasticsearch.

Logstash configuration file example:

input {
  file {
    path => "/var/log/nginx/access.log"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "webapp-access-%{+YYYY.MM.dd}"
  }
}

Visualization panel

Create a visualization panel in Kibana that displays the number of requests per minute and response time.

Select the "Visualize" tab and then select the "Create a new visualization" button.
Select the "Line chart" visualization type and select the "webapp-access-*" index mode.
Add two aggregations in the visual designer, select the "Y-axis" tab, and select "Aggregation" as "Count" and "Average", corresponding to the number of requests and response time respectively.
Select the "X-axis" tab in the visual designer, and select "Date Histogram" as "Aggregation", "Field" as "@timestamp", and "Interval" as "1m".
Select the "Apply changes" button in the visual designer and preview the results.

report generation

Use Kibana's report generator to create a report in PDF format that contains the previous day's web application access.

Select the "Reporting" tab and then select the "Create a new report" button.
Select the "webapp-access-*" index mode in the report designer and select the "Line chart" visual component.
Select the "Export" tab in the report designer and select the "PDF" format.
Select the "Generate report" button in the report designer and wait for the report generation to complete.
Select the Download button in the report designer and download the generated report.

The above is an example implementation of using ELK for data visualization, including using Logstash to collect web application access logs and using Kibana to create visualization panels and reports.

[Teach you how to understand ELK] Data visualization tools and techniques

Guess you like