Zun Du and Fake Du, are Chengdu programmers also 996?

1. Why use ES Kibana

The most important thing in offline data testing is data verification. One part needs to test the correctness of the data stored in es, and the other part needs to verify the correctness of the interface's value logic from es. In order to verify the correctness of the es value logic, you need to use Kibana, which can help test students execute es data queries more quickly and efficiently, greatly improving testing efficiency.

2. What is ES and Kibana

What we usually call ELK refers to Elasticsearch, Logstash and Kibana. The combination of these three technologies is a very clever design in the field of big data. It is a very typical MVC idea, model persistence layer, view layer and control layer. .

Logstash serves as the control layer and is responsible for collecting and filtering data.

Elasticsearch serves as the data persistence layer and is responsible for storing data. It is a real-time distributed storage, search, and analysis engine that is suitable for all types of data, including text, numbers, geospatial, structured and unstructured data. Compared with Mysql, it is better at retrieving millions of data.

The Kibana we are talking about this time plays the role of the view layer. It is a web interface for log analysis provided by Logstash and ElasticSearch. It can be used to perform various operations such as efficient search, visualization, and analysis of logs. It is an open source data analysis and visualization platform that can be used together with the Elasticsearch search engine. You can use Kibana to search, view, and interact with data stored in the Elasticsearch index. You can also use Kibana to display data in charts, tables, maps, etc., to achieve advanced data analysis and visualization purposes.

This time we will introduce the use of SQL query and Visualize (visualized data) functions in Dev Tools in Kibana.

3. SQL query in Dev Tools

The difference between ES and MYSQL:

Mysql	ElasticSearch
Database	index
Table	Type
Row	Document
Column	Field
Schema	Mapping
Index	Everything is indexed
SQL	Query DSL
SELECT * FROM ...	GET http://...
UPDATE table SET...	PUT http://...

The insert, delete, update, and select operations in the database are equivalent to the PUT/POST, delete, update, and GET operations in ES. For these complex queries, es can be implemented using Query DSL.

POST /index111/_search
{
  "query": {
    "bool":
        {
        "must":
          [
            {"term":{"user": "张三"}},
            {"term":{"timeStamp": "2022-08-04 00:00:00"}}
          ]
        }
  }
}

But in comparison, we are more familiar with sql statements, so es also provides the development of sql statements, allowing us to implement ES queries through sql statements. Before es version 6.3, the development of sql statements was not supported. If you need to use sql statements to develop es data queries, then we need to manually install the plug-in ourselves.

But after version 6.3, es comes with a sql plug-in installed, which is integrated under _xpack. We can directly implement data query in es through sql statements.

The following are the steps to query using sql statements:

1. Enter Dev Tools - Console (Console)

POST /_xpack/sql
{

  "query": "select * from index111 "

}

1. Enter the above statement, the default return format is json

2. You can customize the returned format. If you want to return text format, add ?format=txt after /_xpack/sql

More complex SQL statements for classification and aggregation calculations are also supported.

Using this function can help test students query ES data more conveniently and quickly, and improve testing efficiency.

4. Visualize (visualized data) use

Using the above sql statement, the data can be queried quickly, but the sql statement cannot be saved. When the next verification returns the content and queries the same data, the sql statement needs to be entered again, and it may be necessary to re-understand the query logic. The problem of extending the return time. Using Visualize (visualized data), the es index content can be aggregated, saved and displayed through various methods such as charts , so that the es data can be browsed more directly. At the same time, product business can also be used for data analysis and creation of data dashboards .

The following are the steps to create a visual chart for a single es index:

1. Enter Visualize-click to create a new view

2. Select which chart style you want to display the data in

3. Select the es index of the application

Note that the index mode needs to be established for the target index first, otherwise the index will not be selected when creating the visualization graph. Path: Management - Create index pattern.

4. Enter the chart setting interface and set the desired data: Metrics and Buckets. Metrics can set aggregation.

You can use Elasticsearch bucket aggregations to specify what information appears in a chart. Bucket aggregation simply divides documents that match your search criteria into different categories, also called buckets, which is the x-axis. For example: data classification analysis data, by using bucket aggregation, you can create multiple classifications and see the data summary under each classification.

Metric has multiple aggregation methods: Count, Sum, Top Hit, Unique Count, which can be selected as needed.

Buckets set what information in your dataset will be retrieved.

If we want to see the total score of each person, Metric can add a score, the measurement unit is SUM, and Buckets can add the name of the person. If we also want to see the scores of each person in other dimensions, we can add another sub-bucket and click on the bottom left add sub buckets

5. After the setting is completed, please note that you need to select the application time range in the upper right corner. The default is the past 15 minutes. There may be no data. After saving, there is no need to use SQL query next time. You can directly view the corresponding indicators of the table. Data.

When displaying a chart, some fields need to be mapped to other fields for display, or statistical data needs to be calculated and aggregated on several fields in the table, such as summing two of the fields. In this case, Management/Index Patterns/Script fields can be used Script fields are used to process the original fields, map the original fields to other data and add a column of script fields for calculation and summation to increase the readability of the chart. For example: Convert the name field to other person name mapping. But if you use script field, you need to ensure the correctness and executability of your script to avoid query error blocking due to script problems.

Script field can use painless language, with syntax like groovy, similar to Java. The following is an example field mapping script

def path = doc['user'].value;
String newUser;
if (path != null) {
    path =path.toString();
    if (path =='张三'){
        newUser = '张同学';
    }
    if (path=='李四'){
        newUser = '李同学';
    }
}
return org

As you can see, Visualize bucket aggregation uses script processing fields, user fields, etc. to directly map other text, which is convenient and intuitive.

5. Summary

The above is the introduction and usage steps of ES Kibana's devtools sql query and Visualize. Kibana also has many other powerful functions. Clever use of these tools can help test students improve testing efficiency and help product business students conduct more advanced data Analysis, I hope this article can help everyone understand and use Kibana better.

Author: JD Logistics Jiang Wenwen

Source: JD Cloud Developer Community Ziyuanqishuo Tech Please indicate the source when reprinting

Analysis of the practical application of ElasticSearch Kibana in testing work | JD Logistics Technology Team

1. Why use ES Kibana

2. What is ES and Kibana

3. SQL query in Dev Tools

4. Visualize (visualized data) use

5. Summary

Guess you like