[Switch] ELK (ElasticSearch, Logstash, Kibana) builds a real-time log analysis platform

This article is reproduced from: https://my.oschina.net/itblog/blog/547250/

Abstract: In the Log4j+Kafka researched some time ago, some people suggested that the logs collected by Kafka should be stored in ES (ElasticSearch, an open source distributed search engine based on Apache Lucene) for easy search and analysis. At that time, I found that the ELK (ElasticSearch, Logstash, Kibana) platform could just realize the functions of log collection, log search and log analysis at the same time, so I went to learn it again. Later, it was found that if you use these three, you can no longer use Kafka to collect logs, and Logstash can help us do it. Of course, although Logstash also supports the use of Kafka as data source input, there is no need to increase the complexity of the system by using these three.

ELK platform introduction

When searching for ELK information, I found this article to be better, so I excerpted a short paragraph:

The following content is from: http://baidu.blog.51cto.com/71938/1676798

Logs mainly include system logs, application logs and security logs. System operation and maintenance and developers can learn server software and hardware information through logs, check errors in the configuration process and the causes of errors. Frequent analysis of the log can understand the server's load, performance security, so as to take timely measures to correct errors.

Usually, logs are distributed and stored on different devices. If you manage dozens or hundreds of servers, you're still looking at logs using the traditional method of logging into each machine in turn. Does this feel cumbersome and inefficient. It is imperative that we use centralized log management, such as the open source syslog, to collect and aggregate logs from all servers.

After centralized management of logs, log statistics and retrieval have become a more troublesome thing. Generally, we can use Linux commands such as grep, awk, and wc to achieve retrieval and statistics, but for higher requirements such as query, sorting, and statistics And the huge number of machines still use this method is inevitably a bit powerless.

The open source real-time log analysis ELK platform can perfectly solve the above problems. ELK consists of three open source tools, ElasticSearch, Logstash and Kiabana. Official website: https://www.elastic.co/products

• Elasticsearch is an open source distributed search engine. Its features are: distributed, zero configuration, automatic discovery, index automatic sharding, index replication mechanism, restful style interface, multiple data sources, automatic search load, etc.

• Logstash is a completely open source tool that can collect, filter, and store your logs for later use (eg, search).

• Kibana is also an open source and free tool, which Kibana can provide Logstash and ElasticSearch with a friendly web interface for log analysis, which can help you aggregate, analyze and search important data logs.

---------------------------- End of excerpt------------------- ------------

Draw a schematic diagram of ELK work:

As shown in the figure: Logstash collects logs generated by AppServer and stores them in the ElasticSearch cluster, while Kibana queries data from the ES cluster to generate graphs and returns them to the Browser.

 

ELK platform construction

system environment

System: Centos release 6.7 (Final)

ElasticSearch: 2.1.0

Logstash: 2.1.1

Kibana: 4.3.0

Java: openjdk version  "1.8.0_65"

Note: Since the operation of Logstash depends on the Java environment, and Logstash 1.5 or later is not lower than java 1.7, it is recommended to use the latest version of Java. Because we only need the Java runtime environment, we can only install JRE, but I still use JDK here, please search and install by yourself.

ELK download: https://www.elastic.co/downloads/

 

ElasticSearch

Configure ElasticSearch:

tar -zxvf elasticsearch-2.1.0.tar.gz
cd elasticsearch-2.1.0

Install the Head plugin (Optional):

./bin/plugin install mobz/elasticsearch-head

Then edit the ES configuration file:

vi config/elasticsearch.yml

Modify the following configuration items:

cluster.name=es_cluster
node.name=node0
path.data=/tmp/elasticsearch/data
path.logs=/tmp/elasticsearch/logs
#当前hostname或IP，我这里是centos2
network.host=centos2
network.port=9200

Leave other options as default, then start ES:

./bin/elasticsearch

It can be seen that its transmission port with other nodes is 9300, and the port for accepting HTTP requests is 9200.

Use ctrl+C to stop. Of course, ES can also be started as a background process:

./bin/elasticsearch &

Then you can open the page localhost:9200 and you will see the following:

Returns the cluster_name and name of the configuration, as well as the installed ES version and other information.

The head plug-in just installed is a plug-in that uses a browser to interact with the ES cluster. It can view the cluster status, the doc content of the cluster, perform searches and ordinary Rest requests. Now you can also use it to open the localhost:9200/_plugin/head page to view the ES cluster status:

As you can see, now, there is no index and no type in the ES cluster, so these two items are empty.

 

Logstash

Logstash functions as follows:

In fact, it is just a collector , we need to specify Input and Output for it (of course Input and Output can be multiple). Since we need to output the Log4j log in the Java code to ElasticSearch, the Input here is Log4j, and the Output is ElasticSearch.

Configure Logstash:

tar -zxvf logstash-2.1.1.tar.gz
cd logstash-2.1.1

Write a configuration file (name and location can be arbitrary, here I put it in the config directory, named log4j_to_es.conf):

mkdir config
vi config/log4j_to_es.conf

Enter the following:

# For detail structure of this file
# Set: https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html
input {
  # For detail config for log4j as input, 
  # See: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
  log4j {
    mode => "server"
    host => "centos2"
    port => 4567
  }
}
filter {
  #Only matched data are send to output.
}
output {
  # For detail config for elasticsearch as output, 
  # See: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
  elasticsearch {
    action => "index"          #The operation on ES
    hosts  => "centos2:9200"   #ElasticSearch host, can be array.
    index  => "applog"         #The index to write data to.
  }
}

The logstash command has only 2 parameters:

So use the agent to start it (use -f to specify the configuration file):

./bin/logstash agent -f config/log4j_to_es.conf

At this point, we can use Logstash to collect logs and save them to ES. Let's take a look at the project code.

 

Java project

As usual, look at the project structure diagram first:

pom.xml, very simple, only uses the Log4j library:

<dependency>
    <groupId>log4j</groupId>
    <artifactId>log4j</artifactId>
    <version>1.2.17</version>
</dependency>

log4j.properties, output Log4j logs to SocketAppender , because the official website says:

log4j.rootLogger=INFO,console

# for package com.demo.elk, log would be sent to socket appender.
log4j.logger.com.demo.elk=DEBUG, socket

# appender socket
log4j.appender.socket=org.apache.log4j.net.SocketAppender
log4j.appender.socket.Port=4567
log4j.appender.socket.RemoteHost=centos2
log4j.appender.socket.layout=org.apache.log4j.PatternLayout
log4j.appender.socket.layout.ConversionPattern=%d [%-5p] [%l] %m%n
log4j.appender.socket.ReconnectionDelay=10000

# appender console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d [%-5p] [%l] %m%n

Note: The port number here needs to be the same as the port number that Logstash listens to, here is 4567.

Application.java, use Log4j's LOGGER to print logs:

package com.demo.elk;

import org.apache.log4j.Logger;

public class Application {
    private static final Logger LOGGER = Logger.getLogger(Application.class);
    public static void main(String[] args) throws Exception {
        for (int i = 0; i < 10; i++) {
            LOGGER.error("Info log [" + i + "].");
            Thread.sleep(500);
        }
    }
}

View ES status and content with the Head plugin

Run Application.java and first look at the console output (of course, this output is only for verification, it is also possible not to output to the console):

Let's take a look at the head page of ES:

Switch to the Browser tab:

Clicking on a document (doc) will display all the information of the document:

As you can see, in addition to the basic message field is our log content, Logstash also adds many fields for us. And it's clearly stated in https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html :

The ES's Head plugin is used above to observe the status and data of the ES cluster, but this is just a simple page for interacting with ES, and cannot generate reports or charts. Next, use Kibana to perform searches and generate charts .

 

Kibana

Placement Kibana:

tar -zxvf kibana-4.3.0-linux-x86.tar.gz
cd kibana-4.3.0-linux-x86
vi config/kibana.yml

Modify the following items (because it is a stand-alone version, the value of host can also be replaced by localhost, which is only for demonstration):

server.port: 5601
server.host: “centos2”
elasticsearch.url: http://centos2:9200
kibana.index: “.kibana”

Dynamic kibana:

./bin/kibana

Open this address with a browser:

In order to use Kibana subsequently, you need to configure at least one Index name or Pattern, which is used to determine the Index in ES during analysis. Here I enter the previously configured Index name applog, Kibana will automatically load the field of the doc under the Index, and automatically select the appropriate field for the time field in the icon:

After clicking Create, you can see that the configured Index name has been added to the left:

Next, switch to the Discover tab. Note that the upper right corner is the time range of the query. If no data is found, then you may need to adjust this time range. Here I choose Today:

Next, you can see the data in ES:

Perform a search to see what:

Click the save button on the right to save the query as search_all_logs. Next, go to the Visualize page, click to create a new vertical bar chart (Vertical Bar Chart), and then select the query search_all_logs just saved, after that, Kibana will generate a bar chart similar to the following figure (there are only 10 logs, and they are in the same time period) , rather ugly, but enough to illustrate the problem :) ):

You can set various parameters of the graph on the left, click the Apply Changes button, and the graph on the right will be updated. Similarly, other types of graphics can be updated in real time.

Click Save on the right to save this graph and name it search_all_logs_visual. Next switch to the Dashboard page:

Click the New button, select the search_all_logs_visual graph you just saved, and the panel will display the graph:

If there is more data, we can add multiple charts on the Dashboard page according to business needs and concerns: column charts, line charts, maps, pie charts, etc. Of course, we can set the update frequency and let the chart update automatically:

If the set time interval is short enough, it is very close to real-time analysis.

At this point, the ELK platform deployment and basic testing have been completed.

 

refer to:

http://baidu.blog.51cto.com/71938/1676798

http://blog.csdn.net/cnweike/article/details/33736429