Use docker deployment elk log collection system

A, elk concept

ELK is Elasticsearch, Logstash, Kibana short, these three are the core suite, but not all.

Elasticsearch : real-time full-text search and analysis engine that provides collection, analysis, storage of data three functions. Elasticsearch REST is an open structure and the like JAVA API provides efficient search capabilities, scalable distributed system. It is built on top of Apache Lucene search engine library.

Logstash : to collect, analyze, log filtering tools. It supports almost any type of log, including system logs, error logs, and custom application log. It can receive log from many sources, these sources include the syslog, messaging (e.g. RabbitMQ) and the JMX, it is possible to output data in a variety of ways, including e-mail, and WebSockets Elasticsearch.

Kibana : Web-based visualization graphical interface for log data search, analysis and visualization is stored in Elasticsearch Index. It uses Elasticsearch REST interface to retrieve the data, not only allows users to create customized dashboard view their own data, but also allows them a special way to query and filter data.

Two, docker installation elk

1, pull mirroring operation 

docker pull sebp/elk

docker run -d -it --name elk -p 5601:5601 -p 9200:9200 -p 5044:5044 sebp/elk

5601 - Kibana web interfaces
9200 - Elasticsearch JSON Interface
5044 - Logstash log reception interface

logstash There are various ways to accept the data, as used herein, logback directly send logs to logstash tcp by way redis may also be used as a message queue to make a transit log data.

2, elasticsearch error resolved

You may encounter when elasticsearch startup error, reported elasticsearch user has permission to memory is too small, at least 262144.

Solution:
sysctl -w vm.max_map_count = 262144

View Results:
sysctl -a | grep vm.max_map_count, display vm.max_map_count = 262144

After modifying the above method, if the restart of virtual machines will fail. Finally, you can add a line in the /etc/sysctl.conf file
vm.max_map_count = 262144, can be permanently modified.

Three, tcp method to collect log

1, the configuration tcp send log

Add rely on services in pom.xml

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
</dependency>

In logback-spring.xml appender added, the transmission logs to logstash

<springProperty scope="context" name="springAppName" source="spring.application.name"/>    

<appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
    <destination>192.168.0.6:5044</destination>
        <!-- <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder" /> -->
        <!-- 日志输出编码 -->
        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <timeZone>UTC</timeZone>
                </timestamp>
                <pattern>
                    <pattern>
                        {
                        　　"severity": "%level",
                        　　"service": "${springAppName:-}",
                        　　"trace": "%X{X-B3-TraceId:-}",
                        　　"span": "%X{X-B3-SpanId:-}",
                       　　 "exportable": "%X{X-Span-Export:-}",
                        　　"pid": "${PID:-}",
                        　　"thread": "%thread",
                        　　"class": "%logger{40}",
                        　　"rest": "%message"
                        }
                    </pattern>
                </pattern>
            </providers>
        </encoder>
    </appender>

    <!-- 日志输出级别 -->
    <root level="INFO">
        <appender-ref ref="STDOUT" />
        <appender-ref ref="FILE"/>
        <appender-ref ref="logstash" />
    </root>

2, arranged to send log logstash elasticsearch

Since sebp / elk in the input logstash defaults filebeat, tcp mode used here, it is first necessary to modify the container into the elk input way for tcp. logstash will use the default configuration file etc / logstash / conf.d / in.

Start elk, into the container:

#docker exec -it elk /bin/bash

Enter /etc/logstash/conf.d/ configuration directory, modify the 02-beats-input.conf configuration file as follows:

input {    
    tcp {         
        port => 5044         
        codec => json_lines
    } 
} 
output{  
    elasticsearch { 
    　　hosts => ["localhost:9200"]
    } 
}

After editing quit, restart elk container

#docker restart elk

Finally, start the service, the log is sent to logstash in, and access to localhost: 5601 can enter kibana interface check.