kafka + elk log collection system

I have nothing to do. When I used esit before, I saw elkthis thing and didn't know what it was. Later, I found out that it was a thing that collects logs in a unified way. I just had this requirement recently, so I will study it. At present, I only built a simple The single-point project, almost the architecture diagram is like this. Of course it is also possible to remove kafkathis layer.

Because I was studying before log4j2, and then saw the kafkaaccessories inside, I wanted to log4j2send it directly to kafkait by the way.

So roughly the following architecture diagram

write picture description here

Of course, it is also possible to appconnect directly Logstash, but if it is distributed on different machines, you need to configure multiple LogStash. Of course, it is also possible to have a high level of proficiency.

The version of my accessories is a little different, it's in the windowsenvironment.

Accessories	Version
log4j2	2.x.x
kafka	0.10.2.0
Logstash	6.1.3
ElasticSearch	5.4.1
Kibana	5.4.2

configuration file
log4j2.xml

<Appenders>
        <Kafka name="kafka" topic="log-test">
            <PatternLayout>
                <pattern>%d{yyyy-MM-dd HH:mm:ss} [%t] %-5level %logger{36} %L - %msg%n</pattern>
            </PatternLayout>
            <Property name="bootstrap.servers">localhost:9092</Property>
        </Kafka>
    </Appenders>

logstash.confinto logstashthe corresponding configdirectory

input {
      kafka {
        bootstrap_servers => ["127.0.0.1:9092"]
        group_id => "logstash1"
        client_id => "logstash1:::one"
        auto_offset_reset => "earliest"
        topics => ["log-test"]
   }
}

output {
  stdout {
   codec => rubydebug { }
  }

  elasticsearch {
    hosts => "127.0.0.1"
    index => "log-%{+YYYY.MM.dd}"
  }
}

Other plugin configurations are default configurations

The construction of ELK clusters has become overwhelming on the Internet.

start command

## kafka.bat
cd /d G:\kafka\kafka
.\bin\windows\kafka-server-start.bat .\config\server.properties

## es.bat
cd /d G:\elk\elasticsearch-5.4.1
.\bin\elasticsearch.bat

## logstash.bat
cd /d G:\elk\logstash-6.1.2\bin
logstash -f ../config/logstash.conf

## kibana.bat
cd /d G:\elk\kibana-5.4.2-windows-x86
./bin/kibana.bat

kafka + elk log collection system

Guess you like