logstash kafka read from the input format of the log es json

Boss allow researchers ELK, after several days of exploration, and finally the process run through.
Version:
Kafka used to live 0.10.2.0
logstash 5.3.0
elasticsearch 5.3.0
modify logstash configuration file
vi logstash.conf

input {
    kafka{
        bootstrap_servers => ["10.10.10.10:9092"]
        group_id => "es"
        topics => ["myTest"]
        codec => json {
                 charset => "UTF-8"
        }
    }
}

output {
        # 处理后的日志落到本地文件
        file {
                path => "/data/logstash/test.log"
                flush_interval => 0
       }
       # 处理后的日志入es
       elasticsearch {
        hosts => ["10.10.10.20:9200"]
        index => "test"
        id => "my_plugin_id"
        document_id => "%{userid}"
        document_type => "mytype"
       }
}

Wherein the input of the codec specified => json json can parse the string kafka
output index specified in the index, document_type specified type, document_id specify the primary key
if there are duplicate primary key will automatically be overwritten!

Published 118 original articles · won praise 25 · Views 150,000 +

Guess you like

Origin blog.csdn.net/lhxsir/article/details/95991529