Boss allow researchers ELK, after several days of exploration, and finally the process run through.
Version:
Kafka used to live 0.10.2.0
logstash 5.3.0
elasticsearch 5.3.0
modify logstash configuration file
vi logstash.conf
input {
kafka{
bootstrap_servers => ["10.10.10.10:9092"]
group_id => "es"
topics => ["myTest"]
codec => json {
charset => "UTF-8"
}
}
}
output {
# 处理后的日志落到本地文件
file {
path => "/data/logstash/test.log"
flush_interval => 0
}
# 处理后的日志入es
elasticsearch {
hosts => ["10.10.10.20:9200"]
index => "test"
id => "my_plugin_id"
document_id => "%{userid}"
document_type => "mytype"
}
}
Wherein the input of the codec specified => json json can parse the string kafka
output index specified in the index, document_type specified type, document_id specify the primary key
if there are duplicate primary key will automatically be overwritten!