Problem Description
Logstash with UTC time, logstash when outputting to elasticsearch per day, because the time zone using utc, resulting in 8:00 the day before to create the index, while the previous data is output to 8:00 yesterday's index
# Logstash use configuration when writing elasticsearch
Output {
elasticsearch {
ID => "logstash - YYYY.MM.DD% {+}"
}
}
. 1
2
. 3
. 4
. 5
. 6
logstash and elasticsearch accordance with UTC time, kibana is in accordance with normal in your time zone display, because kibana can configure the time zone information.
Solve the problem
Add a field
This field is generated using fileter logstash implemented by a variety of ways, the new timestamp field is generated as follows
# ruby 插件实现
ruby {
code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)"
}
# Grok regular matches obtained from the log
Grok {
match => { "Message" => "% {TIMESTAMP_ISO8601: timestamp}"}
}
# Date time the log is rewritten @timestamp
#date {
# match => [ "Message", "^ \ [(<timestamp> \. 4 {D} -? \ D {2} - \ {2} D \ D 2} {: \ D {2}:. \ D {2} \ {D}. 3) "]
# target =>" @timestamp "
# TimeZone =>" + 08: 00 "
#}
. 1
2
. 3
. 4
. 5
. 6
. 7
. 8
. 9
10
. 11
12 is
13 is
14
15
16
the data conversion plug mutate extract the desired date field
# Convert convert to string type, gsub process only string types of data, with a regular match, finally obtained the desired date
a mutate {
Convert => [ "timestamp", "string"]
gsub => [ "timestamp", "T (? [\ S \ S] *) the Z "," "]
gsub => [" timestamp "," - "." ",]
}
. 1
2
. 3
. 4
. 5
. 6
configure output
elasticsearch {
the hosts => [ "localhost: 9200"]
index => "log -% {timestamp}"
}
. 1
2
. 3
. 4
NOTE: use grok for regular matching is a method that can be used can be found here
https: // github .com / logstash-plugins / logstash-Patterns-Core / BLOB / Master / Patterns / Patterns Grok-
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
Grok test platform
http://grokdebug.herokuapp.com/
----------------
Disclaimer: This article is the original article CSDN bloggers "40kuai", following the CC 4.0 by-sa copyright agreement, reproduced, please attach the original source link and this statement.
Original link: https: //blog.csdn.net/u012881331/article/details/88313045