Logstash millisecond timestamp to date and replace original @timestamp with business log timestamp

question

When using Kibana to observe the log troubleshooting problem, it is found that there are many groups of @timestamp data. As shown in the following

detailed observation of the internal data, it is found that the log data has a timestamp field that stores the millisecond-level timestamp of the business log. After comparing with the @timestamp data It was found that the time of the two did not match. After analysis, it is known that @timestamp is sorted according to the time when logstash inserts es data, and the data comes in batches. The time of each batch may be the same, resulting in a series of problems described above.

solution

To solve this problem, we can use the date attribute in the filter in logstash to convert the date, that is, use the timestamp field in the business log to replace the @timestamp time generated by logstash itself.

The usage of the date attribute is as follows:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html

date {
    match => [ "biz_time", "yyyyMMdd HH:mm:ss.SSS" ]
    target => "new_time"
}

The simple use configuration reference is shown above, where match matches the field that processes the input log data and the corresponding data type, and target indicates the field to be matched and mapped, which can be an existing field or a new field.
It should be noted that you must strictly match the date type of the corresponding attribute of the input log. If you are like 2023-01-01 12:12, you need to specify this date format. If it is a timestamp, you need to specify it as UNIX Or UNIX_MS, the former is a second-level timestamp and the latter is a millisecond-level timestamp, which needs to be strictly corresponding, otherwise it will not be able to match the conversion.

Example: The following is the business log information of the project. I want to use the timestamp field in the red box as @timestamp time

The reference configuration is as follows:

filter{
    
    
    date{
    
    
        match=>["timestamp",  "UNIX_MS" ]
        target=>"@timestamp"
    }
}

After modifying the configuration, restart logstash, observe the startup log, and then look at the index data

reference

[How to convert timestamp to date field data]
https://discuss.elastic.co/t/how-to-convert-a-unix-timestamp-field-to-a-new-date-field/50966

[Date conversion in logstash]
https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html