Logstash is a powerful data processing tool, which can realize data transmission, format processing, format output, and powerful plug-in functions, which are often used for log processing.
Three stages of Logstash work:
1. The input data input terminal can receive source data from anywhere. file, redis, beat (receive events from Filebeat)
2.filter data transfer layer, mainly for format processing, data type conversion, data filtering, field addition, modification, etc.
grok: Through regular parsing and structuring of any text, grok is currently the best way for logstash to parse unstructured log data into structured and queryable
3.output is the last stage of logstash work, responsible for outputting data to a specified location, compatible with most applications
elasticsearch: Send data to elasticsearch for easy query, analysis, and graphing.
file: write event data to disk file
mongodb: Send event data to high-performance NoSQL mongodb for permanent storage, query, analysis, and big data sharding.
redis: send data to redis-server, often used for temporary caching in the middle layer