Logstash notes

First, the concept

Elasticsearch is the current mainstream large distributed data storage and search engines can provide a powerful full-text search capabilities for users, is widely used in log retrieval, site-wide search and other fields. Logstash as Elasicsearch common real-time data acquisition engine may collect data from different sources, and the data is output to a variety of post-processing the output source, it is an important part of Elastic Stack.

Logstash data processing process includes: Inputs , the Filters , Outputs of three parts, may additionally be used in the Inputs and Outputs Codecs data format processing. Four parts are present in the form of plug-ins, user defined profile pipeline, the need to input settings, filter, output, codec plug, to achieve a particular data acquisition, data processing, data output etc.

The Inputs : used to obtain data, such as a common plug-in file, syslog, redis, beats from data sources

The Filters : for processing data such as format conversion, the data fork and the like, such as common plug grok, mutate, drop, clone, geoip etc.

The Outputs : a data output, such as a common plug elastcisearch, file, graphite, statsd etc.

Codecs : codecs is not a separate process, but rather a means for converting data input and output plug-in and the like, for the data encoding process, such as the common plug json, multiline

• (1) Input per one thread start, acquiring data from the corresponding data source
• (2) Input data is written to a queue: The default is memory bounded queue (stops unexpectedly can cause data loss). In order to prevent the number of lost Logstash provides two features: Persistent Queues : through the queue on the disk to prevent data loss Dead Letter Queues : Save event can not be processed (only supports Elasticsearch as the output source)
• (3) Logstash have multiple Pipeline worker, each worker would take Pipeline a batch of data from the queue, and then executing the filter output (the number of worker and the amount of data processed by each configuration determination)

Second, the collection log

1, console input and output console

[root@hdp-1 bin]# ./logstash -e input {stdin} putput {stdout}

2, for the configuration file in order to start logstash

 bin/logstash -f console.conf  

3, monitor file output to the console

input {
    file{
        path => "/root/apps/logstash-5.6.16/data.txt"
    }
 }
output {
     stdout {}
 }

4, to output console messages collected es

input { 
	file{
        path => "/root/apps/logstash-5.6.16/testFile.txt"
    }
 }
output {
  elasticsearch { hosts => ["hdp-4:9200"] }
  stdout { codec => rubydebug }
}