I. Introduction
Logstash is an open source server-side data processing pipeline, support collect data from different sources, data exchange means send data to a different storage libraries.
Logstash project was born on August 2, 2009. Its author is the world-famous operation and maintenance engineers Jordan Cisse (JordanSissel), Jordan Cisse was famous virtual hosting provider DreamHost employees, also released through great software packaging tool fpm.
2013, Logstash was acquired Elasticsearch company.
Second, the installation
Logstash out of the box, so you just download the zip file, and then after decompression can use it.
$ wget https://artifacts.elastic.co/downloads/logstash/logstash-7.3.0.tar.gz $ tar -zxvf logstash-7.3.0.tar.gz $ cd logstash-7.3.0
三、Hello World
Logstash first started learning a form of output "Hello World" is.
Run the following command, stdin{}it represents the standard input from the input information; -erepresents a specified configuration command line; and codec=>rubydebugrepresents outputs the result to the console.
$ bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
At this point in the command line input Hello World, then Enter will see the following results:
{ "@version" => "1", "@timestamp" => 2019-08-09T08:52:29.790Z, "message" => "Hello World ", "host" => "chengshengzhangdeMacBook-Pro.local" }
As shown above, Logstash add some extra information, such as @timestamp time, host the host name and so on.
Fourth, running processes
Used Linux programmers are aware, there is a pipe symbol Linux, similar to the following command:
$ Cat randdata | awk ' {print $ 2} ' | sort | uniq -c | tee sortdata
Logstash like character as a pipe, may be entered, filtered, and then output to a different repository. Logstash use different threads to achieve these functions, data stream in the form of events between threads, and Logstash can handle multiple rows of data.
Pluggable framework Logstash used, there are already more than 200 plug-ins. Input plug into (input), plug-in encoding (codec), the filter plug (filter), the output plug (output).
Logstash entire process of processing data: input | decode | filter | encode | output.
- input plug-docking different data sources
- Initials codec plug-ins, codec from the coder / decoder two words abbreviation, is mainly used to decode, encode events, codec makes it easier Logstash and other custom data formats docking, the data format is converted into the corresponding event Logstash
- filter plug-in can parse the various events, such as the structuring unstructured data, parse out the IP address from geographical coordinates
- output plug butt different storage layers, for example ElasticSearch, HDFS, Kafka like.
Fifth, the command-line arguments
5.1、-e
Which means execution . In our "Hello World" when this parameter has been used up. In fact, you can not write any specific configuration, run directly bin/logstash -e '' achieve the same effect. The default value of the parameter is the following:
input {
stdin { }
}
output {
stdout { }
}
5.2, - config or -f
Which means the file . Real operation, we will write a long configuration, and may even exceed shell can support 1024 characters in length. Therefore, we will solidify the configuration to a file, then run in the form of such a bin / logstash -f agent.conf.
In addition, logstash also provides a handy little feature that we plan and write configuration. You can directly bin/logstash -f /etc/logstash.d/ run. logstash will automatically read /etc/logstash.d/ all the text files in a directory and then splice into a complete big profile in his memory, go perform.
5.3, - configtest or -t
Which means testing . Used to test Logstash to read the configuration file syntax whether the normal resolution. Logstash configuration syntax is defined in the grammar.treetop. Particularly the use of the readers on the way to read a directory mentioned, in particular, to test in advance.
5.4, - log or -l
Which means the log . Logstash default log output to standard error. Production environment you can bin/logstash -l logs/logstash.log be unified storage log commands.
5.5、--filterworkers 或 -w
Which means that the worker thread . Logstash run multiple threads. You can use bin/logstash -w 5 this as a way to force Logstash filter run five threads plugin.
Note: Logstash currently does not support multi-threaded input plug-ins. The multi-threaded output plug-ins need to configure the internal settings, this command line parameter is only used to set the filter plug!
Tip: Logstash currently does not support monitoring and management of the filter thread. If filterworker hang up, Logstash no filter will be in a state of dead. In this case the use of filter / ruby write their own code is to be noted that it is easy to run into NoMethodError: undefined method '*' for nil: NilClass error. The need to properly handle, early to judge.
5.6, - pluginpath or -P
You can write your own plug-ins, and then bin/logstash --pluginpath /path/to/own/plugins load them.
5.7、--verbose
Output some debugging log.
Tips : If you use less than Logstash version 1.3.0, you can use bin/logstash -v instead.
5.8、--debug
More output debug log.
Tips: If you use less than Logstash version 1.3.0, you can use bin/logstash -vv instead.
VI. References