Notes on Logstash

Install Logstash

1. Installation prerequisites: Logstash requires java. Therefore, you need to install java7 or above first. You can refer to here for java using oracle or open source Openjdk .
2. To start the installation, my system is Ubuntu 16.04. It is found that after installing it in the form of deb or through the configuration repository. It is not easy to find the executable file of Logstash, and an error will be reported when executing the following command. So I download and unzip the tar. To install it in the way of gz package, see here for the download address of the tar.gz installation package . Download and decompress it to complete.
3. Run is run by:

cd logstash-5.0.0-alpha3/
bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
    
    

     
     
1
     
     
2
    
    

Then you will find that the terminal is waiting for input. Type Hello and press Enter

--- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0
--- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1
Pipeline main started
Hello
{
    "@timestamp" => 2016-06-28T05:46:23.334Z,
      "@version" => "1",
          "host" => "QT-jiakunliu",
       "message" => "Hello"
}
    
    

     
     
1
     
     
2
     
     
3
     
     
4
     
     
5
     
     
6
     
     
7
     
     
8
     
     
9
     
     
10
    
    

grammar

(1) Section (section): Logstash uses {} to define a section. Regions can include plugin region definitions, and you can define multiple plugins in a region. In the plugin area, you can define key-value pair settings. Examples are as follows:

input {
    stdin {}
    syslog {}
}
    
    

     
     
1
     
     
2
     
     
3
     
     
4
    
    

(2) Data type: Logstash supports a small number of data value types:
bool

debug=>true
    
    

     
     
1
    
    

string

host=>"hostname"
    
    

     
     
1
    
    

number

port => 514
    
    

     
     
1
    
    

array

match => ["datetime", "UNIX", "ISO8601"]
    
    

     
     
1
    
    

hash

options => {
    key1 => "value1",
    key2 => "value2"
}
    
    

     
     
1
     
     
2
     
     
3
     
     
4
    
    

(3) Field reference (field reference): The field is a property of the Logstash::Event object. We mentioned earlier that an event is like a hash, so you can imagine a field like a key-value pair. If you want to use the field value in the Logstash configuration, just write the field name in square brackets [], this is called field reference. For nested fields (that is, multi-dimensional hash table, or hash of hash), the field name of each layer can be written in []. The array of logstash also supports reverse subscripting and variable interpolation

[geoip][location][0]
[geoip][location][-1]
"the longitude is %{[geoip][location][0]}"
    
    

     
     
1
     
     
2
     
     
3
    
    

(4) Conditional
expressions support the following operators:

• ==(equal to), !=(not equal to), <(less than), >(greater than), <=(less than or equal to), >=(greater than or equal to)
• =~ (matching regular), !~ (not matching regular)
• in (includes), not in (does not include)
• and (and), or (or), nand (not and), xor (not or)
• ()(compound expression), !()(invert the result of compound expression)

Typically, you will use field references in expressions. In order to show the full variety of expressions as much as possible, here is a fictitious example:

if "_grokparsefailure" not in [tags] {
} else if [status] !~ /^2\d\d/ or ( [url] == "/noc.gif" nand [geoip][city] != "beijing" ) {
} else {
}
    
    

     
     
1
     
     
2
     
     
3
     
     
4
    
    

(5) Command line parameters
Logstash provides a shell script called logstash for quick and easy operation. It supports parameters:

• -e
  means execute. We have already used this parameter in "Hello". In fact, you can directly run bin/logstash -e " without writing any specific configuration to achieve the same effect. Because the default value of this parameter is as follows:

input {
    stdin { }
}
output {
    stdout { }
}
    
    

     
     
1
     
     
2
     
     
3
     
     
4
     
     
5
     
     
6
    
    

• --config or -f
  means file. In real use, we would write very long configurations, perhaps even longer than the 1024 characters supported by the shell. So we must solidify the configuration into the file, and then run it in the form of bin/logstash -f agent.conf. In addition, logstash also provides a small function for us to plan and write the configuration. You can run it directly with bin/logstash -f /etc/logstash.d/. logstash will automatically read all *.conf text files in the /etc/logstash.d/ directory, and then splicing it into a complete large configuration file in its own memory, and then execute it.
  Note that
  logstash lists all files in a directory in alphabetical order. The filter and output of the logstash configuration section are executed sequentially, so the order is very important. For users who use multi-file management, it is recommended to name the configuration files by numbering. Meanwhile, in the configuration, strictly use if judgment to limit the actions of different logs.
• --configtest or -t
  means test. It is used to test whether the configuration file syntax read by Logstash can be parsed normally. The Logstash configuration syntax is defined with grammar.treetop. Especially readers who use the reading directory method mentioned in the previous article, especially test in advance .
• --log or -l
  means log. Logstash outputs logs to standard error by default. In a production environment, you can run the bin/logstash -l logs/logstash.log command to store logs uniformly.
• --pipeline-workers or -w
  Number of pipeline threads to run filter and output. The default is the number of CPU cores.
• --pipeline-batch-size or -b
  Each Logstash pipeline thread, the maximum number of logs that can be accumulated before executing specific filter and output functions. The default is 125 entries. The bigger the better the performance, but also the more JVM memory is consumed.
• --pipeline-batch-delay or -u
  Each Logstash pipeline thread, wait up to a few milliseconds when packing batch logs. The default is 5 ms. +
• --pluginpath or -P
  can write your own plugins and load them with bin/logstash --pluginpath /path/to/own/plugins .
• --verbose
  output certain debug logs.
• --debug
  output more debug logs.
  (6) Setting file: Starting from Logstash 5.0, the $LS_HOME/config/logstash.yml file has been added, and all command line parameters can be set through YAML files.

Grok regular capture

Use regular expressions to match text fragments:

(?<field_name>the pattern here)
    
    

     
     
1
    
    

The regular expression syntax is as follows:

\s+(?<request_time>\d+(?:\.\d+)?)\s+
    
    

     
     
1
    
    

Note that \s means matching whitespace characters (space, tab, line feed, form feed and carriage return); (?:pattern) means matching pattern but not getting the matching result, which means that this is a non-acquiring match and will not be stored for later use.

Install Logstash

1. Installation prerequisites: Logstash requires java. Therefore, you need to install java7 or above first. You can refer to here for java using oracle or open source Openjdk .
2. To start the installation, my system is Ubuntu 16.04. It is found that after installing it in the form of deb or through the configuration repository. It is not easy to find the executable file of Logstash, and an error will be reported when executing the following command. So I download and unzip the tar. To install it in the way of gz package, see here for the download address of the tar.gz installation package . Download and decompress it to complete.
3. Run is run by:

cd logstash-5.0.0-alpha3/
bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
  
  

   
   
1
   
   
2
  
  

Then you will find that the terminal is waiting for input. Type Hello and press Enter

--- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0
--- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1
Pipeline main started
Hello
{
    "@timestamp" => 2016-06-28T05:46:23.334Z,
      "@version" => "1",
          "host" => "QT-jiakunliu",
       "message" => "Hello"
}
  
  

   
   
1
   
   
2
   
   
3
   
   
4
   
   
5
   
   
6
   
   
7
   
   
8
   
   
9
   
   
10
  
  

grammar

(1) Section (section): Logstash uses {} to define a section. Regions can include plugin region definitions, and you can define multiple plugins in a region. In the plugin area, you can define key-value pair settings. Examples are as follows:

input {
    stdin {}
    syslog {}
}
  
  

   
   
1
   
   
2
   
   
3
   
   
4
  
  

(2) Data type: Logstash supports a small number of data value types:
bool

debug=>true
  
  

   
   
1
  
  

string

host=>"hostname"
  
  

   
   
1
  
  

number

port => 514
  
  

   
   
1
  
  

array

match => ["datetime", "UNIX", "ISO8601"]
  
  

   
   
1
  
  

hash

options => {
    key1 => "value1",
    key2 => "value2"
}
  
  

   
   
1
   
   
2
   
   
3
   
   
4
  
  

(3) Field reference (field reference): The field is a property of the Logstash::Event object. We mentioned earlier that an event is like a hash, so you can imagine a field like a key-value pair. If you want to use the field value in the Logstash configuration, just write the field name in square brackets [], this is called field reference. For nested fields (that is, multi-dimensional hash table, or hash of hash), the field name of each layer can be written in []. The array of logstash also supports reverse subscripting and variable interpolation

[geoip][location][0]
[geoip][location][-1]
"the longitude is %{[geoip][location][0]}"
  
  

   
   
1
   
   
2
   
   
3
  
  

(4) Conditional
expressions support the following operators:

• ==(equal to), !=(not equal to), <(less than), >(greater than), <=(less than or equal to), >=(greater than or equal to)
• =~ (matching regular), !~ (not matching regular)
• in (includes), not in (does not include)
• and (and), or (or), nand (not and), xor (not or)
• ()(compound expression), !()(invert the result of compound expression)

Typically, you will use field references in expressions. In order to show the full variety of expressions as much as possible, here is a fictitious example:

if "_grokparsefailure" not in [tags] {
} else if [status] !~ /^2\d\d/ or ( [url] == "/noc.gif" nand [geoip][city] != "beijing" ) {
} else {
}
  
  

   
   
1
   
   
2
   
   
3
   
   
4
  
  

(5) Command line parameters
Logstash provides a shell script called logstash for quick and easy operation. It supports parameters:

• -e
  means execute. We have already used this parameter in "Hello". In fact, you can directly run bin/logstash -e " without writing any specific configuration to achieve the same effect. Because the default value of this parameter is as follows:

input {
    stdin { }
}
output {
    stdout { }
}
  
  

   
   
1
   
   
2
   
   
3
   
   
4
   
   
5
   
   
6
  
  

• --config or -f
  means file. In real use, we would write very long configurations, perhaps even longer than the 1024 characters supported by the shell. So we must solidify the configuration into the file, and then run it in the form of bin/logstash -f agent.conf. In addition, logstash also provides a small function for us to plan and write the configuration. You can run it directly with bin/logstash -f /etc/logstash.d/. logstash will automatically read all *.conf text files in the /etc/logstash.d/ directory, and then splicing it into a complete large configuration file in its own memory, and then execute it.
  Note that
  logstash lists all files in a directory in alphabetical order. The filter and output of the logstash configuration section are executed sequentially, so the order is very important. For users who use multi-file management, it is recommended to name the configuration files by numbering. Meanwhile, in the configuration, strictly use if judgment to limit the actions of different logs.
• --configtest or -t
  means test. It is used to test whether the configuration file syntax read by Logstash can be parsed normally. The Logstash configuration syntax is defined with grammar.treetop. Especially readers who use the reading directory method mentioned in the previous article, especially test in advance .
• --log or -l
  means log. Logstash outputs logs to standard error by default. In a production environment, you can run the bin/logstash -l logs/logstash.log command to store logs uniformly.
• --pipeline-workers or -w
  Number of pipeline threads to run filter and output. The default is the number of CPU cores.
• --pipeline-batch-size or -b
  Each Logstash pipeline thread, the maximum number of logs that can be accumulated before executing specific filter and output functions. The default is 125 entries. The bigger the better the performance, but also the more JVM memory is consumed.
• --pipeline-batch-delay or -u
  Each Logstash pipeline thread, wait up to a few milliseconds when packing batch logs. The default is 5 ms. +
• --pluginpath or -P
  can write your own plugins and load them with bin/logstash --pluginpath /path/to/own/plugins .
• --verbose
  output certain debug logs.
• --debug
  output more debug logs.
  (6) Setting file: Starting from Logstash 5.0, the $LS_HOME/config/logstash.yml file has been added, and all command line parameters can be set through YAML files.

Grok regular capture

Use regular expressions to match text fragments:

(?<field_name>the pattern here)
  
  

   
   
1
  
  

The regular expression syntax is as follows:

\s+(?<request_time>\d+(?:\.\d+)?)\s+
  
  

   
   
1
  
  

Note that \s means matching whitespace characters (space, tab, line feed, form feed and carriage return); (?:pattern) means matching pattern but not getting the matching result, which means that this is a non-acquiring match and will not be stored for later use.