ELK study notes (2) [original]

In ELK study notes (1) [original] , the introduction of Logstash is introduced. This article analyzes Logstash and imports the logs into ES.
1.
   There are many kinds of input input, such as stdin, file, log4j and so on. Here we only focus on analyzing file, next time we will analyze log4j.
  

input{
  file {
		codec => multiline {
			 pattern => "^\["
			 negate => true
			 what => "previous"
			 charset => "UTF-8"		
		}
    path => ["/usr/local/apache-tomcat-gtw/logs/catalina.*"]
    start_position => "beginning"
    discover_interval => 2
    sincedb_path => "/usr/share/logstash/conf/logstash_gateway/config/tomcat_sincedb.txt"
		sincedb_write_interval => 2
  }
}

  codec uses multi-line mode to split the log
  path log path
  start_position means start from the beginning
  discover_interval to scan the folder for log file changes every two seconds
  sincecedb_path record the number of lines read from the log sincecedb_write_interval
  record location information every two seconds

2.
  filterPrevious In the filter, split is mainly used to parse the log. This article mainly uses json template for matching.
 

filter {
   mutate{  
        gsub => [ "message", "\[", "" ]  
        gsub => [ "message", "]", "" ]  
        remove_field => [ "@version" ]
   }
   grok {
       	patterns_dir => "/usr/share/logstash/conf/logstash_gateway/logstash-patterns"
       	match => {
			"message" => "%{DATETIME:datetime} %{APP:app} %{LOGLEVEL:level} %{JAVACLASS:class} %{METHOD:method}"
		}
	}
	if [level] == 'DEBUG' {
                mutate {
                        replace => ["level_code",1000]
			convert => {
				"level_code" => "integer"
			}
                }
        }
	if [level] == 'INFO' {
		mutate {
			replace => ["level_code",2000]
			convert => {
				"level_code" => "integer"
			}
		}
	}
	if [level] == 'WARN' {
                mutate {
                        replace => ["level_code",3000]
			convert => {
				"level_code" => "integer"
			}            
                }               
        }    
	if [level] == 'ERROR' {
                mutate {
                        replace => ["level_code",4000]  
			convert => {
				"level_code" => "integer"
			}                
                }               
        }    
}

  gsub replaces the [] in the message, because [ is used for multi-line matching, which has no effect in the log.
  remove_field removes unwanted fields @Version fields are automatically added by
  logstash, no need for grok to use regular expressions to match logs
  patterns_dir regular expression file path
  %{DATETIME:datetime} Indicates that the first field is a time type and is converted to a datetime field .
  %{APP:app} Indicates that the second field is the APP type (custom regular), which is converted into an app field. The following analogy
  is followed by defining level_code according to the log level so that report statistics
 
can be
  output to ES and console respectively.

output {
	elasticsearch {
      		hosts => ["192.168.3.140"]
		index => "gateway"
		template => '/usr/share/logstash/conf/logstash_gateway/template/logstash-gateway.json'
		template_name => 'logstash-gateway'
		template_overwrite => true
		flush_size => 20000
                idle_flush_time => 10
   	}
	stdout{
		codec => rubydebug
		codec => plain{charset=>"UTF-8"}
	}
}

  hosts search engine address
  index index name
  template template file
  template_name the template name in the template file
  template_overwrite is set to true, flush_size can be updated when the template is changed, and the
  index is added every 20000
  idle_flush_time exceeds 10s, if it has not reached 20000, also refresh the index