Logstash: processing a plurality of input

Logstash: processing a plurality of input

Pipleline Logstash entire divided into three parts:

• input plugins: extract data. This can come from a log file, TCP or UDP listener, one (such as syslog or IRC) protocol specific number of plug-ins, and even queuing system (such as Redis, AQMP or Kafka). This stage uses metadata about the events marking the source of incoming events.
• filter plug-ins: Plug-rich and data conversion
• output plug-in: The event has been handled is loaded into other content, such as ElasticSearch or other documents database, or queuing system, such as Redis, AQMP or Kafka. It can also be configured to communicate with the API. May be connected to the output Logstash something like PagerDuty.

input here can support multiple input, multiple simultaneous worker can handle the filter and output:

Logstash profile

Logstash configuration file as follows:

# cat multi-input.conf

    input {
      file {
        path => "/Users/liuxg/data/multi-input/apache.log"
        start_position => "beginning"
        sincedb_path => "/dev/null"
        # ignore_older => 100000
        type => "apache"
      }
    }
     
    input {
      file {
        path => "/Users/liuxg/data/multi-input/apache-daily-access.log"
        start_position => "beginning"
        sincedb_path => "/dev/null"
        type => "daily"
      }
    }
     
    filter {
        grok {
            match => {
                "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
            }
        }
     
        if[type] == "apache" {
            mutate {
                add_tag => ["apache"]
            }
        }
     
        if [type] == "daily" {
            mutate {
                add_tag => ["daily"]
            }
        } 
    }
     
     
    output {
        stdout {
            codec => rubydebug
        }
     
        if "apache" in [tags] {
            elasticsearch {
                index => "apache_log"
                template => "/Users/liuxg/data/apache_template.json"
                template_name => "apache_elastic_example"
                template_overwrite => true
          } 
        }
     
        if "daily" in [tags] {
            elasticsearch {
                index => "apache_daily"
                template => "/Users/liuxg/data/apache_template.json"
                template_name => "apache_elastic_example"
                template_overwrite => true
          } 
        }   
    }

To facilitate the description of the problem, two input. Which correspond to different log files. For both input, also used to represent a different type: apache and daily. Despite their format is the same, they use the same common a grok filter, but we still want to deal with them separately. To do this, add a tag. You can also add a field to distinguish. In the output section, the tag according to the output filter portion is provided to them in a different index.

Run Logstash

You can run the following command:

./bin/logstash -f ~/data/multi-input/multi-input.conf

When running this example, the above command needs to be changed according to their stowed position change multi-input.conf file.

The event can be seen daily output according to the first to be processed and the results displayed. Then began the apache data processing. In practical applications, we may have different data sources, such as data from a port other beats of listening.

We can see the final index data in Kibana in.