[Get nèng - Logstash] Beginners (b) - Use Logstash parse Apache Web logs (using Filebeat data collection)

This series of blog is Logstash learning applications, some examples from official documents.
Reference:
official documents
use the log parsing Logstash

I. Introduction

Example from the official website, in this section, you create a Logstash conduit which use Filebeat Apache Web logs to get as input, parses the logs to create specific named fields from the log, and writes the parsed data Elasticsearch clusters.

II. To achieve

2.1 download and install

Download: portal
decompression can (need to install JDK8)

Configuration inputs Filebeat 2.2

1. Prepare the log file
Download: Here
unpacked to E: \ \ elk \ beat \ directory technology (can be any directory)

2. Install Filebeat
default Logstash installation includes Beats input plug-ins. Beats input plug-in enables Logstash can receive from Elastic Beats frame events

• Download the installation is very simple, you can download the zip package and extract, Download: Portal

• Modify filebeat.yml After unpacking, replace the contents with the following lines. Ensure that the paths point to logstash-tutorial.log you downloaded earlier example Apache log files.

filebeat.prospectors:
- type: log
  paths:
    - E:\technology\elk\beat\logstash-tutorial.log\*
output.logstash:
  hosts: ["localhost:5044"]

Filebeat will attempt to connect to port 5044
absolute path to the file Filebeat processing

• Filebeat running, then start Filebeat after logstash start, otherwise Filebeat will be prompted to connect less than 5044

./filebeat -e -c filebeat.yml -d "publish"

2.3 Configuration logstash

1. Write logstash profile
in logstash-6.0.0 \ bin create the configuration file (can be any directory, you need to specify the directory at startup)
filebeat_std.conf

input {
    beats {
        port => "5044"
    }
}
output {
    stdout { codec => rubydebug }
}

2. Start logstash
CD Logstash to the root directory, the command execution start (windows into the terminal by cmd)

cd logstash-6.0.0
bin/logstash -f filebeat_std.conf

• -e indicates that the specified pipeline configuration directly at startup
• -f expressed using a configuration file to start
• -config.test_and_exit option parsing the configuration file and report any errors
• -config.reload.automatic option allows to automatically reload the configuration so you do not have to stop every time you modify the configuration file and restart Logstash

3. Test
starts successfully filebeat displays the following:

logstash terminal displays:

2.3 Grok filter plug-in analytical Web log editor

We can find each row of data is stored in the message field, if we want to further split, parsing log messages you want to create a specific named fields from the log, you need to use Grok filter plug-ins.
1. Write logstash profile
revise filebeat_std.conf, add filter
filebeat_std.conf

input {
    beats {
        port => "5044"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}
output {
    stdout { codec => rubydebug }
}

2. Start logstash
shown below:

2.4 Geoip filter insert Enhanced Data Editor

In addition to parsing log data for better search, the filter plug-in can also obtain supplemental information from existing data in
bold style

input {
    beats {
        port => "5044"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
	geoip {
        source => "clientip"
    }
}
output {
    stdout { codec => rubydebug }
}

Restart logstash and filebeat, shown below:
(need to re-edit the next log file, otherwise it will not re-read after filebeat head read)

Es output to 2.5

Modify filebeat_std.conf

input {
    beats {
        port => "5044"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
	geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

Source Address

CLOUD-LOGSTASH-IT : LOGSTASH tutorial source code. CSDN logstash Bowen in this series.

---

Recommended items

CLOUD-IT : IT service management platform, integrated basic services, middleware services, alarm monitoring services.
CLOUD-ACTIVITI6-IT : the Activiti tutorial source code. CSDN Activiti Bowen in this series.
CLOUD-elasticsearch-IT : elasticsearch tutorial source code. CSDN elasticsearch Bowen in this series.
CLOUD-KAFKA-IT : the Spring integration kafka tutorial source code. Bowen in this CSDN kafka series.
-CLOUD-KAFKA-IT CLIENT : Client tutorial source Kafka used to live. Bowen in this CSDN kafka series.

Open source project, continuously updated, the Star ~ like please