ELK study notes (6) [original]

In ELK study notes (4) [original] , we introduced the method of using log4j to input logs to logstash through tcp. This method is easier to use in the early stage of promotion, because it is relatively simple to use, but there is a problem, that is, when the pressure increases, the problem of java.net.SocketException: Broken pipe will occur. At the same time, the official website also prompts that the log4j input plugin is deprecated, and it is recommended to use filebeat instead. For details, see: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
If you have high requirements for log collection, it is best to use filebeat.
Overall architecture diagram:

In ELK study notes (2) [original] , we introduced the file input using logstash, which is very similar to filebeat. It's just that logstash is written in java, which consumes more resources. filebeat is lighter, has better performance, and is easier to install. Beats include:
Packetbeat (collects network traffic data);
Topbeat (collects data such as CPU and memory usage at the system, process and file system level);
Filebeat (collects file data);
Winlogbeat (collects Windows event log data).
Here only focus on filebeat.
1. Download and install

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.1.1-x86_64.rpm
sudo rpm -vi filebeat-6.1.1-x86_64.rpm

2. Modify the configuration
#/etc/filebeat/filebeat.yml

filebeat.prospectors:
    -
      paths:
        - /var/log/logstash/*.log
      input_type: log
      exclude_lines: ['DEBUG']
      multiline:
        pattern: '^\['
        negate: true
        match: after
output.logstash:
    hosts: ["logstash.zjportdns.gov.cn:5000"]

3. Start

sudo service filebeat start

4. Set up self-start
vi /etc/rc.local

sudo service filebeat start

5. logstash configuration

input{
  beats {
     type => "beat"
     port => 5000
  }
}

For the complete configuration, see the attachment (for compatibility with log4j, some conversions have been made to the filter)