Elastic Beats Introduction

Places to learn: Concepts, usage, module

Elastic Beats Introduction

Elastic Stack traditionally consists of three major components (Elasticsearch, Logstash and Kibana) composition, already out of this combination, it can also be combined with the fourth element named "Beats" - the transport for a different use cases log those series. Online now there is a saying called the ELKB, B here refers to the beats.

In centralized logging, the data conduit comprises three main stages: the polymerization, processing, and storage. ELK in the stack, traditionally, the first two phases are stacks workload Logstash duties. Perform these tasks need to pay a certain price. Due to the inherent problems associated with Logstash design, performance problems become frequent, especially in the case of complex piping requires a lot of processing. The idea of ​​outsourcing part of the responsibility Logstash have emerged, especially in the transfer of data extraction task to other tools.

Beats in the end what is it

Beats is lightweight (resource efficient, no dependence, small), and open-source collection of the log transmission program, the log transmission program acts as a proxy in the infrastructure installed on different servers, for collecting a log or index (metrics) . These can be log files (Filebeat), more and more network data Beats (Packetbeat), server metrics (Metricbeat) or Elastic and community development can collect any other type of data. After collection, the data is sent directly to the Logstash Elasticsearch or otherwise processed. Beats Go framework built on top of libbeat called, the framework for data forwarding, which means that the community has been to develop and contribute new Beats.

Filebeat

As the name suggests, Filebeat for collecting and transferring the log files, it is also the most common Beat. One fact Filebeat so efficient that it handles back pressure - so, if Logstash busy, Filebeat will slow down their reading rate, and speed up the tempo after the end of deceleration.
Filebeat can be installed on virtually any operating system, including a Docker container mounting, it also comes with a particular platform (e.g. Apache, MySQL, Docker, etc.) inside the module, wherein the default configuration comprises Kibana objects and these platforms.

Packetbeat

Packetbeat network packet analyzer is first introduced beat. Packetbeat capture network traffic between servers, and thus can be used for performance monitoring applications.
Packetbeat can be installed on the server monitored, it may be mounted on its own dedicated server. Packetbeat track network traffic, protocol decode and record the data for each transaction. Packetbeat Supported protocols include: DNS, HTTP, ICMP, Redis , MySQL, MongoDB, Cassandra and so on.

Metricbeat

Metricbeat is a very popular beat, it collects and reports a variety of system-level metrics variety of systems and platforms. Metricbeat also supports internal module collects statistics from a particular platform. You can use these modules and metricsets called the index set to configure the frequency metrics Metricbeat collected and which specific indicators to be collected.

Heartbeat

Heartbeat for "uptime monitoring" of. Essentially, Heartbeat detection function is service to check whether they are accessible, for example, it can be used to verify the uptime service meets your SLA. You need to do is provide the URL and uptime metrics for Heartbeat list to be sent directly to Elasticsearch or Logstash to send to your stack before indexing.

Auditbeat

Auditbeat users and processes can be used to audit activity on the Linux server. Similar to other traditional systems auditing tools (systemd, auditd), Auditbeat be used to identify security vulnerabilities - file changes, configuration changes, and other malicious behavior.

Winlogbeat

Winlogbeat Windows only be of interest to system administrators or engineers, because it is designed to collect Windows event logs designed to beat. It can be used to analyze security events, installed updates.

function Beat

Functionbeat is defined as "serverless" sender, it can be deployed to collect data and send it to a functional stack ELK. Functionbeat designed for monitoring cloud environments, has been set up tailor-made for the Amazon, can be deployed as Amazon Lambda functions to collect data from Amazon CloudWatch, Kinesis and SQS.

How to use the beats

Now we show you how to use metricbeat beats. Others can follow this method to do.

Installation metricbeat

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.3.2-linux-x86_64.tar.gz
tar xzvf metricbeat-7.3.2-linux-x86_64.tar.gz

Metricbeat profile

Once we have installed Metricbeat, we can see all the files in the root directory of the following installation directory:

$ ls -F
LICENSE.txt         metricbeat*
NOTICE.txt          metricbeat.reference.yml
README.md           metricbeat.yml
data/               metricbeat.yml.org
fields.yml          module/
kibana/             modules.d/
logs/

In this catalog, we can see two file configuration file:

metricbeat.yml：这是一个默认的配置文件
metricbeat.reference.yml：这是一个完整的样本配置文件

Only the first using a configuration file.

When configuring Metricbeat, you need to specify the module to run. Metricbeat use modules to collect metrics. Each logic module defines the basic data collected from a particular service (e.g. Redis or MySQL). And a module configured to acquire the data metricsets composition. For example Redis module, use the Info Metricset and Keyspace Metricset in this module. They share a channel and a host connected.

1. Enable module to run

If you accept the default configuration does not enable other modules, the Metricbeat collect only System indicators.

# 路径：metricbeat-7.3.2-darwin-x86_64/modules.d

# ls -a
.                   kibana-xpack.yml.disabled
..                  kibana.yml.disabled
aerospike.yml.disabled          kubernetes.yml.disabled
apache.yml.disabled         kvm.yml.disabled
aws.yml.disabled            logstash-xpack.yml.disabled
beat-xpack.yml.disabled         logstash.yml.disabled
beat.yml.disabled           memcached.yml.disabled
ceph.yml.disabled           mongodb.yml.disabled
cockroachdb.yml.disabled        mssql.yml.disabled
consul.yml.disabled         munin.yml.disabled
coredns.yml.disabled            mysql.yml.disabled
couchbase.yml.disabled          nats.yml.disabled
couchdb.yml.disabled            nginx.yml.disabled
docker.yml.disabled         oracle.yml.disabled
dropwizard.yml.disabled         php_fpm.yml.disabled
elasticsearch-xpack.yml.disabled    postgresql.yml.disabled
elasticsearch.yml.disabled      prometheus.yml.disabled
envoyproxy.yml.disabled         rabbitmq.yml.disabled
etcd.yml.disabled           redis.yml.disabled
golang.yml.disabled         system.yml
graphite.yml.disabled           traefik.yml.disabled
haproxy.yml.disabled            uwsgi.yml.disabled
http.yml.disabled           vsphere.yml.disabled
jolokia.yml.disabled            windows.yml.disabled
kafka.yml.disabled          zookeeper.yml.disabled

There in a subfolder called modules.d of Metricbeat. It contains all of the default display all the modules metricbeat can support, in case of default, it is enabled only system.yml state. The other is in the disabled state.

We can get all the current module is supported by the following command in the metricbeat into the installation directory:

./metricbeat modules list

If we want to open a module, we can do this:

./metricbeat modules enable apache mysql

So we opened the apache and mysql module. If we want to close the module, you can use the following command:

./metricbeat modules disable apache mysql

1. Configuration output

Metricbeat supports a variety of output, but usually you will be sent directly to the event Logstash Elasticsearch or otherwise processed.

output.elasticsearch
  hosts: ["127.0.0.1:9200"]

1. If you plan to use the sample Kibana dashboard Metricbeat accompanying configure Kibana endpoint. If Kibana Elasticsearch and running on the same host, you can skip this step.

setup.kibana:
  host: "127.0.0.1:5601"

1. If Elasticsearch and Kibana protected, please set up a certificate metricbeat.yml configuration file before running the command set up and start Metricbeat.

output.elasticsearch:
  hosts: ["myEShost:9200"]
  username: "filebeat_internal"
  password: "YOUR_PASSWORD" 
setup.kibana:
  host: "mykibanahost:5601"
  username: "my_kibana_user"  
  password: "YOUR_PASSWORD"

1. Run the test command:

./metricbeat test config -e

Currently only set up a system module.

1. Set Kibana module

We run the following commands to configure Kibana dashboard:

./metricbeat setup

Show results:

$ ./metricbeat setup
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
 
Loaded dashboards

This operation will produce a corresponding index pattern for us in Kibana, the production and the corresponding sample template dashboard for our use.

Run Metricbeat

In the previous step we have configured our Metricbeat, we can now begin to run Metricbeat up.

./metricbeat -e

You can take a look to the Kibana data. Metricbeat selected as the beginning of the index, we can see the data continues to grow, indicating that metricbeat is constantly collecting data.

Click the Discover icon, and then select the Index pattern metricbeat, you can see a data collected.

Click the Dashboard icon, and then search for the System Metricbeat dashboard display

Shown above is the System Overview, we can also click on the "Host Overview":

With this simple configuration, we can use our understanding of the entire system.