Elasticsearch data acquisition solutions

(Verbatim) Address: Ali cloud Elasticsearch data acquisition solutions: https://help.aliyun.com/document_detail/141794.html

Outline

For data analysis and search for, Elasticsearch everywhere. Developers can use Elasticsearch community and find a wide variety of use cases, from the application and site search, logging, infrastructure monitoring, APM and security analysis, to name a few. Although there are free solutions for these use cases, developers first need to provide data to Elasticsearch.

This article provides several common methods aliyun ES will collect data.

Elastic Beats
Logstash
Language Client
Kibana Development Tools

Elasticsearch provides flexible RESTful API, used to communicate with client applications. Therefore, REST call is used to collect data, perform a search and data analysis, as well as manage the cluster and its index. In fact, all of the above methods rely on the collected data to a REST API Elasticsearch.

Elastic Beats

Elastic Beats is a lightweight set of data acquisition, data can easily be transmitted to Elasticsearch services. Because it is so lightweight, Beats will not generate too much overhead operation, therefore, data can be collected and run on a device with limited hardware resources (e.g., IoT device, an embedded device or edge device). If you need to collect data, but no resources to run resource-intensive data collector, then Beats will be your best choice. This data collection everywhere (across all networked devices), allowing you to quickly detect an abnormal condition to respond to such problems and security incidents system-wide and so on.

Of course, not limited to Beats systems with limited resources, they can also be used in systems with more available hardware resources.

Beats a variety of styles, you can collect different types of data:

Filebeat
enables you to read from the source file provided in the form, the pre-processing and transmission of data. Although most users Filebeat to read log files, but it also supports non-binary file format. Filebeat also supports a variety of other data sources, including TCP / UDP, the container, and the Redis Syslog. 254 module with rich, it can easily log format for Apache, MySQL and Kafka and other common applications, collecting, and analyzing the corresponding data.
Metricbeat
can be collected and pre-treatment systems and service indicators. The system indicators include relevant information about running processes, and data CPU / memory / disk / network utilization aspects. The module 186 may be used to collect data from many different services, including Kafka, Palo Alto Networks, Redis like.
Packetbeat
can be collected and pre-real-time network data to support application monitoring, security and network performance analysis. In addition, Packetbeat also supports DHCP, DNS, HTTP, MongoDB, NFS and TLS protocols.
Winlogbeat
captures event logs from the Windows operating system, including application events, hardware events, as well as security and system events.
Auditbeat
can detect changes to critical files, and collect events from the Linux audit framework. Different modules 118 simplifies its deployment, use cases mainly in the safety analysis.
Heartbeat
using a probe to monitor the availability of systems and services. Therefore, Heartbeat is useful in many scenarios, such as infrastructure monitoring and security analysis. ICMP, TCP and HTTP protocols are supported.
Functionbeat
can collect logs and metrics from no server environments (such as AWS Lambda) in.

You can refer to Ali cloud Elasticsearch by Beats build visualization system operation and maintenance , learning to use the Beats. Other Beats to use similar.

Logstash

Logstash is a powerful and flexible tool that can read, process and transmit any type of data. Logstash provides many features that are not yet available, or executed by Beats cost is too high, for example, by performing a lookup to external data sources to enrich the document. Either way, Logstash this power and flexibility comes at a price. Further, Logstash hardware requirements are also significantly higher than Beats. Strictly speaking, Logstash should not normally deployed on low-resource devices. Therefore, when the function Beats insufficient to meet the requirements of a particular use case, it may be used as alternatives Logstash.

A common pattern is a combination of architecture and Logstash up Beats: Beats use to collect data and perform data processing tasks can not be performed using Beats Logstash.

Logstash Overview

Ali cloud Elasticsearch provides Logstash services. Aliyun Logstash Service server as a data processing pipeline, providing 100% compatible open source Logstash function capable of dynamically acquiring data from multiple sources, data conversion, and stores the data into the selected position. Through the input, and an output filter plug, Logstash can be processed and converted to any type of event.

Logstash tasks are performed by the event processing pipeline, wherein each duct comprises at least one of the following:
the input
reading data from a data source. Official support a variety of data sources, including file, http, imap, jdbc, kafka , syslog, tcp and udp.

Filter
processing data in various ways and rich. In many cases, you first need to parse unstructured log line is more structured format. Thus, among other functions, basic Logstash also the regular expression, there is provided a parsing CSV, JSON, key / value pairs separated by the filter unstructured data, unstructured data and complex (Grok filter ). Logstash also offers more filters, by performing a DNS lookup, add geographical information about IP addresses, or by custom to perform a lookup table of contents or index Elasticsearch to enrich data. , The data can be converted by a variety of these additional filters, e.g. rename, delete, copy, and data field values (a mutate filter).
The output
will be enriched parsed and data writing data receiver, the final stage of the pipeline process Logstash. Although there are many export plug-ins are available, but this article focuses on how to use Elasticsearch output, data will be collected to Elasticsearch service.
Example conduit Logstash

The following provides an example Logstash conduit which can be:

Elastic blog to read RSS feeds.
By copying / renaming fields, remove special characters and HTML tags to perform some simple data preprocessing.
The documents collected Elaticsearch.

Reference arranged Pipeline by Kibana , aliyun Logstash conduit arranged in the following examples.

input { 
  rss { 
    url => "/blog/feed" 
    interval => 120 
  } 
} 
filter { 
  mutate { 
    rename => [ "message", "blog_html" ] 
    copy => { "blog_html" => "blog_text" } 
    copy => { "published" => "@timestamp" } 
  } 
  mutate { 
    gsub => [  
      "blog_text", "<.*?>", "",
      "blog_text", "[\n\t]", " " 
    ] 
    remove_field => [ "published", "author" ] 
  } 
} 
output { 
  stdout { 
    codec => dots 
  } 
  elasticsearch { 
    hosts => [ "https://<your-elsaticsearch-url>" ] 
    index => "elastic_blog" 
    user => "elastic" 
    password => "<your-elasticsearch-password>" 
  } 
}

hosts need to be replaced <corresponding to the network address aliyun Elasticsearch instance>: 9200; password needs to be replaced to correspond aliyun Elasticsearch access password.

The Logstash cloud Elasticsearch associated with Ali.
See detailed operation pipeline management configuration .

In Kibana console, view the index data.

POST elastic_blog/_search

Detailed operations, please refer to the task and see the results .

Language Client

In some cases, the best data acquisition and integration of custom application code. For this reason, we recommend that you use an officially supported Elasticsearch clients. These clients are abstract low-level details of the data collection library, allowing you to concentrate on the actual work of a particular application. Java, JavaScript, Go, .NET, PHP, Perl, Python and Ruby have official client. All details and code examples for your chosen language, see the documentation can be. If your application is not written in the languages listed above, you can find relevant documents in the client's contribution to the community.

Kibana Development Tools

We recommend that you use the console Kibana develop, develop and debug Elasticsearch request. Kibana tools disclosed generic full power and flexibility Elasticsearch REST API, while the technical details of the underlying abstract HTTP request. You can use Kibana development tools, will add the original JSON document to Elasticsearch in.

PUT my_first_index/_doc/1 
{ 
    "title" :"How to Ingest Into Elasticsearch Service",
    "date" :"2019-08-15T14:12:12",
    "description" :"This is an overview article about the various ways to ingest into Elasticsearch Service" 
}

Description In addition to Kibana development tools, you can also use other tools, generic REST interface provided by Elasticsearch, Elasticsearch communicate with and collect documents. For example, a curl is often used as a tool of last resort, for the development, debugging, or custom scripts to integrate with.

to sum up

The collected data to a Elasticsearch services too numerous to mention. You need to use a particular embodiment, and environmental requirements, select the appropriate method or means to collect data.

Beats provides a convenient, lightweight out of the box solutions, data can be collected from many different sources and collection. Beats module packaged together for many common database, the operating system, the container environment, Web server, caching, a data acquisition, resolution, and visualization of the configuration index. These modules can be implemented to provide data to experience five minutes dashboard. Because Beats is lightweight, it is ideal for resource-constrained embedded devices, such as IoT device or firewall.
Logstash is a flexible tool for reading, converting and gathering data, provides a number of filters, input and output plug. If for some functions Beats use case not enough, a common architecture pattern is used to collect data Beats, Logstash by further processing, and then collected in Elasticsearch.
When you need to collect data directly from the application, it is recommended to use an officially supported client libraries.
When you need to Elasticsearch request for development or debugging, it is recommended that you use Kibana development tools.