One picture per text
Overview
Current distributed system log collection, log analysis, log processing, visualization hot technology of course non-stack embodiment ELK ( E lasticSearch, L ogstash, K ibana) must go from L → E → K constitute a Data Pipeline pipe:
Logstash : Connect with data sources to collect, filter and process your logs, transactions or other data
ElasticSearch : is an open source, distributed RESTful search engine, which can be roughly understood as a place for data storage in ELK
Kibana : Analyze and render Elasticsearch data into visual reports for efficient analysis
And in my previous article " Building a Log Center for Docker Containerized Applications Using ELK " , I used ELK to build a data pipeline to serve as a log center for Docker containerized applications.
Note: This article was first published on the My public account CodeSheep , you can long press or scan the caution below to subscribe ↓ ↓ ↓
Why talk about Logstash first
As the data source "direct butt" of Logstash, the position data in the pipeline ELK most distal end , its main role is to collect, analyze filter outputs a variety of structured or unstructured raw data (typically as log data), the original The burden of data from "disorder to order" falls on Logstash's shoulders, so its role is very important.
Speaking of Logstash, I have to say the plug-in mechanism , almost all of its functions are realized by plug-ins, so it is flexible and easy to use:
Regarding data collection , Logstash provides input plugins to support various data sources
Regarding data analysis , Logstash provides a filter plug-in to support fancy processing of input raw data
Regarding data output , Logstash also provides various output plug-ins to support the output of the result data to various places, such as standard consoles, files, various databases including ElasticSearch, etc.
Logstash plugin management
The Logstash plug-in is developed using Ruby. Starting from the very early version 1.5.0+, the plug-in module and core module of Logstash have been maintained separately. The plug-in uses the RubyGems package manager to manage and maintain it. So the Logstash plugin is essentially a self-contained RubyGems.
RubyGems (gems for short) is a Ruby packaging system for packaging Ruby components. It provides a standard format for distributing Ruby programs and libraries, as well as a tool to manage package installation.
All Logstash plugins can be searched on the website rubygems.org:
Common operations on plug-ins are as follows:
Install plugin
Can be installed online:
bin/plugin install [插件名称]
当然也可以将插件提前下载到本地，然后本地安装：
bin/plugin install path/logstash-xxx-x.x.x.gem
卸载插件
bin/plugin uninstall [插件名称]
更新插件
bin/plugin update [插件名称]
其会将插件更新到最新的版本
Logstash 插件语法结构
Logstash 插件的定义其实使用的就是一套其自定义的 DSL语法，我还是习惯用图来说明吧：
从图中可以看出主要包含以下几大部分内容：
1. 需要的依赖
该部分一般会用require语法引入如下依赖：
require "logstash/XXX/base" require "logstash/namespace"
前者引入 特定类型插件的依赖
后者引入 模块命名空间
2. 类定义
需要用 class语法给每一个插件定义一个类，后面我会用实际代码说明
3. 配置插件名字
通过 config_name 语法来给插件取一个名字，这个名字将会用到 Logstash.conf 配置文件的插件配置之中
4. 配置选项设置
可以使用 config 语法来按需定义任意个配置项。可以设置配置选项的名字、数据类型、默认值以及是否为必选项：
举例：
config :percentage, :validate => :number, :default =>100
:percentage：定义配置项的名字:validate：配置指定参数的数据类型，如此处为 number类型:default：指定配置项的默认值:required：用于指定配置项是否必选
5. 插件方法
每一种类型的插件都需要实现一些方法，如下表所示：
| 插件类型 | 插件方法 |
|---|---|
| 输入插件 | register、 run |
| 过滤器插件 | register、 filter |
| 输出插件 | register、 receive |
| 编解码插件 | register、 encode、 decode |
The business processing function of the Logstash plugin comes from the implementation of the business logic of the above plugin method!
Well, the theoretical part is summed up here, let's take an example with a source code defined by the Logstash plug-in !
An example analysis of a Logstash plugin definition file
Let's take the source code of a Logstash filter plugin logstash-filter-example given by the official website of the Logstash plugin as an example for analysis. Although the sparrow is small, it has all the internal organs! The code analysis has been marked in the figure and will not be repeated.
Of course, the example here is an introductory example. After all, it is impossible to give a too complicated Logstash plug-in source code in a limited article. Comparing the source code with the content of the previous section, I think it should not be difficult to understand the source structure of Logstash's plug-in.
It is planned to show an example of customizing and developing a Logstash plug-in that meets specific needs based on specific data requirements.
Postscript
My personal blog: www.codesheep.cn
If you are interested, you can also take time to read the author's article on containerization and microservices:
Use the K8S technology stack to create a personal private cloud series of articles
Use TICK to build a Docker container visual monitoring center
Explain Nginx server configuration from a configuration list
Docker container visualization monitoring center construction
Use ELK to build Docker containerized application log center
On the more pragmatic, able to read, reproducible original article to make public number CodeSheep , subscribe ⬇️⬇️⬇️
