[Teach you through ELK] Logstash plug-in development example implementation

Yuxian: CSDN content partner, CSDN new star mentor, full-stack creative star creator, 51CTO (Top celebrity + expert blogger), github open source enthusiast (go-zero source code secondary development, game back-end architecture https: https://github.com/Peakchen)

 

Logstash is a stream processing tool for collecting, processing and outputting data. Logstash plugins are an extensibility mechanism that can add new functionality and data processing to Logstash in a modular fashion. Plug-ins can be used to implement various functions such as data input, filtering, output, and codec.

Logstash plug-ins are written in Ruby language, developed based on Logstash plug-in API, and can be packaged and distributed through RubyGems. Plug-ins may include input plug-ins, filter plug-ins, codec plug-ins, output plug-ins, and so on.

The input plug-in is used to obtain data from data sources, such as obtaining data from files, networks, MQ, etc.

Filtering plug-ins are used to process data, such as converting, filtering, and aggregating data.

The codec plug-in is used to encode and decode data, such as parsing and encoding data in formats such as JSON, CSV, and XML.

The output plugin is used to output data to the target, such as output to file, network, MQ, etc.

Various data processing and analysis scenarios can be realized by using the Logstash plug-in. For example:

• Data collection and processing: Use input plugins to collect data from various data sources, use filter plugins to process and transform the data, and finally use output plugins to output the data to the target.
• Data aggregation and analysis: Use filtering plug-ins to aggregate and analyze data, such as calculating average values, statistics, etc.
• Data cleaning and conversion: Use filtering plug-ins to clean and convert data, such as filtering invalid data, converting data formats, etc.
• Real-time monitoring and early warning: Use the input plug-in to obtain data from the data source, and use the filter plug-in to monitor and warn the data in real time, such as detecting abnormal data, early warning of high load, etc.

Logstash plug-in development requires mastering the Ruby language and Logstash plug-in API. You can refer to the official Logstash documentation and examples for learning and practice.

Here are some relevant literature and material links:

• Logstash official documentation: Logstash Reference [8.9] | Elastic
• Logstash Plugin Development Guide: https://www.elastic.co/guide/en/logstash/current/_developing_plugins.html
• Logstash plugin API documentation: Documentation for logstash-core-plugin-api (2.1.28)
• Logstash plugin sample code: https://github.com/logstash-plugins/

Logstash plugins are an extensibility mechanism that can add new functionality and data processing to Logstash in a modular fashion. The following is a simple Logstash plug-in development example, which implements a custom filter plug-in to convert input strings to uppercase.

1. Create plugin directory

First, you need to create a plug-in directory to store the code and configuration files of the plug-in. You can create a directory named logstash in the root directory mypluginto store custom plugins.

logstash/
  |- bin/
  |- data/
  |- myplugin/
  |   |- lib/
  |   |- myplugin.gemspec
  |   |- Gemfile
  |   |- Gemfile.lock
  |- ... 

Under mypluginthe directory, the following files and directories need to be created:

• lib/: Used to store plugin code.
• myplugin.gemspec: Plug-in metadata file, including plug-in name, version, dependencies and other information.
• Gemfileand Gemfile.lock: Dependency libraries for managing plugins.

1. Write plugin code

lib/Create a file named in the directory for logstash/filters/myplugin.rbwriting plugin code. The following is a simple plugin code example to convert input strings to uppercase:

require "logstash/filters/base"
require "logstash/namespace"

class LogStash::Filters::MyPlugin < LogStash::Filters::Base
  config_name "myplugin"

  # 过滤器的配置项
  config :field, :string, :default => "message"

  public
  def register
    # TODO：初始化插件
  end

  public
  def filter(event)
    # 获取输入字符串
    input = event.get(@field)

    # 转换为大写
    output = input.upcase

    # 更新事件数据
    event.set(@field, output)

    # 传递事件到下一个过滤器
    filter_matched(event)
  end
end

In the code, LogStash::Filters::MyPluginthe class inherits LogStash::Filters::Basethe class to indicate that this is a filter plugin. config_nameThe attribute specifies the name of the plug-in, and configthe attribute specifies the configuration item of the plug-in. Here, a fieldstring-type configuration item named is specified to specify the input field name.

registerThe method is used to initialize the plugin, and some initialization code can be added here.

filterThe method is used to process the event data, here get the input string, convert to uppercase, update the event data, pass the event to the next filter.

1. Define plugin metadata

Define the metadata of the plug-in in myplugin.gemspecthe file, including plug-in name, version, dependencies and other information. Here is an example:

Gem::Specification.new do |s|
  s.name          = "logstash-filter-myplugin"
  s.version       = "1.0.0"
  s.author        = "Your Name"
  s.email         = "[email protected]"
  s.summary       = "Logstash filter plugin for converting input string to uppercase"
  s.description   = "A Logstash filter plugin that converts the input string to uppercase."
  s.homepage      = "https://github.com/your-github-account/logstash-filter-myplugin"
  s.license       = "Apache-2.0"
  s.platform      = Gem::Platform::RUBY
  s.requires      = [["logstash", ">= 6.0"]]
  s.add_runtime_dependency "logstash-core-plugin-api", ">= 0.1.0", "< 2.0.0"
  s.add_development_dependency "logstash-devutils", "~> 0.8"
  s.files         = Dir["lib/**/*", "spec/**/*", "Gemfile", "Gemfile.lock", "myplugin.gemspec", "LICENSE.txt", "NOTICE.txt", "README.md"]
  s.require_paths = ["lib"]
end

In the metadata, you need to specify the plugin name, version, author, dependencies and other information.

1. install plugin

Inside mypluginthe directory, run the following command to install the plugin:

bin/logstash-plugin install --no-verify

This will install the plugin and add it to Logstash's list of plugins.

1. configure plugin

In the Logstash configuration file, specify the name and configuration items of the plugin. Here is an example:

input {
  stdin {
  }
}

filter {
  myplugin {
    field => "message"
  }
}

output {
  stdout {
  }
}

In this example, stdinthe plugin is used to read data from standard input, mypluginthe plugin is used to convert the input string to uppercase, and stdoutthe plugin is used to output the processed data to standard output.

1. run Logstash

Start Logstash by running the following command:

bin/logstash -f myconfig.conf

This will read the configuration file myconfig.confand start Logstash, which will start processing data.

1. test plugin

Enter some string on the command line, for example hello world, and press Enter. Logstash will read the incoming data and pass it to stdinthe plugin, which will then be passed to mypluginthe plugin for processing, and finally output to stdout.

The output should look like this:

HELLO WORLD

This indicates that the plugin has successfully converted the input string to uppercase.