fluentd Fliter plugin应用

一： record_transformer Filter Plugin

在实际环境中，需要将fluentd接收到的数据项operation_time(UTC，格式如20180716120000)时间转为为其他时区(如UTC+9)的时间，根据其需求，可采用fluentd中的record_transformer Filter Plugin完成。fluentd conf文件如下：

# built-in TCP input
<source>
  @type http
  port 8901
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s
  add_remote_addr  true
</source>

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<filter videoad.*>
  @type parser
  format json
  key_name data
  hash_value_field data
  reserve_data true
</filter>

<filter timeon.behavior>
  @type parser
  format json
  key_name data
  hash_value_field data
  reserve_data true
</filter>

<filter timeon.behavior>
  @type record_transformer
  enable_ruby
  <record>
    t_time ${(Time.utc(record["data"]["operation_time"][0,4],record["data"]["operation_time"][4,2],record["data"]["operation_time"][6,2],record["data"]["operation_time"][8,2],record["data"]["operation_time"][10,2])+record["data"]["time_zone"][1,2].to_i*60*60).strftime("%Y%m%d%H%M%S")}
  </record>
</filter>
<filter videoad.*>
  @type record_transformer
  enable_ruby
  <record>
    t_time ${(Time.utc(record["data"]["operation_time"][0,4],record["data"]["operation_time"][4,2],record["data"]["operation_time"][6,2],record["data"]["operation_time"][8,2],record["data"]["operation_time"][10,2])+record["data"]["time_zone"][1,2].to_i*60*60).strftime("%Y%m%d%H%M%S")}
  </record>
</filter>


# Kafka buffered output
<match **>
  @type kafka_buffered
  brokers       kafka-a:9092,kafka-b:9092,kafka-c:9092
  buffer_type   file
  buffer_path   /var/log/fluent/buffer-01/*.buffer

  # file buffer can store at most 68.3h(1m * 4096) or 32G(8M * 4096) data 
  flush_interval 10s
  buffer_chunk_limit 8m
  buffer_queue_limit 4096

  retry_wait          1s
  max_retry_wait      60s
  disable_retry_limit true

  # ruby-kafka producer options
  max_send_retries    1
  required_acks       1
  compression_codec   gzip
  output_data_type json
  output_include_time true
  num_threads   1
</match>

<system>
  # loglevel: trace, debug, info, warn, error
  #log_level warn 
  log_level debug
</system>

利用@type record_transformer和enable_ruby配置之后，可在${}中利用ruby 表达式对数据中的数据进行时区的转化，其中用到了Time类，处理逻辑为根据解析operation_time字符串转化为Time的UTC时间，在根据time_zone字段加上时区的秒数，生成新的字段t_time。

fluentd Fliter plugin应用

猜你喜欢