E-mail of the author of the article: [email protected] Address: Huizhou, Guangdong
▲ Purpose of this chapter
⚪ Understand the concept and configuration parameters of Interceptor;
⚪ Master the use of Interceptor;
⚪ Master the Host Interceptor of Interceptor;
⚪ Master Static Interceptor of Interceptor;
⚪ Master the Interceptor's UUID Interceptor;
⚪ Master the Search And Replace Interceptor of Interceptor;
⚪ Master Regex Filtering Interceptor of Interceptor;
⚪ Master the Custom Interceptor of Interceptor;
一、Timestamp Interceptor
1 Overview
1. Timestamp Interceptor is to add a timestamp field in the headers to mark the time when the data is collected.
2. Timestamp Interceptor combined with HDFS Sink can realize data storage by day.
2. Configuration Properties
| Attributes |
explain |
| type |
timestamp |
3. Case
1. Write the format file and add the following content:
a1.sources = s1
a1.channels = c1
a1.sinks = k1
a1.sources.s1.type = netcat
a1.sources.s1.bind = 0.0.0.0
a1.sources.s1.port = 8090
# Name the Interceptor
a1.sources.s1.interceptors = i1
# Specified Timestamp Interceptor
a1.sources.s1.interceptors.i1.type = timestamp
a1.channels.c1.type = memory
a1.sinks.k1.type = logger
a1.sources.s1.channels = c1
a1.sinks.k1.channel = c1
2. Start Flume:
../bin/flume-ng agent -n a1 -c ../conf -f in.conf -
Dflume.root.logger=INFO,console
4. Data is stored on a daily basis
1. Write the format file and add the following content:
a1.sources = s1
a1.channels = c1
a1.sinks = k1
a1.sources.s1.type = netcat
a1.sources.s1.bind = hadoop01
a1.sources.s1.port = 8090
a1.sources.s1.interceptors = i1
a1.sources.s1.interceptors.i1.type = timestamp
a1.channels.c1.type = memory
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://hadoop01:9000/flumedata/date=%Y-%m-%d
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.rollInterval = 3600
a1.so