Flume: official case for monitoring port data

Official case of monitoring port data

1. Case requirements:

Use Flume to listen to a port, collect the port data, and print it to the console.

2. Requirements analysis:

3. Implementation steps:

① Software environment configuration
(1) Install netcat tool
[atguigu@hadoop102 ~]$ sudo yum install -y nc

(2) Determine whether port 44444 is occupied
[atguigu@hadoop102 ~]$ sudo netstat -nlp | grep 44444

(3) Create a job folder in the flume directory and enter the job folder.
[atguigu@hadoop102 flume]$ mkdir -p job/simpleCase
[atguigu@hadoop102 flume]$ cd job/simpleCase

② Write the configuration file (type it by yourself)
and create the Flume Agent configuration file flume-1-netcat-logger.conf in the job/simpleCase folder. Add the following content
[atguigu@hadoop102 simpleCase]$ vim flume-1-netcat-logger .conf

#Name the components on this agent
a1.sources = r1                                      # 为a1的Source组件命名为r1，多个组件用空格间隔
a1.sinks = k1                                        # 为a1的Sink组件命名为k1，多个组件用空格间隔
a1.channels = c1                                    # 为a1的Channel组件命名为c1，多个组件用空格间隔

# Describe/configure the source
a1.sources.r1.type = netcat                      # 配置r1的类型
a1.sources.r1.bind = localhost                  # 配置r1的绑定地址（注意localhost和hadoop102的区别）
a1.sources.r1.port = 44444                       # 配置r1的监听端口

# Describe the sink
a1.sinks.k1.type = logger                        # 配置k1的类型为logger,输出到控制台

# Use a channel which buffers events in memory
a1.channels.c1.type = memory                    # 配置c1的类型为memory
a1.channels.c1.capacity = 1000                 # 配置c1的容量为1000个事件
a1.channels.c1.transactionCapacity = 100     # 配置c1的事务容量为100个事件

# Bind the source and sink to the channel
a1.sources.r1.channels = c1                    # 配置r1的channel属性，指定r1连接到那个channel
a1.sinks.k1.channel = c1                        # 配置k1的channel属性，指定k1连接到那个channel

Note: The configuration file comes from the official manual http://flume.apache.org/FlumeUserGuide.html

③ Deploy and run flume listening port
·The first way of writing:
[atguigu@hadoop102 flume]$ bin/flume-ng agent --conf conf/ --name a1 --conf-file job/simpleCase/flume-1-netcat-logger .conf -Dflume.root.logger=INFO,console

·The second way of writing:
[atguigu@hadoop102 flume]$ bin/flume-ng agent -c conf/ -n a1 -f job/simpleCase/flume-1-netcat-logger.conf -Dflume.root.logger=INFO, console

·Parameter description:
–conf/-c: indicates that the configuration file is stored in the conf/ directory
–name/-n: indicates that the agent is named a1
–conf-file/-f: specifies that the configuration file to be read is in job/simpleCase The flume-1-1netcat-logger.conf file in the folder.
-Dflume.root.logger=INFO,console: -D means to dynamically modify the flume.root.logger parameter attribute value when flume is running, and set the console log printing level to INFO level. Log levels include: log, info, warn, error.

④ Test
(1) Use the netcat tool to send content to the 44444 port of this machine
[atguigu@hadoop102 flume]$ nc localhost 44444
hello
atguigu

(2) Observe the data received on the Flume listening page
...
2018-09-07 21:17:48,494 (SinkRunner-ProllingRunner-DefaultSinkProcessor) [INFO – org.apache.flume.sink.Sink.process(LoggerSink.java:95 )] Event:{headers:{} body: 68 65 6c 6F 0D hello.}

Thinking: nc hadoop102 44444, can flume receive it?