Connect one: https://www.cnblogs.com/jhxxb/p/11579518.html
Use Flume1 a port monitor, which sink in the sink group and respectively abutting Flume2 Flume3, using Failover Sink Processor , implement the functions of the failover.
First, create a profile
1.flume-netcat-flume.conf
Configuration 1 and a Channel netcat source, a sink group (2 th sink), respectively, and supplied to flumeflume-console1 flumeflume-console2.
# Name the components on this agent a1.sources = r1 a1.channels = c1 a1.sinkgroups = g1 a1.sinks = k1 k2 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = 127.0.0.1 a1.sources.r1.port = 4444 # Sink Group a1.sinkgroups.g1.processor.type = failover a1.sinkgroups.g1.processor.priority.k1 = 5 a1.sinkgroups.g1.processor.priority.k2 = 10 a1.sinkgroups.g1.processor.maxpenalty = 10000 # Describe the sink a1.sinks.k1.type = avro a1.sinks.k1.hostname = h136 a1.sinks.k1.port = 4141 a1.sinks.k2.type = avro a1.sinks.k2.hostname = h136 a1.sinks.k2.port = 4142 # Describe the channel a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinkgroups.g1.sinks = k1 k2 a1.sinks.k1.channel = c1 a1.sinks.k2.channel = c1
2.flume-flume-console1.conf
Configuration Up Flume output of the Source, is output to the local console.
# Name the components on this agent a2.sources = r1 a2.sinks = k1 a2.channels = c1 # Describe/configure the source a2.sources.r1.type = avro a2.sources.r1.bind = h136 a2.sources.r1.port = 4141 # Describe the sink a2.sinks.k1.type = logger # Describe the channel a2.channels.c1.type = memory a2.channels.c1.capacity = 1000 a2.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a2.sources.r1.channels = c1 a2.sinks.k1.channel = c1
3.flume-flume-console2.conf
Configuration Up Flume output of the Source, is output to the local console.
# Name the components on this agent a3.sources = r1 a3.sinks = k1 a3.channels = c2 # Describe/configure the source a3.sources.r1.type = avro a3.sources.r1.bind = h136 a3.sources.r1.port = 4142 # Describe the sink a3.sinks.k1.type = logger # Describe the channel a3.channels.c2.type = memory a3.channels.c2.capacity = 1000 a3.channels.c2.transactionCapacity = 100 # Bind the source and sink to the channel a3.sources.r1.channels = c2 a3.sinks.k1.channel = c2
Second, the test
1. failover
Since the flume-netcat-flume.conf send data to the other two, i.e. flume-flume-console1.conf and flume-flume-console2.conf the server to receive data, you need to be started before the flume-netcat-flume.conf.
cd /opt/apache-flume-1.9.0-bin bin/flume-ng agent --conf conf/ --name a3 --conf-file /tmp/flume-job/group2/flume-flume-console2.conf -Dflume.root.logger=INFO,console bin/flume-ng agent --conf conf/ --name a2 --conf-file /tmp/flume-job/group2/flume-flume-console1.conf -Dflume.root.logger=INFO,console bin/flume-ng agent --conf conf/ --name a1 --conf-file /tmp/flume-job/group2/flume-netcat-flume.conf -Dflume.root.logger=INFO,console
After starting, since the flume-netcat-flume.conf configuration console1 higher priority than Console2, it will console1 priority connection.
Send a message to the monitoring port
yum -y install nc nc 127.0.0.1 4444 123456
And can be seen only netcat console1 and receive data, then the end console1 off console1 fault simulation, this time to connect Console2 netcat automatically, then send message and is only netcat Console2 the received data.
2. Load Balancing
Use Load balancing Sink completed, failover and almost, but not only when a connection console1, but switching between console1 and console2.
To modify the flume-netcat-flume.conf configuration
# Name the components on this agent a1.sources = r1 a1.channels = c1 a1.sinkgroups = g1 a1.sinks = k1 k2 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = 127.0.0.1 a1.sources.r1.port = 4444 # Sink Group a1.sinkgroups.g1.sinks = k1 k2 a1.sinkgroups.g1.processor.type = load_balance a1.sinkgroups.g1.processor.backoff = true a1.sinkgroups.g1.processor.selector = random # Describe the sink a1.sinks.k1.type = avro a1.sinks.k1.hostname = h136 a1.sinks.k1.port = 4141 a1.sinks.k2.type = avro a1.sinks.k2.hostname = h136 a1.sinks.k2.port = 4142 # Describe the channel a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinkgroups.g1.sinks = k1 k2 a1.sinks.k1.channel = c1 a1.sinks.k2.channel = c1
Testing and failover same, but randomly and sent to a console1 console2 wherein when transmitting a message with NC, not fixed.