本人采用双节点的方式
其中两个节点都存活时 ：两个节点做负载均衡使用
其中一个节点宕机 ： 一个节点承担从前两个节点的流量 （做到高可用）
channel 直接对接kafka 节省资源
其中配置为 （两份）
tier1.sources = source1 #对应sources名字
tier1.channels = kafka-mobile-channel #对应channel 名字
tier1.sources.source1.type = avro
tier1.sources.source1.bind = 0.0.0.0
tier1.sources.source1.port = 44444
tier1.sources.source1.channels = kafka-mobile-channel
tier1.sources.source1.selector.type = multiplexing
tier1.sources.source1.selector.header = topic
tier1.sources.source1.selector.mapping.mobile = kafka-mobile-channel
tier1.channels.kafka-mobile-channel.type = org.apache.flume.channel.kafka.KafkaChannel
tier1.channels.kafka-mobile-channel.parseAsFlumeEvent = false #用了配置是否后面要解析 Flume 头信息内容
tier1.channels.kafka-mobile-channel.kafka.topic = tomcat-mobile
tier1.channels.kafka-mobile-channel.kafka.consumer.group.id = flume-tomcat-mobile
tier1.channels.kafka-mobile-channel.kafka.consumer.auto.offset.reset = earliest
tier1.channels.kafka-mobile-channel.kafka.bootstrap.servers = ZW0804-hadoop-89:9092,ZW0804-hadoop-90:9092,ZW0804-hadoop-91:9092
他的上游配置为
agent
collector.sources = taildir-source
collector.channels = file-channel
collector.sinks = avro-forward-sink-node2 avro-forward-sink-node3
source
collector.sources.taildir-source.type = TAILDIR
collector.sources.taildir-source.channels = file-channel
collector.sources.taildir-source.positionFile = /var/log/flume-ng/taildir_position.json
collector.sources.taildir-source.filegroups = f1
collector.sources.taildir-source.filegroups.f1 = /tmp/nginx/.+.log
collector.sources.taildir-source.fileHeader = true
collector.sources.taildir-source.interceptors = topic UUID
collector.sources.taildir-source.interceptors.topic.type = static
collector.sources.taildir-source.interceptors.topic.key = topic
collector.sources.taildir-source.interceptors.topic.value = we-user
collector.sources.taildir-source.interceptors.topic.preserveExisting = false
collector.sources.taildir-source.interceptors.UUID.type=org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder
collector.sources.taildir-source.interceptors.UUID.headerName=key
collector.sources.taildir-source.interceptors.UUID.prefix=NODE_
collector.sources.taildir-source.interceptors.UUID.preserveExisting=false
collector.sources.taildir-source.skipToEnd = true
channel
collector.channels.file-channel.type=file
collector.channels.file-channel.checkpointDir = /var/log/flume-ng/file-channel/checkpoint #channel 的备份文件方式
collector.channels.file-channel.dataDirs = /var/log/flume-ng/file-channel/data #数据存储路径
sink 采用分发的方式
collector.sinks.avro-forward-sink-node2.type = avro
collector.sinks.avro-forward-sink-node2.channel = file-channel
collector.sinks.avro-forward-sink-node2.hostname = node2 #对应 负载均衡的ip
collector.sinks.avro-forward-sink-node2.port = 44444
collector.sinks.avro-forward-sink-node3.type = avro
collector.sinks.avro-forward-sink-node3.channel = file-channel
collector.sinks.avro-forward-sink-node3.hostname = node3 #对应 负载均衡的ip
collector.sinks.avro-forward-sink-node3.port = 44444
load balance
collector.sinkgroups = g1
collector.sinkgroups.g1.sinks = avro-forward-sink-node2 avro-forward-sink-node3
collector.sinkgroups.g1.processor.type = load_balance
collector.sinkgroups.g1.processor.backoff = true