Apache Hadoop生态-目录汇总-持续更新

系统环境：centos7

Java环境：Java8

1：Kafka Channel

Kafka Channel：数据存储在kafka里，存储在磁盘，如果sink是kafka的情况，采用Kafka Channel会少一步sink

# 1：定义组件
file_flume_kafka.sources = r1
file_flume_kafka.channels = c1
file_flume_kafka.sinks = k1

# 2：定义source
这里主要介绍Channel顾这里省略，到source模块查看写法

# 3：定义channel
file_flume_kafka.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
file_flume_kafka.channels.c1.kafka.bootstrap.servers = 192.168.5.103:9092,192.168.5.87:9092,192.168.5.114:9092
file_flume_kafka.channels.c1.kafka.topic = project_v4_topic_log
    ## 设置消费者组,保证每次消费时能够获取上次对应的Offset
file_flume_kafka.channels.c1.consumer.group.id = file_flume_kafka
    #设置不用flume组装的header
file_flume_kafka.channels.c1.parseAsFlumeEvent = false

# 4：定义sink
....
这里主要介绍Channel顾这里省略，到sink模块查看写法

# 5：定义关联关系
file_flume_kafka.sources.r1.channels = c1
file_flume_kafka.sinks.k1.channel = c1

2：File Channel

File Channel 存储在磁盘。因此在程序关闭或机器宕机的情况下不会丢失数据。

# 1：定义组件
kafka_flume_hdfs.sources = r1
kafka_flume_hdfs.channels = c1
kafka_flume_hdfs.sinks = k1

# 2：定义source
这里主要介绍Channel顾这里省略，到source模块查看写法

# 3：定义channel
kafka_flume_hdfs.channels.c1.type = file
    #file channel数据存储的路径
kafka_flume_hdfs.channels.c1.dataDirs = /usr/local/flume-1.9.0/data/project_v4
    #file checkpointDir路径
kafka_flume_hdfs.channels.c1.checkpointDir = /usr/local/flume-1.9.0/checkpointDir/project_v4

# 4：定义sink
....
这里主要介绍Channel顾这里省略，到sink模块查看写法

# 5：定义关联关系
kafka_flume_hdfs.sources.r1.channels = c1
kafka_flume_hdfs.sinks.k1.channel = c1

3：Memory Channel

Memory Channel 存储在内存是内存中的队列。

Memory Channel 在不需要关心数据丢失的情景下适用，数据有丢失的可能。

# 1：定义组件
a3.sources = r3
a3.sinks = k3
a3.channels = c3

# 2：定义source
这里主要介绍Channel顾这里省略，到source模块查看写法

# 3：定义channel
a3.channels.c3.type = memory
a3.channels.c3.capacity = 1000
a3.channels.c3.transactionCapacity = 100

# 4：定义sink
....
这里主要介绍Channel顾这里省略，到sink模块查看写法

# 5：定义关联关系
a3.sources.r3.channels = c3
a3.sinks.k3.channel = c3