flume collects multiple folder logs

In flume1.6 and before, if you want to monitor multiple files in multiple directories, you can use Filelistener. After flume1.7, TAILDIR has been added to monitor file changes.

Reference configuration:

#配置Agent a1 的组件
a1.sources=r1
a1.sinks=s1
a1.channels=c1
 
#描述/配置a1的source1
a1.sources.r1.type=TAILDIR
#偏移量文件
a1.sources.r1.positionFile = /opt/module/flume/data/taildir_position.json
#文件的组，可以定义多种
a1.sources.r1.filegroups = f1 f2
#第一组监控的是test1文件夹中的什么文件：.log文件
a1.sources.r1.filegroups.f1 = /opt/module/flume/data/test1/.*log
#第二组监控的是test2文件夹中的什么文件：以.txt结尾的文件
a1.sources.r1.filegroups.f2 = /opt/module/flume/data/test2/.*txt
 
#描述sink
a1.sinks.s1.type=file_roll
a1.sinks.s1.sink.directory=/home/work/rolldata
a1.sinks.s1.sink.rollInterval=0
 
#描述内存channel
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
 
#位channel 绑定 source和sink
a1.sources.r1.channels=c1
a1.sinks.s1.channel=c1