场景:flume处理数据到HDFS,文件名如:/data/flume/logs201804160805
代码:sink端
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://hadoop:9000/data/flume/logs/%Y%m%D%H%M
a1.sinks.k1.hdfs.batchSize=10
a1.sinks.k1.hdfs.useLocalTimeStamp = true //在此处添加这个参数报错解决
a1.sinks.k1.hdfs.filePrefix=page-views
a1.sinks.k1.hdfs.fileType= CompressedStream
a1.sinks.k1.hdfs.writeFormat=Text
a1.sinks.k1.hdfs.codeC=bzip2
a1.sinks.k1.hdfs.round=true
a1.sinks.k1.hdfs.roundValue=1
a1.sinks.k1.hdfs.roundUnit=minute
总结
每个文件的记录,都需要时间戳,该参数调节为使用本地时间
- hdfs.useLocalTimeStamp
Use the local time (instead of the timestamp from the event header) while replacing the escape sequences.