kafka消息通过logstash导入HDFS

1.安装Logstash
1、下载安装包
2、解压
tar -xzf 文件名
3、下载插件logstash-output-webhdfs-master
4、解压到logstash
5、 安装Gem并更新

# yum install -y gem
# gem -v
# gem update --system
# gem -v

6、编辑Gemfile
添加：gem “logstash-output-webhdfs”, :path => “/home/es/logstash-6.4.2/logstash-output-webhdfs-master”
7、到logstash 目录下执行
bin / logstash-plugin install logstash-output-webhdf --no-verify
./bin/logstash-plugin install --local logstash-output-webhdfs
8、配置/logstash-6.4.2/config 中的 kafka-logstash-hdfs.conf文件：

input {
    kafka {
        bootstrap_servers => ["10.2.5.37:9092"]  //集群地址
        group_id => "logstash-group1"  
                topics => ["ISEVehicleToBMS"] //主题
                consumer_threads => 1
                decorate_events => true
                codec => "json"
                }
}
output {
        webhdfs {
            workers => 2
            host => "10.2.5.203"
            port => 50070
            user => "root"
            path => "/home/logstash/dt=%{+Y}-%{+M}-%{+d}/logstash-%{+H}.log"
            flush_size => 500
            compression => "snappy"
            idle_flush_time => 10
            retry_interval => 0.5
        }
    }

9、运行
./bin/logstash -f ./config/kafka-logstash-hdfs.conf
10、查看
http://10.2.5.203:50070/home/logstash/

11、查看hdfs信息
curl -i “http://10.2.5.203:50070/webhdfs/v1/?user.name=hadoop&op=LISTSTATUS”

HTTP/1.1 200 OK Cache-Control: no-cache Expires: Wed, 27 Feb 2019
06:20:06 GMT Date: Wed, 27 Feb 2019 06:20:06 GMT Pragma: no-cache
Expires: Wed, 27 Feb 2019 06:20:06 GMT Date: Wed, 27 Feb 2019 06:20:06
GMT Pragma: no-cache Content-Type: application/json Set-Cookie:
hadoop.auth=“u=hadoop&p=hadoop&t=simple&e=1551284406236&s=i/qTx/YV7REcR4PPs1ClFhSesmk=”;
Path=/; Expires=???, 27-??-2019 16:20:06 GMT; HttpOnly
Transfer-Encoding: chunked Server: Jetty(6.1.26)

{“FileStatuses”:{“FileStatus”:[

]}}