Kafka stream data to the Apache NiFi HBase

I. Description

In large data platform business scenarios, real-time processing kafka stream data to a necessary capacity; after this post will attempt to access the memory through the data and then the processing Apache NiFi Kafka HBase

II. Having set out

Ⅰ). Arrangement ConsumeKafka_0_10

Tests used kafka0.10 version, so NiFi also choose to release

　　

a). Select ConsumeKafka_0_10

Search Processor in ConsumeKafka_0_10

　　

b). arrangement ConsumeKafka_0_10

1.Kafka Brokers: hostname1:9092,hostname2:9092:hostname3:9092
2.Topic Name(s): entry_index_nifi
3.Group ID: entry_index_nifi

　　

 

Ⅱ). Configuration PutHBaseJSON

a). Select PutHBaseJSON

In the search for PutHBaseJSON Processor

　　

 

b). Configuration PutHBaseJSON

1.HBase Client Service: select the matching version of HBaseClient
2.Table Name: Configure storage HBase table
3.Row Identifier Field Name: Configuration RowKey value
4.Column Family: Column cluster configuration

　　

 

c). Select HBase_1_1_2_ClientService

Search HBase_1_1_2_ClientService in the Processor

　　

 

d). Configuration HBase_1_1_2_ClientService

1.Zookeeper Quorum: hostname1:2181,hostname2:2181,hostname3:2181
2.Zookeeper Client Port: 2181
3.Zookeeper ZNode Parent: /hbase

　　

 

e). Activate HBase_1_1_2_ClientService

Ⅲ). Start Service

可以点击选择单个Processor启动，也可以在空白处点击流程启动

　　

 

Ⅳ).验证结果

a).Kafka源数据

./bin/kafak-console-consumer.sh --zookeeper hostname1:2181,hostname2:2181,hostname3:2181 ---topic entry_index_nifi

　　

b).HBase入库数据

scan 'kafka.entry_index_nifi',{LIMIT=>10}

　　

三.HBase命令

## 创建表空间
create_namespace 'kafka'

## 查看表空间
list_namespace_tables 'kafka'

## 创建表
create 'kafka.entry_index_nifi','cf1'

## 查看表数据
scan 'kafka.entry_index_nifi',{LIMIT=>10}