Comprehensive data acquisition and storage ----- comprehensive treatment experiments kafka, flume, and a hive of data

In turn enter the command ssh service is turned on, open mysql, start hdfs, start yarm, start Zookeepr

 Enter the hive into the hive

Create a table to put the data: create table kafkatest (id int, name string, age int) clustered by (id) into 2 buckets stored as orc tblproperties ( 'transactional' = 'true');

Then enter exit; to exit the hive

Restart a terminal, input cd ~ / bigdata / kafka_2.11-1.0.0 kafka enter the directory.

输入:nohup bin/kafka-server-start.sh config/server.properties >~/bigdata/kafka_2.11-1.0.0/logs/server.log 2>&1 &

Background start kafka.

Input bin / kafka-topics.sh --create --zookeeper localhost: 2181 --replication-factor 1 --partitions 1 --topic kafkatest create a kafka the topic.

Then start a terminal, enter: cd ~ / bigdata / apache-flume-1.9.0-bin

Flume into the directory, enter: bin/flume-ng agent --conf conf/ --conf-file conf/kafkatohstart flume.

 Back to the terminal (second terminal opened), the input of the next directory kafka: bin / kafka-console-producer.sh --broker-list localhost: 9092 --topic kafkatest start kafka the consumer.

向启动好的kafka中输入1,a,3(1对应hive表中的id,a对应name,3对应age,可以进行适当更换)。发现flume的终端在执行任务。
最后启动一个新的终端,输入hive,输入select * from kafkatest;可以查看到刚才键入的数据。

输入exit;退出hive。

Guess you like

Origin blog.csdn.net/weixin_44961794/article/details/91127737