2.7 , and Sqoop integration
Sqoop supports additional import targets beyond HDFS and Hive. Sqoop can also import records into a table in HBase.
We have previously learned how to use Sqoop in Hadoop import and export data clusters work and relational databases, let's learn about the use of Sqoop in HBase and RDBMS dumps data.
Related parameters:
| parameter |
description |
| --column-family <family> |
Sets the target column family for the import Set target column group introduced. |
| --hbase-create-table |
If specified, create missing HBase tables Whether to automatically create a non-existent HBase table (which means, do not need to manually advance in HBase established in the first table) |
| --hbase-row-key <col> |
Specifies which input column to use as the row key.In case, if input table contains composite key, then <col> must be in the form of a comma-separated list of composite key attributes. mysql value which column as HBase a rowkey , if rowkey a key combination, are separated by commas. (Note: Avoid rowkey repetition) |
| --hbase-table <table-name> |
Specifies an HBase table to use as the target instead of HDFS. Specifies the data to be imported into HBase which of the tables. |
| --hbase-bulkload |
Enables bulk loading. Allow bulk import form. |
1) Case
Objectives: The RDBMS extracted data into HBase in
Achieved in stages:
( 1) Configuration Sqoop -env.sh , add the following:
| export HBASE_HOME=/home/admin/modules/hbase-1.3.6 |
( 2 ) In Mysql a new database DB_library , a table book
| CREATE DATABASE db_library; CREATE TABLE db_library.book( id int(4) PRIMARY KEY NOT NULL AUTO_INCREMENT, name VARCHAR(255) NOT NULL, price VARCHAR(255) NOT NULL); |
(3) inserting some data into the table
| INSERT INTO db_library.book (name, price) VALUES('Lie Sporting', '30'); INSERT INTO db_library.book (name, price) VALUES('Pride & Prejudice', '70'); INSERT INTO db_library.book (name, price) VALUES('Fall of Giants', '50'); |
(4) perform Sqoop operations to import data
| $ bin/sqoop import \ --connect jdbc:mysql://hadoop001:3306/db_library\ --username root \ --password root\ --table book \ --columns "id,name,price" \ --column-family "info" \ --hbase-create-table \ --hbase-row-key "id" \ --hbase-table "hbase_book" \ --num-mappers 1 \ --split-by id |
尖叫提示：sqoop1.4.6只支持HBase1.0.1之前的版本的自动创建HBase表的功能
解决方案：手动创建HBase表
| hbase> create 'hbase_book','info' |
(5) 在HBase中scan这张表得到如下内容
| hbase> scan ‘hbase_book’ |
思考：尝试使用复合键作为导入数据时的rowkey。