Hadoop集群客户端安装Sqoop以及数据迁移

将软件包下载后解压,下载地址:https://mirrors.cnnic.cn/apache/

mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop-hadoop

cd /root/sqoop-hadoop/conf

vim sqoop-env.sh(改文件不存在需要新建)

# Hadoop Environment Variables
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
#zookper
export ZOOCFGDIR=/root/zookeeper-3.4.12
#hbase  
export HBASE_HOME=/root/hbase-2.0.0
#HIVE
export HIVE_HOME=/root/hive2
export HCAT_HOME=/root/hive2/hcatalogs

vim /etc/proflie

添加:

#SQOOP_HOME
export SQOOP_HOME=/root/sqoop-hadoop
export PATH=$PATH:$SQOOP_HOME/bin

复制: cp ../MysqlPackages/mysql-connector-java-6.0.6.jar ./lib/

mysql用户授权:

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '12345';

 GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' IDENTIFIED BY '12345';

创建数据库:

mysql> create database sqoop;
Query OK, 1 row affected (0.00 sec)

mysql> use sqoop;
Database changed
mysql>

创建表插入数据:

create table dept(id int,name varchar(20),primary key(id));

insert into dept values(610213,'ApplicationsCould');

insert into dept values(610215,'ApplicationsBigData');

insert into dept values(590108,'SoftwareTech');

查看MySql数据库:

sqoop list-databases --connect jdbc:mysql://node63:3306/ --username root --password 12345

查看数据库的表:

sqoop list-tables --connect jdbc:mysql://node63:3306/sqoop --username root --password 12345

MySql表导入HDFS:

sqoop import --connect jdbc:mysql://node63:3306/sqoop --username root --password 12345 --table dept -m 1 --target-dir /sqoop/dept

通过网页可以看到导入成功:

在终端输出:hdfs dfs -cat /sqoop/dept/part-m-00000

清空mysql中的数据库表,从hdfs导出:

mysql> use sqoop
Database changed
mysql> truncate dept;
Query OK, 0 rows affected (0.22 sec)

sqoop export --connect jdbc:mysql://node63:3306/sqoop --username root --password 12345 --table dept -m 1 --export-dir /sqoop/dept

再查询dept表:

sqopp增量导入HDFS:

首先再mysql表中添加数据:insert into dept values(590101,'ComputerTech');

然后:sqoop import --connect jdbc:mysql://node63:3306/sqoop --username root --password 12345 --table dept -m 1 --target-dir /sqoop/dept --incremental append --check-column id

命令行输出查看:hdfs dfs -cat /sqoop/dept/part-m-00001

将Mysql的dept表导入hive:

首先看hive中没有表:

sqoop import --connect jdbc:mysql://node63:3306/sqoop --username root --password 12345 --table dept -m 1 --hive-import

默认将表文件保存在/user/root/下

如果报如错误:

先删除:hdfs dfs -rm -r -f /user

再将hive 里面的lib下的hive-exec-**.jar 放到sqoop 的lib 下可以解决以下问题

cp ../../hive2/lib/hive-exec-2.3.3.jar ./

登陆hive,可以看到表已经导入了:

清空Mysql的dept表,将hive的数据导入。

sqoop export --connect jdbc:mysql://node63:3306/sqoop --username root --password 12345 --table dept -m 1 --export-dir /hive/warehouse/dept --input-fields-terminated-by '\0001'

再次在mysql中查询:

Mysql与Hbase之间的数据转移:

启动hbase(start-hbase.sh)

输入hbase shell进入到 hbase命令行:

sqoop import --connect jdbc:mysql://node63:3306/sqoop --username root --password 12345 --table 'dept' --hbase-create-table --hbase-table hbase_dept --column-family col_family --hbase-row-key id

创建hbase表:create 'hbase_dept','col_family'

有与版本不兼容问题,导入失败了。可以将mysql数据导入到hdfs,在从hdfs导入hbase,

sqoop1.99之后的版本只能这样做。

猜你喜欢

转载自blog.csdn.net/qq_25948717/article/details/81167631