sqoop数据迁移 - 安装介绍

一 简介

sqoop是apache旗下一款“Hadoop和关系数据库服务器之间传送数据”的工具。
导入数据:MySQL,Oracle导入数据到Hadoop的HDFS、HIVE、HBASE等数据存储系统;
导出数据:从Hadoop的文件系统中导出数据到关系数据库mysql等

在这里插入图片描述
在这里插入图片描述
站在Apache立场看待数据流转问题,可以分为数据的导入导出:
Import:数据导入。RDBMS----->Hadoop
Export:数据导出。Hadoop---->RDBMS

二 安装

安装sqoop的前提是已经具备java和hadoop的环境

2.1 下载并解压

下载地址
http://archive.cloudera.com/cdh5/cdh/5/
sqoop1版本详细下载地址
http://archive.cloudera.com/cdh5/cdh/5/sqoop-1.4.6-cdh5.14.0.tar.gz

sqoop2版本详细下载地址
http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.5-cdh5.14.0.tar.gz

这里使用sqoop1的版本,下载之后上传到/export/softwares目录下,然后进行解压:

cd /export/softwares
tar -zxvf sqoop-1.4.6-cdh5.14.0.tar.gz -C ../servers/

2.2 修改配置文件

cd sqoop1.4.6/conf/
mv sqoop-env-template.sh sqoop-env.sh
vim sqoop-env.sh

添加:

export HADOOP_COMMON_HOME=/export/servers/hadoop-2.7.5
export HADOOP_MAPRED_HOME=/export/servers/hadoop-2.7.5
export HIVE_HOME=/export/servers/apache-hive-2.1.1-bin

2.3 添加jar包

jar包连接:

链接:https://pan.baidu.com/s/1ESaXp1-RvUlW08Qi6sMkLg
提取码:yamc

第一个mysql的驱动jar包:
这里是从别的地方拷贝的:

cp /export/servers/apache-hive-2.1.1-bin/lib/mysql-connector-java-5.1.38.jar /export/servers/sqoop1.4.6/lib/

第二个java的jsonjar包:

扫描二维码关注公众号,回复: 10863155 查看本文章

本地上传,上面下载即可!

三 启动

测试一下,执行命令:

bin/sqoop list-databases \
 --connect jdbc:mysql://192.168.77.120:3306/ \
 --username root --password 123456

这个是获取mysql库中,所有的数据库,执行后,应该显示:

[root@node01 sqoop1.4.6]# bin/sqoop list-databases \
>  --connect jdbc:mysql://192.168.77.120:3306/ \
>  --username root --password 123456
Warning: /export/servers/sqoop1.4.6/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /export/servers/sqoop1.4.6/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
20/04/05 19:15:33 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
20/04/05 19:15:33 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
20/04/05 19:15:34 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
hive
mysql

或者查看版本:

bin/sqoop-version

显示:

[root@node01 sqoop1.4.6]# bin/sqoop-version
Warning: /export/servers/sqoop1.4.6/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /export/servers/sqoop1.4.6/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
20/04/05 19:20:19 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6

安装成功!!!

发布了107 篇原创文章 · 获赞 20 · 访问量 2万+

猜你喜欢

转载自blog.csdn.net/beishanyingluo/article/details/105331192