一 简介
sqoop是apache旗下一款“Hadoop和关系数据库服务器之间传送数据”的工具。
导入数据:MySQL,Oracle导入数据到Hadoop的HDFS、HIVE、HBASE等数据存储系统;
导出数据:从Hadoop的文件系统中导出数据到关系数据库mysql等
站在Apache立场看待数据流转问题,可以分为数据的导入导出:
Import:数据导入。RDBMS----->Hadoop
Export:数据导出。Hadoop---->RDBMS
二 安装
安装sqoop的前提是已经具备java和hadoop的环境
2.1 下载并解压
下载地址
http://archive.cloudera.com/cdh5/cdh/5/
sqoop1版本详细下载地址
http://archive.cloudera.com/cdh5/cdh/5/sqoop-1.4.6-cdh5.14.0.tar.gz
sqoop2版本详细下载地址
http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.5-cdh5.14.0.tar.gz
这里使用sqoop1的版本,下载之后上传到/export/softwares目录下,然后进行解压:
cd /export/softwares
tar -zxvf sqoop-1.4.6-cdh5.14.0.tar.gz -C ../servers/
2.2 修改配置文件
cd sqoop1.4.6/conf/
mv sqoop-env-template.sh sqoop-env.sh
vim sqoop-env.sh
添加:
export HADOOP_COMMON_HOME=/export/servers/hadoop-2.7.5
export HADOOP_MAPRED_HOME=/export/servers/hadoop-2.7.5
export HIVE_HOME=/export/servers/apache-hive-2.1.1-bin
2.3 添加jar包
jar包连接:
链接:https://pan.baidu.com/s/1ESaXp1-RvUlW08Qi6sMkLg
提取码:yamc
第一个mysql的驱动jar包:
这里是从别的地方拷贝的:
cp /export/servers/apache-hive-2.1.1-bin/lib/mysql-connector-java-5.1.38.jar /export/servers/sqoop1.4.6/lib/
第二个java的jsonjar包:
扫描二维码关注公众号,回复:
10863155 查看本文章
本地上传,上面下载即可!
三 启动
测试一下,执行命令:
bin/sqoop list-databases \
--connect jdbc:mysql://192.168.77.120:3306/ \
--username root --password 123456
这个是获取mysql库中,所有的数据库,执行后,应该显示:
[root@node01 sqoop1.4.6]# bin/sqoop list-databases \
> --connect jdbc:mysql://192.168.77.120:3306/ \
> --username root --password 123456
Warning: /export/servers/sqoop1.4.6/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /export/servers/sqoop1.4.6/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
20/04/05 19:15:33 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
20/04/05 19:15:33 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
20/04/05 19:15:34 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
hive
mysql
或者查看版本:
bin/sqoop-version
显示:
[root@node01 sqoop1.4.6]# bin/sqoop-version
Warning: /export/servers/sqoop1.4.6/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /export/servers/sqoop1.4.6/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
20/04/05 19:20:19 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
安装成功!!!