Sqoop principle

1 Overview

Sqoop is a tool under Apache that "transfers data between Hadoop and relational databases"

• Import data: Import data from MySQL and Oracle to Hadoop's HDFS, HIVE, HBASE and other data storage systems
• Export data: Export data from Hadoop file system to relational database

2. Working mechanism of Sqoop

• Translate import and export commands into MapReduce program implementation
• The translated MapReduce is mainly customized for Inputformat and outputformat.

3. Comparison of Sqoop1 and Sqoop2 architectures

Sqoop has evolved into two different architectures during its development.

3.1. Sqoop1 architecture

• The version number is 1.4.x for sqoop1
• In terms of architecture: sqoop1 uses the sqoop client to submit directly
• Access method: CLI console access
• Security: Specify the user database name and password in the command or script

3.2. Sqoop2 architecture

• The version number is 1.99x for sqoop2
• In terms of architecture: sqoop2 introduces sqoop server to implement centralized management of connectors.
• Access method: REST API, JAVA API, WEB UI and CLI console.

Compare	Sqoop1	Sqoop2
Architecture	Just use a Sqoop client	Introduced Sqoop Server centralized management Connector, rest api, web UI, and introduced security mechanism
deploy	Deployment is simple, installation requires root permissions, and the connector must conform to the JBDC model	The architecture is slightly complex and the configuration and deployment are cumbersome.
use	The command method is error-prone, the format is tightly coupled, cannot support all data types, and the security mechanism is not perfect, such as password exposure.	Multiple interaction methods, command line, WebUI, rest API, Connector centralized management, all connections are installed on Sqoop Server, perfect permission management mechanism, Connector is standardized, only responsible for reading and writing data

4. Sqoop installation and deployment

Sqoop installation is very simple, you can use it after unzipping and making simple modifications.

4.1. Step 1: Download the installation package

https://mirrors.bfsu.edu.cn/apache/sqoop/1.4.7

4.2. Step 2: Upload and decompress

Upload the installation package we downloaded to the /kkb/soft path of the ==node03== server, and then decompress it.

cd /opt/soft/
tar -xzvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/install

4.3. Step 3: Modify the configuration file

cd /opt/install/sqoop-1.4.7.bin__hadoop-2.6.0/conf/
mv sqoop-env-template.sh sqoop-env.sh
vim sqoop-env.sh

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/kkb/install/hadoop-3.1.4

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/kkb/install/hadoop-3.1.4

#set the path to where bin/hbase is available
export HBASE_HOME=/kkb/install/hbase-2.2.2

#Set the path to where bin/hive is available
export HIVE_HOME=/kkb/install/apache-hive-3.1.2

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/kkb/install/apache-zookeeper-3.6.2-bin

4.4. Step 4: Add two necessary jar packages

Sqoop requires two additional dependent jar packages. Add the two jar packages in the courseware materials to the lib directory of sqoop.

cd /opt/soft
cp java-json.jar mysql-connector-java-5.1.38.jar /opt/install/sqoop-1.4.7.bin__hadoop-2.6.0/lib/

4.5. Step 5: Configure Sqoop environment variables

Modify and add as follows

sudo vim /etc/profile	
export SQOOP_HOME=/kkb/install/sqoop-1.4.7.bin__hadoop-2.6.0
export PATH=:$SQOOP_HOME/bin:$PATH

Make environment variables effective

source /etc/profile

4.6. Sqoop help has warning logs

Execute the command `sqoop help` and there will be a warning log.

solution

[hadoop@node03 bin]$ pwd
/opt/install/sqoop-1.4.7.bin__hadoop-2.6.0/bin
# 搜索HCAT_HOME,将下图红框内容注释掉
[hadoop@node03 bin]$ vim configure-sqoop

4.7. Sqoop help has errors

There is an error when running sqoop help:

• • Error: Main class org.apache.hadoop.hbase.util.GetJavaProperty cannot be found or cannot be loaded

This error does not affect sqoop; it can be ignored

• • If you have mysophobia, you can see the following ways to solve it

4.7.1. Solution 1

The simple way is to directly replace the hbase files in the directory `/kkb/install/hbase-2.2.2/bin` of the three nodes in the cluster with the modified hbase files.

Just restart the hbase cluster

4.7.2. Solution 2

node01 modifies the hbase command file

[hadoop@node01 bin]$ cd /opt/install/hbase-2.2.2/bin/
[hadoop@node01 bin]$ vim hbase

Add the content of the first red box; pay attention to the indentation

    # Needed for GetJavaProperty check below
    add_to_cp_if_exists "${HBASE_HOME}/hbase-server/target/classes"

Comment out the content of the second red box

Continue to search for the hbase file and locate the following yellow box content

Add the code of ① in the figure below at line 322 (the first yellow box) in the figure above; == Pay attention to the indent ==

  temporary_cp=
  for f in "${HBASE_HOME}"/lib/hbase-server*.jar; do
    if [[ ! "${f}" =~ ^.*\-tests\.jar$ ]]; then
      temporary_cp=":$f"
    fi
  done

Modify the content of the second yellow box in the above picture to the code in the picture below; == Pay attention to the indentation ==

  HADOOP_JAVA_LIBRARY_PATH=$(HADOOP_CLASSPATH="$CLASSPATH${temporary_cp}" "${    HADOOP_IN_PATH}" \

Save and exit

Distribute hbase to two other nodes

[hadoop@node01 bin]$ pwd
/opt/install/hbase-2.2.2/bin
[hadoop@node01 bin]$ scp hbase node02:$PWD
[hadoop@node01 bin]$ scp hbase node03:$PWD

Restart hbase

5. Sqoop data import

5.1. List all databases

• Command line view help

sqoop help
sqoop list-databases --help

List all databases on node03 host

sqoop list-databases --connect jdbc:mysql://node03:3306/ --username root --password 123456

View all data tables under a certain database (replace the database name hive with one of your own database names)

sqoop list-tables --connect jdbc:mysql://node03:3306/hive --username root --password 123456

5.2. Prepare table data

There is a library userdb in mysql with three tables: emp, emp_add and emp_conn

• Table emp:

id	name	deg	salary	dept
1201	gopal	manager	50,000	TP
1202	manisha	Proof reader	50,000	TP
1203	khalil	php dev	30,000	AC
1204	Prasanth	php dev	30,000	AC
1205	kranthi	admin	20,000	TP

• Table emp_add:

id	hno	street	city
1201	288A	vgiri	jublee
1202	108I	aoc	sec-bad
1203	144Z	pgutta	hyd
1204	78B	old city	sec-bad
1205	720X	hitec	sec-bad

• Table emp_conn:

id	phono	email
1201	2356742	[email protected]
1202	1661663	[email protected]
1203	8887776	[email protected]
1204	9988774	[email protected]
1205	1231231	[email protected]

• The table creation statement is as follows:

CREATE DATABASE /*!32312 IF NOT EXISTS*/`userdb` /*!40100 DEFAULT CHARACTER SET utf8 */;

USE `userdb`;

DROP TABLE IF EXISTS `emp`;

CREATE TABLE `emp` (
  `id` INT(11) DEFAULT NULL,
  `name` VARCHAR(100) DEFAULT NULL,
  `deg` VARCHAR(100) DEFAULT NULL,
  `salary` INT(11) DEFAULT NULL,
  `dept` VARCHAR(10) DEFAULT NULL,
  `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `update_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `is_delete` BIGINT(20) DEFAULT '1'
) ENGINE=INNODB DEFAULT CHARSET=latin1;

INSERT  INTO `emp`(`id`,`name`,`deg`,`salary`,`dept`) VALUES (1201,'gopal','manager',50000,'TP'),(1202,'manisha','Proof reader',50000,'TP'),(1203,'khalil','php dev',30000,'AC'),(1204,'prasanth','php dev',30000,'AC'),(1205,'kranthi','admin',20000,'TP');

DROP TABLE IF EXISTS `emp_add`;

CREATE TABLE `emp_add` (
  `id` INT(11) DEFAULT NULL,
  `hno` VARCHAR(100) DEFAULT NULL,
  `street` VARCHAR(100) DEFAULT NULL,
  `city` VARCHAR(100) DEFAULT NULL,
  `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `update_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `is_delete` BIGINT(20) DEFAULT '1'
) ENGINE=INNODB DEFAULT CHARSET=latin1;

INSERT  INTO `emp_add`(`id`,`hno`,`street`,`city`) VALUES (1201,'288A','vgiri','jublee'),(1202,'108I','aoc','sec-bad'),(1203,'144Z','pgutta','hyd'),(1204,'78B','old city','sec-bad'),(1205,'720X','hitec','sec-bad');

DROP TABLE IF EXISTS `emp_conn`;
CREATE TABLE `emp_conn` (
  `id` INT(100) DEFAULT NULL,
  `phno` VARCHAR(100) DEFAULT NULL,
  `email` VARCHAR(100) DEFAULT NULL,
  `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `update_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `is_delete` BIGINT(20) DEFAULT '1'
) ENGINE=INNODB DEFAULT CHARSET=latin1;

INSERT  INTO `emp_conn`(`id`,`phno`,`email`) VALUES (1201,'2356742','[email protected]'),(1202,'1661663','[email protected]'),(1203,'8887776','[email protected]'),(1204,'9988774','[email protected]'),(1205,'1231231','[email protected]');

5.3. Import database table data to HDFS

• Before using the sqoop command to import and export data, you must first start the hadoop cluster.
• The following command is used to import HDFS from emp table in MySQL database server.

sqoop import --connect jdbc:mysql://node03:3306/userdb --password 123456 --username root --table emp -m 1

• If executed successfully, you will get the following output.

• In order to verify the data imported in HDFS, use the following command to view the imported data

hdfs dfs -ls /user/hadoop/emp

• There are parameters above --m 1, which means that only one map task will be started to import data.
• If you want to open multiple map tasks, you need to add them to the command --split-by column-name, as follows, where the number of maps is 4

sqoop import --connect jdbc:mysql://node03:3306/userdb --password 123456 --username root --table emp -m 4 --split-by id

5.4. Import HDFS specified directory

• When importing table data into HDFS using Sqoop import tool, we can specify the target directory.
• Use the parameter --target-dir to specify the export destination.
• Use the parameter --delete-target-dir to determine whether the export directory exists. If it exists, delete it.

sqoop import --connect jdbc:mysql://node03:3306/userdb --username root --password 123456 --delete-target-dir --table emp --target-dir /sqoop/emp -m 1

• View exported data

hdfs dfs -text /sqoop/emp/part-m-00000

• It will separate the data and fields of the emp_add table with commas (,).

1201,gopal,manager,50000,TP
1202,manisha,Proof reader,50000,TP
1203,khalil,php dev,30000,AC
1204,prasanth,php dev,30000,AC
1205,kranthi,admin,20000,TP

5.5. Import to the hdfs specified directory and specify the separator between fields

sqoop import --connect jdbc:mysql://node03:3306/userdb --username root --password 123456 --delete-target-dir --table emp --target-dir /sqoop/emp2 -m 1 --fields-terminated-by '\t'

• View file contents

hdfs dfs -text /sqoop/emp2/part-m-00000

5.6. Import the relational table into Hive

1) Step one: copy the jar package

To import the data from our mysql table directly into the hive table, we need to copy a jar package of hive called hive-exec-3.1.2.jar to the lib directory of sqoop.

cd /kkb/install/apache-hive-3.1.2/lib/
cp hive-exec-3.1.2.jar /kkb/install/sqoop-1.4.7.bin__hadoop-2.6.0/lib/

2) Step 2: Prepare hive database and tables

• Import the data from our mysql into the hive table

hive (default)> create database sqooptohive;
hive (default)> use sqooptohive;
hive (sqooptohive)> create external table emp_hive(id int,name string,deg string,salary int ,dept string) row format delimited fields terminated by '\001';

3) Step 3: Start importing

sqoop import --connect jdbc:mysql://node03:3306/userdb --username root --password 123456 --table emp --fields-terminated-by '\001' --hive-import --hive-table sqooptohive.emp_hive --hive-overwrite --delete-target-dir --m 1

4) Step 4: View hive table data

select * from emp_hive;

5.7. Import relational tables into hive and automatically create hive tables

• We can also import our mysql table directly into the hive table through commands

sqoop import --connect jdbc:mysql://node03:3306/userdb --username root --password 123456 --table emp_conn --hive-import -m 1 --hive-database sqooptohive

Through this command, we can directly pour the data and table structure in our mysql table into hive.

5.8. Import mysql table data into hbase

Special note: The latest version 1.4.7 of sqoop1 is currently only compatible with hbase 1.x

• First you need to open the hbase cluster

1) Step one: Modify the sqoop configuration file

• To import and export HBase data from sqoop, you need to modify the sqoop configuration file sqoop-env.sh.

cd /opt/install/sqoop-1.4.7.bin__hadoop-2.6.0/conf
vim sqoop-env.sh

Modify the HBASE_HOME value as follows:

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/kkb/install/hadoop-3.1.4

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/kkb/install/hadoop-3.1.4

#set the path to where bin/hbase is available
export HBASE_HOME=/kkb/install/hbase-2.2.2

#Set the path to where bin/hive is available
export HIVE_HOME=/kkb/install/apache-hive-3.1.2

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/kkb/install/apache-zookeeper-3.6.2-bin

2) Step 2: Create a database and database table in mysql and insert data

• Create database table

CREATE DATABASE IF NOT EXISTS library;
USE library;
CREATE TABLE book(
id INT(4) PRIMARY KEY NOT NULL AUTO_INCREMENT, 
NAME VARCHAR(255) NOT NULL, 
price VARCHAR(255) NOT NULL);

• Insert data

INSERT INTO book(NAME, price) VALUES('Lie Sporting', '30'); 
INSERT INTO book (NAME, price) VALUES('Pride & Prejudice', '70'); 
INSERT INTO book (NAME, price) VALUES('Fall of Giants', '50'); 

3) Step 3: Import the data from the mysql table into the HBase table

• Execute the following command to import the data in the mysql table into HBase

sqoop import \
--connect jdbc:mysql://node03:3306/library \
--username root \
--password 123456 \
--table book \
--columns "id,name,price" \
--column-family "info" \
--hbase-create-table \
--hbase-row-key "id" \
--hbase-table "hbase_book" \
--num-mappers 1 \
--split-by id

4) Step 4: View table data in HBase

• Enter the shell client of hbase and view the data through scan

hbase(main):057:0> scan 'hbase_book'
ROW           COLUMN+CELL                            
 1            column=info:name, timestamp=1550634017823, value=Lie Sporting   
 1            column=info:price, timestamp=1550634017823, value=30       
 2            column=info:name, timestamp=1550634017823, value=Pride & Prejudice 
 2            column=info:price, timestamp=1550634017823, value=70       
 3            column=info:name, timestamp=1550634017823, value=Fall of Giants  
 3            column=info:price, timestamp=1550634017823, value=50

5.9. Import table data subsets

• We can import tables using the Sqoop import tool, a subset of the "where" clause. It executes the corresponding SQL query in the respective database server and stores the results in the target directory of HDFS.
• The syntax of where clause is as follows.

 --where <condition>  

• Search according to the conditions, and use the --where parameter to find all the data whose city field value is sec-bad in the table emp_add and import it to hdfs.

bin/sqoop import \
--connect jdbc:mysql://node03:3306/userdb \
--username root --password 123456 --table emp_add \
--target-dir /sqoop/emp_add -m 1 --delete-target-dir \
--where "city = 'sec-bad'"

5.10. SQL statement search and import into HDFS

• We can also specify our sql statement through the –query parameter, and filter our data for import through the sql statement.

bin/sqoop import \
--connect jdbc:mysql://node03:3306/userdb --username root --password 123456 \
--delete-target-dir -m 1 \
--query 'select phno from emp_conn where 1=1 and  $CONDITIONS' \
--target-dir /sqoop/emp_conn

• View hdfs data content

hdfs dfs -text /sqoop/emp_conn/part*

Notice:

1. 1. When using sql statements to search, you cannot add the parameter --table
   2. And a where condition must be added,
   3. And the where condition must be followed by the string $CONDITIONS.
   4. And this sql statement must use single quotes, not double quotes.

5.11. Incremental import

• In actual work, data import often only requires the import of incremental data. It is not necessary to import all the data in the table into hive or hdfs. There will definitely be duplicate data, so we Generally, some fields are selected for incremental import. In order to support incremental import, sqoop also takes this situation into consideration and supports incremental import of data.
• Incremental import is a technique of importing only rows from newly added tables.
• It requires adding 'incremental', 'check-column', and 'last-value' options to perform incremental import.
• The following syntax is used for Sqoop import command increment options.

--incremental <mode>  
--check-column <column name>  
--last-value <last check column value>     

1) The first incremental import is implemented using the above options

• Import all data with ID greater than 1202 in the emp table
• Note: When incrementally importing, you must not add the parameter --delete-target-dir, otherwise an error will be reported.

bin/sqoop import \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456 \
--table emp \
--incremental append \
--check-column id \
--last-value 1202 \
-m 1 \
--target-dir /sqoop/increment

• View data content

hdfs dfs -text /sqoop/increment/part*

2) The second type of incremental import is implemented through the --where condition

• Or we can use --where to control the selection of data, which will be more accurate.

bin/sqoop import \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456 \
--table emp \
--incremental append \
--where "create_time > '2018-06-17 00:00:00' and is_delete='1' and create_time < '2018-06-17 23:59:59'" \
--target-dir /sqoop/incement2 \
--check-column id \
--m 1

• Assignment: How to implement incremental import into hive table? ? ?
• Interview question: How to solve data reduction? ? ?

6. Sqoop data export

6.1. Export data files from HDFS to RDBMS database

• The target table must exist in the target database before exporting.
• The default operation is to insert data from the file into the table using the INSERT statement
• In update mode, an UPDATE statement is generated to update table data.
• The data is in the following directory/sqoop/emp in HDFS, and the data content is as follows

1201,gopal,manager,50000,TP,2018-06-17 18:54:32.0,2018-06-17 18:54:32.0,1
1202,manisha,Proof reader,50000,TP,2018-06-15 18:54:32.0,2018-06-17 20:26:08.0,1
1203,khalil,php dev,30000,AC,2018-06-17 18:54:32.0,2018-06-17 18:54:32.0,1
1204,prasanth,php dev,30000,AC,2018-06-17 18:54:32.0,2018-06-17 21:05:52.0,0
1205,kranthi,admin,20000,TP,2018-06-17 18:54:32.0,2018-06-17 18:54:32.0,1

6.1.1. Step 1: Create mysql table

use userdb;

CREATE TABLE `emp_out` (
  `id` INT(11) DEFAULT NULL,
  `name` VARCHAR(100) DEFAULT NULL,
  `deg` VARCHAR(100) DEFAULT NULL,
  `salary` INT(11) DEFAULT NULL,
  `dept` VARCHAR(10) DEFAULT NULL,
  `create_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `update_time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `is_delete` BIGINT(20) DEFAULT '1'
) ENGINE=INNODB DEFAULT CHARSET=utf8;

6.1.2. Step 2: Execute the export command

Use kkb to export data and export hdfs data to mysql.

bin/sqoop export \
--connect jdbc:mysql://node03:3306/userdb \
--username root --password 123456 \
--table emp_out \
--export-dir /sqoop/emp \
--input-fields-terminated-by ","

6.1.3. Step 3: Verify mysql table data

6.2. Export data from Hbase to mysql

• Export the data in the hbase_book table to mysql
• Note: sqoop does not support us to directly export data from HBase, so we can export it through the following conversion: Hbase→hive external table→hive internal table→through sqoop→mysql

6.2.1. Step 1: Create hive external table

• Enter the hive client, create an hive external table, and map the hbase_book table in hbase.

CREATE EXTERNAL TABLE course.hbase2mysql (id int,name string,price int)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
"hbase.columns.mapping" =
":key,info:name, info:price"
)
TBLPROPERTIES( "hbase.table.name" = "hbase_book",
"hbase.mapred.output.outputtable" = "hbase2mysql");

6.2.2. Step 2: Create hive internal table and insert external table data into the internal table

• Enter the hive client and execute the following command to create the hive internal table and insert the data from the external table into the hive internal table.

CREATE TABLE course.hbase2mysqlin(id int,name string,price int);

6.2.3. Step 3: Insert external table data into the internal table

• Enter the hive client and execute the following command to insert the hive external table data into the hive internal table.

insert overwrite table course.hbase2mysqlin select * from course.hbase2mysql;

6.2.4. Step 4: Clear mysql table data

• Enter the mysql client and execute the following command to clear the mysql table data

use library;
TRUNCATE TABLE book;

6.2.5. Step 5: Execute sqoop to export hive internal table data to

bin/sqoop export --connect jdbc:mysql://node03:3306/library --username root --password 123456 --table book --export-dir /user/hive/warehouse/course.db/hbase2mysqlin --input-fields-terminated-by '\001' --input-null-string '\\N' --input-null-non-string '\\N';

Then query the book table data of mysql

mysql> select * from book; 

7. Sqoop common commands and parameters

7.1. List of common commands

• Here is a list of some common parameters used in Sqoop operations for reference. Those who need in-depth study can refer to the source code of the corresponding class.

serial number	Order	kind	illustrate
1	import	ImportTool	Import data into the cluster
2	export	ExportTool	Export cluster data
3	coding	CodeGenTool	Get data from a certain table in the database, generate Java and package Jar
4	create-hive-table	CreateHiveTableTool	Create Hive table
5	eval	EvalSqlTool	View SQL execution results
6	import-all-tables	ImportAllTablesTool	Import all tables under a database into HDFS
7	job	JobTool	Used to generate a sqoop task. After generation, the task will not be executed unless a command is used to execute the task.
8	list-databases	ListDatabasesTool	List all database names
9	list-tables	ListTablesTool	List all tables under a database
10	merge	MergeTool	Combine data in different directories in HDFS and store it in the specified directory
11	metastore	MetastoreTool	Record the metadata information of the sqoop job. If the metastore instance is not started, the default metadata storage directory is: ~/.sqoop. If you want to change the storage directory, you can change it in the configuration file sqoop-site.xml.
12	help	HelpTool	Print sqoop help information
13	version	VersionTool	打印sqoop版本信息

7.2、命令&参数详解

• 刚才列举了一些Sqoop的常用命令，对于不同的命令，有不同的参数，让我们来一一列举说明。首先来我们来介绍一下公用的参数，所谓公用参数，就是大多数命令都支持的参数。

7.2.1、公用参数：数据库连接

序号	参数	说明
1	--connect	连接关系型数据库的URL
2	--connection-manager	指定要使用的连接管理类
3	--driver	JDBC的driver class
4	--help	打印帮助信息
5	--password	连接数据库的密码
6	--username	连接数据库的用户名
7	--verbose	在控制台打印出详细信息

7.2.2、公用参数：import

序号	参数	说明
1	--enclosed-by <char>	给字段值前后加上指定的字符
2	--escaped-by <char>	对字段中的双引号加转义符
3	--fields-terminated-by <char>	设定每个字段是以什么符号作为结束，默认为逗号
4	--lines-terminated-by <char>	设定每行记录之间的分隔符，默认是\n
5	--mysql-delimiters	Mysql默认的分隔符设置，字段之间以逗号分隔，行之间以\n分隔，默认转义符是\，字段值以单引号包裹。
6	--optionally-enclosed-by <char>	给带有双引号或单引号的字段值前后加上指定字符。

7.2.3、公用参数：export

序号	参数	说明
1	--input-enclosed-by <char>	对字段值前后加上指定字符
2	--input-escaped-by <char>	对含有转移符的字段做转义处理
3	--input-fields-terminated-by <char>	字段之间的分隔符
4	--input-lines-terminated-by <char>	行之间的分隔符
5	--input-optionally-enclosed-by <char>	给带有双引号或单引号的字段前后加上指定字符

7.2.4、公用参数：hive

序号	参数	说明
1	--hive-delims-replacement <arg>	用自定义的字符串替换掉数据中的\r\n和\013 \010等字符
2	--hive-drop-import-delims	在导入数据到hive时，去掉数据中的\r\n\013\010这样的字符
3	--map-column-hive <map>	生成hive表时，可以更改生成字段的数据类型
4	--hive-partition-key	创建分区，后面直接跟分区名，分区字段的默认类型为string
5	--hive-partition-value <v>	导入数据时，指定某个分区的值
6	--hive-home <dir>	hive的安装目录，可以通过该参数覆盖之前默认配置的目录
7	--hive-import	将数据从关系数据库中导入到hive表中
8	--hive-overwrite	覆盖掉在hive表中已经存在的数据
9	--create-hive-table	默认是false，即，如果目标表已经存在了，那么创建任务失败。
10	--hive-table	后面接要创建的hive表,默认使用MySQL的表名
11	--table	指定关系数据库的表名

公用参数介绍完之后，我们来按照命令介绍命令对应的特有参数。

7.2.5、命令&参数：import

• 将关系型数据库中的数据导入到HDFS（包括Hive，HBase）中，如果导入的是Hive，那么当Hive中没有对应表时，则自动创建。

1） 命令

如：导入数据到hive中

$ bin/sqoop import \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456 \
--table emp \
--hive-import

如：增量导入数据到hive中，mode=append

append导入：

$ bin/sqoop import \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456 \
--table emp \
--num-mappers 1 \
--fields-terminated-by "\t" \
--target-dir /user/hive/warehouse/emp \
--check-column id \
--incremental append \
--last-value 3

易错提醒：append不能与--hive-等参数同时使用（Append mode for hive imports is not yet supported. Please remove the parameter --append-mode）

如：增量导入数据到hdfs中，mode=lastmodified

先在mysql中建表并插入几条数据：

mysql> create table company.staff_timestamp(id int(4), name varchar(255), sex varchar(255), last_modified timestamp DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP);
mysql> insert into company.staff_timestamp (id, name, sex) values(1, 'AAA', 'female');
mysql> insert into company.staff_timestamp (id, name, sex) values(2, 'BBB', 'female');

先导入一部分数据：

$ bin/sqoop import \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456 \
--table emp_conn \
--delete-target-dir \
--m 1

再增量导入一部分数据：

mysql> insert into company.staff_timestamp (id, name, sex) values(3, 'CCC', 'female');
$ bin/sqoop import \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456 \
--table emp_conn \
--check-column last_modified \
--incremental lastmodified \
--last-value "2018-0-28 22:20:38" \
-m 1 \
--append

易错提醒：使用lastmodified方式导入数据要指定增量数据是要--append（追加）还是要--merge-key（合并）

易错提醒：--incremental lastmodified模式下，last-value指定的值是会包含于增量导入的数据中。

2）参数

序号	参数	说明
1	--append	将数据追加到HDFS中已经存在的DataSet中，如果使用该参数，sqoop会把数据先导入到临时文件目录，再合并。
2	--as-avrodatafile	将数据导入到一个Avro数据文件中
3	--as-sequencefile	将数据导入到一个sequence文件中
4	--as-textfile	将数据导入到一个普通文本文件中
5	--boundary-query <statement>	边界查询，导入的数据为该参数的值（一条sql语句）所执行的结果区间内的数据。
6	--columns <col1, col2, col3>	指定要导入的字段
7	--direct	直接导入模式，使用的是关系数据库自带的导入导出工具，以便加快导入导出过程。
8	--direct-split-size	在使用上面direct直接导入的基础上，对导入的流按字节分块，即达到该阈值就产生一个新的文件
9	--inline-lob-limit	设定大对象数据类型的最大值
10	--m或–num-mappers	启动N个map来并行导入数据，默认4个。
11	--query或--e <statement>	将查询结果的数据导入，使用时必须伴随参--target-dir，--hive-table，如果查询中有where条件，则条件后必须加上$CONDITIONS关键字
12	--split-by <column-name>	按照某一列来切分表的工作单元，不能与--autoreset-to-one-mapper连用（请参考官方文档）
13	--table <table-name>	关系数据库的表名
14	--target-dir <dir>	指定HDFS路径
15	--warehouse-dir <dir>	与14参数不能同时使用，导入数据到HDFS时指定的目录
16	--where	从关系数据库导入数据时的查询条件
17	--z或--compress	允许压缩
18	--compression-codec	指定hadoop压缩编码类，默认为gzip(Use Hadoop codec default gzip)
19	--null-string <null-string>	string类型的列如果null，替换为指定字符串
20	--null-non-string <null-string>	非string类型的列如果null，替换为指定字符串
21	--check-column <col>	作为增量导入判断的列名
22	--incremental <mode>	mode：append或lastmodified
23	--last-value <value>	指定某一个值，用于标记增量导入的位置

7.2.6、命令&参数：export

从HDFS（包括Hive和HBase）中奖数据导出到关系型数据库中。

1）命令

如：

$ bin/sqoop export \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456 \
--table emp_add \
--export-dir /user/company \
--input-fields-terminated-by "\t" \
--num-mappers 1

2）参数

序号	参数	说明
1	--direct	利用数据库自带的导入导出工具，以便于提高效率
2	--export-dir <dir>	存放数据的HDFS的源目录
3	-m或--num-mappers <n>	启动N个map来并行导入数据，默认4个
4	--table <table-name>	指定导出到哪个RDBMS中的表
5	--update-key <col-name>	对某一列的字段进行更新操作
6	--update-mode <mode>	updateonly allowinsert(默认)
7	--input-null-string <null-string>	请参考import该类似参数说明
8	--input-null-non-string <null-string>	请参考import该类似参数说明
9	--staging-table <staging-table-name>	创建一张临时表，用于存放所有事务的结果，然后将所有事务结果一次性导入到目标表中，防止错误。
10	--clear-staging-table	如果第9个参数非空，则可以在导出操作执行前，清空临时事务结果表

7.2.7、命令&参数：codegen

将关系型数据库中的表映射为一个Java类，在该类中有各列对应的各个字段。如：

$ bin/sqoop codegen \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456 \
--table emp_add \
--bindir /home/admin/Desktop/staff \
--class-name Staff \
--fields-terminated-by "\t"

序号	参数	说明
1	--bindir <dir>	指定生成的Java文件、编译成的class文件及将生成文件打包为jar的文件输出路径
2	--class-name <name>	设定生成的Java文件指定的名称
3	--outdir <dir>	生成Java文件存放的路径
4	--package-name <name>	包名，如com.z，就会生成com和z两级目录
5	--input-null-non-string <null-str>	在生成的Java文件中，可以将null字符串或者不存在的字符串设置为想要设定的值（例如空字符串）
6	--input-null-string <null-str>	将null字符串替换成想要替换的值（一般与5同时使用）
7	--map-column-java <arg>	数据库字段在生成的Java文件中会映射成各种属性，且默认的数据类型与数据库类型保持对应关系。该参数可以改变默认类型，例如：--map-column-java id=long, name=String
8	--null-non-string <null-str>	在生成Java文件时，可以将不存在或者null的字符串设置为其他值
9	--null-string <null-str>	在生成Java文件时，将null字符串设置为其他值（一般与8同时使用）
10	--table <table-name>	对应关系数据库中的表名，生成的Java文件中的各个属性与该表的各个字段一一对应

7.2.8、命令&参数：create-hive-table

生成与关系数据库表结构对应的hive表结构。

命令：

$ bin/sqoop create-hive-table \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456 \
--table emp_add \
--hive-table emp_add

参数：

序号	参数	说明
1	--hive-home <dir>	Hive的安装目录，可以通过该参数覆盖掉默认的Hive目录
2	--hive-overwrite	覆盖掉在Hive表中已经存在的数据
3	--create-hive-table	默认是false，如果目标表已经存在了，那么创建任务会失败
4	--hive-table	后面接要创建的hive表
5	--table	指定关系数据库的表名

7.2.9、命令&参数：eval

可以快速的使用SQL语句对关系型数据库进行操作，经常用于在import数据之前，了解一下SQL语句是否正确，数据是否正常，并可以将结果显示在控制台。

命令：

$ bin/sqoop eval \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456 \
--query "SELECT * FROM emp"

参数：

序号	参数	说明
1	--query或--e

7.2.10、命令&参数：import-all-tables

可以将RDBMS中的所有表导入到HDFS中，每一个表都对应一个HDFS目录

命令：

$ bin/sqoop import-all-tables \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456 \
--warehouse-dir /all_tables

参数：

序号	参数	说明
1	--as-avrodatafile	这些参数的含义均和import对应的含义一致
2	--as-sequencefile
3	--as-textfile
4	--direct
5	--direct-split-size <n>
6	--inline-lob-limit <n>
7	--m或—num-mappers <n>
8	--warehouse-dir <dir>
9	-z或--compress
10	--compression-codec

7.2.11、命令&参数：job

用来生成一个sqoop任务，生成后不会立即执行，需要手动执行。

命令：

$ bin/sqoop job \
 --create myjob -- import-all-tables \
 --connect jdbc:mysql://node03:3306/userdb \
 --username root \
 --password 123456
$ bin/sqoop job \
--list
$ bin/sqoop job \
--exec myjob

易错提醒：注意import-all-tables和它左边的--之间有一个空格

易错提醒：如果需要连接metastore，则--meta-connect jdbc:hsqldb:hsql://node03:16000/sqoop

参数：

序号	参数	说明
1	--create <job-id>	创建job参数
2	--delete <job-id>	删除一个job
3	--exec <job-id>	执行一个job
4	--help	显示job帮助
5	--list	显示job列表
6	--meta-connect <jdbc-uri>	用来连接metastore服务
7	--show <job-id>	显示一个job的信息
8	--verbose	打印命令运行时的详细信息

易错提醒：在执行一个job时，如果需要手动输入数据库密码，可以做如下优化

<property>
	<name>sqoop.metastore.client.record.password</name>
	<value>true</value>
	<description>If true, allow saved passwords in the metastore.</description>
</property>

7.2.12、命令&参数：list-databases

命令：

$ bin/sqoop list-databases \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456

参数：与公用参数一样

7.2.13、命令&参数：list-tables

命令：

$ bin/sqoop list-tables \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456

参数：与公用参数一样

7.2.14、命令&参数：merge

将HDFS中不同目录下面的数据合并在一起并放入指定目录中

数据环境：

new_staff
1       AAA     male
2       BBB     male
3       CCC     male
4       DDD     male
old_staff
1       AAA     female
2       CCC     female
3       BBB     female
6       DDD     female

易错提醒：上边数据的列之间的分隔符应该为\t，行与行之间的分割符为\n，如果直接复制，请检查之。

命令：

创建JavaBean：

$ bin/sqoop codegen \
--connect jdbc:mysql://node03:3306/userdb \
--username root \
--password 123456 \
--table emp_conn \
--bindir /home/admin/Desktop/staff \
--class-name EmpConn \
--fields-terminated-by "\t"

开始合并：

$ bin/sqoop merge \
--new-data /test/new/ \
--onto /test/old/ \
--target-dir /test/merged \
--jar-file /home/admin/Desktop/staff/EmpConn.jar \
--class-name Staff \
--merge-key id

结果：

1	AAA	MALE
2	BBB	MALE
3	CCC	MALE
4	DDD	MALE
6	DDD	FEMALE

参数：

序号	参数	说明
1	--new-data <path>	HDFS 待合并的数据目录，合并后在新的数据集中保留
2	--onto <path>	HDFS合并后，重复的部分在新的数据集中被覆盖
3	--merge-key <col>	合并键，一般是主键ID
4	--jar-file <file>	The jar package introduced during the merge is generated through the Codegen tool.
5	--class-name <class>	The corresponding table name or object name, the class is included in the jar package
6	--target-dir <path>	The directory where the merged data is stored in HDFS

7.2.15. Command & parameters: metastore

The metadata information of Sqoop job is recorded. If the service is not started, the default job metadata storage directory is ~/.sqoop, which can be modified in sqoop-site.xml.

Order:

For example: start the metastore service of sqoop

$ bin/sqoop metastore

parameter:

serial number	parameter	illustrate
1	--shutdown	Close metastore

8. Sqoop job

• Note: Sqoop job--run pre-defined data import and export tasks according to the specified process

8.1. Grammar

• Following is the syntax to create Sqoop job.

$ sqoop job (generic-args)  (job-args)    [-- [subtool-name] (subtool-args)]    
$ sqoop-job (generic-args)  (job-args)    [-- [subtool-name] (subtool-args)]   

8.2. Create job (--create)

• Here we create a job called myjob, which can import data from RDBMS table to HDFS.

bin/sqoop job --create myjob2 -- import --connect jdbc:mysql://node03:3306/userdb --username root --password 123456 --table emp --delete-target-dir

• This command creates a job that imports from the employee table of the db library to an HDFS file.

8.3. Verification job (--list)

• The '--list'  parameter is used to verify saved jobs. The following command is used to verify the list of saved Sqoop jobs.

bin/sqoop job --list

• It shows the list of saved jobs.

• Check jobs (--show) '--show'

Parameters are used to check or verify a specific job and its details. The following commands and sample output are used to verify a job named myjob.

$ sqoop job --show myjob

• It shows the tools and their selection, which are used in the job myjob.

8.4. Execute job (--exec)

• The '--exec'  option is used to execute the saved job. The following command is used to execute a saved job called myjob.

bin/sqoop job --exec myjob2

Article reprinted from: 1. Introduction to Sqoop - Zhihu