sqoop2 1.99.5 installation +hue

First know what sqoop does.
sqoop is a tool that extracts relational database data to Hadoop. At the same time, it also supports importing the results of hive, pig and other queries into relational databases for storage. Personally, I feel that after using an ecosystem, the same series of products can be used. For example, if you choose cloudera, it is best to use the things under its ecosystem.

wget  http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.5-cdh5.8.0.tar.gz

2. Unzip the sqoop installation file 

tar -zxvf sqoop2-1.99.5-cdh5.8.0.tar.gz

3. Modify sqoop-server's reference to hadoop shared jar, where you can specify environment variables, which are directly a HADOOP_HOME or split after 1.99.7

vi server/conf/catalina.properties

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/data/hdp/2.3.6.0-3796/hadoop/*.jar,/data/hdp/2.3.6.0-3796/hadoop/lib/*.jar,/data/hdp/2.3.6.0-3796/hadoop-hdfs/*.jar,/data/hdp/2.3.6.0-3796/hadoop-hdfs/lib/*.jar,/data/hdp/2.3.6.0-3796/hadoop-mapreduce/*.jar,/data/hdp/2.3.6.0-3796/hadoop-mapreduce/lib/*.jar,/data/hdp/2.3.6.0-3796/hadoop-yarn/*.jar,/data/hdp/2.3.6.0-3796/hadoop-yarn/lib/*.jar

Here is the lib corresponding to your own hadoop configuration
4. Modify mapreduce.configuration.directory to correspond to your hadoop configuration file directory
such as : /home/liuyazhuang/hadoop/hadoop/etc/hadoop

vi server/conf/sqoop.properties

At the same time, if you need to modify the output path of the log, you can
5. Start sqoop without configuring environment variables, find the corresponding bin path and test it.

./sqoop.sh server start

6. Import the mysql database

cp mysql-connector-java-5.1.6.jar /data/hdp/2.3.6.0-3796/sqoop2/sqoop2-1.99.5-cdh5.8.2/server/lib

Restart sqoop to enter the client side of sqoop

./sqoop.sh client

Test whether http://123.59.135.103:12000/sqoop/version can be accessed Configure the server
for the client

set server --host node01.soyoung.com --port 12000 --webapp sqoop  

### Import the mysql database [note that the configurations of version 1 and 2 are different here]
1. Create the mysql database corresponding to sqoop

create database sqoop
 		grant all privileges on sqoop to sqoop01@'node01.soyoung.com' identified by 'sqoop09876';

1. Create a hadoop connection

create link --cid 3
           Name: hdfs_link
           HDFS URI: hdfs://name-ha:8020

        
2. Create a mysql connection
 
         

sqoop:000> create link --cid 1  
		Name: sqoop01
		JDBC Driver Class: com.mysql.jdbc.Driver
		JDBC Connection String: jdbc:mysql://node01.soyoung.com:3306/sqoop
		username:sqoop01
		password：sqoop09876
		When entry#protocol -tcp appears, press Enter directly

show link
         +----+-----------+--------------+------------------------+---------+
          | Id |   Name    | Connector Id |     Connector Name     | Enabled |
          +----+-----------+--------------+------------------------+---------+
          | 1  | mysql_link   | 1            | generic-jdbc-connector | true    |
          | 2  | hadoop    | 2            | kite-connector         | true    |
          | 3  | hdfs_link | 3            | hdfs-connector         | true    |
          +----+-----------+--------------+------------------------+---------+
3.建立job(mysql到hdfs)

create job -f 1 -t 3
		and fill in the corresponding value
		show job
		start job --jid 1

4. Create a job (hdfs to mysql)

create job -f 3 -t 1	
		Fill in the required value
		show job
                status job -name mysql_to_hdfs_002 [version 1.99.7]
		start job --jid 3

sqoop1 brings the data on hive into mysql [this is the use of sqoop1, see this is more familiar 2 here is no longer]
It is executed on sqoop1 at the moment of execution, so there is no such command on sqoop2.

sqoop export jdbc:mysql://node01.soyoung.com/sqoop --username sqoop01 --password sqoop09876 --table l_user --hcatalog-database tmp --hcatalog-table l_employee

7. Configure sqoop to hue:

./sqoop.sh client

Execute show version
sqoop to complete the configuration. Specifically, the hue.ini file that needs to be modified on hue needs to be modified. Wait for Brother Zhang to come to coordinate the restart of hue.
Start hue, hue depends on apache. hue supervisor
8. Use sqoop client:

./sqoop.sh client  

implement

show version

The configuration of sqoop is completed. Specifically, the hue.ini file of hue needs to be modified to use hue.



Related Documentation: http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.5-cdh5.8.0/CommandLineClient.html
http://gethue.com/hadoop-tutorials-series-ii-8- how-to-transfer-data/#