Sqoop installation
The prerequisite for installation is to have Java and Hadoop environment
.
1. Installation 1. Download and unzip
1) Download address: http://mirrors.hust.edu.cn/apache/sqoop/1.4.6/
2) Unzip the installation package
tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
2. Modify the configuration file The configuration file of
Sqoop is similar to most big data frameworks, in the conf directory under the root directory of sqoop.
1) Switch to sqoop,,, conf below, change the name of sqoop-env.sh
mv sqoop-env-template.sh sqoop-env.sh
2) Open sqoop-env.sh, add things
vi sqoop-env.sh
add:
export HADOOP_COMMON_HOME=/usr/hadoop/hadoop-2.8.5
export HADOOP_MAPRED_HOME=/usr/hadoop/hadoop-2.8.5
export HBASE_HOME=/usr/hadoop/hbase-1.3.6
export HIVE_HOME=/usr/hadoop/apache-hive-1.2.2-bin
export ZOOCFGDIR=/usr/hadoop/zookeeper-3.4.6
export ZOOKEEPER_HOME=/usr/hadoop/zookeeper-3.4.6
save and exit
3) Configure the environment
vi /etc/profile
export SQOOP_ HOME=/usr/hadoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha
export PATH= P A T H : PATH: PATH:SQOOP_HOME/bin
source /etc/profile
4) Copy the JDBC driver
Copy the jdbc driver to the lib directory of ssqoop, such as:
cp mysql-connector-java-5.1.48-bin.jar /usr/hadoop sqoop-1.4.6.bin__hadoop-2.0.4-alpha/lib
5) Verify Sqoop
We can verify whether the sqoop configuration is correct through a certain command:
sqoop version
has some warnings, accompanied by the output of the help command:
6) To test whether Sqoop can successfully connect to the database, execute the command
sqoop help list-databases --connect jdbc:mysql://master:3306/ --username root --password 8811
Second, the simple use case of Sqoop
1) Import data
In Sqoop, the concept of "import" refers to: transferring data from a non-big data cluster (RDBMS) to a big data cluster (HDFS, HIVE, HBASE), called: import, that is, use import keyword.
First open the mysql service: service mysqld start to
view the status: service mysqld status
Login: mysql -u root -p8811
Start
1. RDBMS to HDFS
- Make sure the Mysql service is started normally
2) Create a new test library in Mysql and insert some data
create database test;
Use test library
use test;
3) Create a table in test and insert 4 pieces of data
create table t_user(id int,name varchar(20),age int);
insert into t_user values(1,‘rod’,20);
insert into t_user values(2,‘tom’,21);
insert into t_user values(3,‘lucy’,22);
insert into t_user values(4,'jet',23);
4) Check the table structure and the data in the table
1 Check the structure desc t_user;
2 Check the data select *from t_user;
After the relational database tables and data are ready, You can write an example.
Create a directory sqoopcrefile under /usr/hadoop, create a file in this directory, and configure data import related information in this file.
import
jdbc:mysql://localhost:3306/text
–username
root
–password
8811
–table
t_user
–columns
id,name,age
–where
id>0
–target-dir
hdfs://master:9000/sqoop
–delete-target-dir
-m
1
–as-textfile
–null-string
Execute Sqoop based on the configuration file, the command format is: sqoop–options-file filename.
According to the running results, it is found that when executing based on sqoop, the bottom layer is running mapreduce. After the execution, you can view the import results. For example, based on the browser, log in to hdfs to view the sqoop directory.
##2) Export data