Big data data migration sqoop installation

1. Introduction to sqoop

        Sqoop is a tool for fast and efficient data transfer between Apache Hadoop and structured data stores such as relational databases. Sqoop has a command line interface that can be used to import data from a relational database into the Hadoop Distributed File System (HDFS), or export data from HDFS to a relational database. Sqoop supports various relational databases, including MySQL, Oracle, PostgreSQL, etc. Sqoop also supports parallel import and export operations for faster processing of large amounts of data. The main goal of Sqoop is to make it easier to integrate Hadoop with relational databases for data analysis and processing.

In short, sqoop is a big data data migration tool.

2. Advantages and disadvantages of sqoop data migration

Sqoop is an open source data migration tool that can quickly import data from a relational database into the Hadoop ecosystem, or quickly export data from the Hadoop ecosystem to a relational database. Its advantages and disadvantages are as follows:

advantage:

1. Reliability: Sqoop ensures data consistency and integrity, and is capable of error handling and retries during the migration process.
2. Efficiency: Sqoop uses parallel processing and data compression technology to improve the efficiency of data migration, and can quickly import large amounts of data into the Hadoop ecosystem.
3. Flexibility: Sqoop supports a variety of data sources and targets, and can import and export data from various relational databases (such as MySQL, Oracle, etc.) and Hadoop storage systems (such as HDFS and HBase, etc.).
4. Ease of use: Sqoop provides simple and easy-to-use command line tools and graphical interfaces, allowing users to easily migrate data.

shortcoming:

1. Does not support real-time data migration: Sqoop is an offline data migration tool and does not support real-time data migration.
2. Complex configuration: The configuration of Sqoop is relatively complex and requires setting many parameters and options, which may require certain technical knowledge and experience.
3. Does not support complex data types: Sqoop does not support complex data types (such as arrays, nested structures, etc.), and special processing is required when migrating these types of data.
4. Does not support non-relational databases: Sqoop only supports relational databases and Hadoop storage systems, and does not support data migration of non-relational databases (such as MongoDB, Cassandra, etc.).

3. Installation of sqoop

3.1. Upload the sqoop installation package to /software and extract it to /opt.

tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/

 3.2. Rename the sqoop decompressed file

mv  /opt/sqoop-1.4.7.bin__hadoop-2.6.0 /opt/sqoop

3.3. Append the sqoop installation path to the end of the environment variable.

export SQOOP_HOME=/opt/sqoop
export PATH=$PATH:$SQOOP_HOME/bin

3.4. Refresh the environment variables and check whether the installation is successful.

source /etc/profile

sqoop version

3.5. Add local hadoop and hive address paths (under the conf file under sqoop)

cp sqoop-env-template.sh sqoop-env.sh

Modify the variables in this executable script and add the following path

export HADOOP_COMMON_HOME=/opt/hadoop
export HADOOP_MAPRED_HOME=/opt/hadoop
export HIVE_HOME=/opt/hive

export HCAT_HOME=/opt/hive/hcatalog

 3.6. If you need to connect to mysql, you need to put the corresponding mysql connection jar into the lib of sqoop.

Sqoop software package download location link: https://pan.baidu.com/s/1v2mn-RMEsb7H7d6gYe6ISw?pwd=asdf 
Extraction code: asdf