Sqoop (SQL to Hadoop) is an open source tool under the Apache Software Foundation for transferring data between Hadoop and relational database servers. Its main purpose is to simplify the process of importing data from relational databases (such as MySQL, Oracle, SQL Server, etc.) to the Hadoop ecosystem (such as Hive, HBase, etc.) or exporting data from Hadoop to relational databases. Sqoop enables data engineers and data scientists to easily move and transform data between Hadoop clusters and traditional relational databases, supporting big data analysis and processing.
Following are the main features and functions of Sqoop:
Data transfer: Sqoop can import table data in a relational database into the Hadoop Distributed File System (HDFS), and can also export data in HDFS to a relational database table. This allows users to run MapReduce jobs on Hadoop clusters or use other Hadoop ecosystem tools to process data.
Parallelism: Sqoop supports parallel data transmission, allowing multiple tables or data partitions to be imported or exported at the same time, thereby improving data transmission efficiency.
Data compression: Sqoop supports compression of transmitted data to reduce network transmission and storage overhead.
Data transformation: Sqoop allows users to perform data transformation operations during data transfer, for example, converting data from specific data types of a relational database to data types supported by Hadoop.
Incremental Import: Sqoop supports incremental import, allowing users to import only the data that has changed since the last import, rather than the entire table.
Third-party connectors: Sqoop provides multiple third-party connectors for connecting to various relational databases. This means you can use Sqoop to integrate with a number of different database systems.
Scheduled tasks: Sqoop can be used with scheduling tools such as Apache Oozie to automate and schedule data transfer jobs.
Supports multiple file formats: Sqoop supports multiple file formats, including text files, Avro, Parquet, etc., to meet different data storage needs.
Sqoop works as follows:
Users use the Sqoop command line tool or the Sqoop client configuration file to specify the data sources and targets to be imported or exported.
Sqoop generates MapReduce jobs based on the user's configuration, which includes data transfer, data transformation, and data loading steps.
The generated MapReduce jobs are executed on the Hadoop cluster to import data from a relational database to HDFS or export from HDFS to a relational database.
Users can monitor the progress and status of jobs and perform necessary error handling or retry operations if needed.
The download, operation and installation methods of Sqoop (SQL to Hadoop) can be divided into the following steps:
1. Download Sqoop:
You can download the latest stable version of Sqoop from Apache Sqoop's official website or Apache's mirror site. Here are some commonly used download links:
Apache Sqoop official website: http://sqoop.apache.org/
Apache mirror site: https://www.apache.org/dyn/closer.lua/sqoop/According
to your needs, choose the operating system and Download link for the platform and download the binary distribution of Sqoop.
2. Install Java:
Sqoop is a Java-based tool, so before installing Sqoop, make sure you have Java installed. Sqoop is generally compatible with Java 8 or higher. You can download and install Java from official websites that provide Java distributions, such as Oracle or OpenJDK.
3. Unzip Sqoop:
After the download is complete, unzip the Sqoop compressed file to a directory of your choice. You can use command line tools or graphical interface tools to decompress. For example, if you unpacked Sqoop into the /opt directory, the unpack command might look like this:
tar -zxvf sqoop-x.y.z.tar.gz -C /opt
4. Configure environment variables:
To facilitate the use of Sqoop commands, you can add Sqoop's bin directory to the system's PATH environment variable. Edit your shell configuration file (such as .bashrc or .zshrc) and add the following line, replacing Sqoop's bin directory path with your actual path:
export PATH=$PATH:/opt/sqoop-x.y.z/bin
Then run the following command to make the configuration take effect:
source ~/.bashrc # 或 source ~/.zshrc，具体取决于您使用的shell
5. Configure Sqoop:
Sqoop requires configuration to connect to relational databases and Hadoop clusters. The main configuration file is sqoop-env-template.sh, which you can copy and rename to sqoop-env.sh, then edit to set the following parameters:
SQOOP_HOME: Points to the Sqoop installation directory.
HADOOP_COMMON_HOME: Points to the Hadoop installation directory.
HADOOP_MAPRED_HOME: Points to the installation directory of Hadoop MapReduce.
HIVE_HOME: If you need to integrate with Hive, point to the Hive installation directory.
6. Use Sqoop:
You can now use the Sqoop command line tool to perform data import and export operations. Sqoop provides a wealth of command options to meet different data transmission needs. Here is a simple example to import data from MySQL to HDFS:
sqoop import --connect jdbc:mysql://localhost/mydb --username myuser --password
mypassword --table mytable --target-dir /user/hadoop/mytable_data
This is just a simple example, Sqoop provides many other options and features to cater for more complex data transfer scenarios.
References
https://blog.csdn.net/a781136776/article/details/80458551
https://attic.apache.org/projects/sqoop.html
https://blog.csdn.net/weixin_50158551/article/details/115771853