Big data data collection project extension - sqoop

Continuing from the previous article, a
summary of data collection projects for big data——hadoop, hive, openresty, frcp, nginx, flume Stage: Data collection is completed, log files are uploaded to hdfs using flume, and partition tables are created using hive . Now add: use sqoop to export the data in hdfs/hive to MySQL. Overview:

insert image description here

show databases;
use ods_news;
show tables;
select * from news_parquet;
-- 筛选出来40条数据，把这个数据存入一个新表,使用sqoop把它导入到MySQL中
select distinct_id,model,network_type,carrier
from news_parquet 
where model is not null and model !="" and network_type !="" and carrier <>"";
-- 创建表
create table useful_info 
row format delimited 
fields terminated by ","
as
(
select distinct_id,model,network_type,carrier
from news_parquet 
where model is not null and model !="" and network_type !="" and carrier <>""
);

-- 查询一下表数据，存在
select * from useful_info;

Data display:
insert image description here
on hdfs:

insert image description here

Write sqoop, export data to MySQL

Export the data in useful_info to the MySQL database on datacollection.
View MySQL databases and tables in datacollection

show databases;

insert image description here

use test1;
show tables;

insert image description here

1. Create a table in MySQL

Note: The export will not automatically create the corresponding table, you need to create it yourself in advance Create
a new table from_sqoop_hdfs in the test1 database

CREATE TABLE from_sqoop_hdfs(
  distinct_id VARCHAR(20),
  model VARCHAR(50),
  network_type VARCHAR(50),
  carrier VARCHAR(50)
) CHARACTER SET utf8;

2. Import the data on hive into the mysql table

Enter the following statement directly on the Linux command line:

sqoop export \
--connect jdbc:mysql://datacollection:3306/test1 \
--username root \
--password 123456 \
--table from_sqoop_hdfs \
--export-dir hdfs://datacollection:8020/user/hive/warehouse/ods_news.db/useful_info/000000_0 \
--input-fields-terminated-by ',' \
--input-lines-terminated-by '\n'

Success! Must take a picture to commemorate! !

insert image description here