hive four other table, partition table, internal, external table, table barrel

Hive four table type internal table, external table, the partition table and the tub Table

I. Overview

Overall Hive There are four tables: External tables, the internal table (management table), the partition table, barrel tables. Corresponding to different requirements. Here mainly on the various exclusions tables, create and load data method.

Second, the specific content

1. Internal Table

Create internal tables and load data

create table emp_inner(
empno int,
ename string,
job string,
mgr int,
hiredate string,
sal double,
comm double,
deptno int
)
row format delimited fields terminated by '\t'
LOCATION '/user/hive/warehouse/hadoop.db/emp';

2. External Table

(1) What it is:

  When a log analysis requires multiple teams together, table created after the analysis done can be deleted. But ordinary table deletion will also delete the data, which would affect the analysis of other teams, and log data can not be easily removed. Therefore, the need for an external table, delete the external table, it does not delete the data on the corresponding hdfs.

(2) create an external table

create EXTERNAL table dept_ext(
deptno int,
dname string,
loc string
)
row format delimited fields terminated by '\t' ;
load data local inpath '/opt/datas/dept.txt' into table dept_ext;

(3) comparing the difference between the outer table and inner table
    to delete the external table, the data will not change, just the metadata mysql is modified, but deletes the internal table (management table), the data will be deleted.

    Summary: The difference between the internal and outer tables hive of
        1:00) to create the table: When you create an internal table, will move to the path of the data warehouse data points; if you create an external table, only records where the data path, the data do not in any position change.
        2) When you delete a table: In the deleted table, the metadata for internal tables and data will be deleted together, and external table only remove metadata, do not delete the data. Such external table relatively more safer, more flexible organization of data to facilitate the sharing of source data

3, a temporary table

(1) What it is

Interim analysis, after closing the hive client, the temporary table will disappear. Primarily for storing intermediate results set is not important, unimportant table.

(2) create a temporary table and load the data

create TEMPORARY table dept_tmp(
deptno int,
dname string,
loc string
)
row format delimited fields terminated by '\t';

load data local inpath '/opt/datas/dept.txt' into table dept_tmp;

(3) View location information

desc formatted dept_tmp;
Location: hdfs://172.19.199.187:8020/tmp/hive/hadoop/68174383-f427-4629-9707-0ab1c9b07726/_tmp_space.db/d872efec-1294-48b0-9071-31cf98d46400
Table Type: MANAGED_TABLE

4, partition table [***]

(1) What it is

Common table: select * from logs where date = '20171209', the process performs: a full table of data query, before the filtering operation.

Partition table: select * from logs where date = '20171209', the process performs: a data path in the corresponding file loaded directly. Suitable for large quantities of data, you can quickly locate the data to be queried by a partition, partition table main role is to improve the query efficiency.

(2) create a partition and load data

create table emp_part(
empno int,
ename string,
job string,
mgr int,
hiredate string,
sal double,
comm double,
deptno int
)partitioned by (`datetime` string)
row format delimited fields terminated by '\t';

load data local inpath '/opt/datas/emp.txt' into table emp_part partition(`datetime`='20171209');
load data local inpath '/opt/datas/emp.txt' into table emp_part partition(`datetime`='20171208');
[Folders are formed on the two hdfs, emp.txt stored inside]
/user/hive/warehouse/hadoop.db/emp_part/datetime=20171208
/user/hive/warehouse/hadoop.db/emp_part/datetime=20171209

search result:
select * from emp_part where `datetime` = '20171209';

(3) Partition and create two load data

create table emp_part2(
empno int,
ename string,
job string,
mgr int,
hiredate string,
sal double,
comm double,
deptno int
)partitioned by (`datetime` string,hour string)
row format delimited fields terminated by '\t';

load data local inpath '/opt/datas/emp.txt' into table emp_part2 partition(`datetime`='20171209',hour='01');

load data local inpath '/opt/datas/emp.txt' into table emp_part2 partition(`datetime`='20171209',hour='02');
[On hdfs, forming a directory]
/user/hive/warehouse/hadoop.db/emp_part2/datetime=20171209/hour=01
/user/hive/warehouse/hadoop.db/emp_part2/datetime=20171209/hour=02

search result:

-》 select * from emp_part2 where `datetime` = '20171209';
View all of the data (ie, twice the emp.txt data) in /user/hive/warehouse/hadoop.db/emp_part2/datetime=20171209
-》 select * from emp_part2 where `datetime` = '20171209' and hour = '01';
View all of the data (ie emp.txt data) in /user/hive/warehouse/hadoop.db/emp_part2/datetime=20171209/hour=01

(4) create an external partition table (deleted when deleting just the metadata, the data will not be deleted)

create EXTERNAL table emp_test(
empno int,
ename string,
job string,
mgr int,
hiredate string,
sal double,
comm double,
deptno int
)
PARTITIONED BY(date string,hour string)
row format delimited fields terminated by '\t';

(5) The method of loading the partition table data

(A) by the load command (specify the partition) directly load data into a partitioned table, select can be queried.                        

load data local inpath '/opt/datas/emp.txt' into table emp_part2 partition(`datetime`='20171209',hour='01');

(B) to manually create the directory /user/hive/warehouse/hadoop.db/emp_part2/datetime=20171209/hour=03, then put the data table select query is a query visible. Then, alter the path to the original database mysql database.

alter table emp_part2 add partition(`datetime`='20171209',hour='03');

5, barrel table

(1) use case

Data have serious data skew, uneven distribution, but relatively speaking, the amount of data in each bucket would be more evenly. Pails join other inquiries between the bucket when there will be optimized.

(2) create and use

First of all,

 set hive.enforce.bucketing = true; 

then,

create table emp_bu(
empno int,
ename string,
job string,
mgr int,
hiredate string,
sal double,
comm double,
deptno int
)CLUSTERED BY(deptno) INTO 4 BUCKETS
row format delimited fields terminated by '\t';

Finally, load data, use the insert

insert overwrite table emp_bu_2 select * from emp;
也可以指定分区写入 insert overwrite