Static partition:
Manually specify the partition to load the data, which is often said to use static partition. But in the daily work with a relatively large number of dynamic partitions.
create:
hive> create table order_mulit_partition(
> order_number string,
> event_time string
> )
> PARTITIONED BY(event_month string, step string)
> row format delimited fields terminated by '\t';
Download Data:
hive> load data local inpath '/opt/data/order_created.txt' overwrite into table order_mulit_partition PARTITION(event_month='201405', step='1');
View data:
hive> select * from order_mulit_partition;
OK
10703007267488 2014-05-01 06:01:12.334+01 NULL 201405 1
10101043505096 2014-05-01 07:28:12.342+01 NULL 201405 1
10103043509747 2014-05-01 07:50:12.33+01 NULL 201405 1
10103043501575 2014-05-01 09:27:12.33+01 NULL 201405 1
10104043514061 2014-05-01 09:03:12.324+01 NULL 201405 1
Dynamic partitioning:
Requirements: according to the different departments as the partitioning guide data into the target table
If the above requirements with a static partition, the amount of data you are not very muddled force? ? So the demand for general use dynamic partitioning to achieve.
1, emp table of contents creation and
create:
hive> create table emp(
> empno int,
> ename string,
> job string,
> mgr int,
> hiredate string,
> sal double,
> comm double,
> deptno int
> )row format delimited fields terminated by '\t';
Download Data:
hive> load data local inpath '/home/hadoop/datas/emp.txt' overwrite into table emp;
Field contents:
hive> select * from emp;
OK
7369 SMITH CLERK 7902 1980-12-17 800.0 NULL 20
7499 ALLEN SALESMAN 7698 1981-2-20 1600.0 300.0 30
7521 WARD SALESMAN 7698 1981-2-22 1250.0 500.0 30
7566 JONES MANAGER 7839 1981-4-2 2975.0 NULL 20
7654 MARTIN SALESMAN 7698 1981-9-28 1250.0 1400.0 30
7698 BLAKE MANAGER 7839 1981-5-1 2850.0 NULL 30
7782 CLARK MANAGER 7839 1981-6-9 2450.0 NULL 10
7788 SCOTT ANALYST 7566 1987-4-19 3000.0 NULL 20
7839 KING PRESIDENT NULL 1981-11-17 5000.0 NULL 10
7844 TURNER SALESMAN 7698 1981-9-8 1500.0 0.0 30
7876 ADAMS CLERK 7788 1987-5-23 1100.0 NULL 20
7900 JAMES CLERK 7698 1981-12-3 950.0 NULL 30
7902 FORD ANALYST 7566 1981-12-3 3000.0 NULL 20
7934 MILLER CLERK 7782 1982-1-23 1300.0 NULL 10
8888 HIVE PROGRAM 7839 1988-1-23 10300.0 NULL NULL
Display Partition:
hive> show partitions order_mulit_partition;
OK
event_month=201405/step=1
2, create the target table
hive> create table emp_dynamic_partition(
> empno int,
> ename string,
> job string,
> mgr int,
> hiredate string,
> sal double,
> comm double)
> PARTITIONED BY(deptno int)
> row format delimited fields terminated by '\t';
3, in the dynamic mode data to the target home table
Before loading to set the following parameters (set to a non strict mode)
hive> set hive.exec.dynamic.partition.mode=nonstrict
Start loading data
hive> insert into table emp_dynamic_partition partition(deptno)
select empno , ename , job , mgr , hiredate , sal , comm, deptno from emp;
注意:The above data is loaded manner and no specific partition, just pointing out the partition field. In the last field must be followed by select your partition field, which would own the value deptno according to the partition.
4. Verify
Have value
hive> select * from emp_dynamic_partition;
OK
7782 CLARK MANAGER 7839 1981-6-9 2450.0 NULL 10
7839 KING PRESIDENT NULL 1981-11-17 5000.0 NULL 10
7934 MILLER CLERK 7782 1982-1-23 1300.0 NULL 10
7369 SMITH CLERK 7902 1980-12-17 800.0 NULL 20
7566 JONES MANAGER 7839 1981-4-2 2975.0 NULL 20
7788 SCOTT ANALYST 7566 1987-4-19 3000.0 NULL 20
7876 ADAMS CLERK 7788 1987-5-23 1100.0 NULL 20
7902 FORD ANALYST 7566 1981-12-3 3000.0 NULL 20
7499 ALLEN SALESMAN 7698 1981-2-20 1600.0 300.0 30
7521 WARD SALESMAN 7698 1981-2-22 1250.0 500.0 30
7654 MARTIN SALESMAN 7698 1981-9-28 1250.0 1400.0 30
7698 BLAKE MANAGER 7839 1981-5-1 2850.0 NULL 30
7844 TURNER SALESMAN 7698 1981-9-8 1500.0 0.0 30
7900 JAMES CLERK 7698 1981-12-3 950.0 NULL 30
8888 HIVE PROGRAM 7839 1988-1-23 10300.0 NULL NULL
There partition (auto-partitioning)
hive> show partitions emp_dynamic_partition;
OK
deptno=10
deptno=20
deptno=30
deptno=__HIVE_DEFAULT_PARTITION__
Time taken: 0.375 seconds, Fetched: 4 row(s)
5, each partition content
deptno=10
782 CLARK MANAGER 7839 1981-6-9 2450.0 \N
7839 KING PRESIDENT \N 1981-11-17 5000.0 \N
7934 MILLER CLERK 7782 1982-1-23 1300.0 \N
deptno=20
7369 SMITH CLERK 7902 1980-12-17 800.0 \N
7566 JONES MANAGER 7839 1981-4-2 2975.0 \N
7788 SCOTT ANALYST 7566 1987-4-19 3000.0 \N
7876 ADAMS CLERK 7788 1987-5-23 1100.0 \N
7902 FORD ANALYST 7566 1981-12-3 3000.0 \N
deptno=30
7499 ALLEN SALESMAN 7698 1981-2-20 1600.0 300.0
7521 WARD SALESMAN 7698 1981-2-22 1250.0 500.0
7654 MARTIN SALESMAN 7698 1981-9-28 1250.0 1400.0
7698 BLAKE MANAGER 7839 1981-5-1 2850.0 \N
7844 TURNER SALESMAN 7698 1981-9-8 1500.0 0.0
7900 JAMES CLERK 7698 1981-12-3 950.0 \N
deptno=__HIVE_DEFAULT_PARTITION__
8888 HIVE PROGRAM 7839 1988-1-23 10300.0 \N
6, the information corresponding to the metadata in mysql:
mysql> select * from TBLS \G;
*************************** 1. row ***************************
TBL_ID: 1
CREATE_TIME: 1561364642
DB_ID: 6
LAST_ACCESS_TIME: 0
OWNER: hadoop
RETENTION: 0
SD_ID: 1
TBL_NAME: order_mulit_partition
TBL_TYPE: MANAGED_TABLE
VIEW_EXPANDED_TEXT: NULL
VIEW_ORIGINAL_TEXT: NULL
*************************** 2. row ***************************
TBL_ID: 2
CREATE_TIME: 1561365034
DB_ID: 6
LAST_ACCESS_TIME: 0
OWNER: hadoop
RETENTION: 0
SD_ID: 3
TBL_NAME: emp_dynamic_partition
TBL_TYPE: MANAGED_TABLE
VIEW_EXPANDED_TEXT: NULL
VIEW_ORIGINAL_TEXT: NULL
*************************** 3. row ***************************
TBL_ID: 6
CREATE_TIME: 1561366290
DB_ID: 6
LAST_ACCESS_TIME: 0
OWNER: hadoop
RETENTION: 0
SD_ID: 6
TBL_NAME: emp
TBL_TYPE: MANAGED_TABLE
VIEW_EXPANDED_TEXT: NULL
VIEW_ORIGINAL_TEXT: NULL
3 rows in set (0.00 sec)
ERROR:
No query specified
mysql> select * from PARTITIONS;
+---------+-------------+------------------+-----------------------------------+-------+--------+
| PART_ID | CREATE_TIME | LAST_ACCESS_TIME | PART_NAME | SD_ID | TBL_ID |
+---------+-------------+------------------+-----------------------------------+-------+--------+
| 1 | 1561364780 | 0 | event_month=201405/step=1 | 2 | 1 |
| 6 | 1561371156 | 0 | deptno=10 | 11 | 2 |
| 7 | 1561371157 | 0 | deptno=__HIVE_DEFAULT_PARTITION__ | 12 | 2 |
| 8 | 1561371158 | 0 | deptno=30 | 13 | 2 |
| 9 | 1561371158 | 0 | deptno=20 | 14 | 2 |
+---------+-------------+------------------+-----------------------------------+-------+--------+
5 rows in set (0.00 sec)
mysql> select * from PARTITION_KEYS;
+--------+--------------+-------------+-----------+-------------+
| TBL_ID | PKEY_COMMENT | PKEY_NAME | PKEY_TYPE | INTEGER_IDX |
+--------+--------------+-------------+-----------+-------------+
| 1 | NULL | event_month | string | 0 |
| 1 | NULL | step | string | 1 |
| 2 | NULL | deptno | int | 0 |
+--------+--------------+-------------+-----------+-------------+
3 rows in set (0.00 sec)
mysql> select * from PARTITION_KEY_VALS;
+---------+----------------------------+-------------+
| PART_ID | PART_KEY_VAL | INTEGER_IDX |
+---------+----------------------------+-------------+
| 1 | 201405 | 0 |
| 1 | 1 | 1 |
| 6 | 10 | 0 |
| 7 | __HIVE_DEFAULT_PARTITION__ | 0 |
| 8 | 30 | 0 |
| 9 | 20 | 0 |
+---------+----------------------------+-------------+
6 rows in set (0.00 sec)
mysql> select * from DBS;
+-------+-----------------------+-------------------------------------------------+---------+------------+------------+
| DB_ID | DESC | DB_LOCATION_URI | NAME | OWNER_NAME | OWNER_TYPE |
+-------+-----------------------+-------------------------------------------------+---------+------------+------------+
| 1 | Default Hive database | file:/hive/warehouse | default | public | ROLE |
| 6 | | hdfs://hadoop000:8020/hive/warehouse/g6_hive.db | g6_hive | NULL | USER |
+-------+-----------------------+-------------------------------------------------+---------+------------+------------+
2 rows in set (0.00 sec)
mysql> select * from COLUMNS_V2;
+-------+---------+--------------+-----------+-------------+
| CD_ID | COMMENT | COLUMN_NAME | TYPE_NAME | INTEGER_IDX |
+-------+---------+--------------+-----------+-------------+
| 1 | NULL | event_time | string | 1 |
| 1 | NULL | order_number | string | 0 |
| 2 | NULL | comm | double | 6 |
| 2 | NULL | empno | int | 0 |
| 2 | NULL | ename | string | 1 |
| 2 | NULL | hiredate | string | 4 |
| 2 | NULL | job | string | 2 |
| 2 | NULL | mgr | int | 3 |
| 2 | NULL | sal | double | 5 |
| 6 | NULL | comm | double | 6 |
| 6 | NULL | deptno | int | 7 |
| 6 | NULL | empno | int | 0 |
| 6 | NULL | ename | string | 1 |
| 6 | NULL | hiredate | string | 4 |
| 6 | NULL | job | string | 2 |
| 6 | NULL | mgr | int | 3 |
| 6 | NULL | sal | double | 5 |
+-------+---------+--------------+-----------+-------------+
17 rows in set (0.00 sec)
mysql> select * from VERSION;
+--------+----------------+-------------------------------------+
| VER_ID | SCHEMA_VERSION | VERSION_COMMENT |
+--------+----------------+-------------------------------------+
| 1 | 1.1.0 | Set by MetaStore [email protected] |
+--------+----------------+-------------------------------------+
1 row in set (0.00 sec)