6.hive parameters and dynamic partition
6.1 Hive parameters
6.1.1 Naming rules
Parameters and variables in hive all start with a namespace and are
referenced through ${}. Among them, variables under system and env must start with a prefix.
6.1.2 hive parameter setting method
- Modify the configuration file ${HIVE_HOME}/conf/hive-site.xml
- When starting hive cli, set it by –hiveconf key=value
hive --hiveconf hive.cli.print.header=true # 查询时显示表头
hive> select * from wc_count;
OK
wc_count.word wc_count.count
andy 3
hello 5
joy 3
mark 1
rose 2
tom 2
Time taken: 3.631 seconds, Fetched: 6 row(s)
Parameters set in this way are valid until the connection is disconnected.
- After entering the cli, use the set command to set
Note: The parameters set in 2 and 3 are only valid in the current session.
6.1.3 hive set command
hive> [root@node4 ~]# hive
hive> select word,count from wc_count;
OK
andy 3
hello 5
joy 3
mark 1
rose 2
tom 2
Time taken: 3.562 seconds, Fetched: 6 row(s)
Use the set command on the hive cli console to query the parameters in hive or set
the set query:
hive> set hive.cli.print.header;
hive.cli.print.header=false
hive> set;
#查询所有的参数
set set parameters
hive> set hive.cli.print.header=true;
hive> set hive.cli.print.header;
hive.cli.print.header=true
Commands for hive history operations:
[root@node4 ~]# ll -a
-rw-r--r-- 1 root root 11590 11月 19 11:07
.hivehistory
There is a .hivehistory file under the current user's home directory/root, which records the executed hive commands:
[root@node4 ~]# vim .hivehistory
create table psn(id int,age int);
insert into psn values(1,18);
create database hivedb1;
......
select * from wc_count;
select word,count from wc_count;
Initial configuration of hive parameters:
in the .hiverc file of the current user's home directory, if there is no one, you can create one and add the configuration of parameters:
[root@node4 ~]# vim .hiverc
set hive.cli.print.header=true;
Disconnect and reconnect the hive client, the parameters will take effect and will always be valid for the current user:
[root@node4 ~]# hive
hive> select id,name,likes from person;
OK
id name likes
1 小明1 ["lol","book","movie"]
2 小明2 ["lol","book","movie"]
3 小明3 ["lol","book","movie"]
4 小明4 ["lol","book","movie"]
5 小明5 ["lol","movie"]
6 小明6 ["lol","book","movie"]
7 小明7 ["lol","book"]
8 小明8 ["lol","book"]
6.2 Dynamic Partitioning
Enable support for dynamic partitions
set hive.exec.dynamic.partition=true;
Default: true #Default supports dynamic partition
hive> set hive.exec.dynamic.partition.mode;
hive.exec.dynamic.partition.mode=strict
#修改为非严格模式
hive>set
hive.exec.dynamic.partition.mode=nostrict;
Default: strict strict mode (for example, creating partitions in seconds for the order table will result in too many partitions, which is generally not allowed in strict mode, but allowed in non-strict mode).
nostrict: Non-strict mode
Case demonstration:
Create original data table
create table person21(
id int,
name string,
age int,
gender string,
likes array<string>,
address map<string,string>
)
row format delimited
fields terminated by ','
collection items terminated by '-'
map keys terminated by ':';
Upload the software /data/person21.txt to the /root/data directory on node4 to load data
hive> load data local inpath
'/root/data/person21.txt' into table person21;
Loading data to table default.person21
OK
Time taken: 0.786 seconds
hive> select * from person21;
OK
person21.id person21.name person21.age
person21.gender person21.likes
person21.address
1 tuhao1 32 man ["lol","book","movie"]
{
"beijing":"xisanqi","shanghai":"pudong"}
2 tuhao2 32 man ["lol","book","movie"]
{
"beijing":"xisanqi","shanghai":"pudong"}
3 tuhao3 12 boy ["lol","book","movie"]
{
"beijing":"xisanqi","shanghai":"pudong"}
4 tuhao4 32 man ["lol","book","movie"]
{
"beijing":"xisanqi","shanghai":"pudong"}
5 tuhao5 12 boy ["lol","movie"]
{
"beijing":"xisanqi","shanghai":"pudong"}
6 tuhao6 32 man ["lol","book","movie"]
{
"beijing":"xisanqi","shanghai":"pudong"}
7 tuhao7 32 man ["lol","book"]
{
"beijing":"xisanqi","shanghai":"pudong"}
8 tuhao8 12 boy ["lol","book"]
{
"beijing":"xisanqi","shanghai":"pudong"}
9 tuhao9 32 man ["lol","book","movie"]
{
"beijing":"xisanqi","shanghai":"pudong"}
Create partition table:
create table person22(
id int,
name string,
likes array<string>,
address map<string,string>
)
partitioned by(age int,gender string)
row format delimited
fields terminated by ','
collection items terminated by '-'
map keys terminated by ':';
Load the data from person21 into the person22 table:
from person21
insert overwrite table person22 partition(age,gender)
select id,name,likes,address,age,gender distribute by age,gender
abbreviated as
from person21
insert overwrite table person22 partition(age,gender)
select id,name,likes,address,age,gender;
Note: The partition field must be written at the end, otherwise it is easy to make mistakes.
Error: FAILED: SemanticException [Error 10096]: Dynamic partition strict mode requires at least one static partition column. To turn this off set hive.exec.dynamic.partition.mode=nonstrict
Solution:
hive> set hive.exec.dynamic.partition=true;
hive> set hive.exec.dynamic.partition.mode=nonstrict;
show partitions person22;#查询某表上的 分区
View table partitions:
hive> show partitions person22;
OK
partition
age=12/gender=boy
age=32/gender=man
Time taken: 0.331 seconds, Fetched: 2 row(s)
Related parameters
set hive.exec.max.dynamic.partitions.pernode;
每一个执行mr节点上，允许创建的动态分区的最大数量(100)
set hive.exec.max.dynamic.partitions;
所有执行mr节点上，允许创建的所有动态分区的最大数量
(1000)
set hive.exec.max.created.files;
所有的mr job允许创建的文件的最大数量(100000)