Repair Hive table partition using MSCK command

Reprint URL: 
http://blog.csdn.net/opensure/article/details/51323220  
http://www.cnblogs.com/chinhr/archive/2007/10/17/927506.html  
http://blog.csdn .net/sparkexpert/article/details/51024392?locationNum=5&fps=1  
http://hadoop.apache.org/docs/r1.0.4/cn/hdfs_shell.html 
My own operation combat: 
1. Create a table first .

hive> create table t427(id int,city string,name string,sex string) 
    > partitioned by (day int)
    > row format delimited                                         
    > fields terminated by '\t';
OK
  • 1
  • 2
  • 3
  • 4
  • 5

2. View the table structure

hive> desc formatted t427;
OK
# col_name              data_type               comment             

id                      int                     None                
city                    string                  None                
name                    string                  None                
sex                     string                  None                

# Partition Information          
# col_name              data_type               comment             

day                     int                     None                

# Detailed Table Information             
Database:               fdm                      
Owner:                  root                     
CreateTime:             Mon May 01 07:22:11 PDT 2017     
LastAccessTime:         UNKNOWN                  
Protect Mode:           None                     
Retention:              0                        
Location:               hdfs://hadoop:9000/warehouse/fdm.db/t427         
Table Type:             MANAGED_TABLE            
Table Parameters:                
        transient_lastDdlTime   1493648531          

# Storage Information            
SerDe Library:          org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe       
InputFormat:            org.apache.hadoop.mapred.TextInputFormat         
OutputFormat:           org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat       
Compressed:             No                       
Num Buckets:            -1                       
Bucket Columns:         []                       
Sort Columns:           []                       
Storage Desc Params:             
        field.delim             \t                  
        serialization.format    \t                  
Time taken: 0.046 seconds
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38

3. Add two partitions

hive> dfs -mkdir hdfs://hadoop:9000/warehouse/fdm.db/t427/day=25;
hive> dfs -mkdir hdfs://hadoop:9000/warehouse/fdm.db/t427/day=26;
  • 1
  • 2

4. Check the partition (Oh, not at this time)

hive> show partitions t427;
OK
Time taken: 0.101 seconds
  • 1
  • 2
  • 3

5. Use the msck command to repair the partition structure

hive> msck repair table t427;
OK
Partitions not in metastore:    t427:day=25     t427:day=26
Repair: Added partition to metastore t427:day=25
Repair: Added partition to metastore t427:day=26
Time taken: 0.18 seconds
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6

6. Check the partition information again, this time is successful.

hive> show partitions t427;  
OK
day=25
day=26
Time taken: 0.049 seconds

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=324580606&siteId=291194637