跨集群hive批量建表

版权声明：欢迎给我留言，多提意见。互相交流、共同进步！ https://blog.csdn.net/qq_31598113/article/details/82262201

【需求】
在集群A建表，需要同步到集群B并修复历史分区。表用到的数据源在云上。

【注意】hive -f时涉及的HDFS操作权限问题
A集群有9个表要同步到B集群
[hive@bigdata-master-1 ~]$ vim tables_ToBeCreated_onOtherCluster

# 9个表是：
[hive@bigdata-master-1 ~]$ cat tables_ToBeCreated_onOtherCluster
tony_db.member_sign_gift
tony_db.member_sign_in
tony_db.member_sign_in_addition
tony_db.​member_sign_in_month
tony_db.​member_sign_user_day_award​
tony_db.​member_sign_user_month_award​
tony_db.member_sign_day_reward
tony_db.member_sign_month_reward
tony_db.​member_sign_recharge_config​

[hive@bigdata-master-1 ~]$ cat tables_ToBeCreated_onOtherCluster | while read tb; do hive -e "SHOW CREATE TABLE $tb" | grep -vE "Logging|OK|seconds" >> create_table.sql; echo finished ____ $tb; done
Logging initialized using configuration in file:/etc/hive/2.5.0.0-1245/0/hive-log4j.properties
OK
Time taken: 3.34 seconds, Fetched: 31 row(s)
finished ____ tony_db.member_sign_gift
Logging initialized using configuration in file:/etc/hive/2.5.0.0-1245/0/hive-log4j.properties
OK
Time taken: 2.78 seconds, Fetched: 19 row(s)
finished ____ tony_db.member_sign_in
... ...(省略其他的日志输出)

# 在每条建表语句末尾添加MSCK REPAIR TABLE XXXX.YYYYYY; 以便修复该表的历史分区
[hive@bigdata-master-1 ~]$ vim create_table.sql

# scp同步到B集群
[root@bigdata-master-1 ~]# grep --color -C 2 master-1 ~/.ssh/authorized_keys
[root@bigdata-master-1 ~]# scp /home/hive/create_table.sql [email protected]:/root/    # 同步到B集群

#在B集群执行建表、修复历史分区
#检查ssh public-key，确保A集群能通过ssh访问此主机
[root@new-cluster-1 ~]# grep --color "UDut1KxWXuXY5GDdt4aQm0CqD9u0SIk24Akjw" ~/.ssh/authorized_keys 
You have new mail in /var/spool/mail/root
[root@new-cluster-1 ~]# ifconfig | grep --color 172
          inet addr:172.31.70.117  Bcast:172.31.70.255  Mask:255.255.255.0
[root@new-cluster-1 ~]# cd ~
[root@new-cluster-1 ~]# ll -ht
total 20K
-rw-r--r--. 1 root root 5.7K Aug 31 20:56 create_table.sql
-rw-r--r--. 1 root root    0 Aug 15 20:00 out
drwxr-xr-x. 2 root root 4.0K Aug 10 19:27 logs
-rw-------. 1 root root  871 Nov 14  2014 anaconda-ks.cfg
-rw-r--r--. 1 root root 3.4K Nov 14  2014 install.log.syslog

#注意HDFS权限问题
[root@new-cluster-1 ~]# chown hdfs:root create_table.sql 
[root@new-cluster-1 ~]# mv create_table.sql /home/hdfs/
[root@new-cluster-1 ~]# su hdfs
[hdfs@new-cluster-1 root]$ cd ~
[hdfs@new-cluster-1 ~]$ pwd
/home/hdfs
[hdfs@new-cluster-1 ~]$ ll -ht
total 8.0K
-rw-r--r--. 1 hdfs root 5.7K Aug 31 20:56 create_table.sql
[hdfs@new-cluster-1 ~]$ hive -f create_table.sql

完成。