Hive version upgrade record (0.13.0 -> 2.3.3)

Others 2020-04-06 10:32:01 views: null

 

 

background:

We have here two sets of online and offline hive, Version: 0.13.0, due to lower version, recently prepared a new hive upgrade version.

First select the next upgrade line hive clusters:

Hive clusters covering wider the line, probably involve about 10 clients, covering all the company's technical department business data. A day by the client or azkaban upload job assignments around 1W.
Meta Store contains the amount partition probably about 700w. Heavy traffic, and more sql change, there are more custom and custom serde udf file in it.

If you do a one-time upgrade the whole amount more prone to problems affecting many sectors, a rollback event of unforeseen circumstances are more complicated.
And a one-time full amount needed to handle the upgrade to upgrade the client and meta middle hive off a longer time, resulting in job fails.

So the final practice measures are:
First, upgrade only  Metastore Server  , and then upgrade each sub-sector part of the hive client ,  hive Thrift Server .

image.png

ready:

Until you are ready to upgrade, the client needs to enter the gray mode :

Close metadata VERISONparity related functions:

<property>
  <name>hive.metastore.schema.verification</name>
  <value>false</value>
  <description>
    Enforce metastore schema version consistency.
    True: Verify that version information stored in is compatible with one from Hive jars.  Also disable automatic
          schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures
          proper metastore schema migration. (Default)
    False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
  </description>
</property>

Disabling JDO framework to automatically update the metadata schema:

<property>
  <name>datanucleus.autoCreateSchema</name>
  <value>false</value>
</property>

<property>
  <name>datanucleus.fixedDatastore</name>
  <value>true</value>
</property>

hive meta store upgrades:

When the hive is no longer a client version check meta information, we again hive meta store upgrade.

Next Analysis: The  hive meta store upgrade scripts:

Sql upgrade process through research found that changing the hive to make a major new version of the library are:

  1. Create a new table.
  2. Add a new field of the original table.
  3. Extension of the original length of the column, such as the table name. Column names and so on.
  4. New Stored Procedure

We can look at it upgrades script

# add acid
# 创建各种事物表
source hive-txn-schema-0.13.0.mysql.sql

# 0.13.0 -> 0.14.0
# create table PART_COL_STATS
# create index PCS_STATS_IDX on PART_COL_STATS
# set version 0.14.0
source upgrade-0.13.0-to-0.14.0.mysql.sql

# 0.14.0 -> 1.1.0
# create table NOTIFICATION_LOG
# create table NOTIFICATION_SEQUENCE
# set version 1.1.0
source upgrade-0.14.0-to-1.1.0.mysql.sql
 
# 1.1.0 -> 1.2.0
# set version 1.2.0
source upgrade-1.1.0-to-1.2.0.mysql.sql 
# 1.2.0 -> 2.0.0
# create procedure RM_TLBS_LINKID
# create procedure RM_PARTITIONS_LINKID
# create procedure RM_LINKID
# ALTER TABLE `COLUMNS_V2` MODIFY `COLUMN_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL;
# ===> `COLUMN_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL
# ALTER TABLE `PART_COL_PRIVS` MODIFY `COLUMN_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL;
# ===>`COLUMN_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# ALTER TABLE `TBL_COL_PRIVS` MODIFY `COLUMN_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL;
# ====>`COLUMN_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# ALTER TABLE `SORT_COLS` MODIFY `COLUMN_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL;
# ===> `COLUMN_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# ALTER TABLE `TAB_COL_STATS` MODIFY `COLUMN_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL;
# ===> `COLUMN_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL
# ALTER TABLE `PART_COL_STATS` MODIFY `COLUMN_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL;
# ===> `COLUMN_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL

# 添加字段 : ALTER TABLE `COMPACTION_QUEUE` ADD `CQ_HIGHEST_TXN_ID` bigint;
# 添加字段 : ALTER TABLE `COMPACTION_QUEUE` ADD `CQ_META_INFO` varbinary(2048);
# 添加字段 : ALTER TABLE `COMPACTION_QUEUE` ADD `CQ_HADOOP_JOB_ID` varchar(32);
# 添加字段 : ALTER TABLE `TXNS` ADD `TXN_AGENT_INFO` varchar(128);
# 添加字段 : ALTER TABLE `TXNS` ADD `TXN_HEARTBEAT_COUNT` int;
# 添加字段 : ALTER TABLE `HIVE_LOCKS` ADD `HL_HEARTBEAT_COUNT` int;
# 添加字段 : ALTER TABLE `TXNS` ADD `TXN_META_INFO` varchar(128);
# 添加字段 : ALTER TABLE `HIVE_LOCKS` ADD `HL_AGENT_INFO` varchar(128);
# 添加字段 : ALTER TABLE `HIVE_LOCKS` ADD `HL_BLOCKEDBY_EXT_ID` bigint;
# 添加字段 : ALTER TABLE `HIVE_LOCKS` ADD `HL_BLOCKEDBY_INT_ID` bigint;
# create table COMPLETED_COMPACTIONS
# create table AUX_TABLE
# set version 2.0.0
source upgrade-1.2.0-to-2.0.0.mysql.sql

# 2.0.0 -> 2.1.0
# create table KEY_CONSTRAINTS
# create table WRITE_SET
# create table
# 添加字段 : TXN_COMPONENTS
# 添加字段 : ALTER TABLE COMPACTION_QUEUE ADD CQ_TBLPROPERTIES varchar(2048);
# 添加字段 : ALTER TABLE COMPLETED_COMPACTIONS ADD CC_TBLPROPERTIES varchar(2048);
# set version : 2.1.0
source upgrade-2.0.0-to-2.1.0.mysql.sql
 

# 2.1.0 -> 2.2.0
# 添加字段 : ALTER TABLE `TBLS` ADD `IS_REWRITE_ENABLED` bit(1);
# 设置字段 : UPDATE `TBLS` SET `IS_REWRITE_ENABLED` = false;
# 修改字段 : ALTER TABLE `TBLS` MODIFY COLUMN `IS_REWRITE_ENABLED` bit(1) NOT NULL;
# 添加字段 : ALTER TABLE `NOTIFICATION_LOG` ADD `MESSAGE_FORMAT` varchar(16);
# 修改字段 : ALTER TABLE `NOTIFICATION_LOG` MODIFY `MESSAGE` longtext; (1.1.0)
# ===> `MESSAGE` mediumtext
# 修改字段 : ALTER TABLE COLUMNS_V2 MODIFY TYPE_NAME MEDIUMTEXT;
# ==> `TYPE_NAME` varchar(4000) DEFAULT NULL
# 修改字段 : ALTER TABLE TABLE_PARAMS MODIFY PARAM_VALUE MEDIUMTEXT;
# ==> `PARAM_VALUE` varchar(4000) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
# 修改字段 : ALTER TABLE SERDE_PARAMS MODIFY PARAM_VALUE MEDIUMTEXT;
# ==> `PARAM_VALUE` varchar(4000) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# 修改字段 : ALTER TABLE SD_PARAMS MODIFY PARAM_VALUE MEDIUMTEXT;
# ==> `PARAM_VALUE` varchar(4000) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# 修改字段 : ALTER TABLE TBLS MODIFY TBL_NAME varchar(256) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL;
# ==> `TBL_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# 修改字段 : ALTER TABLE NOTIFICATION_LOG MODIFY TBL_NAME varchar(256) CHARACTER SET latin1 COLLATE latin1_bin;
# ==> `TBL_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# 修改字段 : ALTER TABLE PARTITION_EVENTS MODIFY TBL_NAME varchar(256) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL;
# ==> `TBL_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# 修改字段 : ALTER TABLE TAB_COL_STATS MODIFY TABLE_NAME varchar(256) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL;
# ==> `TBL_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# 修改字段 : ALTER TABLE PART_COL_STATS MODIFY TABLE_NAME varchar(256) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL;
# ==> `TABLE_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL
# 修改字段 : ALTER TABLE COMPLETED_TXN_COMPONENTS MODIFY CTC_TABLE varchar(256) CHARACTER SET latin1 COLLATE latin1_bin;
# ==> CTC_TABLE varchar(128)
# 修改字段 : ALTER TABLE COLUMNS_V2 MODIFY COLUMN_NAME varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL;
# ==> `COLUMN_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
# 修改字段 : ALTER TABLE PART_COL_PRIVS MODIFY COLUMN_NAME varchar(767) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL;
# ==> `COLUMN_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# 修改字段 : ALTER TABLE TBL_COL_PRIVS MODIFY COLUMN_NAME varchar(767) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL;
# ==> `COLUMN_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# 修改字段 : ALTER TABLE SORT_COLS MODIFY COLUMN_NAME varchar(767) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL;
# ==> `COLUMN_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# 修改字段 : ALTER TABLE TAB_COL_STATS MODIFY COLUMN_NAME varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL;
# ==> `COLUMN_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# 修改字段 : ALTER TABLE PART_COL_STATS MODIFY COLUMN_NAME varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL;
# ==> `COLUMN_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL
# set version 2.2.0
source upgrade-2.1.0-to-2.2.0.mysql.sql

# 2.2.0 -> 2.3.0
# 添加索引 ： CREATE INDEX TC_TXNID_INDEX ON TXN_COMPONENTS (TC_TXNID);
# set version : 2.3.0
source upgrade-2.2.0-to-2.3.0.mysql.sql

We need to pay attention to the 2.1.0  upgrade to  2.2.0 scripts strategy, the TBLSaddition of a non-empty list IS_REWRITE_ENABLED, and the value of all the fields in the old version of the table is set  false.
But for legacy clients in the creation table, and did not fill in this field, causing the client to write information into the database when the dollar caused the failure: non-empty-dependent .

So we need to modify this script to upgrade 037-HIVE-14496.mysql.sql:

ALTER TABLE `TBLS` ADD `IS_REWRITE_ENABLED` bit(1) DEFAULT false;

This avoids the conflict is not empty rely on older versions may hive metastore upgrade.

hive client upgrade:

After the completion of the upgrade of the hive metastore can be more than a hive-2.3.3 client. We use the sub-sector update the client environment, isolation upgrade strategy to complete the upgrade.

The main problems found in the upgrade process are:

  1. The new hive handling of keywords changed:

For example:  select timestamp from table_name; This script needs to be handled separately keyword <code> select `timestamp` from table_name ; </ code>

About the hive version reserved keyword, refer to:  LanguageManual DDL - the Apache Hive - the Apache Software Foundation

  1. hive existing database connection pool problem:

I upgraded during the database connection pool used for the original hive BoneCP , after the discovery of the connection pool to establish a link can not be perceived hang up after the event with the upgrade, resulting in a query if there is no operation for a long time to re-connect the disconnected abnormal accident happens.

After debugging I chose the DBCP connection pool, and set up the test to survive sql. To avoid the problem of the link is broken.

<property>
  <name>datanucleus.connectionPoolingType</name>
  <value>DBCP</value>
  <description>Uses a BoneCP connection pool for JDBC metastore</description>
 </property>
<property>
  <name>datanucleus.connectionPool.testSQL</name>
  <value>select 1</value>
 </property>
<property>
  <name>datanucleus.connectionPool.maxPoolSize</name>
  <value>4</value>
 </property>
<property>
  <name>datanucleus.connectionPool.minIdle</name>
  <value>2</value>
 </property>
  1. Presence .hive-staging_hive_date-time_ XXX Hive object data file:

An intermediate data storage location:

<!-- tmp dir -->
 <property>
  <name>hive.exec.scratchdir</name>
  <value>/tmp/hive</value>
  <description>Scratch space for Hive jobs</description>
 </property>
<property>
  <name>hive.exec.local.scratchdir</name>
  <value>/tmp/${user.name}</value>
  <description>Local scratch space for Hive jobs</description>
 </property>
<property>
  <name>hive.exec.stagingdir</name>
  <value>/tmp/hive-staging/.hive-staging</value>
 </property>

So far, hive upgrade is complete. And successfully completed on-line.

java start
Published 86 original articles · won praise 267 · Views 1.77 million +
His message board concerns

Guess you like

Origin blog.csdn.net/javastart/article/details/104519156
Recommended
Ranking
Linux关机和重启详解（shutdown、halt、poweroff、reboot、init）
Netty work notes 0007---NIO's three core component relationships
Knife4j tutorial
2021.10.29，内容:什么时候用接口和抽象类
How to solve the problem that changing the memory frequency causes the computer to become unusable?
SpringMVC Tutorial - Controller
linux learning skills -Linux 25 transport Vega paid special privileges and facl extension
Financial quarterly report evaluation report data automatic generation 1.0
Agile Development Series - The Values of Agile Development
scrapy achieve browsercookie Middleware
Daily
More
2024-05-19(0)
2024-05-18(31)
2024-05-17(6)
2024-05-16(23)
2024-05-15(5)
2024-05-14(9)
2024-05-13(8)
2024-05-12(28)
2024-05-11(32)
2024-05-10(34)

Code World

© 2018 codetd.com