Apollo Configuration Center 1.0 to 2.0 upgrade table data ETL

Table of contents

0. Demand

1. Export version 1.0 data

2. Convert case

3. Import data into the 2.0 version library


0. Demand

        Import existing Apollo 1.0 table data into the Apollo 2.0 table. The target library has created the Apollo 2.0 library table structure. Version 2.0 has four more tables than version 1.0:
ApolloConfigDB.AccessKey
ApolloConfigDB.ServiceRegistry
ApolloPortalDB.SPRING_SESSION
ApolloPortalDB.SPRING_SESSION_ATTRIBUTES


1. Export version 1.0 data

        Only export data without tombstones. Some tables in the new version add a deletion timestamp field, and set a unique index on this field that combines other fields. Only export data with IsDeleted=0 to prevent errors that violate unique index constraints when inserting a new version library.

# 导出 apolloconfigdb 库中所有包含 IsDeleted = 0 的表数据
mysqldump -uroot -p123456 -S /data/18251/mysqldata/mysql.sock --set-gtid-purged=off --no-create-db --no-create-info --skip-opt --complete-insert --quick --extended-insert apolloconfigdb app appnamespace audit cluster commit grayreleaserule item namespace namespacelock release releasehistory serverconfig --where="IsDeleted=0" > apolloconfigdb_IsDeleted_0.sql

# 导出 apolloconfigdb 库中不包含 IsDeleted 字段的表数据
mysqldump -uroot -p123456 -S /data/18251/mysqldata/mysql.sock --set-gtid-purged=off --no-create-db --no-create-info --skip-opt --complete-insert --quick --extended-insert apolloconfigdb instance instanceconfig releasemessage > apolloconfigdb_no_IsDeleted.sql

# 导出 apolloportaldb 库中所有包含 IsDeleted = 0 的表数据
mysqldump -uroot -p123456 -S /data/18251/mysqldata/mysql.sock --set-gtid-purged=off --no-create-db --no-create-info --skip-opt --complete-insert --quick --extended-insert apolloportaldb app appnamespace consumer consumerrole consumertoken favorite permission role rolepermission serverconfig userrole --where="IsDeleted=0" > apolloportaldb_IsDeleted_0.sql

# 导出 apolloportaldb 库中不包含 IsDeleted 字段的表数据
mysqldump -uroot -p123456 -S /data/18251/mysqldata/mysql.sock --set-gtid-purged=off --no-create-db --no-create-info --skip-opt --complete-insert --quick --extended-insert apolloportaldb authorities consumeraudit users > apolloportaldb_no_IsDeleted.sql


2. Convert case

        In reality, the library table names of 1.0 may all be lowercase, while the library table names of 2.0 may be case-sensitive.

# 库名
sed -i 's/USE `apolloconfigdb`/USE `ApolloConfigDB`/g;' apolloconfigdb_IsDeleted_0.sql
sed -i 's/USE `apolloconfigdb`/USE `ApolloConfigDB`/g;' apolloconfigdb_no_IsDeleted.sql

sed -i 's/USE `apolloportaldb`/USE `ApolloPortalDB`/g;' apolloportaldb_IsDeleted_0.sql
sed -i 's/USE `apolloportaldb`/USE `ApolloPortalDB`/g;' apolloportaldb_no_IsDeleted.sql

# 表名
regexp='s/INSERT INTO `app`/INSERT INTO `App`/g;'
regexp=$regexp's/INSERT INTO `appnamespace`/INSERT INTO `AppNamespace`/g;'
regexp=$regexp's/INSERT INTO `audit`/INSERT INTO `Audit`/g;'
regexp=$regexp's/INSERT INTO `cluster`/INSERT INTO `Cluster`/g;'
regexp=$regexp's/INSERT INTO `commit`/INSERT INTO `Commit`/g;'
regexp=$regexp's/INSERT INTO `grayreleaserule`/INSERT INTO `GrayReleaseRule`/g;'
regexp=$regexp's/INSERT INTO `item`/INSERT INTO `Item`/g;'
regexp=$regexp's/INSERT INTO `namespace`/INSERT INTO `Namespace`/g;'
regexp=$regexp's/INSERT INTO `namespacelock`/INSERT INTO `NamespaceLock`/g;'
regexp=$regexp's/INSERT INTO `release`/INSERT INTO `Release`/g;'
regexp=$regexp's/INSERT INTO `releasehistory`/INSERT INTO `ReleaseHistory`/g;'
regexp=$regexp's/INSERT INTO `serverconfig`/INSERT INTO `ServerConfig`/g;'

sed -i "${regexp}" apolloconfigdb_IsDeleted_0.sql

regexp='s/INSERT INTO `instance`/INSERT INTO `Instance`/g;'
regexp=$regexp's/INSERT INTO `instanceconfig`/INSERT INTO `InstanceConfig`/g;'
regexp=$regexp's/INSERT INTO `releasemessage`/INSERT INTO `ReleaseMessage`/g;'

sed -i "${regexp}" apolloconfigdb_no_IsDeleted.sql

regexp='s/INSERT INTO `app`/INSERT INTO `App`/g;'
regexp=$regexp's/INSERT INTO `appnamespace`/INSERT INTO `AppNamespace`/g;'
regexp=$regexp's/INSERT INTO `consumer`/INSERT INTO `Consumer`/g;'
regexp=$regexp's/INSERT INTO `consumerrole`/INSERT INTO `ConsumerRole`/g;'
regexp=$regexp's/INSERT INTO `consumertoken`/INSERT INTO `ConsumerToken`/g;'
regexp=$regexp's/INSERT INTO `favorite`/INSERT INTO `Favorite`/g;'
regexp=$regexp's/INSERT INTO `permission`/INSERT INTO `Permission`/g;'
regexp=$regexp's/INSERT INTO `role`/INSERT INTO `Role`/g;'
regexp=$regexp's/INSERT INTO `rolepermission`/INSERT INTO `RolePermission`/g;'
regexp=$regexp's/INSERT INTO `serverconfig`/INSERT INTO `ServerConfig`/g;'
regexp=$regexp's/INSERT INTO `userrole`/INSERT INTO `UserRole`/g;'

sed -i "${regexp}" apolloportaldb_IsDeleted_0.sql

regexp='s/INSERT INTO `authorities`/INSERT INTO `Authorities`/g;'
regexp=$regexp's/INSERT INTO `consumeraudit`/INSERT INTO `ConsumerAudit`/g;'
regexp=$regexp's/INSERT INTO `users`/INSERT INTO `Users`/g;'

sed -i "${regexp}" apolloportaldb_no_IsDeleted.sql


3. Import data into the 2.0 version library

mysql -uroot -p123456 -h127.0.0.1 -P3306 -D ApolloPortalDB < /var/lib/mysql/apolloportaldb_no_IsDeleted.sql > apolloportaldb_no_IsDeleted.log 2>&1
mysql -uroot -p123456 -h127.0.0.1 -P3306 -D ApolloConfigDB < /var/lib/mysql/apolloconfigdb_no_IsDeleted.sql > apolloconfigdb_no_IsDeleted.log 2>&1
mysql -uroot -p123456 -h127.0.0.1 -P3306 -D ApolloPortalDB < /var/lib/mysql/apolloportaldb_IsDeleted_0.sql > apolloportaldb_IsDeleted_0.log 2>&1
mysql -uroot -p123456 -h127.0.0.1 -P3306 -D ApolloConfigDB < /var/lib/mysql/apolloconfigdb_IsDeleted_0.sql > apolloconfigdb_IsDeleted_0.log 2>&1

Guess you like

Origin blog.csdn.net/wzy0623/article/details/132036977