将本地PostgreSQL迁移到AWS RDS PostgreSQL

问题描述

昨天在火车上帮客户处理一个PostgreSQL迁移的问题，客户计划将本地的PostgreSQL数据库导出，然后导入到已经创建完毕的RDS PostgreSQL中，但是遇到很多报错：
类似如下：

You are now connected to database "xxxx" as user "root".
SET
SET
SET
SET
SET
SET
ERROR:  permission denied for database aaa
ERROR:  schema "xxx" does not exist
ERROR:  permission denied for database aaa
ERROR:  schema "xxxxx" does not exist
ERROR:  permission denied for database aaa
ERROR:  schema "xxxx" does not exist
CREATE EXTENSION
ERROR:  must be owner of extension plpgsql

以及类似如下的错误：

psql:smart2.sql:3143: invalid command \N
psql:smart2.sql:3144: invalid command \N
psql:smart2.sql:3145: invalid command \N
psql:smart2.sql:3146: invalid command \N
psql:smart2.sql:3147: invalid command \N
psql:smart2.sql:3148: invalid command \N
psql:smart2.sql:3149: invalid command \N
psql:smart2.sql:3150: invalid command \N
psql:smart2.sql:3151: invalid command \N
psql:smart2.sql:3152: invalid command \N
psql:smart2.sql:3153: invalid command \N
psql:smart2.sql:3269: invalid command \N
psql:smart2.sql:3270: invalid command \N
psql:smart2.sql:3271: invalid command \N
psql:smart2.sql:3272: invalid command \N
psql:smart2.sql:3273: invalid command \N
psql:smart2.sql:3274: invalid command \.
psql:smart2.sql:3281: ERROR:  syntax error at or near "2"
LINE 1: 2 2017-10-24 08:34:25.013362 FORBIDDEN 
        ^
psql:smart2.sql:3288: ERROR:  permission denied for relation xxx
psql:smart2.sql:3289: invalid command \.
psql:smart2.sql:3296: ERROR:  permission denied for sequence xxxxxxx
psql:smart2.sql:3303: ERROR:  permission denied for relation xxxxxx

psql:smart2.sql:3733: ERROR:  must be owner of relation xxxxx
psql:smart2.sql:3741: ERROR:  must be owner of relation xxxxxx

问题分析

找客户拿到了pg_dump生成的sql脚本看了一下，发现确实存在一些问题，而这些问题也是我之前没有关注到的。

• pg_dump并不会导出role，可以使用pg_dumpall -r单独将role导出；
• 在RDS PostgreSQL里出于安全上的考虑，并没有superuser 和replication 角色，最大权限的角色就是rds_superuser
• 本地PostgreSQL我们通常会安装在postgre用户下，这个时候在pg中对应的就会有一个postgres超级用户（客户的sql脚本里就有这个用户），在RDS PostgreSQL中缺省没有此用户

解决方案

• 修改01：

原导出sql形式：
CREATE ROLE campaigncenter;
ALTER ROLE campaigncenter WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB LOGIN NOREPLICATION PASSWORD 'md551420aa22858a6506739fddd0e97c1eb';

第二条sql语句由于权限问题在RDS PostgreSQL里执行会报错：
调整为：
CREATE ROLE campaigncenter WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB LOGIN NOREPLICATION PASSWORD 'md551420aa22858a6506739fddd0e97c1eb';

• 修改02

ALTER DATABASE xxx OWNER TO postgres;
ALTER TABLE aaaa OWNER TO postgres;
ALTER TABLE aaaaa OWNER TO postgres;

脚本里大部分的对象schema都是postgres，这个在postgresql里属于超级superuser

由于RDS PostgreSQL里没有postgre用户，可以将其调整为RDS 的 master user，比如root：
ALTER DATABASE xxx OWNER TO root;
ALTER TABLE aaaa OWNER TO root;
ALTER TABLE aaaaa OWNER TO root;

调整之后的效果

调整完成后大部分sql语句都可以正常执行了。

postgres=> \i role2.sql
ALTER DATABASE
DROP DATABASE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
GRANT ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
DROP ROLE
CREATE ROLE
ALTER ROLE
postgres=> 
postgres=> 
postgres=> 
postgres=> 
postgres=> \i smart2.sql
SET
SET
SET
SET
SET
SET
psql:smart2.sql:14: ERROR:  database "smart_dealer_bff" does not exist
psql:smart2.sql:15: ERROR:  database "smart_dealer_bff" does not exist
CREATE DATABASE
ALTER DATABASE
psql (9.6.9, server 9.6.11)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
You are now connected to database "smart_dealer_bff" as user "root".
SET
SET
SET
SET
SET
SET
CREATE EXTENSION
psql:smart2.sql:41: ERROR:  must be owner of extension plpgsql
CREATE EXTENSION
psql:smart2.sql:55: ERROR:  must be owner of extension pgcrypto
SET
CREATE SEQUENCE
ALTER TABLE
SET
SET
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE
ALTER TABLE
CREATE SEQUENCE
ALTER TABLE
CREATE TABLE

.....
.....
.....

ALTER TABLE
COPY 1
COPY 0
 setval 
--------
      1
(1 row)

COPY 0
 setval 
--------
      1
(1 row)

COPY 0
 setval 
--------
      1
(1 row)

........



COPY 1
 setval 
--------
      1
(1 row)

COPY 73
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
.......
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
CREATE INDEX
CREATE INDEX
.......
CREATE INDEX
CREATE INDEX
CREATE INDEX
ALTER TABLE
ALTER TABLE
psql:smart2.sql:4636: WARNING:  no privileges could be revoked for "public"
REVOKE
psql:smart2.sql:4637: WARNING:  no privileges could be revoked for "public"
REVOKE
psql:smart2.sql:4638: WARNING:  no privileges were granted for "public"
GRANT
psql:smart2.sql:4639: WARNING:  no privileges were granted for "public"
GRANT