Author: Han high PG Laboratory (Highgo PG Lab) - Xiao Yu
This article belongs to the first part of a series of articles on pg logical replication. The author plans to divide it into three parts to introduce the principle, construction and maintenance of logical replication. Interested friends can continue to pay attention to the introduction of the principles involved and the pg version on which the operation records are based. For pg12.4.
1. Application scenarios of logical replication
When we use the postgresql database as a production database, we often use streaming replication to implement hot backup of the database. When the original primary database fails, the standby database can quickly switch to the primary database to reduce business downtime. Streaming replication is data synchronization at the physical level of the database. In actual use scenarios, there are some unsatisfied requirements. For example, a certain business table of a provincial company needs to collect the business data of all prefectures and cities, or a basic table of the headquarters is updated. The company needs to synchronize this basic table in real time. For the above scenarios, physical replication cannot be met. At this time, we can use logical replication to meet these requirements.
Logical replication usage scenarios:
- Specify the replication requirements of the library or part of the table
- Aggregate data from multiple database instances into the same target database
- Distribute data from one library to multiple different libraries
- Copy between different versions
- Table synchronization between different library names
Second, the principle of logical replication
Logical replication is based on the logical analysis of WAL logs. We can understand WAL logs as ORACLE's redo logs. The principle of logical replication is that the main library on the publishing side parses the WAL log of the table in the Publication into a certain format and sends it to the subscription of the standby library. After the subscription receives the parsed WAL log, it replays the application, thereby realizing the synchronization of the table data.
You can see that the two important parts related to logical analysis are Publication and Subscription.
Publication (publishing) is created on a readable and writable business database instance, and the database for which Publication has been created is called a publishing node. For the functions and limitations of the publishing node, see Part Three for details. The publishing node is responsible for the logical analysis of the WAL log. This point must be clearly distinguished.
Subscription (subscription) is created on the logical replication database. The database that has created the Subscription becomes a subscription node. When a subscription is created, a logical replication slot for transmitting the parsed WAL log is created on the publishing node by default, and the subscription node passes The logical replication slot obtains the WAL data changes sent by the publishing node. Therefore, when the logical replication is interrupted due to network reasons, the publishing node will always save the WAL logs not applied by the subscribing node. If the business is busy, the disk of the publishing node is likely to be full. Everyone must pay attention to the problem.
Three, logical copy construction steps
1. An
example of creating a logical replication user on a publishing node : create user logicalrep replication login encrypted password'logicalrep'; must have streaming replication permission
2. Release node create release
Create release node syntax
Command: CREATE PUBLICATION Description: define a new publication Syntax: CREATE PUBLICATION name [ FOR TABLE [ ONLY ] table_name [ * ] [, ...] | FOR ALL TABLES ] [ WITH ( publication_parameter [= value] [, ... ] ) ] where option can be: PUBLISH INSERT | NOPUBLISH INSERT | PUBLISH UPDATE | NOPUBLISH UPDATE | PUBLISH DELETE | NOPUBLISH DELETE 默认发布insert,update,delete。
举例：create publication pub1 for table t1
修改发布节点语法
Command: ALTER PUBLICATION
Description: change the definition of a publication
Syntax:
ALTER PUBLICATION name WITH ( option [, ... ] ) where option can be:
PUBLISH INSERT | NOPUBLISH INSERT
| PUBLISH UPDATE | NOPUBLISH UPDATE
| PUBLISH DELETE | NOPUBLISH DELETE
ALTER PUBLICATION name ADD TABLE [ ONLY ] table_name [ * ] [, ...]ALTER PUBLICATION name SET TABLE [ ONLY ] table_name [ * ] [, ...]ALTER PUBLICATION name DROP TABLE [ ONLY ] table_name [ * ] [, ...]ALTER PUBLICATION name SET ( publication_parameter [= value] [, ... ] )ALTER PUBLICATION name OWNER TO { new_owner | CURRENT_USER | SESSION_USER }ALTER PUBLICATION name RENAME TO new_name
发布者小结
①目前仅仅支持发布表，不允许发布其他对象。
②同一张表，可以发布多次。
③在同一个数据库中，可以创建多个publication，但是不能重名，通过系统表查看已创建的publication
④允许使用all tables发布所有表。
⑤一个publication允许有多个订阅者。
⑥目前publication仅支持insert, update, delete。允许发布时，选择发布insert、update、delete，比如只发布insert，而不发布update, delete。当发布了表的update, delete时，表必须设置replica identity，即如何标示OLD TUPLE，通过pk或者uk或者full。如果设置了nothing，则执行update,delete时会报错。
⑦create publication或者alter publication，发布或者修改发布内容中添加或者删除表时，都是事务级别，不会出现复制了部分事务的情况。⑧发布者需要设置wal_level=logical，同时开启足够的worker，设置足够大的replication slot，设置足够多的sender。
⑨发布者的pg_hba.conf需要设置replication条目，允许订阅者连接。
⑩发布者的数据库中，必须有replication角色的用户，或者超级用户，并且订阅者要使用它通过流复制协议连接到发布者。
3、订阅端创建订阅表
创建表结构同发布端表结构相同的空表 注意主键约束，没有的话发布端的update delete等操作会报错
例：create table t1 (id int primary key,name text)
4、订阅端创建订阅
创建订阅语法
Command: CREATE SUBSCRIPTION
Description: define a new subscription
Syntax:
CREATE SUBSCRIPTION subscription_name CONNECTION 'conninfo' PUBLICATION { publication_name [, ...] } [ WITH ( subscription_parameter [= value] [, ... ] ) ]
举例：create subscription sub1 connection ‘host=192.168.230.81 port=5866 dbname=postgres user=logicalrep password=logicalrep’ publication pub1;
修改订阅语法
Command: ALTER SUBSCRIPTION
Description: change the definition of a subscription
Syntax:
ALTER SUBSCRIPTION name WITH ( option [, ... ] ) ] where option can be:
SLOT NAME = slot_name
ALTER SUBSCRIPTION name CONNECTION 'conninfo'ALTER SUBSCRIPTION name SET PUBLICATION publication_name [, ...] [ WITH ( set_publication_option [= value] [, ... ] ) ]ALTER SUBSCRIPTION name REFRESH PUBLICATION [ WITH ( refresh_option [= value] [, ... ] ) ]ALTER SUBSCRIPTION name ENABLE
ALTER SUBSCRIPTION name DISABLE
ALTER SUBSCRIPTION name SET ( subscription_parameter [= value] [, ... ] )ALTER SUBSCRIPTION name OWNER TO { new_owner | CURRENT_USER | SESSION_USER }ALTER SUBSCRIPTION name RENAME TO new_name
Subscriber summary
①You must use the super user to create a subscription
②The subscriber needs to connect to the publisher through the streaming replication protocol, and at the same time, you need to create a replication slot in the publisher.
③In the same database, multiple subscriptions can be created, and these subscriptions can be connected from one or more publishers.
④ The same table of subscribers cannot accept multiple publications from the same source.
⑤ For each subscription, a slot needs to be created on the publishing end, which can be specified using slot name = ?, or the default subscription name.
⑥ When creating a subscription or alter subscription, you can use enable to enable the subscription, or use disable to suspend the subscription.
⑦If you want to delete the subscription completely, use drop subscription. Note that after deleting the subscription, the local table will not be deleted, and the data will not be cleared, just not receiving the upstream information of the subscription.
⑧When creating a subscription, the publishing side table will not be automatically created, so the table needs to be created on the subscription side first.