Use Docker to build Postgresql active and standby clusters

Meaning of this article

• Postgresql cluster, small and hot standby, medium read and write separation, can be used as a distributed storage infrastructure;
• Using Docker to build, you can isolate the differences between development, testing, and production environments, realize the testability of the build script itself, and further achieve the goal of DevOps.

Why not go further with a container orchestration system like Docker Swarm or K8S? Postgresl cluster is strong state, and container orchestration is suitable for stateless services. Allowing K8S to perform automatic failover will cause more failures in the underlying data of Postgesql. Even if K8S state migration is turned off, strong state communication is required between PG servers. , brings a lot of unnecessary arrangement and configuration, which is more than taking off your pants and farting.

Precondition

• Docker: 20+
• Postgresql: 13+ images

Environment construction

Step1 Create a virtual network

In order not to be confused with other network environments, we can first create a Docker virtual network

docker network create pg-cluster-network

This article demonstrates the establishment of a PG cluster on one machine, and the PG in the production environment will be built on two machines, so this step can be ignored.

Step2 Build the main database

1. Create the main library mirror

## 使用postgres官方镜像创建主库容器
docker run -d \
  --restart=unless-stopped \
  --name=pg-cluster-pg1 \
  --network-alias=pg1 \
  --network pg-cluster-network \
  -e POSTGRES_DB=test \
  -e POSTGRES_USER=test \
  -e POSTGRES_PASSWORD=123456 \
  postgres:13-alpine

2. Open remote synchronization permission to the standby database

 # 增加备库远程同步权限配置
 docker exec -it pg-cluster-pg1 \
  sh -c "echo host replication test all md5 >> /var/lib/postgresql/data/pg_hba.conf"

3. Enable sync backup (optional)

Synchronous backup has a great impact on the performance of the main database. Only when the transaction consistency requirements of the backup are absolutely guaranteed, after opening and closing, when the main database crashes, there is a very small possibility that some transactions are not synchronized {.is-warning}

 # 开启同步备份
 docker exec -it pg-cluster-pg1 \
  sh -c "echo synchronous_standby_names = \'*\' >> /var/lib/postgresql/data/postgresql.conf"

4. reload configuration

 # 重载配置文件，以便生效
 docker exec -it -u postgres pg-cluster-pg1 sh -c "pg_ctl reload"

5. Create backup slot

 # 创建备份槽，以便备用服务器接入
 docker exec -it pg-cluster-pg1 \
  psql -U test -c "select * from pg_create_physical_replication_slot('standby1_slot');"

Step3 Build a backup database

1. Create data volumes to synchronize data from the main library, modify configuration

 docker volume rm pg-cluster-pg2-data
 docker volume create pg-cluster-pg2-data

2. Create a base backup from the main repository

  # 使用pg_basebackup创建数据库基础备份到数据卷，-R参数会自动生成热备配置到postgresql.auto.conf中
 docker run -it --rm \
  --network pg-cluster-network \
  -v pg-cluster-pg2-data:/var/lib/postgresql/data \
  postgres:13-alpine \
  pg_basebackup -R -h pg1 -U test -X stream -P -S standby1_slot -D /var/lib/postgresql/data

During execution, you will be prompted to enter the main database backup account password. After the input is completed, the backup progress will be displayed until 100%. Otherwise, it will fail. You need to check the main database permission configuration and network connection.

3. Start the backup repository

 docker run -d \
  --restart=unless-stopped \
  --name=pg-cluster-pg2 \
  --network-alias=pg2 \
  --network pg-cluster-network \
  -v pg-cluster-pg2-data:/var/lib/postgresql/data \
  postgres:13-alpine

4. Verify the synchronization status. You can query the insertion status in the main database to verify whether the standby database is connected.

docker exec -it pg-cluster-pg1
  psql -U test -c "SELECT slot_name, slot_type, active, wal_status from pg_replication_slots;"
# slot_name     slot_type active  wal_status
# standby1_slot physical  t       reserved

Test data synchronization

1. Build pgadmin for testing

 docker run -d \
  --restart=unless-stopped \
  --name=pg-cluster-pgadmin \
  --network pg-cluster-network \
  -p 8888:80 \
  -e PGADMIN_DEFAULT_EMAIL=admin \
  -e PGADMIN_DEFAULT_PASSWORD=123456 \
  dpage/pgadmin4:snapshot

After the pgadmin container is created, you can configure the connection to pg1 and pg2 on http://localhost:8888. Because the network alias is set, the two database servers can use pg1 and pg2 instead of ip to directly access the pgadmin account/password: admin/ 123456

2. Create a test table in the main library and insert data Execute the following sql in the pg1.test library

CREATE TABLE userInfo(
    userName VARCHAR(20),
    age INT,
    PRIMARY KEY(userName)
);
INSERT INTO userInfo VALUES('张三',10);

Execute the following query in pg2.test

SELECT * FROM userInfo;

If you see the data just inserted in pg1.test, it means that the master and backup have been set up successfully

Recovery

When the main library crashes, failover or recovery is required

Please perform the following operations with caution, otherwise it may cause the primary and secondary data to be out of sync {.is-warning}

Troubleshooting solution 1: transfer to the standby database

The standby database is read-only by default and can be used to share the read-only load. It can be verified by the following sql. Normally, the database will be reported as read-only.

-- 在 pg2.test中执行
UPDATE userInfo SET age=4;
-- ERROR:  cannot execute UPDATE in a read-only transaction
-- SQL 状态: 25006

When the main database hangs up, you can quickly execute commands in the standby database to convert the standby database into the main database: Method 1: Use SQL

-- 在pg2.test中执行：
SELECT pg_promote(true,60);
-- 参数1：等待提升完成
-- 参数2：等待60秒

Method 2: Use the pg_ctl command

docker exec -it -u postgres pg-cluster-pg2 \
 pg_ctl promote

After the standby database is converted to the main database, another backup database needs to be built to ensure the next failover.

Troubleshooting solution 2: main library recovery

Step 1: The main database restores data from the standby database in reverse (optional)

In most cases, if the host is powered off, etc., you only need to start the main database and connect it to the network to complete the recovery. If and only when the main database file system is damaged, data is lost, and data needs to be recovered from the standby database.

1. Open slots in the standby database to the primary database

 # 在原备库中创建备份槽，以便原主库接入并恢复数据
 docker exec -it pg-cluster-pg2 \
  psql -U test -c "select * from pg_create_physical_replication_slot('main_slot');"

2 Restore data from backup repository

 # 创建docker数据卷
 docker volume rm pg-cluster-pg1-data
 docker volume create pg-cluster-pg1-data
 # 使用pg_basebackup从备份库恢复数据到数据卷，-R参数会自动生成热备配置到postgresql.auto.conf中
 docker run -it --rm \
  --network pg-cluster-network \
  -v pg-cluster-pg1-data:/var/lib/postgresql/data \
  postgres:13-alpine \
  pg_basebackup -R -h pg2 -U test -X stream -P -S main_slot -D /var/lib/postgresql/data

3. Configure the main database to be automatically upgraded to the main database after data synchronization is completed

If the standby database has been upgraded to the primary database, do not perform this operation! Otherwise, it may cause data inconsistency between the master and backup! {.is-warning}

After the main library is started, you can manually execute pg_promote to upgrade, and you do not need to perform this section. If the data is large and the recovery time is long, consider using the following methods

 # 配置升级触发文件
 docker run -it --rm \
  --network pg-cluster-network \
  -v pg-cluster-pg1-data:/var/lib/postgresql/data \
  postgres:13-alpine \
  sh -c "echo promote_trigger_file = \'standby.signal\' >> /var/lib/postgresql/data/postgresql.conf"

When the main library starts and recovers successfully, it will automatically switch to the main library state to provide normal read and write services.

Step 2 Start the main library

 docker run -d \
  --restart=unless-stopped \
  --name=pg-cluster-pg1 \
  --network-alias=pg1 \
  --network pg-cluster-network \
  -v pg-cluster-pg1-data:/var/lib/postgresql/data \
  postgres:13-alpine

After the main database is started, verify that the data is not lost, that is, the recovery of the main database has been completed. If the data is found to be lost, please go to step 1

{{o.name}}

{{m.name}}