Synchronous replication delay PostgreSQL flow test (ii)

1 replication master synchronizing stream from the measured SR

Built environment

Server | Role
| - |: -: | -: |
10.10.56.16 | Master
10.10.56.17 | slave1
10.10.56.19 | slave2

16 Check status

pocdb=# SELECT client_addr,application_name,sync_state FROM pg_stat_replication;
 client_addr | application_name | sync_state
-------------+------------------+------------
 10.10.56.17 | slave1           | sync
 10.10.56.19 | slave2           | potential
(2 rows)

pocdb=#

When greater than two machines from the library synchronization status potential, may be upgraded to represent master

Query User

pocdb=# \du+
                                          List of roles
 Role name |                         Attributes                         | Member of | Description
-----------+------------------------------------------------------------+-----------+-------------
 postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}        |
 repl      | Replication                                                | {}        |

pocdb=#

16 Create a synchronization test table synctestand sequence

pocdb=# create table synctest (id bigint primary key ,number bigint,date timestamp default now());
CREATE TABLE

pocdb=# create sequence seq_synctest increment by 1 minvalue 1 maxvalue 99999999999999 cache 50 no cycle;
CREATE SEQUENCE
pocdb=#

View table size

pocdb=# \d+
                           List of relations
 Schema |     Name     |   Type   |  Owner   |    Size    | Description
--------+--------------+----------+----------+------------+-------------
 public | seq_synctest | sequence | postgres | 8192 bytes |
 public | synctest     | table    | postgres | 0 bytes    |
(2 rows)

pocdb=#

At this time, the server automatically synchronize the tables 17 and 19, respectively, to query

16 server to write into a script

postgres@clw-db1:/pgdata/10/poc/scripts> vi bench_script_for_insert_20180717.sql
postgres@clw-db1:/pgdata/10/poc/scripts> cat bench_script_for_insert_20180717.sql
\set number random(1, 100000000000000000000000)
INSERT INTO synctest(id,number) VALUES (nextval('seq_synctest'),:number);
postgres@clw-db1:/pgdata/10/poc/scripts>

16,17,19 write latency monitoring script

postgres@clw-db1:/pgdata/10/poc/scripts> cat monior_syncSR_relay.sh
#!/bin/bash

/opt/pgsql-10/bin/psql pocdb<<EOF
select now();
select client_addr, application_name, write_lag, flush_lag, replay_lag \
from pg_stat_replication where usename='l_repl' and application_name='slave1';
 \q
EOF
postgres@clw-db1:/pgdata/10/poc/scripts>

Start pgbench pressure-measurement

postgres@clw-db1:/pgdata/10/poc/scripts> /opt/pgsql-10/bin/pgbench -T 1200 -j 600 -c 500  -f  bench_script_for_insert_20180717.sql pocdb

Slave1 write data query script

for i in {1..1000000000}
do
/pgdata/10/poc/scripts/monior_syncSR_relay_slave1.sh >> syncSR_relay_slave1_result
sleep 10
done

for i in {1..1000000000}
do
/pgdata/10/poc/scripts/monior_syncSR_relay_slave2.sh >> syncSR_relay_slave2_result
sleep 10
done

postgres@clw-db2:/pgdata/10/poc/scripts> cat query_count_slave1.sh
#!/bin/bash

/opt/pgsql-10/bin/psql pocdb<<EOF
select now();
select max(id) from synctest;
 \q
EOF
postgres@clw-db2:/pgdata/10/poc/scripts>

for i in {1..1000000000}
do
/pgdata/10/poc/scripts/query_count_result.sh >> query_count_sum
sleep 10
done

Insert the number of query script

postgres@clw-db1:/pgdata/10/poc/scripts> cat query_count_result.sh
#!/bin/bash

/opt/pgsql-10/bin/psql pocdb<<EOF
select now();
select max(id) from synctest;
 \q
EOF
postgres@clw-db1:/pgdata/10/poc/scripts>

Write memory, network, IO performance monitoring script

/home/super/pgsoft/nmon_x86_64_sles11 -f  -c 150  -s 10

17 number of queries script

for i in {1..1000000000}
do
/pgdata/10/poc/scripts/query_count_slave1.sh >> query_count_slave1_sum
sleep 10
done

17 Performance Monitoring script

/home/pgsoft/nmon_x86_64_sles11 -f  -c 150  -s 10

19 number of queries script

for i in {1..1000000000}
do
/pgdata/10/poc/scripts/query_count_slave2.sh >> query_count_slave2_sum
sleep 10
done

19 Performance Monitoring

/home/pgsoft/nmon_x86_64_sles11 -f  -c 150  -s 10

-16 test results

could not connect to server: Resource temporarily unavailable
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
transaction type: bench_script_for_insert_20180717.sql
scaling factor: 1
query mode: simple
number of clients: 500
number of threads: 500
duration: 1200 s
number of transactions actually processed: 9723149
latency average = 61.736 ms
tps = 8098.953578 (including connections establishing)
tps = 8099.988751 (excluding connections establishing)
postgres@clw-db1:/pgdata/10/poc/scripts>

pocdb=# \d+
                           List of relations
 Schema |     Name     |   Type   |  Owner   |    Size    | Description
--------+--------------+----------+----------+------------+-------------
 public | seq_synctest | sequence | postgres | 8192 bytes |
 public | synctest     | table    | postgres | 377 MB     |
(2 rows)

pocdb=#

A master-slave synchronization test results 2

测试背景: Server memory 200G 10.10.56.16,10.10.56.17,10.10.56.19 CPU 8 core network transmission 3M / s

测试方法: 20 minutes 600 threads 500 to keep client connections 16masterto write data server

测试结果: CPU usage 80%, disk the I / O 23MB/s, TPS: 7419, the amount of data: 1880万,
synchronization delay of about 3毫秒about

Pressure measurement script (-T 1200 seconds -j 600 concurrent client connections -c thread connection number 500 -f output pressure measurement results)

/opt/pgsql-10/bin/pgbench -T 1200 -j 600 -c 500  -f  bench_script_for_insert_20180717.sql pocdb

The first test (2 1 master-slave synchronization SR)

transaction type: bench_script_for_insert_20180717.sql
scaling factor: 1
query mode: simple
number of clients: 500
number of threads: 500
duration: 1200 s
number of transactions actually processed: 8906813
latency average = 67.391 ms
tps = 7419.410395 (including connections establishing)
tps = 7420.536021 (excluding connections establishing)
postgres@clw-db1:/pgdata/10/poc/scripts>

The second test (2 1 master-slave synchronization SR)

postgres@clw-db1:/pgdata/10/poc/scripts> /opt/pgsql-10/bin/pgbench -T 1200 -j 600 -c 500  -f bench_script_for_insert_20180717.sql pocdb

query mode: simple
number of clients: 500
number of threads: 500
duration: 1200 s
number of transactions actually processed: 8239738
latency average = 72.848 ms
tps = 6863.613524 (including connections establishing)
tps = 6865.035476 (excluding connections establishing)
postgres@clw-db1:/pgdata/10/poc/scripts>

Master query from the data delay, the main observation parameters (write_lag, flush_lsn, replay_lag)

pocdb=# select * from pg_stat_replication;
-[ RECORD 1 ]----+------------------------------
pid              | 23564
usesysid         | 16393
usename          | repl
application_name | slave2
client_addr      | 10.10.56.19
client_hostname  |
client_port      | 52820
backend_start    | 2018-05-16 17:43:23.726216+08
backend_xmin     |
state            | streaming
sent_lsn         | 1/7D4FF030
write_lsn        | 1/7D4FF030
flush_lsn        | 1/7D4FEF78
replay_lsn       | 1/7D4FEF78
write_lag        | 00:00:00.003057
flush_lag        | 00:00:00.003057
replay_lag       | 00:00:00.003057
sync_priority    | 2
sync_state       | potential
-[ RECORD 2 ]----+------------------------------
pid              | 23562
usesysid         | 16393
usename          | repl
application_name | slave1
client_addr      | 10.10.56.17
client_hostname  |
client_port      | 33647
backend_start    | 2018-05-16 17:43:17.371715+08
backend_xmin     |
state            | streaming
sent_lsn         | 1/7D524110
write_lsn        | 1/7D523F30
flush_lsn        | 1/7D523570
replay_lsn       | 1/7D523570
write_lag        | 00:00:00.000329
flush_lag        | 00:00:00.000329
replay_lag       | 00:00:00.000329
sync_priority    | 1
sync_state       | sync

pocdb=# select * from pg_stat_replication;
-[ RECORD 1 ]----+------------------------------
pid              | 23564
usesysid         | 16393
usename          | repl
application_name | slave2
client_addr      | 10.10.56.19
client_hostname  |
client_port      | 52820
backend_start    | 2018-05-16 17:43:23.726216+08
backend_xmin     |
state            | streaming
sent_lsn         | 1/7EB37850
write_lsn        | 1/7EB36F00
flush_lsn        | 1/7EB34EE0
replay_lsn       | 1/7EB34EE0
write_lag        | 00:00:00.000841
flush_lag        | 00:00:00.000841
replay_lag       | 00:00:00.000841
sync_priority    | 2
sync_state       | potential
-[ RECORD 2 ]----+------------------------------
pid              | 23562
usesysid         | 16393
usename          | repl
application_name | slave1
client_addr      | 10.10.56.17
client_hostname  |
client_port      | 33647
backend_start    | 2018-05-16 17:43:17.371715+08
backend_xmin     |
state            | streaming
sent_lsn         | 1/7EB37F80
write_lsn        | 1/7EB27B08
flush_lsn        | 1/7EB27B08
replay_lsn       | 1/7EB27B08
write_lag        | 00:00:00.001525
flush_lag        | 00:00:00.003983
replay_lag       | 00:00:00.012568
sync_priority    | 1
sync_state       | sync

Query volume data

pocdb=# \d+
                           List of relations
 Schema |     Name     |   Type   |  Owner   |    Size    | Description
--------+--------------+----------+----------+------------+-------------
 public | seq_synctest | sequence | postgres | 8192 bytes |
 public | synctest     | table    | postgres | 377 MB     |
(2 rows)

pocdb=# select max(id) from synctest;
   max
----------
 18812501
(1 row)

pocdb=#

Single Library test

48 server to create the database pocdband tablessynctest

postgres=# create database pocdb;
CREATE DATABASE

pocdb=# create table synctest (id bigint primary key ,number bigint,date timestamp default now());
CREATE TABLE
pocdb=# create sequence seq_synctest increment by 1 minvalue 1 maxvalue 99999999999999 cache 50 no cycle;
CREATE SEQUENCE
pocdb=# \d+
                           List of relations
 Schema |     Name     |   Type   |  Owner   |    Size    | Description
--------+--------------+----------+----------+------------+-------------
 public | seq_synctest | sequence | postgres | 8192 bytes |
 public | synctest     | table    | postgres | 0 bytes    |
(2 rows)

pocdb=#

48 Performance Monitoring

/home/postgres/pgsoft/nmon_x86_64_sles11 -c 180 -f -s 10

48 number of queries script

for i in {1..100000000}
do
/pgdata/10/scripts/query_count_result.sh >> query_count_slave2_sum
sleep 10
done

48 pressure test script

 /opt/pgsql-10/bin/pgbench -T 1200 -j 800 -c 400  -f bench_script_for_insert_20180717.sql pocdb

Single-instance database test

测试背景: Server: 10.10.56.17 cpu: 8-core Memory: 128GB
测试过程: Perform 20 minutes into the database on the test server, 500 thread count 500 client connections
- 测试结果: Write to the database a total of 35 million, with an average response time 17.125ms, TPS is 29000, CPU occupancy rate of 95% Disk Write: 35MB / s, network transfer rate: 3MB / s

17 pressure measurement results in one database server

transaction type: bench_script_for_insert_20180717.sql
scaling factor: 1
query mode: simple
number of clients: 500
number of threads: 500
duration: 1200 s
number of transactions actually processed: 35047438
latency average = 17.125 ms
tps = 29197.286164 (including connections establishing)
tps = 29205.049038 (excluding connections establishing)
postgres@clw-db2:/pgdata/estest10/scripts>

查询数据
“`
pocdb1=# \d+
List of relations
Schema | Name | Type | Owner | Size | Description
——–+————–+———-+———-+————+————-
public | seq_synctest | sequence | postgres | 8192 bytes |
public | synctest | table | postgres | 1747 MB |
(2 rows)

pocdb1=# select max(id) from synctest;

max

35947438
(1 row)

pocdb1=#
“`

From a single master repository performance comparison

Results Analysis: Pants higher write performance of about 22 times from a master, and the TPS based single library of about 4 times from the higher rate of a single disk write library, only the above test data for reference .

Test from node goes down

Test procedure: kill off from the node, the analog unplanned downtime, the master node to write data
Will be rammed live when synchronous replication stream, it will affect the main library from node goes down, the main library for writing and updating: test results.
kill off 17 slave1, the main library insert data, will live ram.

pocdb=# insert into synctest(age,date)values(nextval(seq_synctest'),28701752,now());
pocdb'#

A master-slave

Test background: Server: 10.10.56.17 cpu: 8-core Memory: 128GB
Test: Perform 20 minutes into the database on the test server, 500 thread count 500 client connections
Test results: Write data: 38,900,000 CPU utilization: 80% of the disk write: 25Mb / s latency: 2 ms, TPS: 8288

Test Results

transaction type: bench_script_for_insert_20180717.sql
scaling factor: 1
query mode: simple
number of clients: 400
number of threads: 400
duration: 1200 s
number of transactions actually processed: 9946727
latency average = 48.258 ms
tps = 8288.726962 (including connections establishing)
tps = 8289.701749 (excluding connections establishing)
postgres@clw-db1:/pgdata/10/poc/scripts>

Insert the amount of data query

pocdb=# \d+
                           List of relations
 Schema |     Name     |   Type   |  Owner   |    Size    | Description
--------+--------------+----------+----------+------------+-------------
 public | seq_synctest | sequence | postgres | 8192 bytes |
 public | synctest     | table    | postgres | 496 MB     |
(2 rows)

pocdb=# select max(id) from synctest;
   max
----------
 38918501
(1 row)

pocdb=#

Query latency observed parameters (write_lag, flush_lag, replay_lag)

pocdb=# select client_addr,usename, application_name, write_lag, flush_lag, replay_lag,sync_state,state from pg_stat_replication;
 client_addr | usename | application_name |    write_lag    |    flush_lag    |   replay_lag    | sync_state |   state
-------------+---------+------------------+-----------------+-----------------+-----------------+------------+-----------
 10.10.56.19 | repl    | slave2           | 00:00:00.002566 | 00:00:00.002566 | 00:00:00.002566 | sync       | streaming
(1 row)

pocdb=# select client_addr,usename, application_name, write_lag, flush_lag, replay_lag,sync_state,state from pg_stat_replication;
 client_addr | usename | application_name |    write_lag    |    flush_lag    |   replay_lag    | sync_state |   state
-------------+---------+------------------+-----------------+-----------------+-----------------+------------+-----------
 10.10.56.19 | repl    | slave2           | 00:00:00.000774 | 00:00:00.000774 | 00:00:00.000774 | sync       | streaming
(1 row)

pocdb=# select client_addr,usename, application_name, write_lag, flush_lag, replay_lag,sync_state,state from pg_stat_replication;
 client_addr | usename | application_name |    write_lag    |    flush_lag    |   replay_lag    | sync_state |   state
-------------+---------+------------------+-----------------+-----------------+-----------------+------------+-----------
 10.10.56.19 | repl    | slave2           | 00:00:00.000648 | 00:00:00.000648 | 00:00:00.000648 | sync       | streaming
(1 row)

pocdb=# select client_addr,usename, application_name, write_lag, flush_lag, replay_lag,sync_state,state from pg_stat_replication;
 client_addr | usename | application_name |    write_lag    |    flush_lag    |   replay_lag    | sync_state |   state
-------------+---------+------------------+-----------------+-----------------+-----------------+------------+-----------
 10.10.56.19 | repl    | slave2           | 00:00:00.000408 | 00:00:00.000408 | 00:00:00.000408 | sync       | streaming
(1 row)

pocdb=# select client_addr,usename, application_name, write_lag, flush_lag, replay_lag,sync_state,state from pg_stat_replication;
 client_addr | usename | application_name |    write_lag    |    flush_lag    |   replay_lag    | sync_state |   state
-------------+---------+------------------+-----------------+-----------------+-----------------+------------+-----------
 10.10.56.19 | repl    | slave2           | 00:00:00.002627 | 00:00:00.002666 | 00:00:00.008609 | sync       | streaming
(1 row)

pocdb=# select client_addr,usename, application_name, write_lag, flush_lag, replay_lag,sync_state,state from pg_stat_replication;
 client_addr | usename | application_name |    write_lag    |    flush_lag    |   replay_lag    | sync_state |   state
-------------+---------+------------------+-----------------+-----------------+-----------------+------------+-----------
 10.10.56.19 | repl    | slave2           | 00:00:00.000593 | 00:00:00.000593 | 00:00:00.000593 | sync       | streaming
(1 row)

pocdb=# select client_addr,usename, application_name, write_lag, flush_lag, replay_lag,sync_state,state from pg_stat_replication;
 client_addr | usename | application_name |    write_lag    |    flush_lag    |   replay_lag    | sync_state |   state
-------------+---------+------------------+-----------------+-----------------+-----------------+------------+-----------
 10.10.56.19 | repl    | slave2           | 00:00:00.003211 | 00:00:00.009313 | 00:00:00.015688 | sync       | streaming
(1 row)

pocdb=# select client_addr,usename, application_name, write_lag, flush_lag, replay_lag,sync_state,state from pg_stat_replication;
 client_addr | usename | application_name |    write_lag    |    flush_lag    |   replay_lag    | sync_state |   state
-------------+---------+------------------+-----------------+-----------------+-----------------+------------+-----------
 10.10.56.19 | repl    | slave2           | 00:00:00.001078 | 00:00:00.001078 | 00:00:00.001078 | sync       | streaming
(1 row)

pocdb=# select client_addr,usename, application_name, write_lag, flush_lag, replay_lag,sync_state,state from pg_stat_replication;
 client_addr | usename | application_name |    write_lag    |    flush_lag    |   replay_lag    | sync_state |   state
-------------+---------+------------------+-----------------+-----------------+-----------------+------------+-----------
 10.10.56.19 | repl    | slave2           | 00:00:00.021852 | 00:00:00.034961 | 00:00:00.035026 | sync       | streaming
(1 row)

pocdb=# select client_addr,usename, application_name, write_lag, flush_lag, replay_lag,sync_state,state from pg_stat_replication;
 client_addr | usename | application_name |    write_lag    |    flush_lag    |   replay_lag   | sync_state |   state
-------------+---------+------------------+-----------------+-----------------+----------------+------------+-----------
 10.10.56.19 | repl    | slave2           | 00:00:00.000829 | 00:00:00.002209 | 00:00:00.00334 | sync       | streaming
(1 row)

pocdb=# select client_addr,usename, application_name, write_lag, flush_lag, replay_lag,sync_state,state from pg_stat_replication;
 client_addr | usename | application_name |    write_lag    |    flush_lag    |   replay_lag    | sync_state |   state
-------------+---------+------------------+-----------------+-----------------+-----------------+------------+-----------
 10.10.56.19 | repl    | slave2           | 00:00:00.054129 | 00:00:00.054134 | 00:00:00.054964 | sync       | streaming
(1 row)

Performance Comparison

Test comparison: 1主1从写the performance and 单库写the performance difference is not large, the amount of data written to about 20 minutes are 37 million 1主两从poor write performance, data only Talia 一半, delay than the 1主1从high point. The above is only the case of test data for reference only and does not reach the limit