Simulation failure and repair of standby master
This document mainly demonstrates the data synchronization of the master node's standby of the GreenPlum database after the exception and the switch of the standby after the master node fails. It is normal to check the status of Standby by the gpstate command before the operation, and the data is also in the synchronized state. At the same time, the existing master (original standby) and the original master are switched again to restore the original state.
1 Check the cluster gpstate -f
[gpadmin@mdw ~]$ gpstate -f 20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- Standby address = smdw 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- Standby data directory = /greenplum/gpdata/master/gpseg-1 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- Standby port = 5432 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- Standby PID = 7593 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- Standby status = Standby host passive 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--WAL Sender State: streaming 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Sync state: sync 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Sent Location: 0/CADFDD8 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Flush Location: 0/CADFDD8 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Replay Location: 0/CADFDD8 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------2 Simulate standby failure
mdw cat .bash_profile export PATH export MASTER_DATA_DIRECTORY=/greenplum/gpdata/master/gpseg-1 source /usr/local/greenplum-db/greenplum_path.sh export PGPORT=5432 export PGDATABASE=archdata sdw3 and smdw vi .bash_profile export MASTER_DATA_DIRECTORY=/greenplum/gpdata/master/gpseg-1 source /usr/local/greenplum-db/greenplum_path.sh export PGPORT=5432 export PGDATABASE=archdata vi .bashr export MASTER_DATA_DIRECTORY=/greenplum/gpdata/master/gpseg-1 source /usr/local/greenplum-db/greenplum_path.sh export PGPORT=5432 export PGDATABASE=archdata smdw： pg_ctl stop -D $MASTER_DATA_DIRECTORY [gpadmin@sdw3 ~]$ pg_ctl stop -D $MASTER_DATA_DIRECTORY waiting for server to shut down.... done server stopped [gpadmin@sdw3 ~]$Check the cluster status on 3 mdw
gpstate -f
[gpadmin@mdw ~]$ gpstate -f 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:- Standby address = smdw 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:- Standby data directory = /greenplum/gpdata/master/gpseg-1 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:- Standby port = 5432 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[WARNING]:-Standby PID = 0 <<<<<<<< 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[WARNING]:-Standby status = Standby process not running <<<<<<<< 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-No entries found. 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------4 Create a table on mdw to test the metadata changes of the master
psql
psql -h 10.102.254.27 -U gpadmin -d archdata [gpadmin@mdw ~]$ psql -h 10.102.254.27 -U gpadmin -d archdata psql (8.3.23) Type "help" for help. archdata=# \c You are now connected to database "archdata" as user "gpadmin". archdata=# archdata=# \d No relations found. archdata=# create table test (id int); NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'id' as the Greenplum Database data distribution key for this table. HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew. CREATE TABLE archdata=# \d List of relations Schema | Name | Type | Owner | Storage --------+------+-------+---------+--------- public | test | table | gpadmin | heap (1 row)5 Start the standby node
[gpadmin@sdw3 ~]$ pg_ctl start -D $MASTER_DATA_DIRECTORY server starting [gpadmin@sdw3 ~]$ 2020-04-19 17:08:36.469615 GMT,,,p29616,th-410839168,,,,0,,,seg-1,,,,,"FATAL","22023","dbid (from -b option) is not specified or is invalid. This value must be >= 0, or >= -1 in utility mode. The dbid value to pass can be determined from this server's entry in the segment configuration; it may be -1 if running in utility mode.",,,,,,,,"PostmasterMain","postmaster.c",1175,
6 Check the cluster status
gpstate -f [gpadmin@mdw ~]$ gpstate -f 20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:- Standby address = smdw 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:- Standby data directory = /greenplum/gpdata/master/gpseg-1 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:- Standby port = 5432 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[WARNING]:-Standby PID = 0 <<<<<<<< 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[WARNING]:-Standby status = Standby process not running <<<<<<<< 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-No entries found. 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- [gpadmin@mdw ~]$7 Try to restart the cluster
gpstop -a gpstart -a [gpadmin@mdw ~]$ gpstop -a 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -a 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-There are 0 connections to the database 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart' 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Master host=mdw 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/greenplum/gpdata/master/gpseg-1 20200420:01:15:45:012279 gpstop:mdw:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process 20200420:01:15:45:012279 gpstop:mdw:gpadmin-[INFO]:-Terminating processes for segment /greenplum/gpdata/master/gpseg-1 20200420:01:15:45:012279 gpstop:mdw:gpadmin-[INFO]:-Stopping master standby host smdw mode=fast 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[WARNING]:-Error occured while stopping the standby master: ExecutionError: 'non-zero rc: 1' occured. Details: 'ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 smdw ". /usr/local/greenplum-db/./greenplum_path.sh; $GPHOME/bin/pg_ctl -D /greenplum/gpdata/master/gpseg-1 -m fast -w -t 120 stop"' cmd had rc=1 completed=True halted=False stdout='' stderr='Warning: the ECDSA host key for 'smdw' differs from the key for the IP address '10.102.254.26' Offending key for IP in /home/gpadmin/.ssh/known_hosts:1 Matching host key in /home/gpadmin/.ssh/known_hosts:9 pg_ctl: PID file "/greenplum/gpdata/master/gpseg-1/postmaster.pid" does not exist Is server running? ' 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown standby process on smdw 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-Targeting dbid [2, 8, 3, 9, 4, 10, 5, 11, 6, 12, 7, 13] for shutdown 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait... 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed 20200420:01:15:47:012279 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed 20200420:01:15:47:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait... 20200420:01:15:47:012279 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:- Segments stopped successfully = 12 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:- Segments with errors during stop = 0 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown 12 of 12 segment instances 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Database successfully shutdown with no errors reported 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpmmon process 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-No leftover gpmmon process found 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes 20200420:01:15:50:012279 gpstop:mdw:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts 20200420:01:15:50:012279 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover shared memory [gpadmin@mdw ~]$ gpstart -a 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -a 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301705051' 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Setting new master era 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Master Started... 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Shutting down master 20200420:01:16:08:012440 gpstart:mdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... .. 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-Process results... 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:- Successful segment starts = 12 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:- Failed segment starts = 0 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:- Skipped segment starts (segments are marked down in configuration) = 0 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-Successfully started 12 of 12 segment instances 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance mdw directory /greenplum/gpdata/master/gpseg-1 20200420:01:16:11:012440 gpstart:mdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active 20200420:01:16:11:012440 gpstart:mdw:gpadmin-[INFO]:-Starting standby master 20200420:01:16:11:012440 gpstart:mdw:gpadmin-[INFO]:-Checking if standby master is running on host: smdw in directory: /greenplum/gpdata/master/gpseg-1 20200420:01:16:12:012440 gpstart:mdw:gpadmin-[WARNING]:-Unable to cleanup previously started standby: 'Warning: the ECDSA host key for 'smdw' differs from the key for the IP address '10.102.254.26' Offending key for IP in /home/gpadmin/.ssh/known_hosts:1 Matching host key in /home/gpadmin/.ssh/known_hosts:9 ' 20200420:01:16:13:012440 gpstart:mdw:gpadmin-[INFO]:-Database successfully started8 Confirm the status again
gpstat -f [gpadmin@mdw ~]$ gpstate -f 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- Standby address = smdw 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- Standby data directory = /greenplum/gpdata/master/gpseg-1 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- Standby port = 5432 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- Standby PID = 30408 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- Standby status = Standby host passive 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--WAL Sender State: streaming 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Sync state: sync 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Sent Location: 0/CB21DB0 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Flush Location: 0/CB21DB0 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Replay Location: 0/CB21DB0 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 状态正常9 Processing if data cannot be synchronized
If the Standby data cannot be automatically synchronized, you need to re-initialize the Standby data, which is to manually synchronize the data in full, because the Master only stores the metadata of the database, which is usually small. This process can be completed quickly, but the Master database Boot to Utility mode, otherwise you will encounter the following error.
[gpadmin@mdw ~]$ gpinitstandby -n 20200420:01:20:23:012710 gpinitstandby:mdw:gpadmin-[INFO]:-Standy master is already up and running. 使用gpstart –m只将Master节点启动到Utility模式。 [gpadmin@mdw ~]$ gpstop -a [gpadmin@mdw ~]$ gpstop -a 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -a 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-There are 0 connections to the database 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart' 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Master host=mdw 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/greenplum/gpdata/master/gpseg-1 20200420:01:21:47:012747 gpstop:mdw:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process 20200420:01:21:47:012747 gpstop:mdw:gpadmin-[INFO]:-Terminating processes for segment /greenplum/gpdata/master/gpseg-1 20200420:01:21:47:012747 gpstop:mdw:gpadmin-[INFO]:-Stopping master standby host smdw mode=fast 20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown standby process on smdw 20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-Targeting dbid [2, 8, 3, 9, 4, 10, 5, 11, 6, 12, 7, 13] for shutdown 20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait... 20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed 20200420:01:21:50:012747 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed 20200420:01:21:50:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait... 20200420:01:21:50:012747 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:- Segments stopped successfully = 12 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:- Segments with errors during stop = 0 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown 12 of 12 segment instances 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Database successfully shutdown with no errors reported 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpmmon process 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-No leftover gpmmon process found 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover shared memory [gpadmin@mdw ~]$ gpstart -m [gpadmin@mdw ~]$ gpstart -m 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -m 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301705051' 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-**************************************************************************** 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-Master-only start requested in a configuration with a standby master. 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-This is advisable only under the direct supervision of Greenplum support. 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-This mode of operation is not supported in a production environment and 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-may lead to a split-brain condition and possible unrecoverable data loss. 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-**************************************************************************** Continue with master-only startup Yy|Nn (default=N): > y > 20200420:01:22:07:012908 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode > 20200420:01:22:08:012908 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information > 20200420:01:22:08:012908 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... > 20200420:01:22:09:012908 gpstart:mdw:gpadmin-[INFO]:-Setting new master era > 20200420:01:22:09:012908 gpstart:mdw:gpadmin-[INFO]:-Master Started... > 此处需要输入Y确认。 将Master启动到Utility模式后，使用gpinitstandby来重新初始化Standby的数据。 [gpadmin@mdw ~]$ gpinitstandby -n [gpadmin@mdw ~]$ gpinitstandby -n 20200420:01:22:35:012938 gpinitstandby:mdw:gpadmin-[INFO]:-Starting standby master 20200420:01:22:35:012938 gpinitstandby:mdw:gpadmin-[INFO]:-Checking if standby master is running on host: smdw in directory: /greenplum/gpdata/master/gpseg-1 20200420:01:22:36:012938 gpinitstandby:mdw:gpadmin-[WARNING]:-Unable to cleanup previously started standby: 'Warning: the ECDSA host key for 'smdw' differs from the key for the IP address '10.102.254.26' Offending key for IP in /home/gpadmin/.ssh/known_hosts:1 Matching host key in /home/gpadmin/.ssh/known_hosts:9 ' 20200420:01:22:37:012938 gpinitstandby:mdw:gpadmin-[INFO]:-Successfully started standby master 重启数据库，Standby的状态就会正常。 [gpadmin@mdw ~]$ gpstop -a [gpadmin@mdw ~]$ gpstart -a [gpadmin@mdw ~]$ gpstate -f ' 20200420:01:23:39:013123 gpstart:mdw:gpadmin-[INFO]:-Database successfully started [gpadmin@mdw ~]$ gpstate -f 20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- Standby address = smdw 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- Standby data directory = /greenplum/gpdata/master/gpseg-1 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- Standby port = 5432 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- Standby PID = 31874 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- Standby status = Standby host passive 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--WAL Sender State: streaming 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Sync state: sync 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Sent Location: 0/CB22200 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Flush Location: 0/CB22200 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Replay Location: 0/CB22010 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
