standby master的模拟故障与修复
本文档主要演示GreenPlum数据库的Master节点的Standby在异常后的数据同步及Master节点故障后，Standby的切换。在操作前通过gpstate命令查看Standby的状态是正常的，数据也在同步状态。同时将现有master（原有的standy）和原有的master再做切换，恢复原状。
1 检查集群gpstate -f
[gpadmin@mdw ~]$ gpstate -f 20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- Standby address = smdw 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- Standby data directory = /greenplum/gpdata/master/gpseg-1 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- Standby port = 5432 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- Standby PID = 7593 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- Standby status = Standby host passive 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--WAL Sender State: streaming 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Sync state: sync 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Sent Location: 0/CADFDD8 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Flush Location: 0/CADFDD8 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Replay Location: 0/CADFDD8 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------2 模拟standby 故障
mdw cat .bash_profile export PATH export MASTER_DATA_DIRECTORY=/greenplum/gpdata/master/gpseg-1 source /usr/local/greenplum-db/greenplum_path.sh export PGPORT=5432 export PGDATABASE=archdata sdw3 and smdw vi .bash_profile export MASTER_DATA_DIRECTORY=/greenplum/gpdata/master/gpseg-1 source /usr/local/greenplum-db/greenplum_path.sh export PGPORT=5432 export PGDATABASE=archdata vi .bashr export MASTER_DATA_DIRECTORY=/greenplum/gpdata/master/gpseg-1 source /usr/local/greenplum-db/greenplum_path.sh export PGPORT=5432 export PGDATABASE=archdata smdw： pg_ctl stop -D $MASTER_DATA_DIRECTORY [gpadmin@sdw3 ~]$ pg_ctl stop -D $MASTER_DATA_DIRECTORY waiting for server to shut down.... done server stopped [gpadmin@sdw3 ~]$3 mdw上检查集群状态
gpstate -f
[gpadmin@mdw ~]$ gpstate -f 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:- Standby address = smdw 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:- Standby data directory = /greenplum/gpdata/master/gpseg-1 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:- Standby port = 5432 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[WARNING]:-Standby PID = 0 <<<<<<<< 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[WARNING]:-Standby status = Standby process not running <<<<<<<< 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-No entries found. 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------4 mdw上创建一个表，测试master的元数据变化
psql
psql -h 10.102.254.27 -U gpadmin -d archdata [gpadmin@mdw ~]$ psql -h 10.102.254.27 -U gpadmin -d archdata psql (8.3.23) Type "help" for help. archdata=# \c You are now connected to database "archdata" as user "gpadmin". archdata=# archdata=# \d No relations found. archdata=# create table test (id int); NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'id' as the Greenplum Database data distribution key for this table. HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew. CREATE TABLE archdata=# \d List of relations Schema | Name | Type | Owner | Storage --------+------+-------+---------+--------- public | test | table | gpadmin | heap (1 row)5 启动standby节点
[gpadmin@sdw3 ~]$ pg_ctl start -D $MASTER_DATA_DIRECTORY server starting [gpadmin@sdw3 ~]$ 2020-04-19 17:08:36.469615 GMT,,,p29616,th-410839168,,,,0,,,seg-1,,,,,"FATAL","22023","dbid (from -b option) is not specified or is invalid. This value must be >= 0, or >= -1 in utility mode. The dbid value to pass can be determined from this server's entry in the segment configuration; it may be -1 if running in utility mode.",,,,,,,,"PostmasterMain","postmaster.c",1175,6 检查集群状态
gpstate -f [gpadmin@mdw ~]$ gpstate -f 20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:- Standby address = smdw 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:- Standby data directory = /greenplum/gpdata/master/gpseg-1 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:- Standby port = 5432 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[WARNING]:-Standby PID = 0 <<<<<<<< 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[WARNING]:-Standby status = Standby process not running <<<<<<<< 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-No entries found. 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- [gpadmin@mdw ~]$7 尝试重启集群
gpstop -a gpstart -a [gpadmin@mdw ~]$ gpstop -a 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -a 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-There are 0 connections to the database 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart' 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Master host=mdw 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/greenplum/gpdata/master/gpseg-1 20200420:01:15:45:012279 gpstop:mdw:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process 20200420:01:15:45:012279 gpstop:mdw:gpadmin-[INFO]:-Terminating processes for segment /greenplum/gpdata/master/gpseg-1 20200420:01:15:45:012279 gpstop:mdw:gpadmin-[INFO]:-Stopping master standby host smdw mode=fast 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[WARNING]:-Error occured while stopping the standby master: ExecutionError: 'non-zero rc: 1' occured. Details: 'ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 smdw ". /usr/local/greenplum-db/./greenplum_path.sh; $GPHOME/bin/pg_ctl -D /greenplum/gpdata/master/gpseg-1 -m fast -w -t 120 stop"' cmd had rc=1 completed=True halted=False stdout='' stderr='Warning: the ECDSA host key for 'smdw' differs from the key for the IP address '10.102.254.26' Offending key for IP in /home/gpadmin/.ssh/known_hosts:1 Matching host key in /home/gpadmin/.ssh/known_hosts:9 pg_ctl: PID file "/greenplum/gpdata/master/gpseg-1/postmaster.pid" does not exist Is server running? ' 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown standby process on smdw 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-Targeting dbid [2, 8, 3, 9, 4, 10, 5, 11, 6, 12, 7, 13] for shutdown 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait... 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed 20200420:01:15:47:012279 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed 20200420:01:15:47:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait... 20200420:01:15:47:012279 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:- Segments stopped successfully = 12 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:- Segments with errors during stop = 0 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown 12 of 12 segment instances 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Database successfully shutdown with no errors reported 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpmmon process 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-No leftover gpmmon process found 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes 20200420:01:15:50:012279 gpstop:mdw:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts 20200420:01:15:50:012279 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover shared memory [gpadmin@mdw ~]$ gpstart -a 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -a 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301705051' 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Setting new master era 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Master Started... 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Shutting down master 20200420:01:16:08:012440 gpstart:mdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... .. 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-Process results... 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:- Successful segment starts = 12 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:- Failed segment starts = 0 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:- Skipped segment starts (segments are marked down in configuration) = 0 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-Successfully started 12 of 12 segment instances 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance mdw directory /greenplum/gpdata/master/gpseg-1 20200420:01:16:11:012440 gpstart:mdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active 20200420:01:16:11:012440 gpstart:mdw:gpadmin-[INFO]:-Starting standby master 20200420:01:16:11:012440 gpstart:mdw:gpadmin-[INFO]:-Checking if standby master is running on host: smdw in directory: /greenplum/gpdata/master/gpseg-1 20200420:01:16:12:012440 gpstart:mdw:gpadmin-[WARNING]:-Unable to cleanup previously started standby: 'Warning: the ECDSA host key for 'smdw' differs from the key for the IP address '10.102.254.26' Offending key for IP in /home/gpadmin/.ssh/known_hosts:1 Matching host key in /home/gpadmin/.ssh/known_hosts:9 ' 20200420:01:16:13:012440 gpstart:mdw:gpadmin-[INFO]:-Database successfully started8 再次确认状态
gpstat -f [gpadmin@mdw ~]$ gpstate -f 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- Standby address = smdw 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- Standby data directory = /greenplum/gpdata/master/gpseg-1 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- Standby port = 5432 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- Standby PID = 30408 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- Standby status = Standby host passive 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--WAL Sender State: streaming 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Sync state: sync 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Sent Location: 0/CB21DB0 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Flush Location: 0/CB21DB0 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Replay Location: 0/CB21DB0 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 状态正常9 如果不能同步数据的处理
如果Standby数据不能自动同步，那么就需要重新初始化Standby的数据，也就是手动全量同步数据，因为Master只存数据库的元数据，通常都很小，这个过程很快即可完成，但是需要将Master数据库启动到Utility模式，否则会遇到下面的错误。
[gpadmin@mdw ~]$ gpinitstandby -n 20200420:01:20:23:012710 gpinitstandby:mdw:gpadmin-[INFO]:-Standy master is already up and running. 使用gpstart –m只将Master节点启动到Utility模式。 [gpadmin@mdw ~]$ gpstop -a [gpadmin@mdw ~]$ gpstop -a 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -a 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-There are 0 connections to the database 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart' 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Master host=mdw 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/greenplum/gpdata/master/gpseg-1 20200420:01:21:47:012747 gpstop:mdw:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process 20200420:01:21:47:012747 gpstop:mdw:gpadmin-[INFO]:-Terminating processes for segment /greenplum/gpdata/master/gpseg-1 20200420:01:21:47:012747 gpstop:mdw:gpadmin-[INFO]:-Stopping master standby host smdw mode=fast 20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown standby process on smdw 20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-Targeting dbid [2, 8, 3, 9, 4, 10, 5, 11, 6, 12, 7, 13] for shutdown 20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait... 20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed 20200420:01:21:50:012747 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed 20200420:01:21:50:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait... 20200420:01:21:50:012747 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:- Segments stopped successfully = 12 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:- Segments with errors during stop = 0 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown 12 of 12 segment instances 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Database successfully shutdown with no errors reported 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpmmon process 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-No leftover gpmmon process found 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover shared memory [gpadmin@mdw ~]$ gpstart -m [gpadmin@mdw ~]$ gpstart -m 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -m 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301705051' 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-**************************************************************************** 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-Master-only start requested in a configuration with a standby master. 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-This is advisable only under the direct supervision of Greenplum support. 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-This mode of operation is not supported in a production environment and 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-may lead to a split-brain condition and possible unrecoverable data loss. 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-**************************************************************************** Continue with master-only startup Yy|Nn (default=N): > y > 20200420:01:22:07:012908 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode > 20200420:01:22:08:012908 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information > 20200420:01:22:08:012908 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... > 20200420:01:22:09:012908 gpstart:mdw:gpadmin-[INFO]:-Setting new master era > 20200420:01:22:09:012908 gpstart:mdw:gpadmin-[INFO]:-Master Started... > 此处需要输入Y确认。 将Master启动到Utility模式后，使用gpinitstandby来重新初始化Standby的数据。 [gpadmin@mdw ~]$ gpinitstandby -n [gpadmin@mdw ~]$ gpinitstandby -n 20200420:01:22:35:012938 gpinitstandby:mdw:gpadmin-[INFO]:-Starting standby master 20200420:01:22:35:012938 gpinitstandby:mdw:gpadmin-[INFO]:-Checking if standby master is running on host: smdw in directory: /greenplum/gpdata/master/gpseg-1 20200420:01:22:36:012938 gpinitstandby:mdw:gpadmin-[WARNING]:-Unable to cleanup previously started standby: 'Warning: the ECDSA host key for 'smdw' differs from the key for the IP address '10.102.254.26' Offending key for IP in /home/gpadmin/.ssh/known_hosts:1 Matching host key in /home/gpadmin/.ssh/known_hosts:9 ' 20200420:01:22:37:012938 gpinitstandby:mdw:gpadmin-[INFO]:-Successfully started standby master 重启数据库，Standby的状态就会正常。 [gpadmin@mdw ~]$ gpstop -a [gpadmin@mdw ~]$ gpstart -a [gpadmin@mdw ~]$ gpstate -f ' 20200420:01:23:39:013123 gpstart:mdw:gpadmin-[INFO]:-Database successfully started [gpadmin@mdw ~]$ gpstate -f 20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- Standby address = smdw 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- Standby data directory = /greenplum/gpdata/master/gpseg-1 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- Standby port = 5432 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- Standby PID = 31874 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- Standby status = Standby host passive 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--WAL Sender State: streaming 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Sync state: sync 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Sent Location: 0/CB22200 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Flush Location: 0/CB22200 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Replay Location: 0/CB22010 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
