The GBase 8a database cluster supports expansion operations from existing clusters, and can simultaneously expand multiple data nodes and management nodes. Expansion is generally caused by space occupation or performance issues. GBase uses the redistribution function to redistribute data to all data nodes to rebalance the data space. This article is a detailed operation steps and manual.
Original: http://www.gbase8.cn/1133
Expansion and shrinkage are two completely opposite results, but the general operation steps are basically similar, but opposite. Please refer to the scaling steps. GBase 8a shrinking operation step example
aims
The goal of this article is to expand a 2-node cluster to 3 nodes, that is, to add 1 management and data compliance node. The status quo is as follows:
2 node cluster
For some precautions of V95 version, please refer to Precautions for new installation of GBase 8a cluster V95
Edit the expansion operation configuration file
The decompression steps of the subpackage will not be introduced. Let's go directly to the gcinstall directory of the installation package and write demo.options. Fill in the user information according to the actual situation.
Note that this article uses the V95 version, its format is different from V8, please refer to the relevant introduction of the installation manual. But the important part is the same. among them
installPrefix is the cluster installation directory, please be consistent with the original node.
coordinatorHost is the IP of the node to be expanded. Separate the
coordinateHostNodeID with a comma when there are more than one. This is only useful for IPV6, ipv4 can be written or not, and is ignored internally.
dataHost The data node IP for this expansion
existCoordinateHost The existing management node IP, multiple comma-delimited
data node IP existDataHost already exists, a plurality of comma separated
dba dbaUser operating system user name, which is v95 in the user-created
dba dbaGroup operating system user name user group
dba operating system user password dbaPwd
The root user password of the rootPwd operating system is not required in V95.
genDBPwd This is the name of V95, which represents the password of the database user root. Please distinguish between database users and operating system users.
[gbase@localhost gcinstall]$ cat demo.options
installPrefix= /opt/gbase
coordinateHost = 10.0.2.203
coordinateHostNodeID = 203
dataHost = 10.0.2.203
existCoordinateHost =10.0.2.102,10.0.2.202
existDataHost =10.0.2.102,10.0.2.202
dbaUser = gbase
dbaGroup = gbase
dbaPwd = 'gbase1234'
rootPwd = '111111'
#rootPwdFile = rootPwd.json
genDBPwd=''
The following is what a V8 version of demo.options looks like. Please try to use root user for loginUser. dbRootPwd is the password of the root user of the database.
[root@localhost gcinstall]# cat demo.options
installPrefix= /opt
coordinateHost = 10.0.2.106
coordinateHostNodeID = 106
dataHost = 10.0.2.106
#existCoordinateHost =
#existDataHost =
loginUser= root
loginUserPwd = '111111'
#loginUserPwdFile = loginUserPwd.json
dbaUser = gbase
dbaGroup = gbase
dbaPwd = 'gbase1234'
rootPwd = '111111'
#rootPwdFile = rootPwd.json
dbRootPwd = ''
#mcastAddr = 226.94.1.39
mcastPort = 5493
[root@localhost gcinstall]#
Stop database service
If the expansion service has management nodes, the database services of all nodes need to be stopped. If you only expand the data node, you can skip this step.
If the service is not stopped and the management node is expanded, the expansion installation step will report this error. Must stop all gcluster nodes before extend gcluster. you can search'still running' in gcinstall.log to find them.
To stop the service, please operate at each node by yourself. This article uses ssh remote operation.
[gbase@localhost gcinstall]$ ssh 10.0.2.102 "gcluster_services all stop"
The authenticity of host '10.0.2.102 (10.0.2.102)' can't be established.
ECDSA key fingerprint is d1:17:f6:1e:3b:97:e4:db:37:d5:9b:cb:00:14:66:b6.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.2.102' (ECDSA) to the list of known hosts.
gbase
[email protected]'s password:
Stopping GCMonit success!
Stopping gcrecover : [ OK ]
Stopping gcluster : [ OK ]
Stopping gcware : [ OK ]
Stopping gbase : [ OK ]
Stopping syncserver : [ OK ]
[gbase@localhost gcinstall]$ ssh 10.0.2.202 "gcluster_services all stop"
[email protected]'s password:
Stopping GCMonit success!
Stopping gcrecover : [ OK ]
Stopping gcluster : [ OK ]
Stopping gcware : [ OK ]
Stopping gbase : [ OK ]
Stopping syncserver : [ OK ]
[gbase@localhost gcinstall]$
Start installation service
Call gcinstall.py to start the installation with the demo.options written earlier.
Please note that if the management node is expanded, you need to copy all the [metadata] of the existing management node to the new management node. The time-consuming depends on the number of tables, disk and network performance. It may take several minutes to several hours.
[gbase@localhost gcinstall]$ ./gcinstall.py --silent=demo.options
*********************************************************************************
Thank you for choosing GBase product!
此处省略许可信息输出...
*********************************************************************************
Do you accept the above licence agreement ([Y,y]/[N,n])? y
*********************************************************************************
Welcome to install GBase products
*********************************************************************************
Environmental Checking on gcluster nodes.
CoordinateHost:
10.0.2.203
DataHost:
10.0.2.203
Are you sure to install GCluster on these nodes ([Y,y]/[N,n])? y
10.0.2.203 start install on host 10.0.2.203
10.0.2.202 start install on host 10.0.2.202
10.0.2.102 start install on host 10.0.2.102
此处省略n多中间日志输出
10.0.2.203 install cluster on host 10.0.2.203 successfully.
10.0.2.202 install cluster on host 10.0.2.202 successfully.
10.0.2.102 install cluster on host 10.0.2.102 successfully.
update and sync configuration file...
Starting all gcluster nodes...
sync coordinator system tables...
adding new datanodes to gcware...
[gbase@localhost gcinstall]$View current status
The new management node has provided services normally, and the data node has not yet participated.
[gbase@localhost gcinstall]$ gcadmin
CLUSTER STATE: ACTIVE
VIRTUAL CLUSTER MODE: NORMAL
=============================================================
| GBASE COORDINATOR CLUSTER INFORMATION |
=============================================================
| NodeName | IpAddress | gcware | gcluster | DataState |
-------------------------------------------------------------
| coordinator1 | 10.0.2.102 | OPEN | OPEN | 0 |
-------------------------------------------------------------
| coordinator2 | 10.0.2.202 | OPEN | OPEN | 0 |
-------------------------------------------------------------
| coordinator3 | 10.0.2.203 | OPEN | OPEN | 0 |
-------------------------------------------------------------
=========================================================================================================
| GBASE DATA CLUSTER INFORMATION |
=========================================================================================================
| NodeName | IpAddress | DistributionId | gnode | syncserver | DataState |
---------------------------------------------------------------------------------------------------------
| node1 | 10.0.2.102 | 5 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node2 | 10.0.2.202 | 5 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node3 | 10.0.2.203 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
Edit the configuration file of the new distribution strategy
Please pay attention to the IP order, try to be the same as the previous order, and put the newly added data node IP at the end.
[gbase@localhost gcinstall]$ cat gcChangeInfo.xml
<?xml version="1.0" encoding="utf-8"?>
<servers>
<rack>
<node ip="10.0.2.102"/>
<node ip="10.0.2.202"/>
<node ip="10.0.2.203"/>
</rack>
</servers>
[gbase@localhost gcinstall]$
Add new distribution strategy
In this step, the metadata of some data nodes will be synchronized, so it will take a little time.
Check the cluster status, and you have seen that a new policy with a distribution Id of 6 has been added, and the old ID is 5. This ID will be used later.
[gbase@localhost gcinstall]$ gcadmin distribution gcChangeInfo.xml p 1 d 1
gcadmin generate distribution ...
NOTE: node [10.0.2.203] is coordinator node, it shall be data node too
copy system table to 10.0.2.203
gcadmin generate distribution successful
[gbase@localhost gcinstall]$ gcadmin
CLUSTER STATE: ACTIVE
VIRTUAL CLUSTER MODE: NORMAL
=============================================================
| GBASE COORDINATOR CLUSTER INFORMATION |
=============================================================
| NodeName | IpAddress | gcware | gcluster | DataState |
-------------------------------------------------------------
| coordinator1 | 10.0.2.102 | OPEN | OPEN | 0 |
-------------------------------------------------------------
| coordinator2 | 10.0.2.202 | OPEN | OPEN | 0 |
-------------------------------------------------------------
| coordinator3 | 10.0.2.203 | OPEN | OPEN | 0 |
-------------------------------------------------------------
=========================================================================================================
| GBASE DATA CLUSTER INFORMATION |
=========================================================================================================
| NodeName | IpAddress | DistributionId | gnode | syncserver | DataState |
---------------------------------------------------------------------------------------------------------
| node1 | 10.0.2.102 | 5,6 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node2 | 10.0.2.202 | 5,6 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node3 | 10.0.2.203 | 6 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
[gbase@localhost gcinstall]$ gcadmin showdistribution
Distribution ID: 6 | State: new | Total segment num: 3
Primary Segment Node IP Segment ID Duplicate Segment node IP
========================================================================================================================
| 10.0.2.102 | 1 | 10.0.2.202 |
------------------------------------------------------------------------------------------------------------------------
| 10.0.2.202 | 2 | 10.0.2.203 |
------------------------------------------------------------------------------------------------------------------------
| 10.0.2.203 | 3 | 10.0.2.102 |
========================================================================================================================
Distribution ID: 5 | State: old | Total segment num: 2
Primary Segment Node IP Segment ID Duplicate Segment node IP
========================================================================================================================
| 10.0.2.102 | 1 | 10.0.2.202 |
------------------------------------------------------------------------------------------------------------------------
| 10.0.2.202 | 2 | 10.0.2.102 |
========================================================================================================================
[gbase@localhost gcinstall]$
Initialize nodedatamap
[gbase@localhost gcinstall]$ gccli
GBase client 9.5.2.17.115980. Copyright (c) 2004-2020, GBase. All Rights Reserved.
gbase> initnodedatamap;
Query OK, 0 rows affected, 5 warnings (Elapsed: 00:00:01.36)
gbase>
Redistribution
This step will redistribute the data to all nodes. The actual time may be minutes, hours, or days. The amount of data redistributed as needed is related to hardware performance.
Note: The database provides a parameter gcluster_rebalancing_random_table_quick_mode. For randomly distributed tables, it will no longer participate in data rebalancing. If you expand for disk space, please turn off this parameter.
gbase> show variables like '%quick%';
+----------------------------------------------+-------+
| Variable_name | Value |
+----------------------------------------------+-------+
| gcluster_rebalancing_random_table_quick_mode | 1 |
+----------------------------------------------+-------+
1 row in set (Elapsed: 00:00:00.00)
gbase> set global gcluster_rebalancing_random_table_quick_mode=0;
Query OK, 0 rows affected (Elapsed: 00:00:00.01)
gbase> show variables like '%quick%';
+----------------------------------------------+-------+
| Variable_name | Value |
+----------------------------------------------+-------+
| gcluster_rebalancing_random_table_quick_mode | 0 |
+----------------------------------------------+-------+
1 row in set (Elapsed: 00:00:00.01)
During the redistribution, you can check the progress through the gclusterdb.rebalancing_status table until all of them are COMPLETED.
gbase> rebalance instance;
Query OK, 21 rows affected (Elapsed: 00:00:00.87)
gbase> select status,count(*) from gclusterdb.rebalancing_status group by status;
+-----------+----------+
| status | count(*) |
+-----------+----------+
| STARTING | 15 |
| RUNNING | 3 |
| COMPLETED | 3 |
+-----------+----------+
3 rows in set (Elapsed: 00:00:00.61)
gbase> select status,count(*) from gclusterdb.rebalancing_status group by status;
+-----------+----------+
| status | count(*) |
+-----------+----------+
| COMPLETED | 21 |
+-----------+----------+
1 row in set (Elapsed: 00:00:00.19)
Check for residue
Check the gbase.table_distribution table to see if there are any old tables with ID 5 left. If so, manually execute a rebalance table library name. Table name, and wait for the redistribution to be completed.
gbase> select count(*) from gbase.table_distribution where data_distribution_id=5;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (Elapsed: 00:00:00.00)
Delete the old nodedatamap
gbase> refreshnodedatamap drop 5;
Query OK, 0 rows affected, 6 warnings (Elapsed: 00:00:01.80)
gbase>
Delete the old distribution strategy
[gbase@localhost gcinstall]$ gcadmin rmdistribution 5
cluster distribution ID [5]
it will be removed now
please ensure this is ok, input [Y,y] or [N,n]: y
gcadmin remove distribution [5] success
[gbase@localhost gcinstall]$ gcadmin showdistribution
Distribution ID: 6 | State: new | Total segment num: 3
Primary Segment Node IP Segment ID Duplicate Segment node IP
========================================================================================================================
| 10.0.2.102 | 1 | 10.0.2.202 |
------------------------------------------------------------------------------------------------------------------------
| 10.0.2.202 | 2 | 10.0.2.203 |
------------------------------------------------------------------------------------------------------------------------
| 10.0.2.203 | 3 | 10.0.2.102 |
========================================================================================================================
[gbase@localhost gcinstall]$
Final Results
It has successfully expanded from a 2-node cluster to a 3-node cluster.
to sum up
The core of the whole step is
1. Install a new node service
2. Add new distribution strategies to utilize new data nodes
3. Redistribute data
4. Delete old information
Accidental handling
During expansion, if there are accidents in certain steps, such as error, downtime, power outage, network disconnection, etc., please refer to