Introduction to MGR
MGR is the abbreviation of mysql Group Replication, and the Chinese name is Mysql group replication. MGR is a brand-new high-availability and high-scalability solution officially launched by MySQL in December 2016, providing a highly available, highly scalable, and highly reliable MySQL cluster Service, currently only supports MYSQL5.7 and mysql8.0 versions.
MGR advantages
1. High consistency: Group replication technology based on native replication and paxos protocol.
2. High fault tolerance: There is an automatic detection mechanism, and when there is a downtime, the problem node will be automatically removed. In a 2N+1 node cluster, as long as the cluster is still alive, the database can be stably provided to the outside world service.
3. High scalability: nodes can be added and removed online.
4. High flexibility: you can switch freely between single-master mode and multi-master mode.
Disadvantages of MGR
MGR technology is relatively new, and its stability needs to be verified by the industry.
Mysql8 Distributed Database MGR Best Practice
The environment prepares
3 virtual machines to build a 3-node MGR cluster, version 8.0.21 for Mysql database.
Create a directory
mkdir -p /data/mysql/mysql8/{data,log,conf,run}/3309
mkdir -p /data/mysql/mysql8/log/3309/{redo,undo,relay}
chown -R mysql:mysql /data/mysql/mysql8/mysql8
chmod 750 /data/mysql/mysql8/{data,log,conf,run}
Configuration parameter
[mysqld]
##basic settings###
server-id=1
port = 3309
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
skip-name-resolve
max_connections=1000
max_user_connections=800
max_allowed_packet=512M
max_connect_errors=100000
datadir = /data/mysql/mysql8/data/3309
socket = /data/mysql/mysql8/run/3309/mysql.sock
pid_file = /data/mysql/mysql8/run/3309/mysql.pid
transaction_isolation = READ-COMMITTED
lower_case_table_names=1
default_time_zone =+8:00
open_files_limit=65535
log_timestamps=system
wait_timeout=900
interactive_timeout=900
##innodb setting##
innodb_buffer_pool_size = 256M
innodb_buffer_pool_instances = 1
innodb_io_capacity=2000
innodb_flush_method=O_DIRECT
innodb_log_group_home_dir = /data/mysql/mysql8/log/3309/redo
innodb_log_file_size = 128M
innodb_log_files_in_group=4
innodb_log_buffer_size = 32M
innodb_undo_directory = /data/mysql/mysql8/log/3309/undo
innodb_undo_tablespaces = 4
innodb_undo_log_truncate=1
innodb_max_undo_log_size=1G
innodb_flush_neighbors=0
innodb_flush_log_at_trx_commit = 1
innodb_print_all_deadlocks = 1
innodb_online_alter_log_max_size=128M
innodb_lock_wait_timeout=10
innodb_file_per_table=ON
innodb_doublewrite=ON
##log settings##
log-error = /data/mysql/mysql8/log/3309/error.log
log-bin = /data/mysql/mysql8/log/3309/mysql_bin.log
slow_query_log = 1
slow_query_log_file = /data/mysql/mysql8/log/3309/mysql_slow_query.log
long_query_time = 10
##replication settings##
gtid-mode=on
enforce-gtid-consistency=true
master-info-repository=TABLE
relay-log-info-repository=TABLE
sync-master-info=1000
relay_log_recovery = 1
relay-log=/data/mysql/mysql8/log/3309/relay/mysql-relay-bin
#binlog
log_bin=/data/mysql/mysql8/log/3309/binlog
expire_logs_days=10
max_binlog_cache_size=1024M
sync_binlog=1
##MGR settings
binlog_checksum = NONE
log_slave_updates = ON
binlog_format=row
#transaction_write_set_extraction ='XXHASH64'
#loose-group_replication_group_name = '38f34157-cbe8-4623-a7bd-054cc5c2de0b'
#loose-group_replication_start_on_boot = off
#loose-group_replication_local_address = '192.168.112.131:10061'
#loose-group_replication_group_seeds ='192.168.112.131:10061,192.168.112.132:10061,192.168.112.135:10061'
#loose-group_replication_bootstrap_group = off
#loose-group_replication_ip_whitelist = '192.168.112.131/24,192.168.112.132/24,192.168.112.135/24'
[client]
port = 3309
socket = /data/mysql/mysql8/run/3309/mysql.sock
What needs to be noted here is that because of the MGR plug-in, mysql is not installed by default, so the configuration parameters of MGR are commented out here. After the database instance is started, start it again.
The server-id of each node needs to be set differently.
Initialize the database
/data/mysql-8.0.21/bin/mysqld --defaults-file=/data/mysql/mysql8/conf/3309/my.cnf --initialize --basedir=/data/mysql-8.0.21/ --datadir=/data/mysql/mysql8/data/3309 --user=mysql --initialize-insecure --ssl --explicit_defaults_for_timestamp
如果要启用SSL安全连接，执行如下命令
/data/mysql-8.0.21/bin/mysql_ssl_rsa_setup --basedir=/data/mysql-8.0.21 --datadir=/data/mysql/mysql8/data/3309
Start and stop the mysql instance
/data/mysql-8.0.21/bin/mysqld_safe --defaults-file=/data/mysql/mysql8/conf/3309/my.cnf &
/data/mysql-8.0.21/bin/mysqladmin -uroot --socket=/data/mysql/mysql8/run/3309/mysql.sock shutdown &
Login instance
/data/mysql-8.0.21/bin/mysql -uroot --socket=/data/mysql/mysql8/run/3309/mysql.sock
MGR plugin installation
[root@localhost] 14:08:44 [(none)]>install plugin group_replication soname 'group_replication.so';
Query OK, 0 rows affected (0.20 sec)
[root@localhost] 14:09:06 [(none)]>show plugins;
+---------------------------------+----------+--------------------+----------------------+---------+
| Name | Status | Type | Library | License |
+---------------------------------+----------+--------------------+----------------------+---------+
| binlog | ACTIVE | STORAGE ENGINE | NULL | GPL |
| mysql_native_password | ACTIVE | AUTHENTICATION | NULL | GPL |
| sha256_password | ACTIVE | AUTHENTICATION | NULL | GPL |
| caching_sha2_password | ACTIVE | AUTHENTICATION | NULL | GPL |
| sha2_cache_cleaner | ACTIVE | AUDIT | NULL | GPL |
| CSV | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MEMORY | ACTIVE | STORAGE ENGINE | NULL | GPL |
| InnoDB | ACTIVE | STORAGE ENGINE | NULL | GPL |
| INNODB_TRX | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMP | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMP_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMPMEM | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMPMEM_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMP_PER_INDEX | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CMP_PER_INDEX_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_BUFFER_PAGE | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_BUFFER_PAGE_LRU | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_BUFFER_POOL_STATS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_TEMP_TABLE_INFO | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_METRICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_FT_DEFAULT_STOPWORD | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_FT_DELETED | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_FT_BEING_DELETED | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_FT_CONFIG | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_FT_INDEX_CACHE | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_FT_INDEX_TABLE | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_TABLES | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_TABLESTATS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_INDEXES | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_TABLESPACES | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_COLUMNS | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_VIRTUAL | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_CACHED_INDEXES | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| INNODB_SESSION_TEMP_TABLESPACES | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
| MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MRG_MYISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| PERFORMANCE_SCHEMA | ACTIVE | STORAGE ENGINE | NULL | GPL |
| TempTable | ACTIVE | STORAGE ENGINE | NULL | GPL |
| ARCHIVE | ACTIVE | STORAGE ENGINE | NULL | GPL |
| BLACKHOLE | ACTIVE | STORAGE ENGINE | NULL | GPL |
| FEDERATED | DISABLED | STORAGE ENGINE | NULL | GPL |
| ngram | ACTIVE | FTPARSER | NULL | GPL |
| mysqlx_cache_cleaner | ACTIVE | AUDIT | NULL | GPL |
| mysqlx | ACTIVE | DAEMON | NULL | GPL |
| group_replication | ACTIVE | GROUP REPLICATION | group_replication.so | GPL |
+---------------------------------+----------+--------------------+----------------------+---------+
45 rows in set (0.00 sec)
After installing the plug-in, remove the MGR parameter comments and restart the mysql instance.
Set up MGR single master mode
. Execute the following commands on all nodes to create a replicated user
set sql_log_bin=0;
create user 'repl'@'%' identified with mysql_native_password by 'repl';
grant replication slave,replication client on *.* to 'repl'@'%';
flush privileges;
set sql_log_bin=1;
Set the master node on the 192.168.112.131 service
set global group_replication_single_primary_mode=on;
set global group_replication_bootstrap_group=ON;
CHANGE MASTER TO MASTER_USER='repl', MASTER_PASSWORD='repl' FOR CHANNEL 'group_replication_recovery';
start group_replication;
set global group_replication_bootstrap_group=OFF;
After setting, you can view the status of MGR replication members
[root@localhost] 14:44:39 [(none)]>select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | c1a60f79-f719-11ea-97db-000c29cc2388 | mysql | 3309 | ONLINE | PRIMARY | 8.0.21 |
+---------------------------+--------------------------------------+----
After the master is set, the remaining 2 slave nodes are set up, and the execution commands are the same.
CHANGE MASTER TO MASTER_USER='repl', MASTER_PASSWORD='repl' FOR CHANNEL 'group_replication_recovery';
start group_replication;
Here, I encountered a big pit, come and have a look
[root@localhost] 14:44:40 [(none)]>select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 188f118c-f71b-11ea-899b-000c29387845 | mysql | 3309 | RECOVERING | SECONDARY | 8.0.21 |
| group_replication_applier | 577b5cdb-f71a-11ea-9f03-000c29231183 | mysql | 3309 | RECOVERING | SECONDARY | 8.0.21 |
| group_replication_applier | c1a60f79-f719-11ea-97db-000c29cc2388 | mysql | 3309 | ONLINE | PRIMARY | 8.0.21 |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
3 rows in set (0.00 sec)
The state of the 2 slave nodes is always in the RECOVERING state, and in the ONLINE state when it is normal. This state lasted for 10 minutes, look at the mysql error log and see the following error message
2020-09-15T14:24:57.555493+08:00 19 [ERROR] [MY-010584] [Repl] Slave I/O for channel 'group_replication_recovery': error connecting to master 'repl@mysql:3309' - retry-time: 60 retries: 1 message: Authentication plugin 'caching_sha2_password' reported error: Authentication requires secure connection. Error_code: MY-002061
2020-09-15T14:24:57.557460+08:00 18 [ERROR] [MY-011582] [Repl] Plugin group_replication reported: 'There was an error when connecting to the donor server. Please check that group_replication_recovery channel credentials and all MEMBER_HOST column values of performance_schema.replication_group_members table are correct and DNS resolvable.'
2020-09-15T14:24:57.557541+08:00 18 [ERROR] [MY-011583] [Repl] Plugin group_replication reported: 'For details please check performance_schema.replication_connection_status table and error log messages of Slave I/O for channel group_replication_recovery.'
2020-09-15T14:25:57.284156+08:00 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'
The key content is error connecting to master'repl@mysql:3309', unable to connect to the master, that is, the network is blocked. Here MGR actually uses the host name to communicate with the cluster members. The key is that my host names are the same. Because the virtual level was cloned.
There are two solutions here.
Solution 1:
Modify the host name and add the correspondence between the host and IP in /etc/hosts on each cluster node of MGR.
Solution 2:
Let the MGR cluster nodes communicate with IP addresses, which can be configured The following 2 parameters
report_host=192.168.112.135
report_port=3309
I use the second option, add the following contents in the my.cnf parameter file respectively
在192.168.112.131服务器上
report_host=192.168.112.131
report_port=3309
在192.168.112.132服务器上
report_host=192.168.112.132
report_port=3309
在192.168.112.135服务器上
report_host=192.168.112.135
report_port=3309
After the addition is complete, restart the Mysql instance and restart the cluster.
Set up the master node
set global group_replication_bootstrap_group=ON;
start group_replication;
set global group_replication_bootstrap_group=OFF;
Set up the slave node
start group_replication;
View cluster status
[root@localhost] 15:02:23 [(none)]>select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 188f118c-f71b-11ea-899b-000c29387845 | 192.168.112.135 | 3309 | ONLINE | SECONDARY | 8.0.21 |
| group_replication_applier | 577b5cdb-f71a-11ea-9f03-000c29231183 | 192.168.112.132 | 3309 | ONLINE | SECONDARY | 8.0.21 |
| group_replication_applier | c1a60f79-f719-11ea-97db-000c29cc2388 | 192.168.112.131 | 3309 | ONLINE | PRIMARY | 8.0.21 |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+
3 rows in set (0.00 sec)
This time the cluster node status is all online.
Single master switch to multi-master mode
Execute on all nodes
stop group_replication;
set global group_replication_single_primary_mode=OFF;
set global group_replication_enforce_update_everywhere_checks=ON;
Select any node to execute
SET GLOBAL group_replication_bootstrap_group=ON;
START GROUP_REPLICATION;
SET GLOBAL group_replication_bootstrap_group=OFF;
Remaining node execution
START GROUP_REPLICATION;
View cluster status
[root@localhost] 17:45:57 [db1]>select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 188f118c-f71b-11ea-899b-000c29387845 | 192.168.112.135 | 3309 | ONLINE | PRIMARY | 8.0.21 |
| group_replication_applier | 577b5cdb-f71a-11ea-9f03-000c29231183 | 192.168.112.132 | 3309 | ONLINE | PRIMARY | 8.0.21 |
| group_replication_applier | c1a60f79-f719-11ea-97db-000c29cc2388 | 192.168.112.131 | 3309 | ONLINE | PRIMARY | 8.0.21 |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+
3 rows in set (0.01 sec)
Multi-master switch to multi-single mode
Execute on all nodes
stop group_replication;
set global group_replication_enforce_update_everywhere_checks=off;
Select the main node database to execute
set global group_replication_single_primary_mode=on;
set global group_replication_bootstrap_group=ON;
start group_replication;
set global group_replication_bootstrap_group=OFF;
Remaining node execution
set global group_replication_single_primary_mode=on;
START GROUP_REPLICATION;
View cluster status
[root@localhost] 17:51:12 [db1]>select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 188f118c-f71b-11ea-899b-000c29387845 | 192.168.112.135 | 3309 | ONLINE | SECONDARY | 8.0.21 |
| group_replication_applier | 577b5cdb-f71a-11ea-9f03-000c29231183 | 192.168.112.132 | 3309 | ONLINE | SECONDARY | 8.0.21 |
| group_replication_applier | c1a60f79-f719-11ea-97db-000c29cc2388 | 192.168.112.131 | 3309 | ONLINE | PRIMARY | 8.0.21 |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+
3 rows in set (0.00 sec)
Reference link
Mysql8 official distributed database MGR best practice
https://mp.weixin.qq.com/s/t9M5NhHjCoOBtULgTQOs6w
Mysql launched the financial-grade distributed database MGR big secret: https://mp.weixin.qq.com/s/Wykr8sczlvPb-1SEAwRQyQ