Combining graphics and text丨GreatSQL MGR + ProxySQL cluster construction plan

Zun Du and Fake Du, are Chengdu programmers also 996?

Preface

ProxySQL

ProxySQL is an open source middleware product based on MySQL. It is a flexible MySQL proxy layer that can realize read and write separation, supports Query routing function, supports dynamically specifying a SQL for caching, and supports dynamic loading (without restarting the ProxySQL service) ), failover and some SQL filtering capabilities.

GreatSQL MGR

GreatSQLIt is a domestic independent open source database suitable for financial-level applications. It has many core features such as high performance, high reliability, high ease of use, and high security. It can be used as an optional replacement for MySQL or Percona Server and used in online production environments. It's completely free and compatible with MySQL or Percona Server.

GreatSQLThe main improvement in high reliability is to make a lot of improvements and improvements to MGR to further enhance the high reliability level of MGR. Including but not limited to the following improvements:

地理标签, improve the data reliability of multi-computer room architecture.
读写节点动态VIP, high availability switching is more convenient.
仲裁节点, achieving higher availability with lower server costs.
快速单主模式, faster and higher performance in single master mode.
智能选主, the high-availability switching master selection mechanism is more reasonable.
全新流控算法, making transactions more stable and avoiding violent jitters.
Optimized the problem that may cause severe performance jitter when nodes join and exit.
Solve the problem that the MGR cluster is blocked when the disk space is full.
Solved the problem of being unable to choose the leader due to long transactions.
Optimize the transaction authentication queue cleaning algorithm to avoid the jitter problem every 60 seconds.
Fixed the long waiting problem during recovery process.

For more details, please go to ➥ https://gitee.com/GreatSQL/GreatSQL-Manual/blob/master/5-enhance/5-2-ha.md

Deployment environment introduction

Deployment architecture diagram

file

GreatSQL MGR cluster implements database replication function and high availability. Proxysql provides access to applications, separates reading and writing on the MGR cluster, detects cluster status, and implements failover.

Deployment environment configuration

Deployment software details

Software name	version number
GreatSQL	8.0.32-24
ProxySQL	2.5.4-58

Deployment environment preparation

This time we use a single-machine multi-instance deployment method. How to deploy a single-machine multi-instance can go to ➥ https://gitee.com/GreatSQL/GreatSQL-Manual/blob/master/6-oper-guide/6-6-multi- instances.md

IP	port	Role
172.17.139.77	3306	MGR01
172.17.139.77	3307	MGR02
172.17.139.77	6032、6033	ProxySQL

$ cat /etc/system-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
$ uname -a
Linux gip 3.10.0-1160.el7.x86_64 #1 SMP Tue Aug 18 14:50:17 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

GreatSQL configuration

#mgr settings
loose-plugin_load_add = 'mysql_clone.so'
loose-plugin_load_add = 'group_replication.so'
loose-group_replication_group_name = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa1"
loose-group_replication_group_seeds = '172.17.139.77:33061,172.17.139.77:33071'
loose-group_replication_start_on_boot = ON
loose-group_replication_bootstrap_group = OFF
loose-group_replication_exit_state_action = READ_ONLY
loose-group_replication_flow_control_mode = "DISABLED"
loose-group_replication_single_primary_mode = ON
loose-group_replication_communication_max_message_size = 10M
loose-group_replication_transaction_size_limit = 3G
loose-group_replication_arbitrator = 0
loose-group_replication_single_primary_fast_mode = 0
loose-group_replication_request_time_threshold = 20000
report_host = "172.17.139.77"

The MGR01 node configuration is as follows

[mysqld@mgr01]
datadir=/data/GreatSQL/mgr01
socket=/data/GreatSQL/mgr01/mysql.sock
port=3306
server_id=103306
log-error=/data/GreatSQL/mgr01/error.log
loose-group_replication_local_address= "172.17.139.77:33061"

The MGR02 node configuration is as follows

[mysqld@mgr02]
datadir=/data/GreatSQL/mgr02
socket=/data/GreatSQL/mgr02/mysql.sock
port=3307
server_id=103317
log-error=/data/GreatSQL/mgr02/error.log
loose-group_replication_local_address= "172.17.139.77:33071"

Build MGR cluster and ProxySQL

Build a GreatSQL MGR cluster

MGR01 instance operation

greatsql> set session sql_log_bin=0;
greatsql> create user 'repl'@'%' identified with mysql_native_password by 'GreatSQL@666';
greatsql> GRANT BACKUP_ADMIN, REPLICATION SLAVE ON *.* TO `repl`@`%`;
greatsql> set session sql_log_bin=1;
greatsql> CHANGE MASTER TO MASTER_USER='repl', MASTER_PASSWORD='GreatSQL@666' FOR CHANNEL 'group_replication_recovery';

Next, you can start the MGR cluster

greatsql> set global group_replication_bootstrap_group=ON;
greatsql> start group_replication;
greatsql> select * from performance_schema.replication_group_members\G
*************************** 1. row ***************************
              CHANNEL_NAME: group_replication_applier
                 MEMBER_ID: 2920447e-35bf-11ee-89a5-00163e566da1
               MEMBER_HOST: 172.17.139.77
               MEMBER_PORT: 3306
              MEMBER_STATE: ONLINE
               MEMBER_ROLE: PRIMARY
            MEMBER_VERSION: 8.0.32
MEMBER_COMMUNICATION_STACK: XCom

MGR02 instance operation

greatsql> set session sql_log_bin=0;
greatsql> create user 'repl'@'%' identified with mysql_native_password by 'GreatSQL@666';
greatsql> GRANT BACKUP_ADMIN, REPLICATION SLAVE ON *.* TO `repl`@`%`;
greatsql> set session sql_log_bin=1;
greatsql> CHANGE MASTER TO MASTER_USER='repl', MASTER_PASSWORD='GreatSQL@666' FOR CHANNEL 'group_replication_recovery';
greatsql> start group_replication;
Query OK, 0 rows affected (5.39 sec)

The MGR cluster was successfully established

greatsql> select * from performance_schema.replication_group_members\G
*************************** 1. row ***************************
              CHANNEL_NAME: group_replication_applier
                 MEMBER_ID: 8f20696c-3b46-11ee-b59e-00163e566da1
               MEMBER_HOST: 172.17.139.77
               MEMBER_PORT: 3306
              MEMBER_STATE: ONLINE
               MEMBER_ROLE: PRIMARY
            MEMBER_VERSION: 8.0.32
MEMBER_COMMUNICATION_STACK: XCom
*************************** 2. row ***************************
              CHANNEL_NAME: group_replication_applier
                 MEMBER_ID: 8ff26a55-3b46-11ee-bd70-00163e566da1
               MEMBER_HOST: 172.17.139.77
               MEMBER_PORT: 3307
              MEMBER_STATE: ONLINE
               MEMBER_ROLE: SECONDARY
            MEMBER_VERSION: 8.0.32
MEMBER_COMMUNICATION_STACK: XCom

Create the accounts required for ProxySQL on the MGR cluster

# proxysql的监控账户
greatsql> create user 'monitor'@'%' identified with mysql_native_password by 'Monitor@666';
greatsql> grant all privileges on *.* to 'monitor'@'%' with grant option;
# proxysql的对外访问账户
greatsql> create user 'proxysql'@'%' identified with mysql_native_password by 'Proxysql@666';
greatsql> grant all privileges on *.* to 'proxysql'@'%' with grant option;

The user authentication method needs to be modified tomysql_native_password

Check to see if it was created successfully

greatsql> select User,Host from mysql.user;
+------------------+-----------+
| User             | Host      |
+------------------+-----------+
| monitor          | %         |
| proxysql         | %         |
| repl             | %         |
| mysql.infoschema | localhost |
| mysql.session    | localhost |
| mysql.sys        | localhost |
| root             | localhost |
+------------------+-----------+

InstallProxySQL

There are detailed installation tutorials in the ProxySQL documentation ➥ https://github.com/sysown/proxysql

Both RPM and yum methods can be installed. This article uses the RPM method to install. If you want to use yum to install, you need to change the yum source.

$ cat > /etc/yum.repos.d/proxysql.repo << EOF
[proxysql]
name=ProxySQL YUM repository
baseurl=https://repo.proxysql.com/ProxySQL/proxysql-2.4.x/centos/\$releasever
gpgcheck=1
gpgkey=https://repo.proxysql.com/ProxySQL/proxysql-2.4.x/repo_pub_key
EOF

Red Hat system needs to \$releaseverchange to7

Then install it directly

$ yum install proxysql

It should be noted here that if GreatSQL is installed using RPM, it will conflict with the dependencies required by ProxySQL!

RPM package download address: https://github.com/sysown/proxysql/releases/tag/v2.5.4

So use the RPM --nodepsoption to force the installation.rpm -ivh proxysql-2.5.4-1-centos7.x86_64.rpm --nodeps

But systemctl start proxysql.servicean error will be reported when starting. You need tolibgnutls.so.28

Install it again at this time and you can start it yum install -y gnutlsagain.systemctl start proxysql.service

Start ProxySQL

$ systemctl start proxysql
# 检查是否启动成功
$ systemctl status proxysql

Check whether the port is open

$ netstat -anlp | grep proxysql
tcp  0  0 0.0.0.0:6033  0.0.0.0:*  LISTEN 1009915/proxysql    
tcp  0  0 0.0.0.0:6032  0.0.0.0:*  LISTEN 1009915/proxysql

6032 is the management port number of ProxySQL
6033 is the port number for external services

The username and password of ProxySQL are both admin by default.

Configure ProxySQL

Administrator login

$ mysql -uadmin -padmin -h 127.0.0.1 -P 6032

You can see that there are some databases available, and ProxySQL converts SHOW DATABASEthe commands into equivalent commands for SQLite3.

greatsql> show databases;
+-----+---------------+-------------------------------------+
| seq | name          | file                                |
+-----+---------------+-------------------------------------+
| 0   | main          |                                     |
| 2   | disk          | /var/lib/proxysql/proxysql.db       |
| 3   | stats         |                                     |
| 4   | monitor       |                                     |
| 5   | stats_history | /var/lib/proxysql/proxysql_stats.db |
+-----+---------------+-------------------------------------+
5 rows in set (0.00 sec)

These databases function as follows:

main: In-memory configuration database Using this database, ProxySQL's configuration can be easily queried and updated in an automated manner. Using LOAD MYSQL USERS FROM MEMORY and similar commands, the configuration stored here can be propagated at runtime to the in-memory data structures used by ProxySQL.
disk: Disk-based "main" image. "main" is not persisted across reboots and can be loaded from the "on-disk" database or the configuration file, depending on the boot flags and the presence of the on-disk database.
stats: Contains runtime metrics collected from the agent's internal functionality. Example metrics include the number of times each query rule was matched, the currently running query, etc.
monitor: Contains monitoring indicators related to the backend server connected to ProxySQL. Example metrics include the minimum and maximum time to connect to or ping the backend server.
myhgm: only enabled in debug builds

Additionally, both types of users access the administration database using these default credentials:

user:admin/password:admin - has read and write access to all tables
user:stats/password:stats - Has read-only access to the statistics table. This is used to pull metrics from ProxySQL without exposing too much of the database

The above access credentials can be configured through variables admin-admin_credentialsand .admin-stats_credentials

For more detailed introduction, please go to "Laoye Tea House MySQL-ProxySQL Middleware (1) [Add on WeChat]" "Laoye Tea House MySQL-ProxySQL Middleware (2) [Add on WeChat]"

To configure monitoring account

greatsql> set mysql-monitor_username='monitor';
greatsql> set mysql-monitor_password='Monitor@666';

The above two sentences are ways to modify variables. They can also be modified using sql statements under the main library.

greatsql>UPDATE global_variables SET variable_value='monitor' WHERE variable_name='mysql-monitor_username';
greatsql>UPDATE global_variables SET variable_value='Monitor@666' WHERE variable_name='mysql-monitor_password';

Configure default group information

greatsql> insert into mysql_group_replication_hostgroups(writer_hostgroup,backup_writer_hostgroup,reader_hostgroup,offline_hostgroup,active,writer_is_also_reader)values(10,20,30,40,1,1);

This SQL statement is used to configure the active/standby and read-write separation of the MGR cluster. mysql_group_replication_hostgroupsInsert the configuration into the table.

writer_hostgroup:Write the host group of the master node (must be greater than 0), here set to 10
backup_writer_hostgroup:The backup is written to the host group of the master node, here it is 20.
reader_hostgroup:The host group of the read-only node, here is 30.
offline_hostgroup:The host group of the offline node, here is 40.
active: Whether to activate this configuration, 1 means activated.
writer_is_also_reader:Whether the writing master node can also be used as a reading node, 1 means yes.

Configure external access users to write group 10

greatsql> insert into mysql_users(username,password,default_hostgroup)values('proxysql','Proxysql@666',10);

The function of this SQL code is to add a MySQL server node to the management of ProxySQL so that ProxySQL can distribute connection requests according to defined rules and policies to achieve load balancing and high availability.

Configure the master node to be defined as write group 10, and the slave node to be defined as read-only group 30

greatsql> insert into mysql_servers(hostgroup_id,hostname,port,weight,max_connections,max_replication_lag,comment)values (10,'172.17.139.77',3306,1,3000,10,'mgr01');
greatsql> insert into mysql_servers(hostgroup_id,hostname,port,weight,max_connections,max_replication_lag,comment)values (30,'172.17.139.77',3306,2,3000,10,'mgr01');
greatsql> insert into mysql_servers(hostgroup_id,hostname,port,weight,max_connections,max_replication_lag,comment)values (30,'172.17.139.77',3307,2,3000,10,'mgr02');

hostgroup_id: Specify the host group to which it belongs. This is a concept used for group management in ProxySQL. Here, it's set to 10.
hostname: Specify the host name or IP address of the MySQL server, here it is '172.17.139.77'.
port: Specify the port number of the MySQL server, here it is 3306.
weight: Specify the weight of the node in load balancing. The higher the weight, the more requests will be assigned to this node. This is set to 1.
max_connections: Specifies the maximum number of connections allowed by this node.
max_replication_lag: Specifies the maximum replication delay (in seconds) that is configured when connecting to a master-slave replica node.
comment: An optional comment or description information, here set to 'mgr01'.

The function of this SQL code is to add a MySQL user to ProxySQL management so that ProxySQL can control user access to the database according to defined user access rules and policies, including routing, load balancing, failover, etc.

Configure the read-write separation parameters. There are two tables related to them mysql_query_rulesand mysql_query_rules_fast_routingyou can configure them yourself here.

where table mysql_query_rules_fast_routingis mysql_query_rulesan extension and fast routing policies and properties are evaluated later (only available in ProxySQL 1.4.7+).

greatsql> insert into mysql_query_rules(rule_id,active,match_digest,destination_hostgroup,apply)values(1,1,^SELECT.*FORUPDATE$,10,1);
greatsql> insert into mysql_query_rules(rule_id,active,match_pattern,destination_hostgroup,apply) values (2,1,'^select',30,1);

active: Whether to enable this rule, 1 means enabled, 0 means disabled
match_patternFields represent setting rules
destination_hostgroupThe field represents the default specified grouping,
applyRepresents the actual execution of application rules

In ProxySQL, rule_idthe sorting function is to control the matching order of rules. When ProxySQL processes a query request, it will rule_idmatch the rules one by one in ascending order until the first matching rule is found. Once a matching rule is found, ProxySQL will process the query request according to the definition of the rule. The purpose of this ordering is to ensure that rules are matched and applied in the expected order, allowing for precise query routing, offloading, and load balancing.

In the above example, select ... for updatethe rule is to ensure that it is rule_idsmaller than the ordinary selectrule rule_idin order to ensure that the rule is matched first select ... for update, rather than the ordinary selectrule.

Because select ... for updateis a special query, it will involve locking operations during execution, which may affect the execution of other queries. By making select ... for updatethe rule_idsmaller, you can ensure that ProxySQL gives priority to matching this special rule when matching query rules, so that the select ... for updatecorresponding routing and processing logic can be applied more accurately during processing.

save causes the memory data to be permanently stored on the disk, and load causes the memory data to be loaded into the runtime to take effect:

greatsql> save mysql users to disk;
greatsql> save mysql servers to disk;
greatsql> save mysql query rules to disk;
greatsql> save mysql variables to disk;
greatsql> save admin variables to disk;
greatsql> load mysql users to runtime;
greatsql> load mysql servers to runtime;
greatsql> load mysql query rules to runtime;
greatsql> load mysql variables to runtime;
greatsql> load admin variables to runtime;

After the loading is completed, you can use select * to query whether the information of each table set is correct.

Verify the monitoring information. The indicators of the ProxySQL monitoring module are all saved in monitor库的log表中. The following is the monitoring of whether the connection is normal. For the monitoring of the connect indicator, there may be many connect_errors in the front. This is because there is an error when the monitoring information is not configured. After configuration, if the connect_error If the result is NULL, it means normal

greatsql> select * from monitor.mysql_server_connect_log;
+---------------+------+------------------+-------------------------+---------------+
| hostname      | port | time_start_us    | connect_success_time_us | connect_error |
+---------------+------+------------------+-------------------------+---------------+
| 172.17.139.77 | 3307 | 1692152694528579 | 2347                    | NULL          |
| 172.17.139.77 | 3306 | 1692152695231434 | 3131                    | NULL          |
+---------------+------+------------------+-------------------------+---------------+
#只截取部分

Monitoring of heartbeat information (monitoring of ping indicators)

greatsql> select * from mysql_server_ping_log limit 10;
+---------------+------+------------------+----------------------+------------+
| hostname      | port | time_start_us    | ping_success_time_us | ping_error |
+---------------+------+------------------+----------------------+------------+
| 172.17.139.77 | 3306 | 1692152694511424 | 132                  | NULL       |
| 172.17.139.77 | 3307 | 1692152694511498 | 125                  | NULL       |
+---------------+------+------------------+----------------------+------------+
#只截取部分

Test read and write separation

Connect through proxysql to see if the read operation is routed to the read group.

$ mysql -uproxysql -pProxysql@666 -P 6033 -h 172.17.139.77 -e "select @@server_id;"
+-------------+
| @@server_id |
+-------------+
|      103317 |
+-------------+
#可以看到已经把路由分配给了读组

Test write operation

greatsql> begin;
Query OK, 0 rows affected (0.00 sec)
greatsql> select User from mysql.user for update;
+------------------+
| User             |
+------------------+
| monitor          |
| proxysql         |
| repl             |
| mysql.infoschema |
| mysql.session    |
| mysql.sys        |
| root             |
+------------------+
7 rows in set (0.00 sec)
greatsql> select @@server_id;
+-------------+
| @@server_id |
+-------------+
|      103306 |
+-------------+
1 row in set (0.00 sec)
#可以看到已经把路由分配给了写组
greatsql> commit;
Query OK, 0 rows affected (0.00 sec)

If you want to view SQL request routing information in ProxySQLstats_mysql_query_digest

greatsql> select hostgroup,schemaname,username,digest_text,count_star from  stats_mysql_query_digest;
+-----------+--------------------+----------+----------------------------------------+------------+
| hostgroup | schemaname         | username | digest_text                            | count_star |
+-----------+--------------------+----------+----------------------------------------+------------+
| 10        | information_schema | proxysql | commit                                 | 1          |
| 10        | information_schema | proxysql | select User from mysql.user for update | 1          |
| 10        | information_schema | proxysql | select * from beta1 for update         | 2          |
| 10        | information_schema | proxysql | begin                                  | 1          |
| 30        | information_schema | proxysql | select @@server_id                     | 2          |
+-----------+--------------------+----------+----------------------------------------+------------+
#只截取部分

count_startCount the number of SQL statements and analyze which SQL statements are frequently executed.

At this point, a GreatSQL MGR + ProxySQL cluster construction solution has been deployed.

end

Although ProxySQL has powerful functions, ProxySQL is not officially native after all, and its cooperation with MGR is not as smooth as GreatSQL-MySQL-Router. For example, additional stored procedures need to be created to monitor changes in MGR. In addition, there are actually quite a lot of bugs in ProxySQL. Of course, if the business volume is not large, or for learning or experimental purposes, it is also possible to choose ProxySQL.

Recommended to use GreatSQL-MySQL-Router, the cooperation with GreatSQL MGR is smoother and more compatible.

Use GreatSQL-MySQL-Routerthe MGR cluster to build a read-write splitting solution: https://gitee.com/GreatSQL/GreatSQL-Manual/blob/master/6-oper-guide/6-3-oper-rw-splitting.md

Enjoy GreatSQL :)

About GreatSQL

GreatSQL is a domestic independent open source database suitable for financial-level applications. It has many core features such as high performance, high reliability, high ease of use, and high security. It can be used as an optional replacement for MySQL or Percona Server and is used in online production environments. , completely free and compatible with MySQL or Percona Server.

Related links: GreatSQL Community Gitee GitHub Bilibili

GreatSQL Community:

Community reward suggestions and feedback: https://greatsql.cn/thread-54-1-1.html

Community blog prize-winning submission details: https://greatsql.cn/thread-100-1-1.html

(If you have any questions about the article or have unique insights, you can go to the official community website to ask or share them~)

Technical exchange group:

WeChat & QQ group:

QQ group: 533341697

WeChat group: Add GreatSQL Community Assistant (WeChat ID: wanlidbc) as a friend and wait for the community assistant to add you to the group.