GTID-based semi-synchronous high-availability MySQL cluster

Prerequisite preparation

Network topology diagram

Prepare 9 centos7.9 virtual machines and assign IP addresses:

CPU name	IP
master	192.168.78.132
slave1	192.168.78.149
slave2	192.168.78.150
delay_backup	192.168.78.148
ansible	192.168.78.139
mysqlrouter1	192.168.78.135
mysqlrouter2	192.168.78.136
promtheus+grafana	192.168.78.147
test	192.168.78.145

Project environment

centos7.9 mysql5.7.41 mysqlrouter8.0.21 keepalived-1.3.5 ansible-2.9.27
sysbench 1.0.17 grafana-9.1.2

project description

The purpose of this project is to build a highly available and efficient MySQL cluster that can separate reading and writing, and at the same time use rsync+sersync to realize remote data backup, do disaster recovery work, and ensure business stability.

Project steps

1. Deploy ansible, configure SSH password-free channels between all machines, and use ansible to quickly build the project environment;

2. Configure master-slave replication on 3 MySQL servers to form a master+2 slave node (semi-synchronous+GTID) cluster to provide database services;

3. Use rsync+sersync to build real-time remote backup of data files, build delayed backup services, and use mysqlrouter to achieve read-write separation to achieve better MySQL cluster performance;

4. Install keepalived and implement read-write separation and high availability on two mysqlrouter servers. Configure 2 instances on keepalived to implement 2 VIPs, each serving as master and backup, to better improve high-availability performance;

5. Build Prometheus+grafana monitoring to achieve data visualization, and use stress testing software (sysbench) to test the performance of the entire MySQL cluster.

1. Install ansible and establish a password-free channel to deploy MySQL

1. Install ansible

[root@ansible ~]# yum install epel-release -y
[root@ansible ~]# yum install ansible -y

2. Establish a secret-free channel

Establish password-free channels with all MySQL servers:

[root@ansible .ssh]# ssh-keygen 	#一直回车进行
[root@ansible .ssh]# ssh-copy-id -i id_rsa.pub 192.168.78.132  		#master
[root@ansible .ssh]# ssh-copy-id -i id_rsa.pub 192.168.78.149 		#slave1
[root@ansible .ssh]# ssh-copy-id -i id_rsa.pub 192.168.78.150		#slave2
[root@ansible .ssh]# ssh-copy-id -i id_rsa.pub 192.168.78.148		#delay_backup

Test connection:

[root@ansible .ssh]# ssh '192.168.78.148'

3. Use ansible to deploy MySQL cluster

Write a host list:

[root@ansible ~]# cd /etc/ansible/
[root@ansible ansible]# ls
ansible.cfg  hosts  playbooks  roles
[root@ansible ansible]# vim hosts 
[mysql]
192.168.78.132          #master
192.168.78.149          #slave1
192.168.78.150          #slave2
192.168.78.148          #delay_backup

Write a playbook, upload the source code package to the remote server, and call the local script to install MySQL:

[root@ansible playbooks]# vim mysql_install.yaml
- hosts: mysql
  remote_user: root
  tasks:
  - name: copy file     #上传MySQL安装包到mysql主机组
    copy: src=/root/mysql-5.7.41-linux-glibc2.12-x86_64.tar.gz dest=/root/
  - name: mysql_install         #安装MySQL
    script: /root/onekey_install_mysql_binary_v3.sh

Check the yaml file syntax and execute:

[root@ansible playbooks]# ansible-playbook --syntax-check mysql_install.yaml 

playbook: mysql_install.yaml
[root@ansible playbooks]# ansible-playbook mysql_install.yaml 

Check whether the installation is successful:

[root@slave1 ~]# ps aux|grep mysqld
root       2521  0.0  0.1  11824  1612 ?        S    11:22   0:00 /bin/sh /usr/local/mysql/bin/mysqld_safe --datadir=/data/mysql --pid-file=/data/mysql/slave1.pid
mysql      2675  0.4 20.6 1610264 205460 ?      Sl   11:22   0:00 /usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql --datadir=/data/mysql --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql --log-error=slave1.err --open-files-limit=8192 --pid-file=/data/mysql/slave1.pid --socket=/data/mysql/mysql.sock --port=3306
root       2784  0.0  0.0 112824   976 pts/0    S+   11:24   0:00 grep --color=auto mysqld
[root@slave1 ~]# netstat -anplut|grep mysqld
tcp6       0      0 :::3306                 :::*                    LISTEN      2675/mysqld    

mysql installation script:

#!/bin/bash

#解决软件的依赖关系
yum  install cmake ncurses-devel gcc  gcc-c++  vim  lsof bzip2 openssl-devel ncurses-compat-libs -y

#解压mysql二进制安装包
tar  xf  mysql-5.7.41-linux-glibc2.12-x86_64.tar.gz

#移动mysql解压后的文件到/usr/local下改名叫mysql
mv mysql-5.7.41-linux-glibc2.12-x86_64 /usr/local/mysql

#新建组和用户 mysql
groupadd mysql
#mysql这个用户的shell 是/bin/false 属于mysql组 
useradd -r -g mysql -s /bin/false mysql

#关闭firewalld防火墙服务，并且设置开机不要启动
service firewalld stop
systemctl  disable  firewalld

#临时关闭selinux
setenforce 0
#永久关闭selinux
sed -i '/^SELINUX=/ s/enforcing/disabled/'  /etc/selinux/config

#新建存放数据的目录
mkdir  /data/mysql -p
#修改/data/mysql目录的权限归mysql用户和mysql组所有，这样mysql用户可以对这个文件夹进行读写了
chown mysql:mysql /data/mysql/
#只是允许mysql这个用户和mysql组可以访问，其他人都不能访问
chmod 750 /data/mysql/

#进入/usr/local/mysql/bin目录
cd /usr/local/mysql/bin/

#初始化mysql
./mysqld  --initialize --user=mysql --basedir=/usr/local/mysql/  --datadir=/data/mysql  &>passwd.txt

#让mysql支持ssl方式登录的设置
./mysql_ssl_rsa_setup --datadir=/data/mysql/

#获得临时密码
tem_passwd=$(cat passwd.txt |grep "temporary"|awk '{print $NF}')
  #$NF表示最后一个字段
  # abc=$(命令)  优先执行命令，然后将结果赋值给abc 

# 修改PATH变量，加入mysql bin目录的路径
#临时修改PATH变量的值
export PATH=/usr/local/mysql/bin/:$PATH
#重新启动linux系统后也生效，永久修改
echo  'PATH=/usr/local/mysql/bin:$PATH' >>/root/.bashrc

#复制support-files里的mysql.server文件到/etc/init.d/目录下叫mysqld
cp  ../support-files/mysql.server   /etc/init.d/mysqld

#修改/etc/init.d/mysqld脚本文件里的datadir目录的值
sed  -i '70c  datadir=/data/mysql'  /etc/init.d/mysqld

#生成/etc/my.cnf配置文件
cat  >/etc/my.cnf  <<EOF
[mysqld_safe]

[client]
socket=/data/mysql/mysql.sock

[mysqld]
socket=/data/mysql/mysql.sock
port = 3306
open_files_limit = 8192
innodb_buffer_pool_size = 512M
character-set-server=utf8

[mysql]
auto-rehash
prompt=\\u@\\d \\R:\\m  mysql>
EOF

#修改内核的open file的数量
ulimit -n 1000000
#设置开机启动的时候也配置生效
echo "ulimit -n 1000000" >>/etc/rc.local
chmod +x /etc/rc.d/rc.local


#启动mysqld进程
service mysqld start

#将mysqld添加到linux系统里服务管理名单里
/sbin/chkconfig --add mysqld
#设置mysqld服务开机启动
/sbin/chkconfig mysqld on

#初次修改密码需要使用--connect-expired-password 选项
#-e 后面接的表示是在mysql里需要执行命令  execute 执行
#set password='Sanchuang123#';  修改root用户的密码为long123#
mysql -uroot -p$tem_passwd --connect-expired-password   -e  "set password='long123#';"


#检验上一步修改密码是否成功，如果有输出能看到mysql里的数据库，说明成功。
mysql -uroot -p'long123#'  -e "show databases;"

2. Configure MySQL semi-synchronization and enable GTID function

1.Export data on master

[root@mysql-master ~]# mysqldump -uroot -p"Sanchuang123#" --all-databases >/backup/all_dbdata.sql
mysqldump: [Warning] Using a password on the command line interface can be insecure.

2. Import master data on all slave servers

scp the data file on the master to the slave server:

[root@mysql-master backup]# scp all_dbdata.sql 192.168.78.149:/root
all_dbdata.sql                                                           100%  888KB  60.0MB/s   00:00    
[root@mysql-master backup]# scp all_dbdata.sql 192.168.78.150:/root
all_dbdata.sql                                                           100%  888KB  91.0MB/s   00:00    
[root@mysql-master backup]# scp all_dbdata.sql 192.168.78.148:/root
all_dbdata.sql                                                           100%  888KB 101.8MB/s   00:00    
[root@mysql-master backup]# 

When importing data files from the server to MySQL:

[root@delay_backup ~]# mysql -uroot -p"long123#" <all_dbdata.sql 
mysql: [Warning] Using a password on the command line interface can be insecure.

3. Semi-sync and GTID configuration

Install the semi-sync plugin on the master machine:

root@(none) 13:37  mysql>install plugin rpl_semi_sync_master SONAME 'semisync_master.so';

Modify the configuration file on the master for permanent configuration and refresh the service:

[root@mysql-master /]# vim /etc/my.cnf
[mysqld]
#开启二进制日志
log_bin
server_id=1
#半同步
rpl_semi_sync_master_enabled=1
rpl_semi_sync_master_timeout=1000  #1 second
#gtid功能
gtid-mode=ON
enforce-gtid-consistency=ON
[root@mysql-master /]# service mysqld restart
Shutting down MySQL.... SUCCESS! 
Starting MySQL. SUCCESS! 

Install the semi-sync plug-in on 3 slave servers:

root@(none) 13:46  mysql>install plugin rpl_semi_sync_slave SONAME 'semisync_slave.so';
Query OK, 0 rows affected (0.01 sec)

Modify the configuration file on the slave server:

[root@slave1 ~]# vim /etc/my.cnf
#二进制日志
log_bin
server_id=2                #！注意：每台slave的id都不一样
expire_logs_days=15     #二进制日志保存15天
#开启gtid功能
gtid-mode=ON
enforce-gtid-consistency=ON
log_slave_updates=ON
#开启半同步，需要提前安装半同步的插件
rpl_semi_sync_slave_enabled=1
[root@slave1 ~]# service mysqld restart

Check whether semi-sync is successfully enabled:

root@(none) 13:44  mysql>show status like 'Rpl_semi_sync_master_status';
+-----------------------------+-------+
| Variable_name               | Value |
+-----------------------------+-------+
| Rpl_semi_sync_master_status | ON    |
+-----------------------------+-------+
1 row in set (0.02 sec)

Create a new authorized user on the master and copy the binary log to the slave:

root@(none) 14:57  mysql>grant replication slave on *.* to 'ly'@'192.168.78.%'identified by '123456';
Query OK, 0 rows affected, 1 warning (1.02 sec)

root@(none) 15:03  mysql>flush privileges;	#刷新
Query OK, 0 rows affected (0.01 sec)

Configure the master info information on slave1 and slave2:
Note: In order to prevent the configuration environment from causing the IO thread to fail to start normally, you need to clear the records on the slave first.

#清除二进制记录与master info的记录
root@(none) 16:07  mysql>reset master;
Query OK, 0 rows affected (0.02 sec)

root@(none) 16:07  mysql>stop slave;
Query OK, 0 rows affected (0.01 sec)

root@(none) 16:07  mysql>reset slave all;
Query OK, 0 rows affected (0.01 sec)

root@(none) 16:08  mysql>CHANGE MASTER TO MASTER_HOST='192.168.78.132',
    ->  MASTER_USER='ly',
    ->  MASTER_PASSWORD='123456',
    ->  MASTER_PORT=3306,
    ->  master_auto_position=1;
Query OK, 0 rows affected, 2 warnings (0.00 sec)

root@(none) 16:11  mysql>start slave;
Query OK, 0 rows affected (0.00 sec)

Check whether the io thread and SQL thread are started:

root@(none) 16:11  mysql>show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.78.132
                  Master_User: ly
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-master-bin.000001
          Read_Master_Log_Pos: 890
               Relay_Log_File: slave1-relay-bin.000002
                Relay_Log_Pos: 1117
        Relay_Master_Log_File: mysql-master-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes

Verify whether semi-synchronization is successful:
Delete a library on the master:

Check on the slave:

4. Delay backup configuration

Enable the binary and log_slave_updates functions on slave1 so that the delayed backup server can get data from slave1.

[root@slave1 ~]# cat /etc/my.cnf
 
[mysqld]
#添加配置
log_slave_updates=ON 
 
#重启服务
[root@slave1 ~]#service mysqld restart

Modify master info on the delayed backup delay_backup server:

root@(none) 16:19  mysql>stop slave;
Query OK, 0 rows affected, 1 warning (0.00 sec)

root@(none) 16:30  mysql>reset slave;
Query OK, 0 rows affected (0.01 sec)

root@(none) 16:31  mysql>reset slave all;
Query OK, 0 rows affected (0.00 sec)

root@(none) 16:31  mysql>CHANGE MASTER TO MASTER_HOST='192.168.78.149',		#slave1的IP
    ->  MASTER_USER='ly',
    ->  MASTER_PASSWORD='123456',
    ->  MASTER_PORT=3306,
    ->  master_auto_position=1;
Query OK, 0 rows affected, 2 warnings (0.02 sec)

root@(none) 16:31  mysql>change master to MASTER_DELAY=1800;		#延迟半个小时进行备份
Query OK, 0 rows affected (0.01 sec)

root@(none) 16:32  mysql>start slave;
Query OK, 0 rows affected (0.00 sec)

Check whether the IO thread and SQL thread start normally:

root@(none) 16:32  mysql>show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.78.149
                  Master_User: ly
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: slave1-bin.000001
          Read_Master_Log_Pos: 1195
               Relay_Log_File: delay_backup-relay-bin.000002
                Relay_Log_Pos: 1105
        Relay_Master_Log_File: slave1-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes

3. Build rsync+sersync to achieve real-time file synchronization

Build rsync+sersync files on the master server and ansible server to achieve off-site backup and real-time synchronization of data to ensure data security and integrity.

1.ansible server operation

Install rsync:

[root@ansible /]# yum install rsync xinetd -y

xinetd is a process that provides nanny services, and rsync is the process it takes care of.
Set up and start up:

[root@ansible /]# vim /etc/rc.d/rc.local 
#添加开机启动
/usr/bin/rsync --daemon --config=/etc/rsyncd.conf
#授权
[root@ansible /]# chmod +x /etc/rc.d/rc.local 
#启动服务
[root@ansible /]# systemctl start xinetd

Setup configuration file rsyncd.conf:

[root@ansible /]# vim /etc/rsyncd.conf
uid = root
gid = root
use chroot = yes
max connections = 0
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
secrets file = /etc/rsync.pass
motd file = /etc/rsyncd.Motd
[back_data]  #配置项名称（自定义）
        path = /backup  #备份文件存储地址
        comment = A directory in which data is stored
        ignore errors = yes
        read only = no
        hosts allow = 192.168.78.132  #允许的IP地址（数据源服务器地址）

Create a user authentication file and set file permissions:

[root@ansible /]# vim /etc/rsync.pass
[root@ansible /]# cat /etc/rsync.pass
ly:ly123
[root@ansible /]# chmod 600 /etc/rsyncd.conf
[root@ansible /]# chmod 600 /etc/rsync.pass

Start rsync and xinetd:

[root@ansible /]# /usr/bin/rsync --daemon --config=/etc/rsyncd.conf
[root@ansible /]# ps aux|grep rsync
root       5330  0.0  0.0 114852   572 ?        Ss   16:51   0:00 /usr/bin/rsync --daemon --config=/etc/rsyncd.conf
root       5332  0.0  0.0 112824   976 pts/0    S+   16:51   0:00 grep --color=auto rsync
[root@ansible /]# service xinetd restart
Redirecting to /bin/systemctl restart xinetd.service
[root@ansible /]# ps aux|grep xinetd
root       5351  0.0  0.0  25044   588 ?        Ss   16:52   0:00 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid
root       5355  0.0  0.0 112824   976 pts/0    S+   16:52   0:00 grep --color=auto xinetd

2.master server operation

Install rsync:

[root@mysql-master /]# yum install rsync xinetd -y

Set up startup:

[root@mysql-master /]# vim /etc/rc.d/rc.local 
/usr/bin/rsync --daemon	--config=/etc/rsyncd.conf
[root@mysql-master /]# chmod +x /etc/rc.d/rc.local

Modify the configuration file and add relevant configuration:

[root@mysql-master /]# vim /etc/rsyncd.conf 
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
motd file = /etc/rsyncd.Motd
[Sync]
comment = Sync
uid = root
gid = root
port= 873

Create an authentication password and start the service:
the password should be consistent with the password in /etc/rsync.pass in the target server

[root@mysql-master /]# cat /etc/passwd.txt 
ly123
[root@mysql-master /]# chmod 600 /etc/passwd.txt 
[root@mysql-master /]# systemctl start xinetd

Test the data synchronization between the data source server master and the backup server ansible, and back up the mysql data directory to the backup server:

[root@mysql-master backup]# rsync -avH --port=873 --progress --delete  /data  [email protected]::back_data --password-file=/etc/passwd.txt

3. Install the sersync tool for real-time synchronization

Modify the default parameters of inotify (inotify is already installed in the kernel by default and does not need to be installed):

[root@mysql-master backup]# vim /etc/sysctl.conf
fs.inotify.max_queued_events=99999999
fs.inotify.max_user_watches=99999999
fs.inotify.max_user_instances=65535

Install sersync:

[root@mysql-master /]# wget http://down.whsir.com/downloads/sersync2.5.4_64bit_binary_stable_final.tar.gz
[root@mysql-master /]# tar xf sersync2.5.4_64bit_binary_stable_final.tar.gz 
[root@mysql-master /]# mv GNU-Linux-x86 /usr/local/sersync

Create rsync:

[root@mysql-master ~]# cd /usr/local/sersync
[root@mysql-master sersync]# ls
confxml.xml  sersync2
[root@mysql-master sersync]# cp confxml.xml confxml.xml.bak
[root@mysql-master sersync]# cp confxml.xml data_configxml.xml
[root@mysql-master sersync]# ls
confxml.xml  confxml.xml.bak  data_configxml.xml  sersync2

Modify the configuration after line 24 of the data_configxml.xml file:

start the service and set startup:

[root@mysql-master sersync]# /usr/local/sersync/sersync2 -d -r -o  /usr/local/sersync/data_configxml.xml
set the system param
[root@mysql-master sersync]# vi /etc/rc.d/rc.local
 /usr/local/sersync/sersync2 -d -r -o  /usr/local/sersync/data_configxml.xml
[root@mysql-master sersync]# chmod +x /etc/rc.d/rc.local 

Test:
master:

backup server:

4. Configure MySQL read-write separation

1.Install mysqlrouter

Download address: mysqlrouter
is installed on mysqlrouter1 and mysqlrouter2:

[root@mysqlrouter1 ~]# rpm -ivh mysql-router-community-8.0.21-1.el7.x86_64.rpm 

2. Modify configuration file

Both mysqlrouter1 and mysqlrouter2 need to be modified:

[root@mysqlrouter1 ~]# cd /etc/mysqlrouter/
[root@mysqlrouter1 mysqlrouter]# ls
mysqlrouter.conf
[root@mysqlrouter1 mysqlrouter]# vim mysqlrouter.conf 
[routing:slaves]
bind_address = 0.0.0.0:8888		#8888端口监听读
destinations = 192.168.78.149:3306,192.168.78.150:3306	#在slave上执行读操作
mode = read-only
connect_timeout = 1

[routing:masters]
bind_address = 0.0.0.0:8889		#8889端口监听写
destinations = 192.168.78.132:3306	#在master是执行写操作
mode = read-write
connect_timeout = 1

Start the mysqlrouter service:

[root@mysqlrouter2 mysqlrouter]# service mysqlrouter start
Redirecting to /bin/systemctl start mysqlrouter.service
[root@mysqlrouter2 mysqlrouter]# netstat -anplut |grep 888
tcp        0      0 0.0.0.0:8888            0.0.0.0:*               LISTEN      1683/mysqlrouter    
tcp        0      0 0.0.0.0:8889            0.0.0.0:*               LISTEN      1683/mysqlrouter 

3. Verify read and write separation

Create 2 test accounts on the master, one for reading data and one for writing data:

root@(none) 21:48  mysql>grant all on *.*  to 'write'@'%' identified by '123456';
Query OK, 0 rows affected, 1 warning (0.00 sec)

root@(none) 21:49  mysql>grant select on *.*  to 'read'@'%' identified by '123456';
Query OK, 0 rows affected, 1 warning (0.01 sec)

Test the effect of reading and writing separation on the client test:
Use the client to connect to the write port of mysqlrouter, test the writing function, and find that reading and writing are normal:

[root@test ~]# mysql -h 192.168.78.135 -P 8889 -uwrite -p'123456'
write@(none) 14:24  mysql>create database db1;
Query OK, 1 row affected (0.00 sec)

write@(none) 14:25  mysql>drop database db1;
Query OK, 0 rows affected (0.02 sec)

write@(none) 14:26  mysql>use ly
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
write@ly 14:26  mysql>show tables;
+--------------+
| Tables_in_ly |
+--------------+
| t1           |
+--------------+
1 row in set (0.01 sec)

write@ly 14:26  mysql>select * from t1;
+----+------+
| id | name |
+----+------+
|  1 | cali |
|  2 | ly   |
|  3 | xx   |
|  4 | ada  |
|  5 | dad  |
|  6 | asd  |
+----+------+
6 rows in set (0.00 sec)

Use a client to connect to the read port of mysqlrouter and test the read function. It is found that the read is normal but cannot be written:

read@(none) 14:56  mysql>create database db2;
ERROR 1044 (42000): Access denied for user 'read'@'%' to database 'db2'
read@(none) 14:56  mysql>use ly
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
read@ly 14:56  mysql>select * from  t1;
+----+------+
| id | name |
+----+------+
|  1 | cali |
|  2 | ly   |
|  3 | xx   |
|  4 | ada  |
|  5 | dad  |
|  6 | asd  |
+----+------+
6 rows in set (0.01 sec)

read@ly 14:57  mysql>insert into t1 values(7,zx);
ERROR 1142 (42000): INSERT command denied to user 'read'@'192.168.78.135' for table 't1'

5. Build keepalived to achieve high availability

1. Install the keepalived software on both mysqlrouters:

[root@mysqlrouter1 ~]# yum install keepalived -y

2. Modify the configuration file:

mysqlrouter1 configuration:

[root@mysqlrouter1 keepalived]# vim keepalived.conf
! Configuration File for keepalived

global_defs {
    
    
   notification_email {
    
    
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   #vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    
    
    state MASTER
    interface ens33
    virtual_router_id 220
    priority 120
    advert_int 1
    authentication {
    
    
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
    
    
        192.168.78.220
    }
}

vrrp_instance VI_2 {
    
    
    state BACKUP
    interface ens33
    virtual_router_id 221
    priority 100
    advert_int 1
    authentication {
    
    
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
    
    
        192.168.78.221
    }
}
[root@mysqlrouter1 keepalived]# service keepalived restart
Redirecting to /bin/systemctl restart keepalived.service

mysqlrouter2 configuration:

[root@mysqlrouter2 keepalived]# vim keepalived.conf
! Configuration File for keepalived

global_defs {
    
    
   notification_email {
    
    
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   #vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    
    
    state BACKUP
    interface ens33
    virtual_router_id 220
    priority 100
    advert_int 1
    authentication {
    
    
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
    
    
        192.168.78.220
    }
}
vrrp_instance VI_2 {
    
    
    state MASTER
    interface ens33
    virtual_router_id 221
    priority 120
    advert_int 1
    authentication {
    
    
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
    
    
        192.168.78.221
    }
}
[root@mysqlrouter2 keepalived]# service keepalived restart
Redirecting to /bin/systemctl restart keepalived.service

Start 2 vrrp instances. Enable 2 vrrp instances on each machine. One is the master and the other is the backup. Enable 2 VIPs. There will be a VIP on each machine. Both VIPs provide services to the outside world. This way It can avoid the situation of single VIP, one is very busy and the other is very idle. Can improve equipment usage.

6. Set up Prometheus+grafana monitoring

1. Install prometheus server

#上传下载的源码包到linux服务器
[root@prometheus ~]# mkdir /prom
[root@prometheus ~]# cd /prom
[root@prometheus prom]# ls
prometheus-2.34.0.linux-amd64.tar.gz
#解压源码包
[root@prometheus prom]# tar xf prometheus-2.34.0.linux-amd64.tar.gz
[root@prometheus prom]# ls
prometheus-2.34.0.linux-amd64  prometheus-2.34.0.linux-amd64.tar.gz
[root@prometheus prom]# mv prometheus-2.34.0.linux-amd64 prometheus
[root@prometheus prom]# ls
prometheus  prometheus-2.34.0.linux-amd64.tar.gz
#临时和永久修改PATH变量，添加prometheus的路径
[root@prometheus prometheus]# PATH=/prom/prometheus:$PATH		#临时
[root@prometheus prometheus]# cat /root/.bashrc				
PATH=/prom/prometheus:$PATH   #添加
#执行prometheus程序
[root@prometheus prometheus]# nohup prometheus  --config.file=/prom/prometheus/prometheus.yml &
[1] 8431
[root@prometheus prometheus]# nohup: 忽略输入并把输出追加到"nohup.out"

2. Make prometheus a service for management

[root@prometheus prometheus]# vim /usr/lib/systemd/system/prometheus.service 
[Unit]
Description=prometheus
[Service]
ExecStart=/prom/prometheus/prometheus --config.file=/prom/prometheus/prometheus.yml
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
[Install]：
WantedBy=multi-user.target
#重新加载systemd相关的服务
[root@prometheus prometheus]# systemctl daemon-reload
[root@prometheus prometheus]#  service prometheus start
[root@prometheus system]# ps aux|grep prometheu
root       7193  2.0  4.4 782084 44752 ?        Ssl  13:16   0:00 /prom/prometheus/prometheus --config.file=/prom/prometheus/prometheus.yml
root       7201  0.0  0.0 112824   972 pts/1    S+   13:16   0:00 grep --color=auto prometheu

3. Execute the Prometheus program

[root@prometheus prometheus]# nohup prometheus  --config.file=/prom/prometheus/prometheus.yml &
[1] 1543
[root@prometheus prometheus]# nohup: 忽略输入并把输出追加到"nohup.out"

[root@prometheus prometheus]# 
[root@prometheus prometheus]# service prometheus restart
Redirecting to /bin/systemctl restart prometheus.service
[root@prometheus prometheus]# netstat -anplut|grep prome
tcp6       0      0 :::9090                 :::*                    LISTEN      1543/prometheus     
tcp6       0      0 ::1:9090                ::1:42776               ESTABLISHED 1543/prometheus     
tcp6       0      0 ::1:42776               ::1:9090                ESTABLISHED 1543/prometheus   

4. Install the exporter program on the node server

Download the node_exporter-1.4.0-rc.0.linux-amd64.tar.gz source code and upload it to the node server:

 wget https://github.com/prometheus/node_exporter/releases/download/v1.4.0/node_exporter-1.4.0.linux-amd64.tar.gz

Unzip it and store it separately in the /node_exporter folder:

1.下载node_exporter-1.4.0-rc.0.linux-amd64.tar.gz源码，上传到节点服务器上
2.解压
[root@mysql-master ~]# ls
anaconda-ks.cfg  node_exporter-1.4.0-rc.0.linux-amd64.tar.gz
[root@mysql-master ~]# tar xf node_exporter-1.4.0-rc.0.linux-amd64.tar.gz
[root@mysql-master ~]# ls
node_exporter-1.4.0-rc.0.linux-amd64         
node_exporter-1.4.0-rc.0.linux-amd64.tar.gz  
单独存放到/node_exporter文件夹
[root@mysql-master ~]# mv node_exporter-1.4.0-rc.0.linux-amd64 /node_exporter
[root@mysql-master ~]# cd /node_exporter/
[root@mysql-master node_exporter]# ls
LICENSE  node_exporter  NOTICE
[root@mysql-master node_exporter]#

# 修改PATH变量
[root@mysql-master node_exporter]# PATH=/node_exporter/:$PATH
[root@mysql-master node_exporter]# vim /root/.bashrc 
[root@mysql-master node_exporter]# tail -1 /root/.bashrc 
PATH=/node_exporter/:$PATH

# 执行node exporter 代理程序agent
[root@mysql-master node_exporter]# nohup node_exporter --web.listen-address 0.0.0.0:8090  &
[root@mysql-master node_exporter]# ps aux | grep node_exporter 
root      64281  0.0  2.1 717952 21868 pts/0    Sl   19:03   0:04 node_exporter --web.listen-address 0.0.0.0:8090
root      82787  0.0  0.0 112824   984 pts/0    S+   20:46   0:00 grep --color=auto node_exporter
[root@mysql-master node_exporter]# netstat -anplut | grep 8090
tcp6       0      0 :::8090                 :::*                    LISTEN      64281/node_exporter 
tcp6       0      0 192.168.17.152:8090     192.168.17.156:43576    ESTABLISHED 64281/node_exporter 

5. Add the exporter program in prometheus server

[root@prometheus prometheus]# vim prometheus.yml 
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "master"
    static_configs:
      - targets: ["192.168.78.132:8090"]
  - job_name: "delay_backup"
    static_configs:
      - targets: ["192.168.78.148:8090"]
  - job_name: "slave1"
    static_configs:
      - targets: ["192.168.78.149:8090"]
  - job_name: "slave2"
    static_configs:
      - targets: ["192.168.78.150:8090"]
  - job_name: "mysqlrouter1"
    static_configs:
      - targets: ["192.168.78.135:8090"]
  - job_name: "mysqlrouter2"
    static_configs:
      - targets: ["192.168.78.136:8090"]
[root@prometheus prometheus]# service  prometheus restart
Redirecting to /bin/systemctl restart prometheus.service

View the effect:

6. Deploy grafana

Grafana is an open source data visualization and analysis platform that supports query and display of multiple data sources.

[root@prometheus grafana]# wget https://dl.grafana.com/enterprise/release/grafana-enterprise-9.1.2-1.x86_64.rpm
[root@prometheus grafana]# yum install grafana-enterprise-9.1.2-1.x86_64.rpm -y
[root@prometheus grafana]# service grafana-server start
Starting grafana-server (via systemctl):                   [  确定  ]
#设置开机启动
[root@prometheus grafana]# systemctl enable grafana-server
Created symlink from /etc/systemd/system/multi-user.target.wants/grafana-server.service to /usr/lib/systemd/system/grafana-server.service.
#查看端口号
[root@prometheus grafana]# netstat -anplut|grep grafana
tcp6       0      0 :::3000                 :::*                    LISTEN      2187/grafana-server 

Enter the website and log in to grafana:
Log in to view http://192.168.78.147:3000/login:

The default username and password are username admin and password admin

Configure the data source of prometheus (click Settings>Add data source>Prometheus), fill in the URL, then save, import the grafana template (click import under Dashboards), fill in the template number, and get the following rendering:

7. Use sysbench to stress test the cluster

1.Install sysbench

[root@test ~]# yum install epel-release -y
[root@test ~]# yum install sysbench -y

2. Test

First connect to the write port of mysqlrouter on test, and create the database sbtest required by sysbench in advance (the default library name used by sysbench).

[root@slave1 node_exporter]# mysql -uwrite -p"123456" -h 192.168.78.135 -P 8889
write@(none) 17:48  mysql>create database sbtest;
Query OK, 1 row affected (0.01 sec)

data preparation:

#It --tables=10means creating 10 test tables, --table_size=100000which means inserting 10,000 rows of data into each table

[root@test ~]# sysbench --mysql-host=192.168.78.135 --mysql-port=8889 --mysql-user=write --mysql-password='123456' /usr/share/sysbench/oltp_common.lua  --tables=10  --table_size=10000 prepare
sysbench 1.0.17 (using system LuaJIT 2.0.4)

Creating table 'sbtest1'...
Inserting 10000 records into 'sbtest1'
Creating a secondary index on 'sbtest1'...
Creating table 'sbtest2'...
Inserting 10000 records into 'sbtest2'
Creating a secondary index on 'sbtest2'...
Creating table 'sbtest3'...
Inserting 10000 records into 'sbtest3'
Creating a secondary index on 'sbtest3'...
Creating table 'sbtest4'...
Inserting 10000 records into 'sbtest4'
Creating a secondary index on 'sbtest4'...
Creating table 'sbtest5'...
Inserting 10000 records into 'sbtest5'
Creating a secondary index on 'sbtest5'...
Creating table 'sbtest6'...
Inserting 10000 records into 'sbtest6'
Creating a secondary index on 'sbtest6'...
Creating table 'sbtest7'...
Inserting 10000 records into 'sbtest7'
Creating a secondary index on 'sbtest7'...
Creating table 'sbtest8'...
Inserting 10000 records into 'sbtest8'
Creating a secondary index on 'sbtest8'...
Creating table 'sbtest9'...
Inserting 10000 records into 'sbtest9'
Creating a secondary index on 'sbtest9'...
Creating table 'sbtest10'...
Inserting 10000 records into 'sbtest10'
Creating a secondary index on 'sbtest10'...

Start running data and perform stress testing:

[root@test sysbench-1.0.17]# sysbench --threads=4 --time=20 --report-interval=5 --mysql-host=192.168.78.135 --mysql-port=8889 --mysql-user=write --mysql-password='123456' /usr/share/sysbench/oltp_read_write.lua  --tables=10  --table_size=10000 run

Some problems encountered during the project:

1. It is easy to make mistakes when modifying the configuration file, resulting in failure to start. If you do not know the specific situation, you can check the log;
2. When setting up delayed backup, I encountered: Starting MySQL... ERROR! The server quit without updating PID file, I looked up the data and found that it was a problem with permissions, ports or configuration, but I tried every time and found that it was wrong, and the logs were ambiguous. Finally, I uninstalled and reinstalled. Maybe I made an error during configuration;
3. keepalived split brain situation, check configuration files and firewall issues;
4. After modifying the host name, mysql restart fails. Kill all mysqld processes and then restart mysql.

Project experience

Planning the architecture of the entire cluster in advance can improve the efficiency of project development. When
running errors, read more error message prompts and logs, which is very helpful.
Have a deeper understanding of master-slave replication based on GTID + semi-synchronization.
Keepalived's split-brain and VIP I also have a deeper understanding and analysis of the drift phenomenon and
realize the convenience and benefits of rsync+sersync data synchronization tool.