mysql双主高可用架构+keepalived+lvs配置说明
1.1. 环境简介
1.1.1、vmvare虚拟机，系统版本CentOS7.5 x86_64位最小化安装，mysql的版本5.7.21，
1.1.2、虚拟机器的ssh端口均为默认22,
1.1.3、虚拟机的iptables全部关闭，
1.1.4、虚拟机的selinux全部关闭，
1.1.5、虚拟机服务器时间全部一致 ntpdate 0.asia.pool.ntp.org
1.1.6、3台机器的ssh端口为22
1.2、此次试验采用的是3台机器，机器具体部署如下：
角色 IP地址（内网） 主机名称 节点机器部署服务 业务用途
Mater 192.168.20.51 db50 master01
写入(keepalived)
Master 192.168.20.20 db51 master02
写入(keepalived)
--------------------------------------------------------------------------------------------
slave(备master) 192.168.10.57 db57 slave
Lvs01 192.168.20.51 lvs51 lvs+keepalived
从库负载均衡
Lvs02 192.168.20.20 lvs52 lvs+keepavlied
从库负载均衡
Mysql write vip： 192.168.20.151
1.3 系统架构图：
1.4 架构实现原理：
能够保证数据的冗余的同时可以做读写分离来分担系统压力，如果是主主复制还可以很好的避免主节点的单点故障。但是MySQL主主复制存在一些问题无法满足我们的实际需要：未提供统一访问入口来实现负载均衡，如果其中master宕掉的话需要手动切换到另外一个master，而不能自动进行切换。
这篇文章下面要介绍如何通过LVS+Keepalived的方式来是实现MySQL的高可用性，同时解决以上问题。
Keepalived
是一个基于VRRP（虚拟路由冗余协议）可用来实现服务高可用性的软件方案，避免出现单点故障。Keepalived一般用来实现轻量级高可用性，且不需要共享存储，一般用于两个节点之间，常见有LVS+Keepalived、Nginx+Keepalived组合。
LVS
(Linux Virtual Server)是一个高可用性虚拟的服务器集群系统。本项目在1998年5月由章文嵩博士成立，是中国国内最早出现的自由软件项目之一。LVS主要用于多服务器的负载均衡，作用于网络层。LVS构建的服务器集群系统中，前端的负载均衡层被称为Director Server；后端提供服务的服务器组层被称为Real Server。通过下图可以大致了解LVS的基础架构。
LVS有三种工作模式，分别是DR（Direct Routing 直接路由）、TUN(Tunneling IP隧道）、NAT（Network Address Translation 网络地址转换）。其中TUN模式能够支持更多的Real Server，但需要所有服务器支持IP隧道协议；DR也可以支持相当的Real Server，但需要保证Director Server虚拟网卡与物理网卡在同一网段；NAT扩展性有限，无法支持更多的Real Server，因为所有的请求包和应答包都需要Director Server进行解析再生，影响效率。 同时，LVS负载均衡有10中调度算法，分别是rr、wrr、lc、wlc、lblc、lblcr、dh、sh、sed、nq（详细介绍本文不在说明）
本文中将利用LVS实现MySQL的读写负载均衡，Keepalived避免节点出现单点故障。
1.5 安装mysql
双Mater和slave都需要安装
wget http://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpm
yum localinstall mysql57-community-release-el7-8.noarch.rpm
yum install mysql-community-server -y
systemctl enable mysqld
systemctl daemon-reload
systemctl start mysqld
systemctl status mysqld
grep 'temporary password' /var/log/mysqld.log
mysql -uroot -p
service mysqld restart
192.168.20.51
vim /etc/my.cnf
[client]
port = 3306
#socket = /tmp/mysql.sock
socket=/var/lib/mysql/mysql.sock
default-character-set=utf8
[mysqld]
port = 3306
socket=/var/lib/mysql/mysql.sock
datadir=/var/lib/mysql
pid-file=/var/run/mysqld/mysqld.pid
log-error=/var/log/mysqld.log
#每个server上不一致，见规划
user = mysql
bind-address = 0.0.0.0
init-connect = 'SET NAMES utf8mb4'
character-set-server = utf8mb4
explicit_defaults_for_timestamp = 1
lower_case_table_names=1
skip-name-resolve
back_log = 300
max_connections = 1000
max_connect_errors = 600000
open_files_limit = 65535
max_allowed_packet = 16M
max_heap_table_size = 32M
tmp_table_size = 32M
read_buffer_size = 16M
read_rnd_buffer_size = 32M
sort_buffer_size = 8M
join_buffer_size = 8M
key_buffer_size = 8M
thread_cache_size = 512
query_cache_type = 0
query_cache_size = 0
#query_cache_limit = 2M
ft_min_word_len = 4
server-id = 1
log_bin = mysql-bin
relay-log=relog
log_bin = mysql-bin
relay-log=relog
relay_log_purge=on
binlog_format = ROW
expire_logs_days = 60
log-slave-updates=1
slave-skip-errors=all
sync_binlog=1
auto_increment_increment=2
auto_increment_offset=2
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=16
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON
gtid-mode=on
enforce-gtid-consistency=true
plugin-load=rpl_semi_sync_master=semisync_master.so
rpl_semi_sync_master_enabled=1
rpl_semi_sync_master_timeout=10000
plugin-load=rpl_semi_sync_slave=semisync_slave.so
rpl_semi_sync_slave_enabled=1
slow_query_log = 1
long_query_time = 1
slow_query_log_file = /var/lib/mysql/mysql-slow.log
performance_schema = 0
skip-external-locking
default_storage_engine = InnoDB
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 128M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 18G
bulk_insert_buffer_size = 8M
myisam_sort_buffer_size = 8M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
interactive_timeout = 28800
wait_timeout = 28800
interactive_timeout = 28800
wait_timeout = 28800
[mysqldump]
quick
max_allowed_packet = 16M
[myisamchk]
key_buffer_size = 8M
sort_buffer_size = 8M
read_buffer = 4M
write_buffer = 4M
192.168.20.20
vim /etc/my.cnf
[client]
port = 3306
#socket = /tmp/mysql.sock
socket=/var/lib/mysql/mysql.sock
default-character-set=utf8
[mysqld]
port = 3306
#socket = /tmp/mysql.sock
socket=/var/lib/mysql/mysql.sock
default_password_lifetime=0
datadir=/var/lib/mysql
pid-file=/var/run/mysqld/mysqld.pid
log-error=/var/log/mysqld.log
#每个server上不一致，见规划
server-id = 2
log-bin=master-log
relay-log=relay-log
relay_log_purge=0
skip_name_resolve=1
innodb_file_per_table=1
user = mysql
bind-address = 0.0.0.0
init-connect = 'SET NAMES utf8mb4'
character-set-server = utf8mb4
explicit_defaults_for_timestamp = 1
lower_case_table_names=1
skip-name-resolve
range_optimizer_max_mem_size=0
back_log = 300
max_connections = 1000
max_connect_errors = 600000
open_files_limit = 65535
max_allowed_packet = 16M
max_heap_table_size = 32M
tmp_table_size = 32M
read_buffer_size = 16M
read_rnd_buffer_size = 32M
sort_buffer_size = 8M
join_buffer_size = 8M
key_buffer_size = 8M
query_cache_type = 0
query_cache_size = 0
query_cache_type = 0
query_cache_size = 0
ft_min_word_len = 4
server-id = 2
log_bin = mysql-bin
binlog_format = ROW
expire_logs_days = 60
log-slave-updates
slave-skip-errors=all
sync_binlog=1
auto_increment_increment=2
auto_increment_offset=1
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=16
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON
gtid-mode=on
enforce-gtid-consistency=true
plugin-load=rpl_semi_sync_master=semisync_master.so
rpl_semi_sync_master_enabled=1
rpl_semi_sync_master_timeout=10000
plugin-load=rpl_semi_sync_slave=semisync_slave.so
rpl_semi_sync_slave_enabled=1
slow_query_log = 1
long_query_time = 1
slow_query_log_file = /var/mysql/data/mysql-slow.log
performance_schema = 0
skip-external-locking
default_storage_engine = InnoDB
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 128M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 6G
bulk_insert_buffer_size = 8M
myisam_sort_buffer_size = 8M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
interactive_timeout = 28800
interactive_timeout = 28800
wait_timeout = 28800
[mysqldump]
quick
max_allowed_packet = 16M
[myisamchk]
key_buffer_size = 8M
sort_buffer_size = 8M
read_buffer = 4M
write_buffer = 4M
192.168.10.57
vim /etc/my.cnf
[client]
port = 3306
socket=/var/lib/mysql/mysql.sock
default-character-set=utf8
[mysqld]
port = 3306
socket=/var/lib/mysql/mysql.sock
default_password_lifetime=0
datadir=/var/lib/mysql
pid-file=/var/run/mysqld/mysqld.pid
#每个server上不一致，见规划
user = mysql
bind-address = 0.0.0.0
init-connect = 'SET NAMES utf8mb4'
character-set-server = utf8mb4
explicit_defaults_for_timestamp = 1
lower_case_table_names=1
skip-name-resolve
range_optimizer_max_mem_size=0
back_log = 500
max_connect_errors = 600000
open_files_limit = 65535
max_allowed_packet = 16M
max_heap_table_size = 32M
tmp_table_size = 32M
read_buffer_size = 16M
read_rnd_buffer_size = 32M
sort_buffer_size = 8M
join_buffer_size = 8M
key_buffer_size = 8M
query_cache_type = 0
query_cache_size = 0
ft_min_word_len = 4
log_bin = mysql-bin
relay-log=relay-log
relay_log_purge=0
binlog_format = ROW
expire_logs_days = 7
server-id = 3
log-slave-updates
slave-skip-errors=all
sync_binlog=1
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=16
master_info_repository=TABLE
slave-parallel-workers=16
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON
binlog-ignore-db=mysql
log-error=/var/log/mysqld.log
#gtid-mode=on
#enforce-gtid-consistency=true
slow_query_log = 1
long_query_time = 1
slow_query_log_file = /var/mysql/data/mysql-slow.log
read-only = 1
performance_schema = 0
skip-external-locking
default_storage_engine = InnoDB
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 128M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 6G
bulk_insert_buffer_size = 8M
myisam_sort_buffer_size = 8M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
interactive_timeout = 28800
wait_timeout = 28800
[mysqldump]
quick
max_allowed_packet = 16M
[myisamchk]
key_buffer_size = 8M
sort_buffer_size = 8M
read_buffer = 4M
write_buffer = 4M
第二：新建用户repl_user设置密码123456
重置root密码：123456
#更改密码
mysql -u root -p
#更改密码策略
set global validate_password_policy=0;
set global validate_password_length=4;
SET PASSWORD = PASSWORD('123456');
GRANT ALL PRIVILEGES ON *.* TO 'root'@'192.168.20.%' IDENTIFIED BY '123456' WITH GRANT OPTION;
flush privileges;
四、半同步复制安装
主从复制授权：
第三：在 3 个 mysql 节点做授权配置(主从复制授权)
Master和备库:
grant replication slave,replication client on *.* to 'repluser'@'192.168.20.%' identified by '123456' ;
flush privileges;
mysql> show master status;
+-------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-------------------+----------+--------------+------------------+-------------------+
| master-log.000005 | 154 | | | |
+-------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
slave（两个从库）
#配置主从复制起点
change master to master_host='192.168.20.251',master_user='repluser',master_password='123456',master_log_file='master-log.000005',master_log_pos=154;
start slave;
show slave status\G
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
mysql> set global read_only=1; #查看slave IO和slave sql是否都正常
#查看主从复制情况
show grants for 'repluser'@'192.168.20.%';
mysql> flush privileges; #刷新权限
Query OK, 0 rows affected (0.00 sec)
删除多余用户
mysql> drop user root@'localhost';
mysql> select user,host from mysql.user;
主从切换后这里需要修改。并查看
3、开启master(server2)
mysql> INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';
mysql> SET GLOBAL rpl_semi_sync_master_enabled = 1;
mysql> show status like 'Rpl_semi_sync_master_status'; ###查询主是否开启成功
+-----------------------------+-------+
| Variable_name | Value |
+-----------------------------+-------+
| Rpl_semi_sync_master_status | ON |
+-----------------------------+-------+
4、开启slave(serve3、4)
mysql> INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';
mysql> SET GLOBAL rpl_semi_sync_slave_enabled = 1;
mysql> STOP SLAVE IO_THREAD;
mysql> START SLAVE IO_THREAD;
mysql> show status like 'Rpl_semi_sync_slave_status'; ###查询从是否开启成功
+----------------------------+-------+
| Variable_name | Value |
+----------------------------+-------+
| Rpl_semi_sync_slave_status | ON |
+----------------------------+-------+
---------------------
参考：https://blog.csdn.net/Dream_ya/article/details/80379184
1.6 ssh授信
配置三台机器的ssh互信（三台都要操作）
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
#测试是否成功
ssh 192.168.20.20 date
1.7 安装配置LVS+keepalived
192.168.20.251（backup_master）和192.168.20.51（slave）
vim /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_tw_recycle = 0
net.ipv4.conf.lo.arp_ignore= 1
net.ipv4.conf.lo.arp_announce= 2
net.ipv4.conf.all.arp_ignore= 1
net.ipv4.conf.all.arp_announce= 2
sysctl -p
[root@db251 ~]# vi /etc/rc.d/init.d/realserver
#!/bin/sh
# chkconfig: - 80 90
# description:realserver
# mysql_vip start realserver
mysql_vip=192.168.20.151
. /etc/rc.d/init.d/functions
case "$1" in
start)
ifconfig lo:0 $mysql_vip netmask 255.255.255.255 broadcast $mysql_vip
/sbin/route add -host $mysql_vip dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1
echo "RealServer Start OK"
;;
stop)
ifconfig lo:0 down
route del $mysql_vip >/dev/null 2>&1
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "RealServer Stoped"
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0
chmod +x /etc/rc.d/init.d/realserver #添加脚本执行权限
chkconfig realserver on #添加开机启动
/etc/rc.d/init.d/realserver start #开启，参数stop为关闭
Lvs01-192.168.20.51,配置文件51和52一致
[root@lvs51 ~]# yum install ipvsadm keepalived gcc gcc-c++ make openssl-devel kernel-devel ncurses-devel -y
[root@lvs51 ~]# lsmod |grep ip_vs
ip_vs 141432 0
nf_conntrack 133053 1 ip_vs
libcrc32c 12644 3 xfs,ip_vs,nf_conntrack
[root@lvs51 ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf-bak
[root@lvs51 ~]# >/etc/keepalived/keepalived.conf
[root@lvs51 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script check_run {
script "/etc/keepalived/mysql_check.sh"
interval 3
}
vrrp_instance VI_1 {
state MASTER #从BACKUP
interface em1
virtual_router_id 51
priority 100 #从90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
check_run
}
virtual_ipaddress {
192.168.20.151/24
}
}
virtual_server 192.168.20.151 3307 {
delay_loop 6
lb_algo wlc
lb_kind DR
#persistence_timeout 120
protocol TCP
real_server 192.168.20.51 3306 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
}
real_server 192.168.20.20 3306 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
}
[root@lvs51 keepalived]# pwd
/etc/keepalived
[root@lvs51 keepalived]# ll
total 12
-rw-r--r-- 1 root root 816 Jan 11 11:05 check_slave.py
-rw-r--r-- 1 root root 1489 Jan 11 11:35 keepalived.conf
-rw-r--r-- 1 root root 3598 Jan 11 10:01 keepalived.conf-bak
主、主备的以下配置一致
vim /etc/keepalived/mysql_check
#!/bin/bash
#########################################################################
# File Name: mysql_check.sh
# Author: LookBack
# Email: admin#dwhd.org
# Version:
# Created Time: 2016年01月08日 星期五 05时12分56秒
#########################################################################
username=root
password=123456
bindir="/usr/bin"
mysql_check() { ${bindir}/mysql -u${username} -p${password} -e "select version();"; }
mysql_pid_check() { ps aux | pgrep mysqld; }
while :; do
[ -n "$(mysql_check)" ] && [ -n "$(mysql_pid_check)" ] && exit 0
[ -n "$(mysql_check)" ] && exit 0 || {
[[ "$i" -gt "4" ]] \
&& break \
|| { let i++ && sleep 0.3 && continue; }
}
done
service keepalived stop
[root@lvs51 keepalived]# systemctl start keepalived
[root@lvs51 keepalived]# systemctl status keepalived
[root@lvs51 keepalived]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.20.151:3307 wlc
-> 192.168.20.51:3306 Route 1 0 0
-> 192.168.20.20:3306 Route 1 0 0
1.8 测试
测试 read vip 负载均衡
分别在从库 db51、 db52、 创建可区分的库，库名分别为
read_one、 read_two
# db51
mysql> create database read_one; Query OK, 1 row affected (0.00 sec)
mysql> show databases like 'read_one';
+---------------------+
| Database (read_one) |
+---------------------+
| read_one |
+---------------------+
1 row in set (0.00 sec)
# db51 操作略
[root@db251 ~]# mysql -uroot -p123456 -h 192.168.20.151 -P3306 -e "show databases"|egrep -v "*schema|mysql|test|Database"
mysql: [Warning] Using a password on the command line interface can be insecure.
read_one
read_two
sys
[root@db251 ~]# mysql -uroot -p123456 -h 192.168.20.151 -P3306 -e "show databases"|egrep -v "*schema|mysql|test|Database"
mysql: [Warning] Using a password on the command line interface can be insecure.
read_one
sys
测试从库故障被剔除，恢复被挂起
将 db52 数据库关闭，观察 lvs 状态
#lvs 未关闭 mysql 时的状态
[root@master1 keepalived]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.20.151:3306 wlc
-> 192.168.20.51:3306 Route 1 0 0
-> 192.168.20.52:3306 Route 1 0 0
[root@master1 keepalived]# msyql -uroot -h 192.168.20.151 -p
-bash: msyql: command not found
[root@master1 keepalived]# mysql -uroot -h 192.168.20.151 -p
Enter password:
^Z
[1]+ Stopped mysql -uroot -h 192.168.20.151 -p
#lvs 关闭 mysql 后的状态
[root@master1 keepalived]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.20.151:3306 wlc
-> 192.168.20.51:3306 Route 1 0 0
注：数据库开启后，会自动加入到节点中
[root@master1 keepalived]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.20.151:3306 wlc
-> 192.168.20.51:3306 Route 1 0 0
-> 192.168.20.52:3306 Route 1 0 0
测试 keepalived 高可用 vip 切换
192.168.20.51
[root@lvs51 keepalived]# systemctl stop keepalived
192.168.20.52
[root@lvs52 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.20.151:3306 wlc
-> 192.168.20.51:3306 Route 1 0 0
-> 192.168.20.52:3306 Route 1 0 0
192.168.20.52
[root@lvs52 keepalived]# systemctl stop keepalived
192.168.20.51
[root@lvs51 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.20.151:3306 wlc
-> 192.168.20.51:3306 Route 1 0 0
-> 192.168.20.52:3306 Route 1 0 0
测试mysql能否远程登录
特别注意：双主数据库需要给192.168.20.151设置登录权限。