Build one of the high-performance database clusters: MySQL master-slave replication (one master, one slave, dual master and dual slave)

I. Overview

1. Database master-slave concepts, advantages, and uses
　　What does master-slave database mean? Master means the master database, and slave means the slave database. The main database provides read and write operations to the outside world, and the slave database provides read operations to the outside world.

Why does the database need a master-slave architecture?
　　High availability, real-time disaster recovery, and failover. For example, if the master database is down, you can switch to the slave database. Read and write are separated to provide query services, reduce the pressure on the main database, improve performance and backup data to avoid affecting business.
　　
2. Database master-slave replication principle
The master-slave replication principle, in short, is carried out in three steps:
① The master database has a binlog binary file, which records all additions, deletions and modifications of SQL statements;
② (binlog thread) The slave database transfers the data of the master database to Copy the SQL statements of the binlog file to its own relay log relaylog;
③ (io thread) redo the log file from the relaylog of the database and execute these SQL statements again.
(SQL execution thread) The detailed master-slave replication process is as follows:

2. Install mysql database

The following work needs to be completed on both the master and slave servers:

1. Create a local working directory : /usr/rdc/mysql-8.0.23 and the folders conf, logs, and data under it; and authorize the working directory

cd /usr/rdc/mysql-8.0.23
chmod -R 777 data

2. Install mysql database (based on docker)

docker search mysql
docker pull mysql:8.0.23

3. Obtain the location of mysql configuration files, log files, and data files

#step 1.启一个该版本mysql的容器
docker run -p 3306:3306 --name mysql -e MYSQL_ROOT_PASSWORD=root -d mysql:8.0.23
#step 2.进入容器查找配置文件、日志文件、数据文件位置
docker ps      #查看容器id
docker exec -it mysql /bin/bash
find / -name "my.cnf"
find / -name "log"

#step 3.退出容器，将容器中/etc/mysql/my.cnf文件拷贝到宿主机目录：/usr/rdc/mysql-8.0.23/conf/
exit
docker cp mysql:/etc/mysql/my.cnf  /usr/rdc/mysql-8.0.23/conf
#step 4.打开配置文件查看
vi /usr/rdc/mysql-8.0.23/conf/my.cnf

It can be seen from the configuration file: the data file location is /var/lib/mysql, the custom configuration file can be placed in the /etc/mysql/conf.d directory of the container, and the log file directory is /var/log. Remember the above three files
. Location, one-to-one correspondence is required when creating a container for data volume mapping below.

#退出容器，并销毁
exit
docker stop mysql
docker rm mysql

2. Configure the master library

1. Edit the my.cnf file in the main server

vi /usr/rdc/mysql-8.0.23/conf/my.cnf

2. Add the following content to the [mysqld] node of the configuration file

[mysqld]
# 设置数据库引擎为INNODB
default-storage-engine=INNODB
# 设置授权访问的加密策略
default_authentication_plugin=mysql_native_password

# 主从复制配置.start
# 服务器ID
server-id=2013306
# 启用二进制日志
log-bin=master-bin
# 设置logbin格式：STATEMENT(同步SQL脚本) / ROW(同步数据行) / MIXED(混合同步)
binlog_format=MIXED
# 设置日志最长保存时间
expire_logs_days=30
# 0-读写，1-只读
read-only=0

# 设置忽略同步的数据库
binlog-ignore-db=information_schema
binlog-ignore-db=mysql
binlog-ignore-db=performance_schema
binlog-ignore-db=sys

# 设置需要同步的数据库
#binlog-do-db=pmonitor
#binlog-do-db=ucoal

# 主从复制配置.end

3. Create a container (mapped data volume)

docker run -p 3306:3306 --privileged=true --restart=always --name mysql -v /usr/rdc/mysql-8.0.23/conf:/etc/mysql/conf.d -v /usr/rdc/mysql-8.0.23/logs:/var/log/mysql -v /usr/rdc/mysql-8.0.23/data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=root -d mysql:8.0.23

4. Enter the container to access mysql

#查看容器id
docker ps

#进入容器
docker exec -it 容器id /bin/bash

#访问mysql数据库
mysql -uroot -p

5. Create an account used by the slave to access the main library

#创建账号
create user 'slave1'@'%' identified by 'slave1';
#授权
grant replication slave on *.* to 'slave1'@'%';
#更新用户密码方案（一定要执行否则无法远程访问）
alter user 'slave1'@'%' identified with mysql_native_password by 'slave1';

#刷新
flush privileges;

6. Restart the container

docker restart 容器id或名称

7. Get the log file name and offset and
execute the following command:

show master status;

Remember the values ​​of the file and position fields , which are required when configuring the slave.

3. Configure the slave library

1. Edit the my.cnf file in the slave server

vi /usr/rdc/mysql-8.0.23/conf/my.cnf

2. Add the following content to the [mysqld] node of the configuration file

[mysqld]
# 设置数据库存储引擎为INNODB
default-storage-engine=INNODB
# 设置授权验证的加密策略
default_authentication_plugin=mysql_native_password

# 主从复制配置.start
# 服务器ID
server-id=2023306

# 启用中继日志
relay-log=slave-relay-bin
relay-log-index=slave-relay-bin.index
# 设置日志最长保存时间
expire_logs_days=30
# 0-读写，1-只读；slave设置为只读（具有super权限的用户除外）
read_only=1

# 开启二进制日志功能，以便本机可以作为其它Slave的Master时使用
log-bin=slave-bin
# 设置logbin格式：STATEMENT(同步SQL脚本) / ROW(同步数据行) / MIXED(混合同步)
binlog_format=MIXED
# 1表示slave将复制事件写进自己的二进制日志
log_slave_updates=1

# 设置允许复制的库
# replicate-do-db=pmonitor-cloud
# replicate-do-db=ucoal

# 设置忽略复制的库
# replicate-ignore-db=mysql
# replicate-ignore-db=information_schema
# replicate-ignore-db=performance_schema

#主从复制配置.end

3. Create a container (mapped data volume)

docker run -p 3306:3306 --privileged=true --restart=always --name mysql -v /usr/rdc/mysql-8.0.23/conf:/etc/mysql/conf.d -v /usr/rdc/mysql-8.0.23/logs:/var/log/mysql -v /usr/rdc/mysql-8.0.23/data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=root -d mysql:8.0.23

4. Enter the container to access mysql

#查看容器id
docker ps

#进入容器
docker exec -it 容器id /bin/bash

#访问mysql数据库
mysql -uroot -p

5. Main library parameter association settings

#先停止从服务器线程
stop slave;

change master to master_host='192.168.0.201', master_port=3306, master_user='slave1', master_password='slave1', master_log_file='master-bin.000001', master_log_pos=156, get_master_public_key=1;

Note: You need to replace the values ​​of master_log_file and master_log_pos in the above instructions with the file and postion queried through show master status in the main library.

#启动从服务器线程
start slave;

6. Restart the container

docker restart mysql

7. Check the slave status

show slave status \G;

Use the above command to check the status. When the two parameters Slave_IO_Running and Slave_SQL_Running are both YES, it means that the cluster status is normal.
Otherwise, you need to troubleshoot based on the error information of Last_IO_Error or Last_SQL_Error.

4. Notes on pitfalls

When we set the permissions of the custom my.cnf configuration file to be too large, it will cause mysql to ignore the loading of the configuration file for security reasons during execution, thus making the custom configuration invalid (including: cluster server-id, The definitions of binlog logs and relaylog logs are all invalid). This problem once troubled Brother Liang for a long time. Later, he discovered the problem by checking the docker log. Make a note below:

0.Bury pit

#设置配置文件权限为777
chmod -R 777 /usr/rdc/mysql-8.0.23/conf

#启动docker容器
docker start mysql

#查看docker日志
docker logs mysql

Found a line of warning: World-writable config file '/etc/mysql/conf.d/my.cnf' is ignored.
This sentence means that the configuration file that everyone can write has been ignored and loaded!

This file happens to be the custom configuration file we mapped when creating the container.
The best solution is to lower the permissions of the my.cnf configuration file and then restart the container.

chmod -R 644 /usr/rdc/mysql-8.0.23/conf/my.cnf
docker restart mysql

However, Brother Liang is a very curious person and wants to see what will happen to the cluster without reducing the permissions of my.cnf, and how to repair it and make it run normally:

1. Check the master-slave synchronization status

Log in to mysql on the slave machine and execute the following instructions:

show slave status \G;

The Slave_IO_Running status is NO. According to the error message, it can be seen that the server-id of the cluster is duplicated.

2. Log in to the main server to check whether the configuration takes effect.

#查看server-id
show variables like 'server_id';
#查看binlog日志文件名称
show global variables like '%log_bin%';

It was found that the server-id and binlog log files were not generated according to the configuration file requirements.

Solution:

#step 1.编辑容器中的配置文件/etc/mysql/my.cnf
vim /etc/mysql/my.cnf

#step 2.加入以下内容
#服务器ID
server-id=2013306

#启用二进制日志
log-bin=master-bin
#设置logbin格式：STATEMENT / ROW / MIXED
binlog_format=ROW

#step 3.重启容器
exit
docker restart mysql

The master server configuration has taken effect:

3. Log in to the slave server to check whether the configuration takes effect.

Similar to configuring the main server, check whether the server-id and relay-bin log files are consistent with the configuration file.

show variables like 'server_id';  
show global variables like '%relay_log%';

The query results are inconsistent with the configuration file.

Solution:

#step 1.编辑容器中的配置文件/etc/mysql/my.cnf
vim /etc/mysql/my.cnf

#step 2.加入以下内容
#服务器ID
server-id=2023306

#开启二进制日志功能，以便本机可以作为其它Slave的Master时使用
log-bin=slave-bin

#启用中继日志
relay-log=slave-relay-bin
relay-log-index=slave-relay-bin.index

#step 3.重启容器
exit
docker restart mysql

View configuration

show variables like 'server_id';
show global variables like '%relay_log%';
show global variables like '%log_bin%';

The configuration has taken effect:

View master-slave replication status

show slave status\G;

Master-slave replication is not started:

manually start it:

start slave;

Found that it cannot be started:

Solution:

#重置slave
reset slave;

#重新关联主服务器
change master to master_host='192.168.0.201', master_port=3306, master_user='slave1', master_password='slave1', master_log_file='master-bin.000001', master_log_pos=156, get_master_public_key=1;

#手动启用从机
start slave;

Perfect solution:

4. Other solutions
If the server IDs in the cluster are repeated, use the following commands to view the server_id on the master and slave machines respectively.

show variables like 'server_id';

It means that the server-id in the slave configuration file does not take effect.
It can be repaired in the following two ways:
① Change the server-id in the my.cnf file to server_id and restart the mysql container;
② Execute the following command on the slave mysql console:

stop slave;
set global server_id = 2023306;
start slave;

After the execution is completed, check the status of the slave machine. The service is normal.

5. Extension – dual master and dual slave configuration

Configure master-slave replication of the database, 4 databases, 2 hosts, 2 slaves.
Step 1: Host configuration.
Find the Mysql configuration file and modify the following content under [mysqld].

[mysqld]
server-id = 101  # 主服务器唯一ID

log-bin=自己本地的路径/data/mysqlbin  # 启用二进制日志，日志的存放地址

binlog_format=STATEMENT  # 二进制日志格式

binlog-ignore-db=mysql # 设置不要复制的数据库

binlog-do-db=需要复制的主数据库名字1  # 设置需要复制的数据库
binlog-do-db=需要复制的主数据库名字2  # 设置需要复制的数据库
binlog-do-db=需要复制的主数据库名字3  # 设置需要复制的数据库

log-slave-updates # 在作为从数据库的时候，有写入操作也要更新二进制日志文件

Another host is also configured in this way. Note that the server-id cannot be repeated.

Step 2: Slave configuration
Modify the my.cnf configuration file

[mysqld]
server-id = 201  # 从服务器唯一ID
relay-log=mysql-relay # 开启中继日志

The other slave machine is also configured in this way. Note that the server-id cannot be repeated.

Step 3: Restart the service and turn off the firewall

After changing the configuration file, restart the Mysql service

Turn off the firewall systemctl stop firewalld

Step 4: Create a user and authorize it to the slave machine

Create users on 2 hosts:

GRANT REPLICATION SLAVE ON *.* TO '用户名'@'从机器数据库IP 或者 % 所有' IDENTIFIED BY '密码';
flush privileges;

Query the status of the host:

show master status;
# 执行上面命令，得到 File(binlog日志) Position(接入点) Binlog_Do_DB(要复制的数据库) Binlog_IgnoreDB()

Configure the host that needs to be copied on the slave machine

Slave 1 copies host 1, slave 2 copies host 2,

Slave 1 executes:

CHANGE MASTER TO MASTER_HOST='主机1IP',MASTER_USER='主机用户',MASTER_PASSWORD='主机密码',
MASTER_LOG_FILE='binlog日志名字',MASTER_LOG_POS=具体的接入点值;

Execute from machine 2:

CHANGE MASTER TO MASTER_HOST='主机2IP',MASTER_USER='主机用户',MASTER_PASSWORD='主机密码',
MASTER_LOG_FILE='binlog日志名字',MASTER_LOG_POS=具体的接入点值;

If the operation fails and needs to be reconfigured, execute the following 2 commands.

stop slave;    # 停止同步操作
reset master;  # 重置主从配置

Then the two slave machines execute the following commands to enable synchronization

start slave;

Step 5: (Here comes the important point) The two hosts copy each other
and host 1 executes

CHANGE MASTER TO MASTER_HOST='主机2IP',MASTER_USER='主机用户',MASTER_PASSWORD='主机密码',
MASTER_LOG_FILE='binlog日志名字',MASTER_LOG_POS=具体的接入点值;

Host 2 executes

CHANGE MASTER TO MASTER_HOST='主机1IP',MASTER_USER='主机用户',MASTER_PASSWORD='主机密码',
MASTER_LOG_FILE='binlog日志名字',MASTER_LOG_POS=具体的接入点值;

Then the two hosts execute:

start slave;

Step 6: Check if successful

show slave status\G;  # 检查状态

If the field below the result is Yes, it means the configuration is successful.

• Slave_IO_Running: Yes
• Slave_SQL_Running: Yes