Comprehensively understand the new features of version 8.1.0: related operations of InnoDB Cluster read-only replicas.
Author: Miguel Araújo Senior Software Engineer/Kenny Gryp MySQL Product Director
Source of this article: Oracle MySQL official website blog
*Produced by Aikeson Open Source Community.
Preface
MySQL's first Innovation version 8.1.0 has been released, and we will launch MySQL InnoDB Cluster read-only replicas .
In MySQL InnoDB Cluster, the main purpose of the replica (Secondray) node is to switch over and replace the primary (primary) node when it fails (high availability). This is accomplished with the MySQL Group Replication (MGR) plug-in. Another use of replica nodes is to reduce the read-only workload of the primary node.
You can now add asynchronous replica nodes to your database topology. These replica nodes can be used for:
- Transfer read traffic from the primary node or other replica nodes to share their read pressure.
- Set up a replica node dedicated to reading.
- Set up replica nodes dedicated to specific purposes such as reporting.
- By adding multiple read replica nodes, expansion beyond the processing capabilities of other replica nodes is achieved.
MySQL InnoDB Cluster manages the operations of these replica nodes, including:
- MySQL Router automatically redirects traffic.
- MySQL Shell is responsible for configuration and initial provisioning (using the InnoDB Clone plug-in ).
- Copy the configuration and any desired topology changes.
Read-only replica nodes can also be used with MySQL InnoDB ClusterSet . In this mode, replica nodes can be added to the main cluster or replica cluster.
MySQL Router identifies these read replica nodes and redirects read traffic to them based on some configuration options. These read replicas will replicate from the master or other replicas, and will automatically reconnect to another member if replication is interrupted or the membership changes (such as the election of a new master).
Create a read-only replica
Adding a read replica node to the cluster is very similar to adding a slave node. The appearance and interaction of the Admin API remain unchanged, ensuring the best user experience.
Assuming you already have a running cluster, you can use this new command to add read replica nodes:
<Cluster>.addReplicaInstance(instance[, options])
mysqlsh-js> cluster.addReplicaInstance("rome4:3306")
Setting up 'rome4:3306' as a Read Replica of Cluster 'rome'.
Validating instance configuration at rome4:3306...
This instance reports its own address as rome4:3306
Instance configuration is suitable.
* Checking transaction state of the instance...
省略……
* Waiting for clone to finish...
NOTE: rome4:3306 is being cloned from rome1:3306
** Stage DROP DATA: Completed
** Clone Transfer
FILE COPY ############################################################ 100% Completed
PAGE COPY ############################################################ 100% Completed
REDO COPY ############################################################ 100% Completed
NOTE: rome4:3306 is shutting down...
* Waiting for server restart... ready
* rome4:3306 has restarted, waiting for clone to finish...
** Stage RESTART: Completed
* Clone process has finished: 4.30 GB transferred in 4 sec (1.08 GB/s)
* Configuring Read-Replica managed replication channel...
** Changing replication source of rome4:3306 to rome1:3306
* Waiting for Read-Replica 'rome4:3306' to synchronize with Cluster...
** Transactions replicated ############################################################ 100%
'rome4:3306' successfully added as a Read-Replica of Cluster 'rome'.As simple as that, a new replica is added to the cluster. Like ordinary cluster members, read-only replicas also support Clone or incremental provisioning.
As usual, you can <Cluster>.status()check the status of the cluster with the command:
mysqlsh-js> cluster.status()
{
"clusterName": "rome",
"defaultReplicaSet": {
"name": "default",
"primary": "rome1:3306",
"ssl": "REQUIRED",
"status": "OK",
"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
"topology": {
"rome1:3306": {
"address": "rome1:3306",
"memberRole": "PRIMARY",
"mode": "R/W",
"readReplicas": {
"rome4:3306": {
"address": "rome4:3306",
"role": "READ_REPLICA",
"status": "ONLINE",
"version": "8.1.0"
}
},
"replicationLag": "applier_queue_applied",
"role": "HA",
"status": "ONLINE",
"version": "8.1.0"
},
"rome2:3306": {
"address": "rome2:3306",
"memberRole": "SECONDARY",
"mode": "R/O",
"readReplicas": {},
"replicationLag": "applier_queue_applied",
"role": "HA",
"status": "ONLINE",
"version": "8.1.0"
},
"rome3:3306": {
"address": "rome3:3306",
"memberRole": "SECONDARY",
"mode": "R/O",
"readReplicas": {},
"replicationLag": "applier_queue_applied",
"role": "HA",
"status": "ONLINE",
"version": "8.1.0"
}
},
"topologyMode": "Single-Primary"
},
"groupInformationSourceMember": "rome1:3306"
}You may notice that the newly added read-only replica runs on rome4:3306the port, which is placed rome1:3306under the cluster master node, which means that it will rome1:3306use as the source node for data synchronization, that is, it will rome1:3306copy data from the node.
Replicating data from the primary node has some advantages, such as reducing potential replication delays. However, it may also increase the load on the master node. In order to provide flexibility and choice according to different scenarios, we have improved this configuration to become a configurable item.
Operating principle: source node identification
As shown above, read-only replicas are synchronized from the master node by default, and if a master-slave switch or failover occurs, it will automatically reconnect to the new master node of the cluster. In other words, it always keeps track of the master node and stays in sync with the cluster.
This is achieved through the asynchronous replication connection failover feature in MySQL replication technology. The replica checks the group replication status and stores a list of potential source services. If the current source service goes offline, it will select a new source from the list.
This source list can be maintained manually or automatically, the latter supporting clustered replication topologies. In this case, the failover mechanism can monitor membership changes and add or remove candidate sources accordingly. Additionally, it can also differentiate between master and slave nodes. This is consistent with the way replica sets replicate the primary cluster in MySQL InnoDB Cluster.
With such a flexible architecture, the Admin API can manage and configure read replicas based on each user's needs.
How it works: Configuring a read-only replica
Select recovery candidates
Replicating from the primary node has the advantage of reducing latency, but may also increase pressure on the primary node.
We understand how the automatic failover mechanism works and understand the different configuration options:
- Prioritize replication from the primary node
- Prioritize a replication from the replica node
- Set up candidate list
You can define it when adding a replica, or change the configuration of an existing replica at any time. The figure below shows that the Rome region replica selects the slave node as the source, and the Brussels replica selects the master node as the source.
As an example, let's add a new replica to the cluster using a specified instance as the source and a predefined list of failover candidates:
mysqlsh-js> cluster.addReplicaInstance("rome5:3306", {replicationSources: ["rome2:3306", "rome3:3306"]})
Setting up 'rome5:3306' as a Read Replica of Cluster 'rome'.
Validating instance configuration at rome5:3306...
This instance reports its own address as rome5:3306
Instance configuration is suitable.
* Checking transaction state of the instance...
NOTE: A GTID set check of the MySQL instance at 'rome5:3306' determined that it is missing transactions that were purged from all cluster members.
NOTE: The target instance 'rome5:3306' has not been pre-provisioned (GTID set is empty). The Shell is unable to determine whether the instance has pre-existing data that would be overwritten with clone based recovery.
The safest and most convenient way to provision a new instance is through automatic clone provisioning, which will completely overwrite the state of 'rome5:3306' with a physical snapshot from an existing cluster member. To use this method by default, set the 'recoveryMethod' option to 'clone'.
Please select a recovery method [C]lone/[A]bort (default Clone): c
* Waiting for the donor to synchronize with PRIMARY...
** Transactions replicated ############################################################ 100%
Monitoring Clone based state recovery of the new member. Press ^C to abort the operation.
Clone based state recovery is now in progress.
NOTE: A server restart is expected to happen as part of the clone process. If the
server does not support the RESTART command or does not come back after a
while, you may need to manually start it back.
* Waiting for clone to finish...
NOTE: rome5:3306 is being cloned from rome2:3306
** Stage DROP DATA: Completed
** Clone Transfer
FILE COPY ############################################################ 100% Completed
PAGE COPY ############################################################ 100% Completed
REDO COPY ############################################################ 100% Completed
NOTE: rome5:3306 is shutting down...
* Waiting for server restart... ready
* rome5:3306 has restarted, waiting for clone to finish...
** Stage RESTART: Completed
* Clone process has finished: 4.30 GB transferred in 6 sec (717.27 MB/s)
* Configuring Read-Replica managed replication channel...
** Changing replication source of rome5:3306 to rome2:3306
* Waiting for Read-Replica 'rome5:3306' to synchronize with Cluster...
** Transactions replicated ############################################################ 100%
'rome5:3306' successfully added as a Read-Replica of Cluster 'rome'.rome5:3306This time, we have completed adding a new read replica running on to the cluster .
This replica uses a fixed list of candidate failures consisting of rome2:3306and rome3:3306, where rome2:3306is the currently running source service. The other members in the list are potential failover candidates, and their order determines the weight, with those at the front having a higher weight and those at the back having a lower weight.
Let's look at the cluster description to get a more intuitive understanding of the current topology:
mysqlsh-js> cluster.describe()
{
"clusterName": "rome",
"defaultReplicaSet": {
"name": "default",
"topology": [
{
"address": "rome1:3306",
"label": "rome1:3306",
"role": "HA"
},
{
"address": "rome2:3306",
"label": "rome2:3306",
"role": "HA"
},
{
"address": "rome3:3306",
"label": "rome3:3306",
"role": "HA"
},
{
"address": "rome4:3306",
"label": "rome4:3306",
"replicationSources": [
"PRIMARY"
],
"role": "READ_REPLICA"
},
{
"address": "rome5:3306",
"label": "rome5:3306",
"replicationSources": [
"rome2:3306",
"rome3:3306"
],
"role": "READ_REPLICA"
}
],
"topologyMode": "Single-Primary"
}
}Likewise, using extended status we can view more information about the current topology:
mysqlsh-js> cluster.status({extended:1})
{
"clusterName": "rome",
"defaultReplicaSet": {
"GRProtocolVersion": "8.0.27",
"communicationStack": "MYSQL",
"groupName": "33cfdab9-3469-11ee-9f3b-d08e7912e4ee",
"groupViewChangeUuid": "33cfe2b0-3469-11ee-9f3b-d08e7912e4ee",
"groupViewId": "16913336945761559:7",
"name": "default",
"paxosSingleLeader": "OFF",
"primary": "rome1:3306",
"ssl": "REQUIRED",
"status": "OK",
"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
"topology": {
"rome1:3306": {
"address": "rome1:3306",
"applierWorkerThreads": 4,
"fenceSysVars": [],
"memberId": "e304af5d-3466-11ee-8d97-d08e7912e4ee",
"memberRole": "PRIMARY",
"memberState": "ONLINE",
"mode": "R/W",
"readReplicas": {
"rome4:3306": {
"address": "rome4:3306",
"applierStatus": "APPLIED_ALL",
"applierThreadState": "Waiting for an event from Coordinator",
"applierWorkerThreads": 4,
"receiverStatus": "ON",
"receiverThreadState": "Waiting for source to send event",
"replicationLag": null,
"replicationSources": [
"PRIMARY"
],
"replicationSsl": "TLS_AES_256_GCM_SHA384 TLSv1.3",
"role": "READ_REPLICA",
"status": "ONLINE",
"version": "8.1.0"
}
},
"replicationLag": "applier_queue_applied",
"role": "HA",
"status": "ONLINE",
"version": "8.1.0"
},
"rome2:3306": {
"address": "rome2:3306",
"applierWorkerThreads": 4,
"fenceSysVars": [
"read_only",
"super_read_only"
],
"memberId": "e6eb91c6-3466-11ee-aca6-d08e7912e4ee",
"memberRole": "SECONDARY",
"memberState": "ONLINE",
"mode": "R/O",
"readReplicas": {
"rome5:3306": {
"address": "rome5:3306",
"applierStatus": "APPLIED_ALL",
"applierThreadState": "Waiting for an event from Coordinator",
"applierWorkerThreads": 4,
"receiverStatus": "ON",
"receiverThreadState": "Waiting for source to send event",
"replicationLag": null,
"replicationSources": [
"rome2:3306",
"rome3:3306"
],
"role": "READ_REPLICA",
"status": "ONLINE",
"version": "8.1.0"
}
},
"replicationLag": "applier_queue_applied",
"role": "HA",
"status": "ONLINE",
"version": "8.1.0"
},
"rome3:3306": {
"address": "rome3:3306",
"applierWorkerThreads": 4,
"fenceSysVars": [
"read_only",
"super_read_only"
],
"memberId": "ea08833f-3466-11ee-b87c-d08e7912e4ee",
"memberRole": "SECONDARY",
"memberState": "ONLINE",
"mode": "R/O",
"readReplicas": {},
"replicationLag": "applier_queue_applied",
"role": "HA",
"status": "ONLINE",
"version": "8.1.0"
}
},
"topologyMode": "Single-Primary"
},
"groupInformationSourceMember": "rome1:3306",
"metadataVersion": "2.2.0"
}Change fault candidates
Changing source configuration is as easy as changing instance options. Cluster.setInstanceOption()Enhanced with a new option replicationSources that allows this operation:
Let's modify the configuration of the first read replica to prioritize synchronization from the second instance instead of the primary instance.
mysqlsh-js> cluster.setInstanceOption("rome4:3306", "replicationSources", "secondary")
Setting the value of 'replicationSources' to 'secondary' in the instance: 'rome4:3306' ...
WARNING: To update the replication channel with the changes the Read-Replica must be reconfigured using Cluster.rejoinInstance().
Successfully set the value of 'replicationSources' to 'secondary' in the cluster member: 'rome4:3306'.Cluster.rejoinInstance()To make the new settings take effect immediately, we need to force the instance to rejoin the cluster using the Admin API's famous command :
mysqlsh-js> cluster.rejoinInstance("rome4:3306")
Rejoining Read-Replica 'rome4:3306' to Cluster 'rome'...
* Checking transaction state of the instance...
The safest and most convenient way to provision a new instance is through automatic clone provisioning, which will completely overwrite the state of 'rome4:3306' with a physical snapshot from an existing cluster member. To use this method by default, set the 'recoveryMethod' option to 'clone'.
The incremental state recovery may be safely used if you are sure all updates ever executed in the cluster were done with GTIDs enabled, there are no purged transactions and the new instance contains the same GTID set as the cluster or a subset of it. To use this method by default, set the 'recoveryMethod' option to 'incremental'.
Incremental state recovery was selected because it seems to be safely usable.
NOTE: User 'mysql_innodb_replica_2506922964'@'%' already existed at instance 'rome1:3306'. It will be deleted and created again with a new password.
** Changing replication source of rome4:3306 to rome2:3306
* Waiting for Read-Replica 'rome4:3306' to synchronize with Cluster...
** Transactions replicated ############################################################ 100%
<strong>Read-Replica 'rome4:3306' successfully rejoined to the Cluster 'rome'.Note: Like a normal instance, Cluster.removeInstance()read replicas can be removed from the cluster using .
Route read requests to replicas
MySQL Router plays a vital role in InnoDB Cluster, so now it is fully aware of the existence of read replicas. The Router sits between the application and the cluster, directing client traffic to the correct destination. It can use read replicas for read-only requests. But slave nodes are also available, so how is it selected?
Configure Router target pool
By default, the behavior of the Router is unchanged, that is, read traffic is directed to the slave nodes of the cluster. However, it is now configurable.
read_only_targets mode accepts the following possible behaviors:
- secondaries : Use only replica members of the target cluster for read-only traffic (default)
- read_replicas : Use only read replicas of the target cluster for read-only traffic
- all : Use all read replicas of the target cluster together with other replica members for read-only traffic
.setRoutingOption()This mode can be configured through another common command , which can be configured at the Router, Cluster or ClusterSet level.
For example:
mysqlsh-js> cluster.setRoutingOption("read_only_targets", "all")
Routing option 'read_only_targets' successfully updated.mysqlsh-js> cluster.routingOptions()
{
"clusterName": "rome",
"global": {
"read_only_targets": "all",
"stats_updates_frequency": null,
"tags": {}
},
"routers": {
"domus::": {}
}
}The diagram below shows a more complex topology, where a ClusterSet consists of a master cluster (Rome) and replica clusters (Brussels) deployed with multiple routers.
The primary cluster is a 3-member cluster with 3 primary members and 3 read replicas. Read replicas are used sourcesList:"secondaryso they replicate from the secondary member of the cluster.
The two routers deployed in this data center use the primary cluster as the target cluster and are configured to use all read targets, ie read_only_targets:all.
The replica cluster in the other data center is also a 3-member cluster, containing 3 read replicas that are configured to replicate from the primary cluster master member. The two routers have different configurations: the first router is configured to use the primary cluster as the target cluster and only use read replicas for read-only targets, ie read_only_targets: read_replicas. Another router is configured to use Brussels as the target cluster and only use the secondary members for read-only targets, ie read_only_targets:secondaries.
Health screening and quarantine
MySQL Router operates as a stateless service, so it relies on InnoDB Cluster membership information for accurate routing. MGR is based on Paxos implementation to provide cluster membership services, defining which servers are online and participating in the group. By utilizing this information, the Router avoids connecting to each member to check its status.
However, when the read replica is also part of the cluster, this information is not available in the group membership information and the Router cannot rely on this mechanism. At the same time, the availability seen among cluster members is different from the router's perspective, and member information may be inaccurate.
To address these challenges, Router implements built-in isolation mechanisms.
Simply put, when the Router fails to connect to a target endpoint due to a new user connection attempt, it puts the target into quarantine. But the target does not stay in isolation forever, after the timeout, the Router performs a health check on the server to determine whether it can be removed from the isolation pool.
The isolation mechanism is configurable in two ways:
- Quarantine Threshold: The number of failed connection attempts to accept until the server is quarantined.
- How frequently quarantined servers should undergo health checks.
By default, the value of both settings is 1, that is, a failed connection will cause the target instance to be quarantined, and a health check will be performed on the quarantined instance every second to see if it can be removed from the quarantine pool.
replication delay
Synchronous replication causing replication delays is a common issue that needs to be considered. If an instance is severely lagging, the DBA may want to hide it and wait for it to come back in sync before exposing it to applications.
Additionally, the DBA may wish to hide read replicas:
- Perform continuous upgrades without impacting incoming traffic
- Make maintenance operations or configuration changes without stopping MySQL
- Exclude it when backing up or generating reports so it doesn't affect other queries
- Configure it as a backup server, excluding any read traffic
As with regular cluster members, this can be accomplished by instructing the Router not to use its label.
Hide copy
To hide an instance from Router traffic, you can use the built-in _hiddentag, which .setInstanceOption()is easy to set up with the command:
mysqlsh-js> cluster.setInstanceOption("rome5:3306", "tag:_hidden", true)This feature is not limited to replicas, secondary instances can also be marked as hidden.
Summarize
Read replicas can scale read-intensive workloads, relieve pressure on other cluster members, and provide more data redundancy.
Thank you for using MySQL!
For more technical articles, please visit: https://opensource.actionsky.com/
About SQLE
SQLE from the Axon open source community is a SQL audit tool for database users and managers that supports multi-scenario audits, standardized online processes, native support for MySQL audits and scalable database types.
SQLE get
| type | address |
|---|---|
| Repository | https://github.com/actiontech/sqle |
| document | https://actiontech.github.io/sqle-docs/ |
| release news | https://github.com/actiontech/sqle/releases |
| Data audit plug-in development documentation | https://actiontech.github.io/sqle-docs/docs/dev-manual/plugins/howtouse |