Problems you may encounter when reading and writing are separated in MySQL

Preface

Read-write separation is a frequently used architecture in MySQL. Through read-write separation, horizontal expansion is achieved. Write and update operations are performed on the source server, and data reading operations are performed from the slave server. By increasing the slave server's The number can greatly enhance the reading ability of the database.

The high-availability architecture in MySQL has shown a trend of becoming more and more complex, but it has evolved from the most basic one master and one slave, so here we will understand the basic principles of master-slave.

First, let’s understand the difference between master-slave, master-slave, and dual-master mode.

double master

There are two main libraries, each of which can be read and written, and data synchronization operations can be performed between the two main libraries.

Master-slave

From the master to the slave, data writing is performed in the master node, data reading is performed in the slave node, and the master database synchronizes data to the slave database.

Active and standby

Primary and secondary databases are only used for data backup. No read and write operations occur. Data reading and writing occur in the primary database, and the primary database will synchronize data to the secondary database.

The data latency discussed below is mainly based on the master-slave mode. The principles of master-slave, master-slave, and dual-master data synchronization are the same and will not be analyzed here.

1. Architecture of separation of reading and writing

Commonly used separation of reading and writing has the following two implementations:

1. The client realizes reading and writing separation;

2. Implement read and write separation based on the intermediate proxy layer.

Implement read and write separation based on client

The client actively performs load balancing and select、insert performs routing classification. Read requests are sent to the reading library, and write requests are forwarded to the writing library.

This method is characterized by better performance, direct implementation in the code, no need for additional hardware support, simple architecture, and more convenient troubleshooting.

Disadvantages need to be embedded in the code and developers need to implement it. Operation and maintenance cannot intervene. For large codes, a lot of code needs to be changed to achieve separation of reading and writing.

Implement read-write separation based on intermediate proxy

The intermediate proxy layer implements read-write separation. There is an intermediate proxy layer proxy between MySQL and the client. The client only connects to the proxy, and the proxy determines the distribution route of the request based on the request type and context.

The architecture with proxy is more client-friendly. The client does not need to pay attention to back-end details. Connection maintenance, back-end information maintenance and other work are all completed by the proxy. But in this case, the requirements for the back-end maintenance team will be higher. Moreover, the proxy also needs to have a highly available architecture. Therefore, the overall architecture with proxy is relatively complicated.

However, that deployment method will encounter the problem of master-slave delay in read-write separation. Because of the master-slave delay, the client has just completed an update transaction and then immediately initiates a query. If the query is from the slave database, it may read The status is the status before the update.

2. How to ensure the consistency of master-slave data in MySQL

The master-slave synchronization of MySQL data is mainly realized through binlog. The slave library uses the binlog on the master library to replay to achieve master-slave synchronization.

Let's take a look at the implementation principle

dump thread，I/O thread，sql thread In master-slave replication, the slave library uses the binlog on the master library to replay to achieve master-slave synchronization. These three threads are mainly used in the replication process.

IO thread: Created when the slave library executes start slave a statement, it is responsible for connecting to the main library, requesting the binlog, receiving the binlog and writing it to the relay-log;

dump thread: It is used for the master library to synchronize binlog to the slave library, and is responsible for responding to slave IO thread requests. The main library will create a connection for each slave library dump thread, and then synchronize the binlog to the slave library;

sql thread: Read relay log and execute commands to update slave data.

Let’s take a look at the copy process:

1. The main library receives the update command, executes the update operation, and generates binlog;

2. The slave library establishes a long connection between the master and slave;

3. The dump_thread of the main library reads the binlog locally and transmits it to the slave library;

4. The slave library obtains the binlog from the main library and stores it locally, which becomes relay log(relay log);

5. The sql_thread thread reads, relay log parses, and executes commands to update data.

have to be aware of is

When the master-slave relationship is initially created, synchronization is specified by the slave library. For example, in a site-based master-slave relationship, if the slave library says "I want to start synchronization from position P of binlog file A", the master library will start from this specified position and then send data backwards.

After the master-slave replication relationship is established, it is the master database that decides "to send data to the slave database." Only if the master database has new logs, it will be sent to the slave database.

Binlog has three formats statement，row and mixed.

1. Statement (Statement-Based Replication, SBR): Every SQL that modifies data will be recorded in the binlog, which records the executed SQL;

Statement mode only records executed SQL and does not need to record changes in each row of data. Therefore, it greatly reduces the amount of binlog logs, avoids a large number of IO operations, and improves system performance.

It is precisely because the Statement mode only records SQL, and if some SQL contains functions, the execution results may be inconsistent.

For example, the uuid() function generates a random string every time it is executed, and the uuid is recorded in the master. After synchronizing to the slave, if it is executed again, another result will be obtained.

Therefore, there will be some data consistency issues when using the Statement format.

2. Row (Row-Based Replication, RBR): does not record SQL statement context information, only records how a certain record has been modified;

The log content in Row format will clearly record the details of each row of data modification, so that there will be no situation where the data in Statement cannot be copied normally.

For example, if there are 100 lines of data that meet the conditions in a modification, the 100 lines of data will be recorded in binlog in detail. Of course, at this time, the content of the binlog file is much more than the first one.

However, the Row format also has a big problem, that is, the amount of logs is too large, especially for operations such as batch update, entire table delete, and alter table. Since the changes in each row of data need to be recorded, a large amount of logs will be generated. Logs can also cause IO performance problems.

3. Mixed (Mixed-Based Replication, MBR): a mixture of Statement and Row.

Because some statement format binlogs may cause master-slave inconsistency, row format must be used.

But the disadvantage of the row format is that it takes up a lot of space. For example, if you use a delete statement to delete 100,000 rows of data, if you use statement, a SQL statement will be recorded in the binlog, occupying dozens of bytes of space. But if you use the binlog in row format, you have to write all the 100,000 records into the binlog. Doing so will not only take up more space, but also consume IO resources while writing binlog, which will affect the execution speed. Therefore, MySQL took a compromise solution, which is to have a mixed format binlog.

The mixed format means that MySQL itself will determine whether this SQL statement may cause master-slave inconsistency. If possible, use the row format, otherwise use the statement format. In other words, the mixed format can take advantage of the advantages of the statment format while avoiding the risk of data inconsistency.

But now more and more scenarios will set MySQL's binlog format to row, such as data recovery in the following scenarios.

Here delete、insert we look at the problem of data recovery from the perspective of three SQL statements: update and update.

1. delete: If the delete operation is performed, the data needs to be restored. The binlog in row format will also save the entire line information of the deleted line. Therefore, if after executing a delete statement, you find that the wrong data has been deleted, you can directly convert the delete statement recorded in the binlog into an insert, and insert the wrongly deleted data back to recover;

2. Insert: If an insert operation is performed, the data needs to be restored. In the row format, all field information will be recorded in the binlog of the insert statement. This information can be used to accurately locate the row that was just inserted. At this time, you can directly convert the insert statement into a delete statement to delete the mistakenly inserted row of data;

3. Update: If an update operation is performed, data recovery is required. The binlog will record the entire row of data before modification and the entire row of data after modification. Therefore, if the update statement is executed by mistake, you only need to swap the two lines of information before and after the event, and then execute it in the database to restore the update operation.

cyclic replication problem

From the principle of binlog above, we can know that master-slave data synchronization in MySQL is to maintain data consistency through binlog.

However, the double M structure may have problems with cyclic replication.

What is the double M architecture, the double M structure and the MS structure? In fact, the difference is just one more line, that is: there is always a master-slave relationship between nodes A and B. In this way, there is no need to modify the master-slave relationship when switching.

The business logic updates a statement in node A, and then sends the generated binlog to node B, and node B will also generate binlog.

If node A is also the slave library of node B, node B will also pass the binlog to node A, and node A will execute the binlog passed by node B. In this way, node A and node B will continuously loop this update statement, which is circular replication.

How to solve it?

1. It is stipulated that the two libraries server id must be different. If they are the same, they cannot be set as a master-slave relationship;

server id 2. A slave receives the binlog and generates a new binlog identical to the original binlog during replay ;

3. After each library receives the log sent from its own main library, it first judges server idthat if it is the same as its own, it means that the log was generated by itself, and then discards the log directly.

In this way, when the double M structure is set, the execution flow of the log will become like this:

1. For the transactions updated from node A, the binlog records all the transactions of A server id;

2. After being transferred to node B and executed once, the binlog generated by node B server id is also that of A server id;

3. Then send it back to node A. If A determines that this log server id is the same as its own, it will no longer process the log. Therefore, the infinite loop is broken here.

3. Master-slave synchronization delay

The master-slave synchronization delay is the difference between the time when the slave library completes execution and the time when the master library completes the execution of the same transaction.

1. Main library A completes a transaction and writes it to binlog. The time recorded is T1;

2. Pass the data to the slave library, and the slave library receives the binlog. The time when the reception is completed is recorded as T2;

3. Slave library B executes and completes the received transaction. This time is recorded as T3.

The master-slave delay time is the time difference between T3-T1.

The value of seconds_behind_master can be obtained through show slave status the command, which indicates how many seconds the current slave library is delayed.

How seconds_behind_master is calculated:

1. The binlog of each transaction has a time field, which is used to record the time of writing to the main database;

2. Get the value of the time field of the currently executing transaction from the library, calculate the difference between it and the current system time, and you can get seconds_behind_master.

Simply put, seconds_behind_master is T3 -T1 the time difference above.

If the time settings of the master and slave machines are inconsistent, will it lead to inaccurate master-slave delays?

The answer is no. When the slave library connects to the main library, it will SELECT UNIX_TIMESTAMP()obtain the current master library time through a function. If it is found that the main library system time is inconsistent with its own, the slave library will actively deduct it when performing seconds_behind_master calculation. This difference.

4. Reasons for master-slave synchronization delay

Possible reasons for master-slave synchronization delay:

1. The performance of the slave database is worse than that of the machine where the main database is located;

Compare the performance of the slave library. If the replication capability of the slave library is lower than that of the master library, then when the master library is under heavy writing pressure, it will cause long-term data delays in the slave library.

2. The pressure on the slave library is high;

Placing a large number of queries on the slave database may cause a large amount of CPU resources to be consumed on the slave database, thereby affecting the synchronization speed and causing master-slave delay.

3. Execution of major affairs;

When a transaction occurs, the master database must wait for the transaction to be completed before it can be written to the binlog. Assume that the transaction being executed is a very large data insertion. This data is transmitted to the slave database, and it also takes a certain amount of time to synchronize the data from the slave database. This will cause data delays on the slave node.

4. Master-slave or master-slave switching occurs when the main library is abnormal.

When the main library is switched, data inconsistency may occur. The master-slave switch has the following two strategies:

Reliability priority strategy:

1. First judge the seconds_behind_master of the slave library, if it is less than a tolerable value, proceed to the next step, otherwise continue to retry this step;

2. Change the main library A to a read-only state, and set readonly to true;

3. Judge the seconds_behind_master of slave library B until the value becomes 0;

4. Change slave library B to a readable and writable state, set readonly to false, and slave library B becomes the new master library;

5. The update business request will be switched to slave library B.

There is an unavailable time during this switching process. After step 2, both the main library A and the slave library B are in the readonly state. At this time, the system is in a non-writable state. Until the readonly state of the slave library B becomes false, at this time Only then can write requests be received normally.

Step 3: Determine seconds_behind_master to be 0. This operation is the most time-consuming. Through the early judgment in step 1, you can ensure that the value of seconds_behind_master is small enough to reduce the waiting time in step 3.

Availability priority strategy:

If steps 4 and 5 are adjusted to be executed at the very beginning, without waiting for the data synchronization of the main database, the connection is directly switched to the slave database B, so that the slave database B can directly read and write, so that the system will have almost no unavailable time.

This strategy can ensure the availability of the service to the greatest extent possible, but data inconsistency will occur because the write request is directly switched to the slave library B, that is, B is set as the new master library, because the latest database has not been synchronized to the B library. After the data becomes the main database, this part of the data is lost.

5. How to deal with master-slave delay

In the face of master-slave delay, there are several solutions:

1. Force the main database solution;

2. sleep program;

3. Determine the master-slave non-delay plan;

4. Cooperate with semi-sync solution;

5. Wait for the main warehouse location plan;

6. Etc. GTID scheme.

Force the main library plan

The forced access to the main database solution is essentially to classify queries, and queries that cannot tolerate synchronization delays are directly processed in the main database.

1. For requests that must get the latest results, they are forced to be sent to the main database.

2. For the request that can read the old data, it is sent to the slave library.

This method is a bit speculative. If all queries are not allowed to appear delayed, it means that all queries will appear in the main database. This is equivalent to giving up the separation of reading and writing. All reading and writing pressure is on the main library, which is equivalent to giving up scalability.

Sleep scheme

After the master database update is completed, the slave database sleeps for a while before reading data. The specific solution is similar to executing a select sleep(1) command.

The assumption of this solution is that in most cases the master-slave delay is within 1 second, and doing a sleep can have a high probability of getting the latest data.

This method is not a reliable solution

1. If a query request can get the correct result from the slave database in 0.5 seconds, it will still wait for 1 second;

2. If the delay exceeds 1 second, expired reading will still occur.

Determine the master-slave no-delay solution

You can check whether there is a delay between the master and the slave. If there is no delay, you can execute the query operation in the slave database. There are three methods below.

1, judgment seconds_behind_master;

Before each query, first determine whether the value of seconds_behind_master is equal to 0. If it is not equal to 0, it means there is still a delay, and the query operation will not be performed until it is equal to 0.

2. Compare sites to ensure no delay between master and slave;

Master_Log_File and Read_Master_Log_Pos represent the latest position of the master library read;
Relay_Master_Log_File and Exec_Master_Log_Pos represent the latest execution point from the library.

If Master_Log_File and

Relay_Master_Log_File、Read_Master_Log_Pos

The two sets of values are exactly the same as Exec_Master_Log_Pos, which means that the received log has been synchronized.

3. Compare the GTID set to ensure no delay between master and slave:

Auto_Position=1 , indicating that the master-slave relationship uses the GTID protocol;
Retrieved_Gtid_Set, is a collection of GTIDs of all logs received from the library;
Executed_Gtid_Set, is a collection of all GTIDs that have been executed from the library.

If the two sets Retrieved_Gtid_Set and Executed_Gtid_Set are the same, it means that the logs received from the library have been synchronized.

What are GTIDs?

The full name of GTID is Global Transaction Identifier, which is the global transaction ID. It is generated when a transaction is submitted and is the unique identifier of the transaction.

It consists of two parts and the format is:

GTID=server_uuid:gno

server_uuid is automatically generated when an instance starts for the first time, and is a globally unique value;
gno is an integer with an initial value of 1. It is assigned to this transaction and incremented by 1 each time a transaction is committed.

In GTID mode, each transaction will have a one-to-one correspondence with a GTID. There are two ways to generate GTID, which are determined by the value of the session variable gtid_next:

1. If gtid_next=automatic, it means using the default value. At this time, MySQL will be server_uuid:gno assigned to this transaction.

a. When recording binlog, record one line first

SET @@SESSION.GTID_NEXT=‘server_uuid:gno’;

b. Add this GTID to the GTID set of this instance.

2. If gtid_next is a specified GTID value, such as set gtid_next='current_gtid’ current_gtid, then there are two possibilities:

a. If current_gtid already exists in the GTID set of the instance, the transaction executed next will be directly ignored by the system;

b. If current_gtid does not exist in the GTID set of the instance, assign this current_gtid to the transaction to be executed next. That is to say, the system does not need to generate a new GTID for this transaction, so gno does not need to be incremented by 1.

A current_gtid can only be used by one transaction. After this transaction is submitted, if you want to execute the next transaction, you need to execute the set command to set gtid_next to another gtid or automatic.

Each MySQL instance maintains a GTID collection that corresponds to "all transactions executed by this instance."

After understanding the concept of GTID, let's look at the usage of GTID-based master-slave replication.

In GTID mode, the syntax of slave library C to be set as master-slave library B is as follows:

We record the GTID set of instance B as set_b at this moment, and the GTID set of instance C as set_c. Next, let's take a look at the current master-slave switching logic.

We execute the command on instance C start slave , and the logic of taking the binlog is as follows:

1. Instance C specifies the main library B, and establishes a connection based on the master-slave protocol.

2. Instance C sends set_c to main library B.

3. Instance B calculates the difference between set_b and set_c, that is, the set of all GITDs that exist in set_b but not in set_c, and determines whether B locally contains all the binlog transactions required for this difference.

a. If it is not included, it means that B has deleted the binlog required by instance C and returns an error directly;

b. If it is confirmed that all are included, B finds the first transaction that is not in set_c from its own binlog file and sends it to C;

Then start from this transaction, read the file later, fetch the binlog in order and send it to C for execution.

This logic contains a design idea: in the master-slave relationship based on GTID, the system believes that as long as the master-slave relationship is established, it must ensure that the logs sent by the master database to the slave database are complete. Therefore, if the log required by instance C no longer exists, B refuses to send the log to C.

This is different from the site-based master-slave protocol. The site-based protocol is determined by the slave library. Which site is specified by the slave library will be sent by the master library without judging the integrity of the log.

Let's take a look at how master-slave switching is implemented in a master-slave switching scenario after the introduction of GTID.

Since there is no need to find a location, slave libraries C and D only need to execute change master commands to point to instance B respectively.

In fact, strictly speaking, it is not that the master-slave switch does not require finding the location, but the work of finding the location is automatically completed within instance B. But since this work is automatic, it is very friendly to developers of HA systems.

After that, the system will be written by the new main library B. The GTID collection format in the binlog generated by main library B is: server_uuid_of_B:1-M.

When setting up GTID master-slave synchronization, if master library A finds that the GTID log to be synchronized has been deleted, A will report an error.

Solution:

Slave library C needs to set gtid_purged before starting synchronization and specify the starting point of GTID synchronization. This setting is required when using backup to build a slave library.

If a separate operation is performed on the slave library, resulting in a lack of GTID on the master library, you can simulate an empty transaction on the master library with the same GTID as on slave library C, so that the master-slave synchronization will not report an error.

Cooperate with semi-sync

MySQL has three synchronization modes, namely:

1. Asynchronous replication: The default replication in MySQL is asynchronous. The master database will immediately return the results to the client after executing the transaction submitted by the client, and does not care whether the slave database has received and processed it. The problem is that if the logs of the master database are not synchronized to the slave database in time, and then the master database is down, then failover is performed and the master is selected from the slave database, the data in the selected master database may be incomplete;

2. Fully synchronous replication: means that when the main library completes a transaction and waits until all slave libraries have also completed the transaction, the main library commits the transaction and returns the data to the client. Because you have to wait for all slave databases to be synchronized to the data in the master database before returning the data, the consistency of the master-slave data can be guaranteed, but the performance of the database will inevitably be affected;

3. Semi-synchronous replication: It is a kind of synchronization between full synchronization and full asynchronous synchronization. The main library needs to wait for at least one slave library to receive and write to the file. The main library does not need to wait for all Relay Log slave libraries to return ACK to the main library. The main library receives the ACK, indicating that the transaction is completed, and returns the data to the client.

The default replication in MySQL is asynchronous, so there will be a certain delay in the synchronization of the master database and the slave database. More importantly, asynchronous replication may also cause data loss. The performance of full synchronous replication is too poor, so from MySQL 5.5 the beginning, MySQL supports semi-sync semi-synchronous replication in the form of a plug-in.

Because this synchronous replication does not need to wait for all slave libraries to be synchronized successfully, the query in the slave library at this time will face two situations:

1. If the query falls on the slave database that responded to the ack, it can ensure that the latest data is read;

2. If the query falls on other slave libraries, they may not have received the latest logs, which will cause the problem of expired reading.

Waiting for the main library location plan

Before understanding the main warehouse location plan, first take a look at the meaning of the following command.

select master_pos_wait(file, pos[, timeout]);

Take a look at some parameters of this command:

1. This command is executed in the slave library;

2. The parameters file and pos refer to the file name and location on the main library;

3. timeout is optional, set to a positive integer N to indicate that this function will wait for up to N seconds.

The normal return of this command is a positive integer M, which represents the number of successfully executed transactions from the beginning of the command execution to the completion of the application of the binlog position represented by file and pos.

If the slave library executes this command, it means that the slave library has completed synchronizing the binlog position represented by file and pos, and the number of synchronized transactions. A normal return of M means that the slave library data has no delay relative to the main library within the timeout time.

In addition to normal returns, there will also be other execution return information.

1. If an exception occurs during slave library synchronization during execution, NULL will be returned;

2. If the wait exceeds N seconds, -1 will be returned;

3. If you find that this position has already been executed when you first start executing, 0 will be returned.

After understanding the function of this command, let's look at the specific execution process of the lower master database location plan. If we perform a data update in the master database and then query it in the slave database, the execution process is:

1. After the execution of the update transaction is completed, it will be executed immediately show master status to obtain the File and Position executed by the current main library;

2. Select a slave database to execute the query;

3. Execute on the slave library select master_pos_wait(File, Position, 1);

4. If the return value is a positive integer >= 0, execute the query statement in this slave library;

5. Otherwise, go to the main database to execute the query.

If the delay of the slave library exceeds the waiting timeout time, then all requests will eventually fall to the main library, and the query pressure will still climb to the main library.

Other GTID schemes

If the GTID mode is turned on, there is also a corresponding solution to wait for the GTID.

MySQL also provides commands to wait for synchronization

select wait_for_executed_gtid_set(gtid_set, 1);

Execution logic:

1. The slave library carries the gtid of the transaction when executing the command. If 0 is returned, it means that the transaction has been synchronized to the slave library normally, and the query can be executed in the slave library;

2. The timeout test returns 1.

In the previous plan, after we execute the transaction, we have to actively go to the main database to execute it show master status. Starting MySQL 5.7.6 from the new version, it is allowed to return the GTID of this transaction to the client after executing an update transaction, so that the solution of waiting for GTID can reduce one query.

The process of waiting for the GTID plan is

1. Execute and complete a transaction in the main library, and directly obtain the GTID of this transaction from the return packet, which is recorded as gtid1;

2. Select a slave database to execute the query statement;

3. Execute from the library select wait_for_executed_gtid_set(gtid1, 1);

4. If the return value is 0, execute the query in the slave library;

5. If the return value is not 0, it means that the transaction has not been synchronized to the slave database, and it is necessary to execute the query in the master database.

Same as the scheme of waiting for the main library location, if the wait times out, you need to initiate a query to the main library.

6. Summary

1. Read-write separation is a frequently used architecture in MySQL. Through read-write separation, the ability to achieve horizontal expansion is achieved. Write and update operations are performed on the source server, and data reading operations are performed from the server. By increasing the size of the slave server, The number of servers can greatly enhance the reading ability of the database;

2. Master-slave synchronization of MySQL data is mainly achieved through binlog. The slave library uses the binlog on the main library to replay to achieve master-slave synchronization;

3. There will be a problem in data synchronization and timely synchronization delay;

4. Possible reasons for master-slave synchronization delay:

1. The performance of the slave database is worse than that of the machine where the main database is located;
2. The pressure on the slave library is high;
3. Execution of major affairs;
4. Master-slave or master-slave switching occurs when the main library is abnormal.

5. How should master-slave delay be handled?

In the face of master-slave delay, there are several solutions: