Table of contents
1 Primary and secondary delays
1.1 Primary and secondary delays
1.2 Sources of primary and secondary delays
1.2.1 There is a performance gap between the main and backup machines
1.2.2 High pressure on backup database
1.3 Troubleshooting ideas for active and standby delays
2 Active/standby switching strategy
2.1 Reliability priority strategy
2.2 Availability priority strategy
2.3 Common switching technologies
Since entering the Internet era, we have moved from stand-alone machines to clusters and then to the current microservice architecture. We rarely use stand-alone architecture to implement business logic. Even if microservices are not used, clusters such as master-standby and master-slave are already part of the business. Necessary abilities.
However, whether it is a master-standby or master-slave architecture, it is actually a strategy implemented for the high availability of the system to prevent the host from abnormally going offline due to certain faults. At this time, the backup or slave instance will become the master through selection or other strategies. Service instance, continue to provide services to the outside world.
Under normal circumstances in MySQL, as long as all binlogs generated by the main database when executing updates are correctly transmitted to the standby database and executed correctly, the standby database can be consistent with the main database data, achieving ultimate consistency. However, eventual consistency cannot meet online performance requirements, and the availability of the cluster needs to be ensured.
1 Primary and secondary delays
1.1 Primary and secondary delays
When a master-backup delay occurs, the time points for data synchronization mainly include:
- Main library A completes a transaction and writes it to binlog. We record this moment as T1;
- Then it is passed to the standby database B. We record the time when the standby database B receives the binlog as T2;
- Standby database B completes this transaction, and we record this time as T3.
The master-slave delay is the difference between the time when the slave database is executed and the time when the master database is executed for the same transaction, which is T3-T1.
Executing show slave status on the standby database will result in seconds_behind_master, which represents the delay time of the standby database. The calculation method is:
- The binlog of each transaction has a time field used to record the writing time of the main database;
- The standby database takes out the value of the time field of the currently executing transaction and calculates the difference from the current system time, which is the value in seconds;
If the system time settings of the active and standby database machines are inconsistent, it will cause the active and standby delay values to be inaccurate. Because when the standby database connects to the primary database, it will obtain the system time of the current primary database by executing the SELECT UNIX_TIMESTAMP() function. If it is found that the system time of the main database is inconsistent with its own, the standby database will automatically deduct this difference when performing seconds_behind_master calculation.
However: if the system time of the main database is modified after the standby database has been connected to the main database, the time will not be automatically corrected when the standby database is synchronized. Therefore, the time correction will only be executed when the connection is established for the first time.
When the network is normal, the time required to transfer logs from the primary database to the standby database is very short, that is, the value of T2-T1 is very small. In other words, under normal network conditions, the main source of delay between the primary and secondary databases is the time difference between the secondary database receiving the binlog and executing the transaction. Therefore, the most direct manifestation of the master-backup delay is that the speed at which the backup database consumes relay logs is slower than the speed at which the primary database produces binlogs.
1.2 Sources of primary and secondary delays
1.2.1 There is a performance gap between the main and backup machines
The performance of the machine where the standby database is located is worse than that of the main database. In this case, the standby database is generally set to "non-double 1" mode [sacrifice some reliability of the standby database, reduce the number of disk writes, and enhance IO capabilities], and trigger during the update process A large number of read operations may cause delays in the primary and secondary servers.
This situation is relatively rare now, because it is now a master-slave deployment, and master-slave switching may occur at any time, so it is generally a symmetrical deployment.
1.2.2 High pressure on backup database
The common reason is that in a read-write separation scenario, the standby database provides external read capabilities, and queries consume a lot of CPU resources, affecting the synchronization speed and causing delays in the active and standby databases.
The measures at this time are:
- One master and multiple slaves use slave libraries to share the pressure;
- Output to external systems, such as Hadoop systems, through binlog to provide statistical query capabilities;
The slave database and the standby database are conceptually similar. In our column, for the convenience of description, I call the database that will be selected as the new primary database during the HA process the standby database, and the others are called slave databases.
1.2.3 Big events
The primary database must wait until the transaction is completed before writing to the binlog and then passing it to the standby database, causing delays in the primary and standby databases.
For example, deleting a large amount of data will cause a large transaction, which generally requires execution in batches. The reason why deletion will cause a large transaction is because regardless of whether there is an index, the storage engine queries and locks data one by one, and returns it to the execution engine, which marks the data for deletion. After all data has been processed, the transaction will be submitted to release the lock.
The other is large table DDL.
1.3 Troubleshooting ideas for active and standby delays
1) Check what the database is doing
pager cat - | grep -v Sleep | sort -rn -k 12 | head -n 20
show full processlist;
select * from information_schema.processlist
where 1=1 order by TIME desc limit 10;2) Check what sql_thread is doing
Check the status on the slave:
show slave status\G;View relay_master_log_file and exec_master_log_pos
Parse binglog logs on the master:
mysqlbinlog -v --base64-output=decode-rows --start-position=exec_master_log_pos relay_master_log_fileIf you find yourself stuck while operating a certain watch:
1) Checklist structure
- No index: stop slave may be stuck. It is recommended to close mysql, add index first after starting, and then start slave
- With index: you can only wait, large transactions need to be split, don’t operate too much data
2) Large transactions: Session replies on M use statement format and use statement-level replication.
3) View MySQL status
- Machine performance (CPU, IO, etc.): The slave library configuration is appropriately higher, and new hardware PCI-E or SSD devices are used.
- Table structure: The design must be reasonable, there must be a primary key, the primary key must be short, and an index must be built for the query field.
- Business program: Appropriate use of cache to reduce database pressure
Analyze the MySQL process and combine it with the source code:
perf top `pidof mysqld`4) Temporary optimization of parameters
- Enable group commit in the main library
- Open writeset from library
- Set sync_binlog=0 && innodb_flush_log_at_trx_commit=2 from the library
5) Check the lock condition
show engine innodb status\G;2 Active/standby switching strategy
2.1 Reliability priority strategy
In the double M structure, the flow of active and standby switching is as follows:
- Determine the current seconds_behind_master (SBM) of standby database B. If it is less than a certain value (such as 5 seconds), continue to the next step, otherwise continue to retry this step; here the master-slave delay time is short, indicating that there are currently no large transactions, and the delay is relatively low, reducing The chance of data being unreliable due to delays;
- Change main library A to read-only status, that is, set readonly to true;
- Determine the value of seconds_behind_master of standby database B until this value becomes 0;
- Change standby database B to a read-write state, that is, set readonly to false;
- Cut the business request to standby database B.
This switching process is generally completed by a dedicated HA system, which we temporarily call the reliability priority process.
There is unavailable time in this switching process. Because after step 2, both main database A and standby database B are in the readonly state, which means that the system is in a non-writable state at this time and cannot be restored until step 5 is completed.
In this unavailable state, the most time-consuming step is step 3, which may take several seconds. This is why you need to make a judgment first in step 1 to ensure that the value of seconds_behind_master is small enough.
2.2 Availability priority strategy
If you directly advance steps 4 and 5, it will ensure that the system is almost unavailable, but it may cause data inconsistency.
In fact, these are C and A in CAP. The MySQL main database responds to the client after writing the binlog, without waiting for the binlog to be synchronized to one or more standby databases. This strategy chooses A between C and A. , C is sacrificed. If the main database goes down, but the last or several transactions of the binlog are not synchronized to the standby database, then the data will be lost after the standby database becomes the main database. Many other NoSQL systems provide users with choices, such as Mongo. Users can set the log to be synchronized to several slaves before responding to the client. The more slaves that are synchronized, the stronger C and the weaker A. For example, synchronize to X slaves. Then respond to the client, then even if any X nodes are down, there will still be 1 node in the cluster with the latest log, it will become the master node, the data will not be lost, and the cluster can still work.
On the premise of meeting data reliability, the availability of the MySQL high-availability system depends on the latency of the primary and secondary servers. The smaller the delay time, the shorter the time required for service recovery when the main database fails, and the higher the availability.
2.3 Common switching technologies
Semi-sync will degenerate into async in the case of network failure and timeout. If the main library happens to be powered off at this time, some binlogs have not been transmitted to the slave library. The slave library cannot judge whether the data is consistent with the main library. If the switch is forced, it may As a result, data is lost. In financial business scenarios, only "artificial intelligence" can be used to switch, and service interruption time is long. AliSQL uses dual-channel replication to make it easier to determine whether the primary and secondary data are consistent. If they are consistent, they can automatically switch. If they are inconsistent, manual data recovery is required.
Related content expansion: (Technology Frontier)
In the past 10 years, when even traditional enterprises began to digitize on a large scale, we found that in the process of developing internal tools, a large number of pages, scenes, components, etc. were constantly repeated. This repetitive work of reinventing the wheel wasted a lot of engineers' time.
To address this type of problem, low-code concretizes certain recurring scenes and processes into components, APIs, and database interfaces to avoid reinventing the wheel. Greatly improves programmers' productivity.
Recommend a software JNPF rapid development platform that all programmers should know. It adopts the industry-leading SpringBoot microservice architecture and supports SpringCloud mode, which improves the platform's expansion foundation and meets the needs of rapid system development, flexible expansion, seamless integration and high-end Comprehensive capabilities such as performance applications; using the front-end and back-end separation model, front-end and back-end developers can work together to take charge of different sectors, which is trouble-free and convenient.
Experience the official website:https://www.jnpfsoft.com/?csdnxx
If you haven’t learned about low-code technology yet, you can quickly experience it and learn it!