Why do we need to optimize the database for read-write separation?

There are the following two problems, so database optimization must be carried out

A single table cannot be too large: MySQL official statement: 20 million data in a single table will reach the bottleneck. , so in order to ensure query efficiency, the size of each table must be controlled.
The query pressure is high: in projects, the query is often the frequency of use > the frequency of additions, deletions and modifications, that is, the frequency of reading > the frequency of writing

Reading is used frequently and writing is used less frequently, so we can consider using one database for reading and one database for writing.

The two-master architecture supported by MySQL usually refers to Master-Slave Replication and Master-Master Replication. They are both solutions used to achieve data backup, load balancing and high availability of the database.

master master mode

In master-master replication, two databases act as masters, and each master database can handle write and read operations. This architecture is often used in scenarios that require high availability and load balancing. However, master-master replication requires more complex configuration and conflict-handling mechanisms because both master databases can be writing at the same time, potentially causing data conflicts.

Method to realize

When reading data, query from either of the two main databases

When writing data

The first one: If one database has changed data, it will automatically be synchronized to another database.

The second type: double writing. After writing one database, write another database as well.

Features of master-master replication:

High availability: Even if one master database fails, the other master database is still available.
Load balancing: Two primary databases handle write operations together, helping to spread the load.
Write conflicts: Since both primary databases can write data, data conflicts may need to be resolved

Summary : Because it may cause data conflicts, it is still flawed and not recommended.

master-slave mode

As shown in the figure above, in master-slave replication, there is a master database (Master) and one or more slave databases (Slave).

The master database handles write operations (INSERT, UPDATE, DELETE, etc.), while the slave database replicates data changes from the master database, usually for read operations.

That is, when the master database is added, deleted, or modified, it is automatically synchronized to the slave database.

Features of master-slave replication:

Read and write separation: The master database handles write operations and the slave database handles read operations, thereby sharing the load.
Data backup: The slave database can be used for disaster recovery and data backup.
Load balancing: Load balancing can be achieved by spreading read traffic to multiple slave databases.
Automatic promotion: When the master database fails, a slave database can be promoted to the new master database.

Summary : It only relieves query pressure, but it does not solve the problem of a single database table being too large. Because the main database still has all the data, and no other database can share some of the pressure for it, it is still not recommended.

Multiple masters and multiple slaves

In order to solve the problem of master-slave mode, we can use the multi-master and multiple mode.

Split the table into several sub-tables, and then use the master-slave mode respectively. This solves the problem that the master-slave mode does not solve the problem that a single database table is too large.

Method to realize

How to check the score sheet specifically? You can use the remainder operation. If you want to split it into 3 sub-tables, then take the remainder of 3. There will be one table for remainder 0, one table for remainder 1, and one table for remainder 2.

Before the program performs operations, it first calculates which library should be used.

Hot and cold separation

If the amount of data is large, hundreds of millions, you can adopt a hot and cold separation strategy.

Hot-Cold Separation (Hot-Cold Separation) is a strategy in database optimization that aims to improve performance by storing and managing data that is accessed less frequently (cold data) and data that is accessed more frequently (hot data) separately. Database performance, cost reduction, and optimized query efficiency. This strategy is typically applied in large database environments where the data sets are very large and there are significant changes in access patterns.

Hot data refers to data that is frequently accessed, such as recent data or commonly used data.
Cold data is data that is accessed relatively infrequently, such as historical data or infrequently used data.

Code to achieve separation of reading and writing

mybatisPlus implements multiple data sources

Multiple data sources | MyBatis-Plus (baomidou.com)

1.Introduce dynamic-datasource-spring-boot-starter.

<dependency>
  <groupId>com.baomidou</groupId>
  <artifactId>dynamic-datasource-spring-boot-starter</artifactId>
  <version>${version}</version>
</dependency>

2. Configure the data source.

Configured in the yaml configuration file, master is the main database, slave_1 is a slave database, that is, one project introduces multiple databases

spring:
  datasource:
    dynamic:
      primary: master #设置默认的数据源或者数据源组,默认值即为master
      strict: false #严格匹配数据源,默认false. true未匹配到指定数据源时抛异常,false使用默认数据源
      datasource:
        master:
          url: jdbc:mysql://xx.xx.xx.xx:3306/dynamic
          username: root
          password: 123456
          driver-class-name: com.mysql.jdbc.Driver # 3.2.0开始支持SPI可省略此配置
        slave_1:
          url: jdbc:mysql://xx.xx.xx.xx:3307/dynamic
          username: root
          password: 123456
          driver-class-name: com.mysql.jdbc.Driver
        slave_2:
          url: ENC(xxxxx) # 内置加密,使用请查看详细文档
          username: ENC(xxxxx)
          password: ENC(xxxxx)
          driver-class-name: com.mysql.jdbc.Driver
       #......省略
       #以上会配置一个默认库master，一个组slave下有两个子库slave_1,slave_2

Use @DS to switch data sources.

For example: different mappers can connect to different databases through annotations

A mapper connects to the master database

Corresponds to the configuration file

datasource:
        master:

Method 1: Manually add annotations (not recommended)

Directly use this annotation of mybatisPlus

@DS can be annotated on methods or classes. At the same time, there is a proximity principle. Annotations on methods take precedence over annotations on classes.

We can add an annotation to a mapper that only has read operations, and add an annotation to a mapper that only has addition, deletion, and modification operations.

Or directly add annotations to the query method and go to "query library", add annotations to the add, delete and modify methods and go to "write library"

advantage:

Simple

shortcoming:

The amount of code changes is too large, and comments need to be added in many places.
There is no way to specify using the conditional constructor method

Method 2: mybatis interceptor (not recommended)

Use the interceptor to switch databases, intercept all sql, and go to the main database when insert, update, and delete appear, and go to the slave database when select appears.

Advantages: less intrusive to the code

Disadvantages: There is no way to guarantee the transaction

Method 3: Middleware

It is best to have minimal code intrusion and ensure transactions.

Option 1: mycat

Deploy mycat and use mycat for data source management

MyCAT is a powerful database middleware that can be used not only for reading and writing separation, table and database partitioning, and disaster recovery management, but also for multi-tenant application development and cloud platform infrastructure, giving your architecture a strong Adaptability and flexibility.

Suitable for large enterprises with dedicated maintenance teams, or large teams.

It’s too troublesome and has to be deployed. I don’t recommend it.

Option 2: sharding-jdbc (recommended)

It is something similar to mybatis. We do not need to deploy other middleware or change the code.

Sharding-JDBC is positioned as a lightweight Java framework. It uses the client to directly connect to the database and provides services in the form of jar packages. There is no proxy layer, no additional deployment, no other dependencies, and the DBA does not need to change the original operation and maintenance method.

Sharding-JDBC directly encapsulates the JDBC API and can be understood as an enhanced version of the JDBC driver. The cost of migrating old code is almost zero.

Suitable for small and medium-sized enterprises or small and medium-sized teams.

For details on how to use sharding-jdbc to separate reading and writing, please see the next episode for explanation.

Database optimization: separation of reading and writing, and code implementation in the SpringBoot project