Use canal to solve Mysql and Redis data synchronization (TCP)

foreword

I wrote an article before " Using canal to solve the problem of Mysql and Redis data synchronization ", which also uses canal to achieve data synchronization between mysql and redis. The difference from this article is that the previous article is based on MQ to achieve data synchronization. This article The article is implemented based on the TCP method.

Analysis of working principle

During the interview, we often hear the interviewer ask such a question: How do you synchronize data between Mysql and Redis? There are many solutions according to different business scenarios. You may say to write the library first and then delete the cache, or delay double deletion or other options. What I want to share with you today is a relatively mature solution - using Canal to synchronize Mysql and Redis data.

I don’t know if you know about Mysql master-slave. According to the 2/8 principle, 80% of the performance problems are on the read. When the read concurrency of our database is large, we can use the Mysql master-slave to share the pressure of reading. Its principle is that all write operations are performed on the master library, and read operations are performed on the slave library. Of course, the master library can also undertake read requests, while the data in the slave library is copied from the master library. Mysql comes with the function of master-slave replication. . As shown below
insert image description here

Master-slave replication steps:

Open the binary-log log file of the Master, and mysql will write all DDL, DML, and TCL into the BinaryLog log file
Master will generate a log dump thread to transfer binlog to the i/o thread of the slave library
Request the binlog of the main library from the i/o thread of the library, and write the obtained binlog log to the relay log (relaylog)
The sql thread of the slave library will read the logs in the relaylog file and parse them into specific operations. Through the consistent operation of the master and slave, the final data consistency will be achieved.

The principle of Canal is to pretend to be Slave to copy SQL statements or data from Binlog.

Mysql and Redis data synchronization scheme

According to the above, we can automatically synchronize the binlog data log files of the database through Canal, and then synchronize the data to Redis, so as to achieve the function of automatic synchronization between Mysql and Redis. It is a pity that Canal has no way to directly synchronize the database to Redis. It supports the following components: mysql, Kafka, ElasticSearch, Hbase, RocketMQ, etc. Of course, Canal specially designed the
insert image description here
client-server mode. The interactive protocol uses protobuf 3.0, and the client side can Use different languages to implement different consumption logic

canal java client: https://github.com/alibaba/canal/wiki/ClientExample
canal c# client: https://github.com/dotnetcore/CanalSharp
canal go client: https://github.com/CanalClient/canal-go
canal Python client: https://github.com/haozi3156666/canal-python

Canal, as a MySQL binlog incremental acquisition and analysis tool, can synchronize data to canal-client, which is our application, through the TCP protocol, so we can use the following scheme to synchronize data
insert image description here

The first choice is to enable Mysql bin-log
Then you need to install canal-server pretending to be a slave to synchronize data in mysql
Write the canal-client client to listen to the canal-server and synchronize the data from the canal-server
Then write the obtained data to Redis

Enable Mysql bin-log log

Find the my.ini configuration file in the Mysql installation directory. I take mysql 5.5 as an example and configure it as follows under mysqld

[mysqld]
#开启bInlog
log-bin=mysql-bin
#给mysql服务指定一个唯一的ID
server-id=1
#以数据的方式写binlog日志 ：statement 是记录SQL，row是记录数据
binlog-format=ROW
#同步的数据库名
#binlog-do-db=canaldb
#忽略的表
binlog-ignore-db=mysql
# 启动mysql时不启动grant-tables授权表
skip-grant-tables

After modification, restart the Mysql service. Note: I specified that the database to be synchronized is canaldb, so a database needs to be created, and an employee table is created as a demonstration, and then
insert image description here
a user is created and provided to canal to link Mysql for data synchronization

flush privileges;
#创建用户cannal
CREATE USER canal IDENTIFIED BY 'canal';
#把所有权限赋予canal，密码也是canal
GRANT ALL PRIVILEGES ON canaldb.user TO 'canal'@'%' identified by "canal";
//GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%' identified by "canal";
#刷新权限
flush privileges;

At this point, the Mysql part is done

Install Canal

Go to the official website to download Canal: https://github.com/alibaba/canal/releases, I am using canal.deployer-1.1.5.tar.gz version insert image description here

After downloading, unzip it, and the directory structure is as follows
insert image description here
Next, modify the instance configuration file: conf/example/instance.properties

#  按需修改成自己的数据库信息
#################################################
...
#我的端口是3307
canal.instance.master.address=192.168.1.20:3307
# username/password,数据库的用户名和密码
...
#刚才开通的mysql的账户密码
canal.instance.dbUsername = canal
canal.instance.dbPassword = canal
...
# 同步的表的规则
# table regex
# 同步所有表
#canal.instance.filter.regex=.*\\..*
# 同步多个表，用逗号隔开
canal.instance.filter.regex=canaldb.employee,canaldb.dept
#################################################

...省略...

Pay attention to the following things here, don’t worry about others

master.address : Mysql address, my port is 3307, the default is 3306
dbUsername: Mysql user opened above
dbPassword: password
ccanal.instance.filter.regex : Tables to be synchronized, multiple tables separated by commas

Then modify the canal configuration file conf/canal.properties

# ...
# 可选项: tcp(默认), kafka, RocketMQ
# 这里使用tcp , 还支持kafka和rocketmq
canal.serverMode = tcp
...省略...

Note here: canal.serverMode = tcp: I take tcp as an example here, which refers to synchronizing data with the tcp protocol instead of synchronizing to mq

After configuration, find the startup.bat in the bin directory of the canal installation directory, double-click to start, and start on linux: startup.sh
insert image description here

write canal-client

Next, we need to integrate the canal-client in the project to synchronize the data in the canal-server, and then write to Redis

Step 1: Import the following dependencies, I used here canal-spring-boot-starterto integrate canal-client

<parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.2.5.RELEASE</version>
    </parent>
    <dependencies>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
        </dependency>
        <!--Canal 依赖-->
        <dependency>
            <groupId>top.javatool</groupId>
            <artifactId>canal-spring-boot-starter</artifactId>
            <version>1.2.1-RELEASE</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.50</version>
        </dependency>
    </dependencies>

Step 2: Configure the canal address and Redis related parameters in yaml

canal:
  server: 127.0.0.1:11111 #canal的地址
  destination: example #默认的数据同步的目的地
spring:
  redis:
    host: 127.0.0.1
    password: 123456

Write startup class

@SpringBootApplication
public class CanalApplication {
    
    
    public static void main(String[] args) {
    
    
        SpringApplication.run(CanalApplication.class,args);
    }
}

Step 3: Configure Redis to realize automatic serialization

//缓存的配置
@Configuration
public class RedisConfig {
    
    

    @Resource
    private RedisConnectionFactory factory;


    //使用JSON进行序列化
    @Bean
    public RedisTemplate<Object, Object> redisTemplate() {
    
    
        RedisTemplate<Object, Object> redisTemplate = new RedisTemplate<>();

        redisTemplate.setConnectionFactory(factory);
        //JSON格式序列化
        GenericFastJsonRedisSerializer serializer = new GenericFastJsonRedisSerializer();
         //key的序列化
        redisTemplate.setKeySerializer(serializer);
        //value的序列化
        redisTemplate.setValueSerializer(serializer);
        //hash结构key的虚拟化
        redisTemplate.setHashKeySerializer(new StringRedisSerializer());
        //hash结构value的虚拟化
        redisTemplate.setHashValueSerializer(serializer);
        return redisTemplate;
    }

}

Step 4: Write the entity class, corresponding to the table of the database to be synchronized

@Data
public class Employee {
    
    
    private Long id;
    private String username;
}

Step 5: Write a data synchronization processor. canal-client provides EntryHandler, which provides insert, delete, and update methods. When the related operations of a certain table are monitored, the corresponding method will be called back to pass the data in. We can get the data and synchronize it to Redis.

@CanalTable("employee")
@Component
@Slf4j
public class EmployeeHandler implements EntryHandler<Employee> {
    
    

	//把数据往Redis同步
    @Autowired
    private RedisTemplate<Object,Object> redisTemplate;

    @Override
    public void insert(Employee employee) {
    
    
        redisTemplate.opsForValue().set("EMP:"+employee.getId(),employee);
    }

    @Override
    public void delete(Employee employee) {
    
    
        redisTemplate.delete("EMP:"+employee.getId());
    }

    @Override
    public void update(Employee before, Employee after) {
    
    
        redisTemplate.opsForValue().set("EMP:"+after.getId(),after);
    }
}

@CanalTable("employee") : the monitored table
EntryHandler<Employee>: After getting the changed data of the employee table, it will be packaged as an Employee entity and delivered to us

The code is written here, and the startup program can see that the canal-client is trying to obtain data from the console.

insert image description here
After starting redis, try to manually modify the data in the employee table of the database, and then use redis-cli to view the data. The following is the data in the table

insert image description here

The following is the data in redis

insert image description here

Well, that’s the end of the article. If you like it, please give it a good review. It’s even better if you accidentally come to a one-click triple link!!!