Docker-compose builds maxwell (realizes synchronous database binlog to kafka)

Docker-compose build maxwell

Maxwell synchronizes MySQL binlog to Kafka, and uses Python consumers to process messages. The architecture is as follows :

+------------+             +------------+              +-----------------+             +-----------------+
|   MySQL    |    binlog   |   Maxwell  |    Kafka     |    Kafka topic   |    Python   |    Consumer     |
|  database  +------------>  connector +------------->  (message queue) +------------>  application(s) |
|            |             |            |              |                 |             |                 |
+------------+             +------------+              +-----------------+             +-----------------+

Maxwell realizes synchronous database binlog preliminary preparation :
Please refer to my previous article:
Build mysql service : Docker-compose build Mysql8.0 (and open binlog)
Build kafka service : Docker-compose build Kafka (and open kafka manage interface)

1. Directory structure

.
└── docker_maxwell
	├── docker-compose.yml
	├── maxwell
    └── conf
    	└── config.properties

2. docker-compose.yml

version: '3.5'
services:
  maxwell:
    image: zendesk/maxwell
    network_mode: "host"
    command: bin/maxwell --config /etc/maxwell/config.properties
    volumes:
      - ./conf:/etc/maxwell/

3. Create a conf folder and add a config.properties file under the conf folder

Note: Replace the IP in the file with your own local IP

daemon=true
# 第一次启动时建议改为debug，可以开到mysql数据与kafka请求，稳定后再改为info
log_level=info

producer=kafka
kafka.bootstrap.servers=IP:9092
# 会往 kafka下主题为'test'的分区下推送数据
kafka_topic=test
#当producer_partition_by设置为table时，Maxwell会将生成的消息根据表名称进行分区，不同的表将会被分配到不同的分区中，默认为database
producer_partition_by=table
client_id=maxwell_1

# mysql login info 需要先在mysql创建maxwell用户
host=IP
port=33106
user=maxwell
password=Abc123654
schema_database=maxwell

4. Create maxwell user in mysql and authorize

Execute the following sql in order
a. Create a maxwell database (after maxwell starts, a table will be built under the database, and some data will be saved when running)

CREATE DATABASE `maxwell`;

b. Create a maxwell user with the password Abc123654 (the default user name and password in the maxwell configuration file), and authorize the management of the database:

GRANT ALL ON maxwell.* TO 'maxwell'@'%' IDENTIFIED BY 'Abc123654';

c. Assign select, replication slave, replication client permissions:

grant select ,replication slave,replication client on *.* to maxwell@'%';

d. Refresh permissions, effective immediately

flush privileges;

5. Prepare test database and table in mysql

# a. 创建toutiao数据库
CREATE DATABASE `toutiao`;
# b.  在创建新闻频道表
CREATE TABLE `news_channel` (
  `channel_id` int unsigned NOT NULL AUTO_INCREMENT COMMENT '频道ID',
  `channel_name` varchar(32) CHARACTER SET utf8mb3 NOT NULL COMMENT '频道名称',
  `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
  `update_time` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间',
  `sequence` int unsigned DEFAULT '0' COMMENT '序号',
  `is_visible` tinyint(1) DEFAULT '0' COMMENT '是否可见',
  `is_default` tinyint(1) NOT NULL DEFAULT '0' COMMENT '是否默认',
  PRIMARY KEY (`channel_id`),
  UNIQUE KEY `channel_name` (`channel_name`)
) ENGINE=MyISAM AUTO_INCREMENT=30 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci COMMENT='新闻频道表';

6. Start the Maxwell service

# 进入docker_maxwell目录下
cd /系统目录/maxwell
# 启动服务
docker compose up -d

7. Mysql binglog synchronization service verification

a. Insert data into the news_channel table

INSERT INTO news_channel
(channel_id, channel_name, create_time, update_time, `sequence`, is_visible, is_default)
VALUES(30, 'android开发', '2023-03-14 08:23:27', '2023-03-14 08:23:27', 0, 0, 0);

b. Execute the following code in python to check whether the synchronization is successful (the IP needs to be replaced with the IP of your own kafka, this code is in the Docker-compose to build Kafka (and open the kafka manage interface) )

import json
from kafka import KafkaConsumer, TopicPartition
k_topic = 'test'

consumer = KafkaConsumer(k_topic, bootstrap_servers=['IP:9092'],
                         enable_auto_commit=False,
                         auto_offset_reset="earliest",
                         group_id='test')
# auto_offset_reset
# 1. 从上一次未消费的位置开始读（则该参数设置为earliest）;
# 2. 从当前时刻开始读之后产生的, 之前产生的数据不再消费（则该参数设置为latest）
for message in consumer:
    print(message)
    print(message.topic)
    print(f"receive: \n  key: {
      
      json.loads(message.key.decode())},\n  value: {
      
      json.loads(message.value.decode())} \n ")
    # 手动commit
    # consumer.commit()

The execution results are as follows (this code is a consumer of kafka, and the script will always be executed if it is not actively closed):