Quickly solve the trouble that binlog is not opened when Maxwell connects to MYSQL

Introduction to Maxwell

• Maxwell official website
• Maxwell is an open source MySQL database change data capture (CDC) tool, which can capture data changes in the MySQL database in real time, convert them into an easy-to-consume format, such as JSON or AVRO, and send them to a message queue (such as RabbitMQ, Kafka, etc.).
• Maxwell can help developers and data engineers build real-time data streams and data pipelines to support real-time analysis, data synchronization, data integration and other application scenarios. It provides simple configuration and scalability, and is compatible with common data processing tools and frameworks (such as Spark, Flink, Kafka Connect, etc.), making it easy to integrate into existing data architectures.

Maxwell uses

With Maxwell, it can be set up and configured as follows:

1. Install Maxwell: You can download the latest version of the binaries from Maxwell's Gitee repository , or install them through a package management tool such as Homebrew.

2. Configure Maxwell: Create a configuration file (usually config.properties) that contains configuration options for Maxwell. You can specify MySQL host and port, database username and password, target message queue, table filtering rules, etc.

3. Start Maxwell: Use the command line or the system service management tool to start Maxwell. For example, Maxwell can be started with the following command: maxwell --config /path/to/config.properties.

4. Listen for MySQL changes: Once Maxwell starts, it will start listening for changes to the MySQL database. When data is inserted, updated or deleted, Maxwell will capture these changes and forward them to the configured message queue.

5. Process changed data: You can use message queue consumers to process changed data sent by Maxwell. Depending on your needs, you can write custom consumer applications to process the data, or use existing data processing tools and frameworks.

Through these steps, you can start using Maxwell to capture and process the change data of the MySQL database to build real-time data streams and data pipelines.

• Note that the specific configuration and usage methods may vary according to your environment and needs. It is recommended to refer to Maxwell's official documents and examples for more detailed settings and configurations.

Maxwell connection to MYSQL shows that binlog is not opened

Confirm whether the binlog of the mysql container is open

• To confirm whether the binlog of the MySQL container is turned on, you can follow the steps below:

1. Enter the MySQL container: use the docker exec command to enter the command line interface of the MySQL container.

   docker exec -it <container_name> bash
   

2. Enter the MySQL command line interface: Use the mysql command to enter the MySQL command line interface.

   mysql -u <username> -p
   

3. Check the binlog status: In the MySQL command line interface, execute the following command to check the binlog status.

   SHOW VARIABLES LIKE 'log_bin';
   

   • If the value in the returned result is "ON", it means that the binlog is enabled; if the value in the returned result is "OFF", it means that the binlog is not enabled.

4. View the location of the binlog file: Execute the following command to view the location of the current binlog file.

   SHOW VARIABLES LIKE 'log_bin_basename';
   

   • The Value in the returned result is the location of the current binlog file.

• If the binlog is not enabled, it can be enabled by modifying the MySQL configuration file binlog. In MySQLa container, configuration files are typically located at /etc/mysql/my.cnfor /etc/mysql/mysql.conf.d/mysqld.cnf. Find the following configuration item and make sure its value is "ON":

  log_bin = ON
  

5. After the modification is complete, exit the mysql container and restart the MySQL container to make the configuration take effect.

   exit
   docker restart <container_name>
   

Interpretation of log information after resolution

2023-06-15 04:49:11 INFO  Maxwell - Starting Maxwell. maxMemory: 431816704 bufferMemoryUsage: 0.25
2023-06-15 04:49:12 INFO  Maxwell - Maxwell v1.40.2 is booting (RabbitmqProducer), starting at Position[BinlogPosition[binlog.000003:24293], lastHeartbeat=1686804505492]
2023-06-15 04:49:12 INFO  MysqlSavedSchema - Restoring schema id 1 (last modified at Position[BinlogPosition[binlog.000002:4762], lastHeartbeat=0])
2023-06-15 04:49:12 INFO  BinlogConnectorReplicator - Setting initial binlog pos to: binlog.000003:24293
2023-06-15 04:49:12 INFO  BinaryLogClient - Connected to mysql:3306 at binlog.000003/24293 (sid:6379, cid:98)
2023-06-15 04:49:12 INFO  BinlogConnectorReplicator - Binlog connected.

According to the logs provided, Maxwell has successfully started and connected to MySQL's binary log (binlog). Here is some important information about the logs:

• Maxwell version: Maxwell version is 1.40.2.
• Start time: Maxwell will start at 04:49:11 on June 15, 2023.
• Memory usage: Maxwell has a maximum memory limit of 431816704 bytes.
• RabbitMQ producer: Maxwell uses RabbitMQ as a message queue, and this log shows that Maxwell is using RabbitmqProducer.
• Position information: Maxwell uses BinlogPosition[binlog.000003:24293] as the initial position at startup.
• Database Schema Recovery: The log also shows that Maxwell is restoring the database schema with ID 1. This schema was last modified at BinlogPosition[binlog.000002:4762].
• Connect to MySQL Binlog: Maxwell successfully connects to MySQL binlog, the binlog file name is binlog.000003, and the initial location is 24293.

This log indicates that Maxwell has successfully started and connected to the MySQL binlog, and is ready to start fetching data changes and sending them to the RabbitMQ queue.