Use TCP to pull Canal data

1 Canal docking with Kafka joint debugging

1.1 Configuration modification

canal.properties

Modify zk:

canal.zkServers = 10.51.50.219:2181

instance.properties

Enable configuration items:

canal.mq.dynamicTopicIt is Canal's MQ dynamic Topic configuration item:

• test_javaedge_01It is the topic of kafka
• test_db.usersDatabases and tables to be monitored
• When test_db.usersthe table changes, Canal will push the changed data to the test_javaedge_01:test_db.usersMQ Topic named.

canal.mq.dynamicTopic=test_javaedge_01:test_db\\.users

Open a consumer

[root@javaedge-kafka-dev bin]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test_javaedge_01

datagrip adds new data:

Consume this data:

2 Use TCP to pull Canal data

Now change the serverMode back to tcp. Restart

javaedge@JavaEdgedeMac-mini deployer % jps
71002 CanalLauncher
javaedge@JavaEdgedeMac-mini deployer %

canal synchronization program

package com.javaedge.canal;

import com.alibaba.fastjson.JSON;
import com.alibaba.otter.canal.client.CanalConnector;
import com.alibaba.otter.canal.client.CanalConnectors;
import com.alibaba.otter.canal.protocol.CanalEntry;
import com.alibaba.otter.canal.protocol.Message;
import com.google.common.base.CaseFormat;

import java.net.InetSocketAddress;
import java.util.HashMap;
import java.util.List;

public class CanalClientApp {
    
    
    public static void main(String[] args) throws Exception {
    
    

        CanalConnector connector = CanalConnectors.newSingleConnector(
                new InetSocketAddress("localhost", 11111),
                "example",
                null, null);

        while (true) {
    
    
            connector.connect();
            connector.subscribe("test_db.users");
            Message message = connector.get(100);
            List<CanalEntry.Entry> entries = message.getEntries();
            if (entries.size()>0) {
    
    
                for (CanalEntry.Entry entry : entries) {
    
    
                    String tableName = entry.getHeader().getTableName();

                    CanalEntry.RowChange rowChange = CanalEntry.RowChange.parseFrom(entry.getStoreValue());
                    List<CanalEntry.RowData> rowDatasList = rowChange.getRowDatasList();
                    CanalEntry.EventType eventType = rowChange.getEventType();

                    if (eventType == CanalEntry.EventType.INSERT) {
    
    
                        for (CanalEntry.RowData rowData : rowDatasList) {
    
    
                            List<CanalEntry.Column> afterColumnsList = rowData.getAfterColumnsList();
                            HashMap<Object, Object> map = new HashMap<>();
                            for (CanalEntry.Column column : afterColumnsList) {
    
    
                                String key = CaseFormat.LOWER_UNDERSCORE.to(CaseFormat.LOWER_CAMEL, column.getName());
                                map.put(key, column.getValue());
                            }
                            System.out.println("tableName=" + tableName + "  map=" + JSON.toJSONString(map));
                        }
                    }
                }

            }

        }

    }
}

Run the program. Operate the user data table and add a new row of data:

Program output:

Obviously, no matter where you want to synchronize the data later, you are completely free!

data link

MySQL -》canal server（tcp）-》canal client-》kafka。