The record about the issue canal. Data synchronization has been using open source Ali's canal, recently used some of the problems encountered in the process, here to record it.
the reason
We use the canal monitor MySQL, then gets sent to the mq (custom format) by the client. Recent data sets colleagues batch update the data once, probably more than 800 million, resulting in a client occur oom, not consumption.
solution
Modify the server side of the filter, remove the monitor of the table, so that client will not obtain the data for this table, and get a lot of data to avoid client server-side. (In fact, the fundamental problem is that the client wrote the rough, the current method does not modify the source code on the basis of a good solution to the current problem, and then follow-up on the client to optimize protocols)
Let us do some custom management, you can modify instance.properties this file in real time. Automatically scans automatically restart after configuration changes and canal server opens the auto scan.
