otter and canal learning summary

1. Can multiple nodes be configured on the same machine? (OK, yes, but it is basically unnecessary. Since the node itself can be configured for use by two channels, and the libraries of two sources can be specified to be synchronized, it is only

necessary to release the energy of a single node, unless it is necessary to ensure two channels. The upgrade of each node does not affect each other)


2. Can different libraries of the same source be synchronized using the same node? (OK, yes, but you must configure two canal names and use two pipelines)


3. Can two libraries from different sources be synchronized with the same node? (Yes, but two canal must be established)


4. Can two tables of a pipeline be two libraries? (Yes) Can it be two sources? (No)


5. What is the load capacity of each node and how much data transmission speed does it support? (Under the default configuration, the memory of 10,000 pieces of data per second occupies about 200M)


6. Configure the automatic master-slave switchover of canal to obtain data. Since the load is kafka, there is no need to configure the master-standby switch of load for the time being.


7. Do I need to install canal after using otter? (No need, otter embeds canal in node, and starts multi-threading through the interface)



8. Node must add nid to node after manager is created, and node configuration must be connected to manager. Configured under otter.properties corresponding to node.


9. Two libraries must be established for the installation of the manager, an otter library to store the business configuration of the manager, including the running logs of nodes, etc., and a binlog library to store the binlog logs of each system to

prevent schmoozing, pay attention to this binlog real-time clearing.


10. Whether real-time data synchronization can be used, the following configuration needs to be verified

Check the format method of binlog
(1) show variables like '%binlog_format%';

(2) Turn on mysql's binlog, and configure

    log-bin=mysql-bin in mysql's my.ini #add this line and it's ok

    binlog-format= ROW #Select row mode

    Note that mysql needs to be restarted

(3) For the configuration user of canal, enable binlog dump permission

    CREATE USER canal IDENTIFIED BY 'canal'; -- If there is a user, it does not need to be created.
    GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
    -- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
    FLUSH PRIVILEGES


; As follows, otherwise there will be a phenomenon that cannot be clicked to start.

You need to ensure that the zk configuration and node configuration are added first.

Operation steps:
1. Add database
    a. Source library jdbc:mysql://10.20.144.25:3306
    b. Target library jdbc:mysql://10.20.144.29:3306
2.
    a. Provide database IP information
3. Add synchronization table information
    a. Source data table test.example
    b. Target data table test.example
4. Add channel
5. Add pipeline
    a. Select node node
    b. Select canal
6. Add synchronization mapping Rules
    a. Define the synchronization relationship between the source table and the target table
7. Start
8. Test data


12. Otter can configure different role users through the system to operate and view the system.