Oracle data synchronization solution databus

• Overview

At present, I know that Oracle-based open source data synchronization projects include yugong, databus, and SymmetricDS. I have tried yugong before, and it is easy to get started. There are two points to pay attention to when using: 1. Materialized view and return table query will have a relatively large impact on the source database; 2. Each table has a separate instance to handle the entire process, and data extraction, conversion, and storage cannot be separated; 3. The purpose of yugong design is to go to IOE and use it for library cutting. After multiple incremental synchronization confirmations, incremental synchronization can be stopped. If continuous incremental synchronization is required, each table needs to provide a thread, and polling continuous synchronization is not supported.

• Introduction to databus

databus is an SCN-based data synchronization project that uses triggers and JOBs to update SCN. Compared with yugong, its biggest advantage lies in the separation of server and client, and provides a complete consumption mechanism. For details, please refer to:

quote

https://github.com/linkedin/databus

quote

http://databus.majunwei.com/document/Connecting_Databus_to_an_Oracle_Database.html

quote

http://blog.sina.com.cn/s/blog_6c9b83990101bbzq.html

• databus use

databus uses gradle as the compilation tool, executes the command: gradle -Dopen_source=true assemble to package, the compilation always fails when packaging: "UTF-8 unmappable characters", I have tried many online gradle support configuration schemes for UTF-8 Will not work. Later, after reading the error message, the coding problems were all concentrated in the annotations of the same class, so I manually changed the annotations of the class (two symbols) and compiled successfully.

1. Create a database account

./createUser.sh [db_name] [db_pwd] 192.168.11.129:1521/ORCL1 [table_space] [/mnt/u001/oracle/data/DB] > /tmp/createUser.out

2. Database add field (txn), view, package (SYNC_ALERT, SYNC_CORE), procedure (COMPILE_ALLOBJECTS), trigger (before update txn), sequence, Jobs (J_CALL_SIGNAL, J_COALESCE_LOG)

./createSchema.sh db_name/[email protected]:1521/ORCL1 [tab, view file location] > /tmp/createSchema.out

Repeated execution will form duplicate data in the sy$sources table and sync_core_settings table, which needs to be deleted. The tab and view files correspond to one table for each table. The tablespace file stores the table space name.

3. Generate EventSchema, enter databus-util-cmdline\databus2-cmdline-tools-pkg, execute gradle assemble, package the EventSchema generation tool, unzip it, and enter bin Folder execution:

./dbus2-avro-schema-gen.sh -namespace com.linkedin.events.example.person -recordName [tablename] \
    -viewName "sy\$person" -avroOutDir /root/databus-master/databus2-example/databus2-example-relay-pkg/schemas_registry -avroOutVersion 1 \
    -javaOutDir /root/databus-master/databus2-example/databus2-example-person/src/main/java \
   -database jdbc:oracle:thin:@192.168.11.129:1521:ORCL1  -userName [db_name] -password [db_pwd]

Execute the above command: generate entity class, .avsc file

4. Write server code: a. Write sources-person.json, each table corresponds to a json file; b. Write the avsc file name generated above in index.schemas_registry; c. Server startup class

5. Write client code: refer to demo

6. Compile and package: enter databus2-example-client-pkg and databus2-example-relay-pkg respectively, run the command: gradle -Dopen_source=true clean assemble test, Copy the tarballs under the distribution folder under databus2-example-client-pkg and databus2-example-relay-pkg under databus-master/build to linux, decompress them, and run ./start-example-relay.sh under bin person and ./start-example-client.sh

summary of person:
1. A txn field needs to be added to each synchronized source table. The .tab file in the demo is a create statement. If the table exists, it can be changed to an update statement
. 2. Provide There are a lot of script files, basically involving database operations, so the execution of the script needs to be in the Linux environment and the system supports sqlplus. At present, no bat file is provided, and it cannot be executed under the window.
3. The communication between client and server provides http, rm, and rmi is closed by default. However, during the test, it was found that the client side reported an error every time it was started, and the rmi link was rejected, which did not affect the use.
4. avsc and entity classes need to be generated through scripts, and batch generation of multiple tables is not supported. Also need to write json file. In short, the development under the window is not very convenient, and the script needs to be refactored to simplify the preparation work.

refer to:

quote

http://blog.csdn.net/guojing505123/article/details/51361281

quote

https://github.com/linkedin/databus/wiki/Databus-2.0-Example