Incremental data collection based on oracle

I have investigated MySQL-based incremental data collection solutions before. Currently, there are Ali's canal and Open Replicator, both of which are open source projects based on MySQL binlog analysis. Open Replicator only provides binlog parsing; canal provides incremental data subscription & consumption based on database incremental log parsing, encapsulated on the basis of binlog analysis, and has a matching canal consumer project open source: Otter (distributed database Synchronization system), there is a relatively complete set of incremental data subscription & consumption scheme.

In addition to MySQL, canal also provides log parsing for some versions of Oracle, but unfortunately only the MySQL part is currently open source. For Oracle incremental data synchronization, the first thing that comes to mind is triggers. Triggers are established on the test table for testing, and data additions, deletions, and changes can be monitored. But the trigger event is in the same thing as the trigger, and the trigger is executed before commit. If there are some non-transactional operations in the trigger, such as: sending change events and data to the outside, so that the uncommitted data has been sent out. At this time, when the transaction is rolled back, the non-transactional operation cannot be undone. Therefore, we must have the trigger listen for changes to the committed data. After investigation, the establishment of triggers on the materialized view can achieve our purpose.

create materialized view log on jd_address;

create materialized view MV_JD_ADDRESS_TEST
refresh fast on commit
as
select * from jd_address;

SELECT * FROM MV_JD_ADDRESS_TEST

Create a materialized view of the table, and incrementally refresh the materialized view when committing.

As mentioned above, data is sent to the outside world. After Oracle8, it provides support for Java, and you can call external programs through Java. There are many ways to introduce it . The calling process is as follows: trigger > stored procedure > java source > external program . Combined with the materialized view mentioned above, it is possible to monitor incremental data changes and synchronize them to external programs in time.

For the use of Java Source in Oracle, authorization is generally required. We can open the Java Source log first, and execute the stored procedure in the command window to call the java source to view the print information. If it is not authorized, an error message will be printed, and the statement that requires authorization will be provided, and the copy can be executed.

SQL> set serveroutput on;
SQL> exec dbms_java.set_output(5000);

The stored procedure call Java Source code is as follows:

create or replace procedure prc_test(str varchar2) as language java name 'Demo.entry(java.lang.String)';

Reference documentation: Oracle CDC provides a tigger-based synchronous mode and a streaming replication-based asynchronous mode, which is also a good solution.

quote

Talking about the data synchronization scheme between Oracle databases: http://blog.csdn.net/leamonjxl/article/details/6695479

quote

Oracle CDC(Change Data Capture)概述： http://blog.csdn.net/chensrao/article/details/6200338

quote

Design and implementation of real-time monitoring oracle database table data changes: http://blog.csdn.net/as339000204/article/details/45390727

quote

Oracle uses Java Source programming: http://www.2cto.com/database/201308/236506.html

quote

Oracle uses java source to call external programs: http://www.cnblogs.com/mellowsmile/archive/2016/09/18/5881109.html