DBSyncer is an open source data synchronization middleware that provides synchronization scenarios such as MySQL, Oracle, SqlServer, PostgreSQL, Elasticsearch(ES), Kafka, File, etc.SQL
It supports uploading plug-ins to customize synchronous conversion services, and provides monitoring full and incremental data statistical charts, application performance warnings, etc.
Features:
Combination-driven , custom library synchronization to library combination, combination between relational database and non-relational database, any matching table synchronization mapping relationship
Real-time monitoring , driving full or incremental real-time synchronization of running status, results, synchronization logs and system logs
Develop plug-ins to customize conversion synchronization logic
design
architecture diagram
Application Scenario
| Connector | data source | target source | Supported versions (including the following) |
|---|---|---|---|
| MySQL | ✔ | ✔ | 5.7.19 or higher |
| Oracle | ✔ | ✔ | 10gR2 -11g |
| SqlServer | ✔ | ✔ | 2008 and above |
| PostgreSQL | ✔ | ✔ | 9.5.25 or higher |
| ES | ✔ | ✔ | 6.0 or higher |
| Kafka | In development | ✔ | 2.10-0.9.0.0 or above |
| File | ✔ | ✔ | *.txt, *.unl |
| SQL | ✔ | Supports the above relational databases | |
| late plan | Redis |
tutorial
Start agent and database services
Prepare a test account, test
Assign the sysadmin role
Assign specific database permissions
PostgreSQL
Monitor incremental events through replication stream technology, and implement parsing of wal logs pgoutputbased on built-in plug-instest_decoding
1. Modify postgresql.confthe file and restart the service
wal_level=logical2. Grant account permissions LOGINandREPLICATION
File
Monitor the file modification time to get the changed file, and read the latest data through the file offset
“https://gitee.com/ghi/dbsyncer/issues/I55EP5
ES
Acquire incremental data regularly.
The account has access rights.
log
It is recommended that MySQL, SqlServer, and PostgreSQL all use logs
timing
Assumed source table data format
preview
driver management
Driver Details
Driver table field relationship configuration
monitor
upload plugin
open source address
https://gitee.com/ghi/dbsyncer
------
We have created a high-quality technical exchange group. When you are with excellent people, you will become excellent yourself. Hurry up and click to join the group and enjoy the joy of growing together. In addition, if you want to change jobs recently, I spent 2 weeks a year ago collecting a wave of face-to-face experience from big factories. If you plan to change jobs after the festival, you can click here to claim it !
recommended reading
Still using open source software to build ChatGPT website? Beware of others spending your money!
True punishment! A few lines of code end the entire educational administration system
··································
Hello, I am DD, a programmer. I have been developing a veteran driver for 10 years, MVP of Alibaba Cloud, TVP of Tencent Cloud. From general development to architect to partner. Along the way, my deepest feeling is that we must keep learning and pay attention to the frontier. As long as you can persevere, think more, complain less, and work hard, it will be easy to overtake on corners! So don't ask me if it's too late to do what I do now. If you are optimistic about something, you must persevere to see hope, not to persevere only when you see hope. Believe me, as long as you stick to it, you will be better than now! If you have no direction yet, you can follow me first, and I will often share some cutting-edge information here to help you accumulate capital for cornering and overtaking.