Alibaba Open Source Project: Distributed Database Synchronization System otter (solving the computer room in China and the United States)

Project background

   Alibaba B2B company, because of the characteristics of the business, the sellers are mainly concentrated in China, and the buyers are mainly concentrated in foreign countries, so the demand for remote computer rooms in Hangzhou and the United States has been derived. At the same time, in order to improve user experience, the structure of the entire computer room is double A. , can be written on both sides, and thus a product such as otter was born.

   The first version of otter can be traced back to 2004~05. The external open source version is the fourth version. The development time has continued from July 2011 to the present. At present, the synchronization requirements of local/remote computer rooms within Alibaba B2B are basically complete. On the otte4.

Current synchronization scale:

600 million data synchronization, 1.5TB
file synchronization (2000w pictures),
involving synchronization between 200+ database instances,
80+ machines cluster size

Project introduction

Name: otter ['ɒtə(r)]

Translation: Otter, data porter

Language : Pure java development

Positioning: Based on incremental log parsing of the database, quasi-real-time synchronization to the mysql/oracle database in the local computer room or across computer rooms.


Working principle


Principle description:
1. Based on Canal open source products, obtain database increments log data. What is Canal, please click
2. Typical management system architecture, manager (web management) + node (worker node)
     a. The manager pushes the synchronization configuration to the node node when it is running
     b. The node node feeds back the synchronization status to the manager
3. Based on zookeeper , which solves distributed state scheduling and allows multiple nodes to work together.

What is canal?

A sub-project of Otter's open source, open source link address: http://github.com/alibaba/canal


What can Otter solve?

1. Heterogeneous database synchronization
   a. mysql -> mysql/oracle. (Currently the open source version only supports mysql incremental, and the target library can be mysql or oracle, depending on the function of canal)
2. Single room synchronization (RTT between databases < 1ms)
   a. Database version upgrade
   b. Data table migration
   c. Asynchronous secondary index
3. Synchronization across computer rooms (for example, Alibaba International Station is the same database in Hangzhou and the US computer room, RTT > 200ms, highlight)
   a. Disaster recovery in the computer room
4. Two-way synchronization
    a. Avoid loopback algorithm (general solution, support most relational databases)
    b. Data consistency algorithm (guarantee the final consistency of data in double A computer room mode, highlights)
5. File synchronization
    a. Site mirroring (copy related pictures while copying data, such as copying product data and copying product pictures at the same time).

Schematic diagram of single-room replication:

Description:
   a. Data on-Fly, as far as possible, do not land, and synchronize data faster . (Turn on the node loadBalancer algorithm, if the Node node S+ETL falls on different Nodes, the data will have a network transmission process)
   b. The node node can have failover / loadBalancer. Diagram of 

cross-machine room replication:

Description:
   a. The data involves network transmission, and the S/E/T/L stages will be scattered on 2 or more Node nodes. The nodes work together through zookeeper (usually Select and Extract are in the Node in one computer room, and Transform/Load falls on the Node in another computer room)
   b. The node node can have failover / loadBalancer. (The Node node in each computer room can be either Cluster, one or more machines)


related terms explained


otter core model relationship diagram







Glossary

Pipeline: the entire process description from the source to the target, mainly composed of some synchronization mapping process
Channel: synchronization channel, composed of a Pipeline in one-way synchronization , In two-way synchronization, there are two Pipelines to form
DateMediaPair: define the mapping relationship according to the business table, such as source table and target table, field mapping, field group, etc.
DateMedia: abstract data medium concept, which can be understood as data table/mq queue definition
DateMediaSource : Abstract data medium source information, supplementary description DateMedia
ColumnPair : Define field mapping relationship
ColumnGroup : Define field mapping group
Node : Work node that handles synchronization process, corresponding to a jvm
otter S/E/T/L stage model



Description: In order to better support the scalability and flexibility of the system, the entire synchronization process is abstracted into Select/Extract/Transform/Load, which are four stages.

Select stage: In order to solve the difference of data sources, such as access to canal Incremental data, you can also access other systems to obtain other data, etc.

Extract/Transform/Load phase: similar to the ETL model of data warehouse, specifically for data join, data transformation, data Load



related implementation introduction
   Otter scheduling model
   Otter data storage algorithm
   Otter two-way loop control
   Otter data consistency
   Otter high availability
   Otter Extensibility
QuickStart
See the page for quick start: QuickStart.

AdminGuide
See the page for admin deploy guide : AdminGuide