MongoDB Series 10: Change Streams Build Real-Time Synchronous Data Streams

Deng Kaibiao's actual combat MongoDB series of articles, very good, praise! Highly recommend!

This article is the 10th article. It mainly describes the practical experience of Change Streams in building real-time synchronous data streams, which is very worth reading.

Previous series of articles:

Kerberos authentication for MongoDB security practice

MongoDB Compass--A must-have management tool for MongoDB DBAs

MongoDB Security Practice Audit

MongoDB security practice SSL protocol encryption

Network security reinforcement of MongoDB security practice

Introduction to MongoDB Indexes

MongoDB storage engine

Incremental update of MongoDB collections

MongoDB Series 9: Migrating MongoDB Data to MySQL

1. Overview of ChangeStreams

MongoDB 's ChangeStreams allow applications to access data changes in real time. Applications can use Change Streams to subscribe to changes to all data on a collection and respond to them immediately. Using Change Streams , you can build real-time data synchronization applications.

2. Characteristics of ChangeStreams

Change Streams can only be turned on in replica sets or sharded clusters. In a sharded cluster, Change Streams operations must be turned on for the mongos route . The storage engine requirement is the WiredTiger storage engine.

3. Change event type of ChangeStreams

· insert event

update event _

· delete event

replace event _

· invalidate event

4. Open ChangeStreams

A ChangeStreams can be opened in Python in the following way :

cursor =db.collection_name.watch()

Among them, the following pipeline can be used to control the output of ChangeStreams :

·$match

·$project

·$addFields

·$replaceRoot

·$redact

5. Build a case of real-time synchronous data flow

This example uses Python to build real-time synchronization from MongoDB 's stream_table collection to mysql 's stream_table table. Among them, the three change events of insert , update and delete of ChangeStreams are monitored respectively.

640?wx_fmt=png

1) Execute the code

640?wx_fmt=png

2) Insert a document into the stream_table collection

In the initial case, both the stream_table collection and stream_table have no data. So first demonstrate the monitoring of the insert change event.

640?wx_fmt=png

At the same time, query whether the stream_table table of mysql has records:

640?wx_fmt=png

Data is synced in real time.

3) update collection stream_table data

640?wx_fmt=png

Similarly, query whether the records of mysql 's stream_table table have changed:

640?wx_fmt=png

Data is updated in real time!

4) The last delete collection stream_table data

640?wx_fmt=png

Finally, query whether the records of mysql 's stream_table table have changed:

640?wx_fmt=png

Data is deleted!

summary:

In the data migration from MongoDB to relational databases, for some scenarios that require high data timeliness, especially the pursuit of real-time data changes, MongoDB Change Streams can effectively solve such scenarios.

MongoDB Series 10: Change Streams Build Real-Time Synchronous Data Streams

Guess you like