MongoShake简介
MongoShake是阿里云以Golang语言编写的通用平台型服务工具，完全开源，它通过读取MongoDB的Oplog操作日志来复制MongoDB的数据以实现特定需求。
基于mongodb oplog的集群复制工具，可以满足迁移和同步的需求，进一步实现灾备和多活功能。可以实现MongoDB数据库间的数据同步，该功能可用于数据分析、灾备和多活等业务场景。
MongoShake还提供了日志数据的订阅和消费功能，可通过SDK、Kafka、MetaQ等方式的灵活对接，适用于日志订阅、数据中心同步、Cache异步淘汰等场景。
项目地址：https://github.com/alibaba/MongoShake
参考文档：https://help.aliyun.com/document_detail/122621.html
mongodb 数据迁移
1、在kubernetes集群中创建2个示例 mongodb 副本集集群，作为迁移源端和目标端。
添加bitnami mongodb helm仓库
helm repo add bitnami https://charts.bitnami.com/bitnami
部署第一个集群
helm upgrade -i mongodb bitnami/mongodb -n mongodb --create-namespace \
--set architecture=replicaset \
--set replicaCount=2 \
--set externalAccess.enabled=true \
--set externalAccess.service.type=NodePort \
--set externalAccess.service.nodePorts[0]='30700' \
--set externalAccess.service.nodePorts[1]='30701'
部署第二个集群
helm upgrade -i mongodb-dest bitnami/mongodb -n mongodb-dest --create-namespace \
--set architecture=replicaset \
--set replicaCount=2 \
--set externalAccess.enabled=true \
--set externalAccess.service.type=NodePort \
--set externalAccess.service.nodePorts[0]='30702' \
--set externalAccess.service.nodePorts[1]='30703'
获取第一个集群的root密码
export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace mongodb-dest mongodb-dest -o jsonpath="{.data.mongodb-root-password}" | base64 -d)
echo $MONGODB_ROOT_PASSWORD
安装mongodb客户端工具，连接到第一个mongodb集群，连接方式为nodeip+nodeport，地址为pod所在节点地址。
[root@Ops-repo mongo-shake]# mongo --host "192.168.0.23:30700,192.168.0.24:30701" --authenticationDatabase admin -u root -p K1IySNf3JM
MongoDB shell version v4.4.4
connecting to: mongodb://192.168.0.23:30700,192.168.0.24:30701/admin?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session {
"id" : UUID("71963472-8169-4248-8253-fbde1d981d39") }
MongoDB server version: 6.0.2
WARNING: shell and server versions do not match
---
The server generated these startup warnings when booting:
2022-11-01T05:14:02.656+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
2022-11-01T05:14:05.060+00:00: vm.max_map_count is too low
2022-11-01T05:14:05.060+00:00: currentValue: 262144
2022-11-01T05:14:05.060+00:00: recommendedMinimum: 1677720
2022-11-01T05:14:05.060+00:00: maxConns: 838860
---
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
rs0:PRIMARY>
迁移源端创建示例数据
> use testdb;
> db.createCollection("users");
> db.myusers.insert({
"name":"jack", age:24});
MongoShake安装
1、安装MongoShake程序
wget https://github.com/alibaba/MongoShake/releases/download/release-v2.8.1-20220922/mongo-shake-v2.8.1.tgz
执行如下命令将MongoShake解压到/opt/mongoshake目录中。
tar zxvf mongo-shake-v2.8.1.tgz -C /opt
mv /opt/mongo-shake-v2.8.1/ /opt/mongo-shake
cd /opt/mongo-shake
2、修改MongoShake的配置文件collector.conf，涉及的主要参数说明如下所示。
说明：密码中不得包含艾特（@）字符，否则会导致连接失败。
mongo_urls 为源端MongoDB实例的ConnectionStringURI格式连接地址。示例如下：
mongo_urls = mongodb://root:[email protected]:30700,192.168.0.24:30701
tunnel.address为目标端MongoDB实例的ConnectionStringURI格式连接地址。示例如下：
tunnel.address = mongodb://root:[email protected]:30702,192.168.0.24:30703
sync_mode 为数据同步的方式，默认值为incr。all表示全量+增量同步，full表示全量同步，incr表示增量同步。
sync_mode = incr
执行迁移，执行下述命令启动同步任务，并打印日志信息。
./collector.linux -conf=collector.conf -verbose
观察打印的日志信息，当出现如下日志时，即代表全量数据同步已完成，并进入增量数据同步模式。
[root@Ops-repo mongo-shake]# ./collector.linux -conf=collector.conf -verbose 2
[2022/11/01 13:45:36 CST] [INFO] log init succ. log.dir[] log.name[collector.log] log.level[info]
[2022/11/01 13:45:36 CST] [INFO] MongoDB Version Source[6.0.2] Target[6.0.2]
[2022/11/01 13:45:36 CST] [WARN]
______________________________
\ \ _ ______ |
\ \ / \___-=O'/|O'/__|
\ MongoShake, Here we go !! \_______\ / | / )
/ / '/-==__ _/__|/__=-| -GM
/ Alibaba Cloud / * \ | |
/ / (o)
------------------------------
if you have any problem, please visit https://github.com/alibaba/MongoShake/wiki/FAQ
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] Close client with mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] Close client with mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701
[2022/11/01 13:45:36 CST] [INFO] Collector startup. shard_by[collection] gids[[]]
[2022/11/01 13:45:36 CST] [INFO] Collector configuration {
"ConfVersion":10,"Id":"mongoshake","MasterQuorum":false,"FullSyncHTTPListenPort":9101,"IncrSyncHTTPListenPort":9100,"SystemProfilePort":9200,"LogLevel":"info","LogDirectory":"","LogFileName":"collector.log","LogFlush":false,"SyncMode":"incr","MongoUrls":["mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701"],"MongoCsUrl":"","MongoSUrl":"","MongoSslRootCaFile":"","MongoSslClientCaFile":"","MongoConnectMode":"secondaryPreferred","Tunnel":"direct","TunnelAddress":["mongodb://root:***@192.168.0.23:30702,192.168.0.24:30703"],"TunnelMessage":"raw","TunnelKafkaPartitionNumber":1,"TunnelJsonFormat":"","TunnelMongoSslRootCaFile":"","FilterNamespaceBlack":[],"FilterNamespaceWhite":[],"FilterPassSpecialDb":[],"FilterDDLEnable":false,"FilterOplogGids":false,"CheckpointStorageUrl":"mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701","CheckpointStorageDb":"mongoshake","CheckpointStorageCollection":"ckpt_default","CheckpointStorageUrlMongoSslRootCaFile":"","CheckpointStartPosition":1,"TransformNamespace":[],"SpecialSourceDBFlag":"","FullSyncReaderCollectionParallel":6,"FullSyncReaderWriteDocumentParallel":8,"FullSyncReaderDocumentBatchSize":128,"FullSyncReaderFetchBatchSize":1024,"FullSyncReaderParallelThread":1,"FullSyncReaderParallelIndex":"_id","FullSyncCollectionDrop":true,"FullSyncCreateIndex":"none","FullSyncReaderOplogStoreDisk":false,"FullSyncReaderOplogStoreDiskMaxSize":256000,"FullSyncExecutorInsertOnDupUpdate":false,"FullSyncExecutorFilterOrphanDocument":false,"FullSyncExecutorMajorityEnable":false,"IncrSyncMongoFetchMethod":"oplog","IncrSyncChangeStreamWatchFullDocument":false,"IncrSyncReaderFetchBatchSize":1024,"IncrSyncOplogGIDS":[],"IncrSyncShardKey":"collection","IncrSyncShardByObjectIdWhiteList":[],"IncrSyncWorker":8,"IncrSyncTunnelWriteThread":8,"IncrSyncTargetDelay":0,"IncrSyncWorkerBatchQueueSize":64,"IncrSyncAdaptiveBatchingMaxSize":1024,"IncrSyncFetcherBufferCapacity":256,"IncrSyncExecutorUpsert":false,"IncrSyncExecutorInsertOnDupUpdate":false,"IncrSyncConflictWriteTo":"none","IncrSyncExecutorMajorityEnable":false,"CheckpointStorage":"database","CheckpointInterval":5000,"FullSyncExecutorDebug":false,"IncrSyncDBRef":false,"IncrSyncExecutor":1,"IncrSyncExecutorDebug":false,"IncrSyncReaderDebug":"","IncrSyncCollisionEnable":false,"IncrSyncReaderBufferTime":1,"IncrSyncWorkerOplogCompressor":"none","IncrSyncTunnelKafkaDebug":"","Version":"improve-2.8.1,9f2515bbe508b2d86fab2ed87f01f3121efc6920,release,go1.15.10,2022-09-22_11:49:22","SourceDBVersion":"6.0.2","TargetDBVersion":"6.0.2","IncrSyncTunnel":"","IncrSyncTunnelAddress":null,"IncrSyncTunnelMessage":"","HTTPListenPort":0,"SystemProfile":0}
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] Close client with mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] Close client with mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701
[2022/11/01 13:45:36 CST] [INFO] GetAllTimestamp biggestNew:{
1667281485 1}, smallestNew:{
1667281485 1}, biggestOld:{
1667279683 1}, smallestOld:{
1667279683 1}, MongoSource:[url[mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701], name[rs0]], tsMap:map[rs0:{7160911711770247169 7160919451301314561}]
[2022/11/01 13:45:36 CST] [INFO] all node timestamp map: map[rs0:{
7160911711770247169 7160919451301314561}] CheckpointStartPosition:{
1 0}
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] rs0 Load exist checkpoint. content {
"name":"rs0","ckpt":7160913867843829761,"version":2,"fetch_method":"oplog","oplog_disk_queue":"","oplog_disk_queue_apply_finish_ts":0}
[2022/11/01 13:45:36 CST] [INFO] rs0 checkpoint using mongod/replica_set: {
"name":"rs0","ckpt":7160913867843829761,"version":2,"fetch_method":"oplog","oplog_disk_queue":"","oplog_disk_queue_apply_finish_ts":0}, ckptRemote set? [true]
[2022/11/01 13:45:36 CST] [INFO] sync mode run incr
[2022/11/01 13:45:36 CST] [INFO] start running with mode[incr], fullBeginTs[0[0, 0]]
[2022/11/01 13:45:36 CST] [INFO] start incr replication
[2022/11/01 13:45:36 CST] [INFO] RealSourceIncrSync[0]: url[mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701], name[rs0], startTimestamp[7160913867843829761]
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30702,192.168.0.24:30703 successfully
监控MongoShake状态，增量数据同步开始后，可以再开启一个命令行窗口，通过如下命令来监控MongoShake。
cd /opt/mongoshake && ./mongoshake-stat --port=9100
说明： mongoshake-stat是一个Python脚本，执行之前请先安装Python 2.7版本。
监控输出示例。
[root@Ops-repo ~]# cd /opt/mongo-shake/
[root@Ops-repo mongo-shake]# ./mongoshake-stat --port=9100
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| log_size_avg | log_size_max | logs_get/sec | logs_repl/sec | logs_success/sec | lsn.time | lsn_ack.time | lsn_ckpt.time | now.time | replset | tps/sec |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| 331.00B | 339.00B | none | none | none | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-11-01 13:21:22 | rs0 | none |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| 331.00B | 339.00B | 0 | 0 | 0 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-11-01 13:21:23 | rs0 | 0 |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| 331.00B | 339.00B | 0 | 0 | 0 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-11-01 13:21:24 | rs0 | 0 |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| 217.00B | 339.00B | 1 | 0 | 0 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-11-01 13:21:25 | rs0 | 0 |
连接目标端副本集验证数据同步成功
[root@Ops-repo bin]# mongo --host "192.168.0.23:30702,192.168.0.24:30703" \
> --authenticationDatabase admin -u root -p o72ntfVFVL
MongoDB shell version v4.4.4
connecting to: mongodb://192.168.0.23:30702,192.168.0.24:30703/admin?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session {
"id" : UUID("18118dec-ac80-401b-bba5-2c8a80bd0066") }
MongoDB server version: 6.0.2
WARNING: shell and server versions do not match
---
The server generated these startup warnings when booting:
2022-11-01T05:18:05.971+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
2022-11-01T05:18:07.811+00:00: vm.max_map_count is too low
2022-11-01T05:18:07.811+00:00: currentValue: 262144
2022-11-01T05:18:07.811+00:00: recommendedMinimum: 1677720
2022-11-01T05:18:07.811+00:00: maxConns: 838860
---
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
rs0:PRIMARY>
rs0:PRIMARY> show dbs;
admin 0.000GB
config 0.000GB
local 0.001GB
testdb 0.000GB
rs0:PRIMARY> use testdb;
switched to db testdb
rs0:PRIMARY> show tables;
users
rs0:PRIMARY> show collections;
users
rs0:PRIMARY>