MongoShake の概要
MongoShake は、Alibaba Cloud によって Golang 言語で書かれた汎用プラットフォーム サービス ツールで、完全にオープン ソースであり、MongoDB の Oplog 操作ログを読み取り、特定の要件を満たすように MongoDB のデータをコピーします。
mongodb oplog に基づくクラスター レプリケーション ツールは、移行と同期のニーズを満たし、さらに災害復旧とマルチアクティブ機能を実現します。MongoDBデータベース間のデータ同期が実現でき、データ分析、ディザスタリカバリ、マルチアクティブなどのビジネスシーンで活用できる機能です。
MongoShake はログ データのサブスクリプションおよび消費機能も提供しており、SDK、Kafka、MetaQ などを通じて柔軟に接続でき、ログ サブスクリプション、データ センターの同期、キャッシュの非同期削除などのシナリオに適しています。
プロジェクトアドレス： https: //github.com/alibaba/MongoShake
参考ドキュメント： https: //help.aliyun.com/document_detail/122621.html
mongodbのデータ移行
1. 移行ソースおよびターゲットとして、kubernetes クラスター内に 2 つのサンプル mongodb レプリカ セット クラスターを作成します。
bitnami mongodb helm リポジトリを追加
helm repo add bitnami https://charts.bitnami.com/bitnami
最初のクラスターをデプロイする
helm upgrade -i mongodb bitnami/mongodb -n mongodb --create-namespace \
--set architecture=replicaset \
--set replicaCount=2 \
--set externalAccess.enabled=true \
--set externalAccess.service.type=NodePort \
--set externalAccess.service.nodePorts[0]='30700' \
--set externalAccess.service.nodePorts[1]='30701'
2 番目のクラスターをデプロイする
helm upgrade -i mongodb-dest bitnami/mongodb -n mongodb-dest --create-namespace \
--set architecture=replicaset \
--set replicaCount=2 \
--set externalAccess.enabled=true \
--set externalAccess.service.type=NodePort \
--set externalAccess.service.nodePorts[0]='30702' \
--set externalAccess.service.nodePorts[1]='30703'
最初のクラスターの root パスワードを取得する
export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace mongodb-dest mongodb-dest -o jsonpath="{.data.mongodb-root-password}" | base64 -d)
echo $MONGODB_ROOT_PASSWORD
mongodb クライアント ツールをインストールし、最初の mongodb クラスターに接続します。接続方法は nodeip+nodeport で、アドレスはポッドが配置されているノードのアドレスです。
[root@Ops-repo mongo-shake]# mongo --host "192.168.0.23:30700,192.168.0.24:30701" --authenticationDatabase admin -u root -p K1IySNf3JM
MongoDB shell version v4.4.4
connecting to: mongodb://192.168.0.23:30700,192.168.0.24:30701/admin?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session {
"id" : UUID("71963472-8169-4248-8253-fbde1d981d39") }
MongoDB server version: 6.0.2
WARNING: shell and server versions do not match
---
The server generated these startup warnings when booting:
2022-11-01T05:14:02.656+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
2022-11-01T05:14:05.060+00:00: vm.max_map_count is too low
2022-11-01T05:14:05.060+00:00: currentValue: 262144
2022-11-01T05:14:05.060+00:00: recommendedMinimum: 1677720
2022-11-01T05:14:05.060+00:00: maxConns: 838860
---
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
rs0:PRIMARY>
ソースを移行してサンプルデータを作成する
> use testdb;
> db.createCollection("users");
> db.myusers.insert({
"name":"jack", age:24});
MongoShakeのインストール
1. MongoShake プログラムをインストールします
wget https://github.com/alibaba/MongoShake/releases/download/release-v2.8.1-20220922/mongo-shake-v2.8.1.tgz
次のコマンドを実行して、MongoShake を /opt/mongoshake ディレクトリに解凍します。
tar zxvf mongo-shake-v2.8.1.tgz -C /opt
mv /opt/mongo-shake-v2.8.1/ /opt/mongo-shake
cd /opt/mongo-shake
2. MongoShake の設定ファイルを変更しますcollector.conf。関連する主なパラメータは次のとおりです。
注: パスワードには Aite（@）文字が含まれていてはなりません。含まれていない場合、接続は失敗します。
mongo_urls は、ConnectionStringURI 形式のソース MongoDB インスタンスの接続アドレスです。例は次のとおりです。
mongo_urls = mongodb://root:[email protected]:30700,192.168.0.24:30701
tunnel.address は、ターゲット MongoDB インスタンスの ConnectionStringURI 形式の接続アドレスです。例は次のとおりです。
tunnel.address = mongodb://root:[email protected]:30702,192.168.0.24:30703
sync_mode はデータ同期の方法で、デフォルト値は incr です。all は完全 + 増分同期を意味し、full は完全同期を意味し、incr は増分同期を意味します。
sync_mode = incr
移行を実行し、次のコマンドを実行して同期タスクを開始し、ログ情報を出力します。
./collector.linux -conf=collector.conf -verbose
出力されたログ情報を確認し、以下のログが表示されたら、完全データ同期が完了し、増分データ同期モードに入ったことを意味します。
[root@Ops-repo mongo-shake]# ./collector.linux -conf=collector.conf -verbose 2
[2022/11/01 13:45:36 CST] [INFO] log init succ. log.dir[] log.name[collector.log] log.level[info]
[2022/11/01 13:45:36 CST] [INFO] MongoDB Version Source[6.0.2] Target[6.0.2]
[2022/11/01 13:45:36 CST] [WARN]
______________________________
\ \ _ ______ |
\ \ / \___-=O'/|O'/__|
\ MongoShake, Here we go !! \_______\ / | / )
/ / '/-==__ _/__|/__=-| -GM
/ Alibaba Cloud / * \ | |
/ / (o)
------------------------------
if you have any problem, please visit https://github.com/alibaba/MongoShake/wiki/FAQ
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] Close client with mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] Close client with mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701
[2022/11/01 13:45:36 CST] [INFO] Collector startup. shard_by[collection] gids[[]]
[2022/11/01 13:45:36 CST] [INFO] Collector configuration {
"ConfVersion":10,"Id":"mongoshake","MasterQuorum":false,"FullSyncHTTPListenPort":9101,"IncrSyncHTTPListenPort":9100,"SystemProfilePort":9200,"LogLevel":"info","LogDirectory":"","LogFileName":"collector.log","LogFlush":false,"SyncMode":"incr","MongoUrls":["mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701"],"MongoCsUrl":"","MongoSUrl":"","MongoSslRootCaFile":"","MongoSslClientCaFile":"","MongoConnectMode":"secondaryPreferred","Tunnel":"direct","TunnelAddress":["mongodb://root:***@192.168.0.23:30702,192.168.0.24:30703"],"TunnelMessage":"raw","TunnelKafkaPartitionNumber":1,"TunnelJsonFormat":"","TunnelMongoSslRootCaFile":"","FilterNamespaceBlack":[],"FilterNamespaceWhite":[],"FilterPassSpecialDb":[],"FilterDDLEnable":false,"FilterOplogGids":false,"CheckpointStorageUrl":"mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701","CheckpointStorageDb":"mongoshake","CheckpointStorageCollection":"ckpt_default","CheckpointStorageUrlMongoSslRootCaFile":"","CheckpointStartPosition":1,"TransformNamespace":[],"SpecialSourceDBFlag":"","FullSyncReaderCollectionParallel":6,"FullSyncReaderWriteDocumentParallel":8,"FullSyncReaderDocumentBatchSize":128,"FullSyncReaderFetchBatchSize":1024,"FullSyncReaderParallelThread":1,"FullSyncReaderParallelIndex":"_id","FullSyncCollectionDrop":true,"FullSyncCreateIndex":"none","FullSyncReaderOplogStoreDisk":false,"FullSyncReaderOplogStoreDiskMaxSize":256000,"FullSyncExecutorInsertOnDupUpdate":false,"FullSyncExecutorFilterOrphanDocument":false,"FullSyncExecutorMajorityEnable":false,"IncrSyncMongoFetchMethod":"oplog","IncrSyncChangeStreamWatchFullDocument":false,"IncrSyncReaderFetchBatchSize":1024,"IncrSyncOplogGIDS":[],"IncrSyncShardKey":"collection","IncrSyncShardByObjectIdWhiteList":[],"IncrSyncWorker":8,"IncrSyncTunnelWriteThread":8,"IncrSyncTargetDelay":0,"IncrSyncWorkerBatchQueueSize":64,"IncrSyncAdaptiveBatchingMaxSize":1024,"IncrSyncFetcherBufferCapacity":256,"IncrSyncExecutorUpsert":false,"IncrSyncExecutorInsertOnDupUpdate":false,"IncrSyncConflictWriteTo":"none","IncrSyncExecutorMajorityEnable":false,"CheckpointStorage":"database","CheckpointInterval":5000,"FullSyncExecutorDebug":false,"IncrSyncDBRef":false,"IncrSyncExecutor":1,"IncrSyncExecutorDebug":false,"IncrSyncReaderDebug":"","IncrSyncCollisionEnable":false,"IncrSyncReaderBufferTime":1,"IncrSyncWorkerOplogCompressor":"none","IncrSyncTunnelKafkaDebug":"","Version":"improve-2.8.1,9f2515bbe508b2d86fab2ed87f01f3121efc6920,release,go1.15.10,2022-09-22_11:49:22","SourceDBVersion":"6.0.2","TargetDBVersion":"6.0.2","IncrSyncTunnel":"","IncrSyncTunnelAddress":null,"IncrSyncTunnelMessage":"","HTTPListenPort":0,"SystemProfile":0}
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] Close client with mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] Close client with mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701
[2022/11/01 13:45:36 CST] [INFO] GetAllTimestamp biggestNew:{
1667281485 1}, smallestNew:{
1667281485 1}, biggestOld:{
1667279683 1}, smallestOld:{
1667279683 1}, MongoSource:[url[mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701], name[rs0]], tsMap:map[rs0:{7160911711770247169 7160919451301314561}]
[2022/11/01 13:45:36 CST] [INFO] all node timestamp map: map[rs0:{
7160911711770247169 7160919451301314561}] CheckpointStartPosition:{
1 0}
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701 successfully
[2022/11/01 13:45:36 CST] [INFO] rs0 Load exist checkpoint. content {
"name":"rs0","ckpt":7160913867843829761,"version":2,"fetch_method":"oplog","oplog_disk_queue":"","oplog_disk_queue_apply_finish_ts":0}
[2022/11/01 13:45:36 CST] [INFO] rs0 checkpoint using mongod/replica_set: {
"name":"rs0","ckpt":7160913867843829761,"version":2,"fetch_method":"oplog","oplog_disk_queue":"","oplog_disk_queue_apply_finish_ts":0}, ckptRemote set? [true]
[2022/11/01 13:45:36 CST] [INFO] sync mode run incr
[2022/11/01 13:45:36 CST] [INFO] start running with mode[incr], fullBeginTs[0[0, 0]]
[2022/11/01 13:45:36 CST] [INFO] start incr replication
[2022/11/01 13:45:36 CST] [INFO] RealSourceIncrSync[0]: url[mongodb://root:***@192.168.0.23:30700,192.168.0.24:30701], name[rs0], startTimestamp[7160913867843829761]
[2022/11/01 13:45:36 CST] [INFO] New session to mongodb://root:***@192.168.0.23:30702,192.168.0.24:30703 successfully
MongoShake のステータスを監視する: 増分データ同期が開始されたら、コマンド ライン ウィンドウを開いて次のコマンドを使用して MongoShake を監視できます。
cd /opt/mongoshake && ./mongoshake-stat --port=9100
説明:mongoshake-statこれは Python スクリプトです。実行する前に Python 2.7 バージョンをインストールしてください。
モニター出力の例。
[root@Ops-repo ~]# cd /opt/mongo-shake/
[root@Ops-repo mongo-shake]# ./mongoshake-stat --port=9100
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| log_size_avg | log_size_max | logs_get/sec | logs_repl/sec | logs_success/sec | lsn.time | lsn_ack.time | lsn_ckpt.time | now.time | replset | tps/sec |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| 331.00B | 339.00B | none | none | none | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-11-01 13:21:22 | rs0 | none |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| 331.00B | 339.00B | 0 | 0 | 0 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-11-01 13:21:23 | rs0 | 0 |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| 331.00B | 339.00B | 0 | 0 | 0 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-11-01 13:21:24 | rs0 | 0 |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| 217.00B | 339.00B | 1 | 0 | 0 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-11-01 13:21:25 | rs0 | 0 |
ターゲット レプリカ セットに接続して、データ同期が成功したことを確認します。
[root@Ops-repo bin]# mongo --host "192.168.0.23:30702,192.168.0.24:30703" \
> --authenticationDatabase admin -u root -p o72ntfVFVL
MongoDB shell version v4.4.4
connecting to: mongodb://192.168.0.23:30702,192.168.0.24:30703/admin?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session {
"id" : UUID("18118dec-ac80-401b-bba5-2c8a80bd0066") }
MongoDB server version: 6.0.2
WARNING: shell and server versions do not match
---
The server generated these startup warnings when booting:
2022-11-01T05:18:05.971+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
2022-11-01T05:18:07.811+00:00: vm.max_map_count is too low
2022-11-01T05:18:07.811+00:00: currentValue: 262144
2022-11-01T05:18:07.811+00:00: recommendedMinimum: 1677720
2022-11-01T05:18:07.811+00:00: maxConns: 838860
---
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
rs0:PRIMARY>
rs0:PRIMARY> show dbs;
admin 0.000GB
config 0.000GB
local 0.001GB
testdb 0.000GB
rs0:PRIMARY> use testdb;
switched to db testdb
rs0:PRIMARY> show tables;
users
rs0:PRIMARY> show collections;
users
rs0:PRIMARY>