Cloud database MongoDB version log clean-up and compact command Detailed

1. Description of the problem:

Today saw the company mongodb oplog a little big, seeing cloud database MongoDB version log clean-up strategy.

MongoDB database frequently deleted in long-term / write data or mass delete a lot of data, will generate a lot of physical space debris.

These pieces will take up disk space and reduce disk utilization.

You can perform all data collection and index defragmentation to rewrite and release unused space, improve disk utilization and query performance.

The following diagram illustrates:

Prerequisites

mongo instance storage engine is WiredTiger.

details

Prohibit the use of user db.repairDatabase commands.
When the log space is too large, it will trigger an automatic clean-up strategy.

Precautions

Before performing this operation, it is recommended to back up the database
This operation will lead to the collection belongs to the database is locked, and the database read and write operations will be blocked, low peak in business operations.
Description: performing physical space recovery command (Compact) the time required for the set amount of data, system load and other factors.

Remove and drop the difference

To remove a MongoDB collection of documents in all, there are two ways

db.collection.remove({}, {multi: true})One by one document removed from the btree, and finally all documents are deleted, but the files will not be recovered physical space
db.collection.drop() A collection of physical files deleted, the space is immediately recovered

multi: Optional, mongodb default is false, only updating the first record found, if this parameter is true, according to the conditions put many records to check out all the updates.

In general:

The new data will be written to use the physical space is not recovered, so in the scene sustained write data, the command does not require frequent execution compact organize physical space debris.

After some scenes, remove a large amount of data, subsequent write may not be much, then if you want to reclaim the space, you need to explicitly call compact.

compact specifically what to do?

Compact action finalized by the storage engine WiredTiger, free space behind the WiredTiger in the implementation of compact, will continue the collection of data files Wang Qianmian write,

Truancate file and then gradually recover physical space. Before each round, compact, WT will first check for compliance comapact conditions.

前面80%的空间里，是否有20%的空闲空间，用于写入文件后面20%的数据，或者
前面90%的空间里，是否有10%的空闲空间，用于写入文件后面10%的数据

如果上面都不满足，说明执行compact肯定无法回收10%的物理空间，此时 compact 就回退出。

所以有时候遇到对一个大集合进行 compact，compact立马就返回ok：1，集合的物理空间也没有变化，就是因为 WiredTiger 认为这个集合没有 compact 的必要。

预估回收的物理空间

1、连接mongo实例parmary或scondary

2、切换至集合所在的数据库。

use <database_name>

3、执行下列命令查询预估回收空间。

db.<collection_name>.stats().wiredTiger["block-manager"]["file bytes available for reuse"]

4、执行结果示例：

整理单节点实例/副本集实例的碎片

1、通过mongo shell连接MongoDB实例的Primary节点

2、切换至集合所在的数据库。

use <database_name>

3、执行db.stats()命令查看碎片整理前数据库占用的磁盘空间。

4、执行以下命令，对某个集合进行碎片整理。

db.runCommand({compact:"<collection_name>",force:true})

5、等待执行，返回{ "ok" : 1 }代表执行完成。

6、碎片整理完毕后，可通过db.stats()命令查看碎片整理后数据库占用的磁盘空间

本案例碎片整理前后的对比如下图所示：

参数说明：

<database_name>：数据库名。
<collection_name>：集合名。
force为可选项，如您需要在副本集实例的Primary节点执行该命令，需要设置force的值为true。
compact操作不会传递给Secondary节点，当实例为副本集实例时，请重复上述步骤通过mongo shell连接至Secondary节点，执行碎片整理命令。

整理分片集群实例的碎片

1、通过mongo shell连接分片集群实例中的任一mongos节点

2、执行db.stats()命令查看碎片整理前数据库占用的磁盘空间。

3、执行以下命令，对Shard节点中的Primary节点进行集合的碎片整理。

db.runCommand({runCommandOnShard:"<Shard ID>","command":{compact:"<collection_name>",force:true}})

4、执行以下命令，对Shard节点中的Secondary节点进行集合的碎片整理。

db.runCommand({runCommandOnShard:"<Shard ID>","command":{compact:"<collection_name>"},queryOptions: {$readPreference: {mode: 'secondary'}}})

参数说明：

<Shard ID>：Shard节点ID。
<collection_name>：集合名。

碎片整理完毕后，可通过db.runCommand({dbstats:1}) 命令查看碎片整理后数据库占用的磁盘空间。