Article Directory
1. How to judge fragmentation
1. Query database space occupation information
1) View the storage space occupied by the database
-- 通过db.stats()函数查询storageSize参数大小
use db
db.stats()
-- 直接查看目标DB物理存储大小
use db
db.runCommand({dbStats : 1,scale : 1073741824}) //scale指定单位为GB
2) View the storage space occupied by the collection
-- 通过db.collname.stats()函数查询storageSize参数大小
use db
db.collname.stats()
-- 直接查看目标DB物理存储大小
use db
db.runCommand({"collStats":"oplog.rs",scale:1048576}) //scale指定单位，单位为MB
2. Check the fragmentation problem, check the reclaimable space of the current collection
db.collname.stats().wiredTiger["block-manager"]["file bytes available for reuse"]
rs0:PRIMARY> db.aa.stats().wiredTiger["block-manager"]["file bytes available for reuse"]
16384
2. How to deal with fragmentation
2.1 compact
1. Grammar
db.runCommand({compact:'dsir',force:true})
2. Permission required for compact
use admin
db.createRole(
{
role: "myCustomCompactRole",
privileges: [
{
resource: { "db" : "<database>" , "collection" : "<collection>" },
actions: [ "compact" ]
}
],
roles: []
}
)
db.grantRoleToUser("myCompactUser", [ "dbAdmin" | "myCustomCompactRole" ] )
3. What are the main operations of compact?
1) Check whether the operated compact collection satisfies the conditions
- In the first 80% of the space, is there 20% free space for writing the 20% of the data behind the file
- In the first 90% of the space, is there 10% free space for writing the 10% of the data behind the file
2) If any of the conditions are met, the engine layer starts to execute compact, blocking the left and right read and write operations on the DB during execution, and writes the data behind the collection file to the free space in front, and then gradually truancate the file to reclaim the physical space.
3) If any of the above conditions is not met, it means that 10% of the physical space cannot be reclaimed by compact execution. Then the collection does not need to be compact at present, and the compact operation is directly exited.
4. The impact of compact
1) Compact a collection will add the exclusive write lock of the DB where the collection is located, which will cause all read and write requests on the DB to be blocked; and the execution time of compact is related to the amount of data in the collection. The larger the amount of data, the longer the execution time of compact. Therefore, it is strongly recommended to execute during the low peak period of the business to avoid affecting the business.
2) If you use the db.killOp() method to terminate the operation, or restart the server before the compression operation is completed, please note the following:
- If logging is enabled, data will remain valid and usable regardless of the status of the compression operation. You may have to rebuild the index manually.
- If logging is not enabled, data validity cannot be guaranteed when the compact operation is interrupted
- In both cases, most of the existing free space in the collection may not be reused. Need to re-execute compact to restore the use of space.
3) The compact operation is independent on each node and will not be transferred to the secondary node with the operation of the primary node;
4) If you want to perform conpact operations on the primary node, you need to identify force: true;
5) When the compact operation is performed on the secondary node, the state of the node will be converted to RECOVERING, during which the business cannot access the node;
2.2 How does compact release space under each storage engine
1、WiredTiger
1) Compact operation will release data and index space to the operating system;
2) Compact execution requires additional space.
2 、 MMAPv1
1) The commpact operation can reduce the data space and rebuild the index, but these spaces will not be returned to the operating system, but will only be left to MongoDB for subsequent data writing;
2) If you want to reclaim the disk space of MMAPv1, you need to perform an initial sync ;
3) The execution of compact operation requires at least 2G of free space in the current instance to execute.
Three, common solutions to deal with fragmentation
3.1 Direct operation
-
Direct compact operation in a single node environment -
In the replica set environment,
1) During the low peak period of the business, execute the compact command on the secondary node;
2) Promote the secondary node that has been
compacted to the primary node; 3) Compact the node that has been downgraded to secodary.
3.2 Rebuild collection
Add a new secondary node, and then upgrade the node to a primary node (in order not to affect the business in the case of a very large amount of data)
3.3 Cyclic processing of single node startup mode
1) Start a secondary node in the replica set as a single node, and execute the compact command to deal with the fragmentation of the collection
2) Restart the node that has been fragmented as a replica set, and the node will be promoted to a primary node
3) Fragment the remaining secondary nodes in the manner of 1)
Document reference:
https://docs.mongodb.com/manual/reference/command/compact/index.html
https://blog.csdn.net/u013162930/article/details/78580083