Reference: https://enterprisezine.jp/dbonline/detail/7098
mongodb has two storage engines
mmap
has always been the default storage engine of mongodb before version 3.2
■ wired tiger
was born in version 2.8 and became the default storage engine for mongodb after version 3.2
Regarding mmap.
mongodb does not manage the use of memory, it hands over these tasks to the OS, which is actually the mmap of the linux system used.
What is mmap?
mmap is a file system that maps files to memory, and a process can quickly read the contents of files that exist in memory.
When the process is just started, the file is not mapped into memory. The first time a process reads a file is from memory, and the file is placed in memory after reading.
After that, the file remains in memory, and the second and subsequent accesses have memory responses.
After that, when the memory is full, there will be an overflow phenomenon, and the mmap algorithm will cause some files to be withdrawn from the memory. This is also known as a page fault.
In mongodb, mongodb and mmap
load the data files in the data directory into memory.
There are files such as database name.0, database name.1... in the data directory.
As follows:
[db]# ls -lh
total 545M
drwxr-xr-x 2 root root 4.0K Nov 27 22:40 2014 journal
-rw------- 1 root root 64M Nov 27 22:40 2014 local .0
-rw------- 1 root root 16M Nov 27 22:40 2014 local.ns
-rwxr-xr-x 1 root root 6 Nov 27 22:40 2014 mongod.lock
-rw--- ---- 1 root root 64M Nov 27 21:15 2014 test.0 ★
-rw------- 1 root root 128M Nov 27 21:15 2014 test.1 ★
-rw----- -- 1 root root 256M Nov 27 21:15 2014 test.2 ★
-rw------- 1 root root 16M Nov 27 21:15 2014 test.ns
Contents of the data file
The content in the data file is the data itself (BSON) and the index.
In the data file, the extent for data and the extent for index are stored with variable length.
http://blog.mongolab.com/2014/01/how-big-is-your-mongodb/
Time to load mongodb into memory
The data and Index will not be loaded into memory at startup,
that is because mmap is used
As an example, take a look at the hoge Collection
> db.hoge.stats()
{
"ns" : "test.hoge",
"count" : 1740681,
"size" : 83552848,
"avgObjSize" : 48,
"storageSize" : 123936768,
"numExtents" : 11,
"nindexes" : 4,
"lastExtentSize" : 37625856,
"paddingFactor" : 1,
"systemFlags" : 0,
"userFlags" : 1,
"totalIndexSize" : 228331152,
"indexSizes" : {
"_id_" : 50846544,
"a_1" : 43807008,
"_id_1_a_1" :66838800,
"a_1__id_1" : 66838800
},
"ok" : 1
}
The data size is 83552848 about 80M, the index size is 228331152 about 220M, look at serverStatus().
db.serverStatus().mem
{
"bits" : 64,
"resident" : 47,
"virtual" : 1482,
"supported" : true ,
"mapped" : 544,
"mappedWithJournal" : 1088
}
Now it actually occupies 47M of memory, which means that the data and indexes of the hoge collection have not been loaded into the memory.
As the query is executed, it will gradually be loaded into memory.
At startup, there is nothing in memory, and as the retrieval progresses, the necessary data and indexes are gradually loaded into memory.
Therefore, the response speed of Mongodb will be a little slow when it is just started.
preheat
If preheating, you can add frequently used data to the memory.
The syntax is as follows
db.runCommand({ touch: "collectionName", data: [true|false], index: [true|false] })
> db.serverStatus().mem
{
"bits" : 64,
"resident" : 48, ←48M after startup
"virtual" : 1482,
"supported" : true,
"mapped" : 544,
"mappedWithJournal" : 1088
}
An example, the previous example, only loads the index into memory
db.runCommand({ touch: "hoge", data: false, index: true })
{
"indexes" : {
"num" : 4,
"numRanges" : 30,
"millis" : 48
},
"ok" : 1
}
> db.serverStatus().mem
{
"bits" : 64,
"resident" : 324,← only load indexes into memory
"virtual" : 1482,
"supported" : true,
"mapped" : 544,
"mappedWithJournal" : 1088
}
clear memory
Check the memory usage of mongodb
top -p (pidof mongod)
or
top
shift+m
Sometimes, for some reason, you may want to free the memory occupied by MongoDB, but as mentioned earlier, memory management is controlled by the virtual memory manager, so usually you can only free memory by restarting the service, you must I don't like this method. Fortunately, you can use MongoDB's built-in closeAllDatabases command to achieve the goal:
use admin <-- must switch to admin collection
db.runCommand({closeAllDatabases:1})
In addition, the cache can also be released by adjusting the kernel parameter drop_caches:
sysctl -w vm.drop_caches=1
You can usually monitor the memory usage of MongoDB through the mongo command line, as follows:
db.serverStatus().mem:
mongostat
Use the top command to record the memory usage of the mongodb process
n Set the number of screen refreshes before exiting.
d Specify the time interval between every two screen information refreshes.
b Arrange the top output into a format suitable for output to a file, you can use this option to create a process log
p Only monitor the status of a process by specifying the monitoring process ID
top -d 300 -b -n 2016 -p 2793 > plog &
The status of process 2793 (mongodb) is refreshed 2016 times (one week) at an interval of 5 minutes and imported into the plog file.