manual compaction
Manually trigger compaction. Manually trigger a higher priority than automatic trigger.
Automatically triggered compaction
Triggering conditions:
imm_! = NULL expressed the need Memtable dump into SSTable, initiated Minor Compaction.
manual_compaction_! = NULL expressed manually initiate Compaction.
versions _-> NeedsCompaction function returns True.
Note: In leveldb in, compaction entry in DBImpl::MaybeScheduleCompaction()
Memtable --> Immutable Memtable
Memtable: Memory SkipList
Simple to understand: the read-write the Memtable becomes readable indelible writing of Memtable.
When the memory Memtable wrote a certain amount of time, it will generate a new wal log and new Memtable.
minor compaction
immutable memtable dump into sstable file.
Immutable Memtable --> Level0
This process is known as Minor compaction .
Immutable Memtable exports the data in a background thread, flush to the disk to form a new sstable file.
Note: This process does not delete data kv. Delete key information is recorded into sstable file.
Level0 files have multiple files, keys range multiple files will be repeated.
minor compaction mainly do two things:
1, construction sstable
2, the new file is written sstable which layer.
Code flow:
Note that the file is not necessarily new to come out in level0. Most number of cases are level0.
Create a new file out of the number of layers in the PickLevelForMemTableOutput computing function. Logic is as follows:
In terms of strategy, we want to try a new compact files pushed to a high level.
Because too many files need to control the level 0, compaction IO and find all consuming.
On the other hand can not be pushed to the high level, the number of lookup control to a certain extent, but if certain range of key updates more frequently, the follow-up to the senior compaction IO consumption is also large.
So PickLevelForMemTableOutput is a trade-off compromise. If the new generation of sstable and Level 0 of sstable there is overlap, sstable newly created directly added to level 0, or according to a certain strategy, even pushing up to Level1 Level 2, but pushed up to Level2.
There is a control parameter: kMaxMemCompactLevel. Determining whether there is overlap to use key data structure described earlier FileMetaData between sstable file.
major compaction
When the number of files in a Level level exceeds a certain threshold value, it will be from the Level of sstable file sstable its file level and a high level + 1 will be a new level file compaction + 1 layer.
Triggering conditions:
bool NeedsCompaction() const {
Version* v = current_;
return (v->compaction_score_ >= 1) || (v->file_to_compact_ != NULL);
}which is
1. Too many number of files or a hierarchy total file size is too large, it will trigger compaction.
Lot number and a layer file. (Refers level0)
Or a level of total file size is too large. Exceeds the limit value.
2. seek too many times, trigger compaction
In addition to level 0, any level of internal documents are in order, but also between the document and orderly. But the key level (n) and the level (n + 1) of the two cross-file may exist. It is because of this cross, to find a key value of time, level (n) lookup failed, and had to go level (n + 1) to find. If you find a number, had to find a file, but could not find, always go to a high level, can be found. This shows the hierarchy of files and folders on stage, key ranges overlap very serious, this is unreasonable, it will lead to decrease in efficiency. Thus, the need to initiate a major compaction level, and to reduce the level of level + 1 overlap.
This is called Seek Compaction. For compaction seek triggered, which seek to document the number of invalid threshold, the file is level n participation compaction files. The compaction size triggered a bit more complicated, it requires a compaction on which key consideration do, where you are, then the key is greater than the first file is the level n participation compaction files.
For n> 0, the primary case of n-level files are only involved in compaction with 1, if n = 0, since between the 0 level file, key may overlap, therefore, according to the selected level of 0 the document, obtained the minimum and maximum key of the key file is responsible for, and find all the key sections have overlapping level 0 documents are added to the war file.
Due to the special nature of level0, so major compaction to be divided into two categories:
Level0 --> Level1
Level0 select a file.
And to find a duplicate key of the level0 level1 file.
Find out all this and then level1 file duplicate key of level0 file.
Level0 files and merge all level1 level1 files into a new file.
Level N --> Level N +1
Selecting a file Level N
Find all Level N +1 and the duplicate key file by the file.
compaction, create a new file Level N + 1.
levels
minor compaction mainly do two things: 1, construction sstable2, the new file is written sstable which layer.