An index in ES consists of one or more shards. When creating an index, you can set the number of primary shards and replica shards. After the primary shard is determined, it cannot be modified (because routing needs to be based on this number) distribution request), and the number of replica fragments can be modified at any time
PUT /myIndex
{
"settings" : {
"number_of_shards" : 2, //该索引有2个分片
"number_of_replicas" : 1 //每个分片都有一个副本
}
}Here I assume that two nodes are established, that is, two ES services are started. shard1 and shard2 are the two primary shards created, and replica1 and replica2 are two replica shards. Generally, in order to achieve high availability, ES will The primary shards and replica shards are stored in different NODE nodes, and this action is automatically completed by ES.
So what are these shards?
shard = Lucene Index
We know that ElasticSearch is a distributed and scalable real-time search and analysis engine, which is a search engine based on the full-text search engine Apache Lucene. Then shards are essentially Lucene Indexes one by one.
There are many small segments in Lucene, and each segment has many data structures inside, such as the often-heard Inverted Index (inverted index), Sorted Fields, Document Values..., the most important of which is inverted indexed.
Inverted Index (inverted index)
The inverted index mainly consists of two parts:
When we search, we will segment the content of the search into words. Then find the corresponding term in the dictionary, so that you can find the search-related file content.
-
Dictionary of ordered data (including the word term and its frequency)
-
Postings corresponding to the word (that is, the file in which the word exists)
Sorted Fields (field lookup)
This structure is used when you want to find documents that contain content with a certain heading. Essentially it is a simple KV collection, which stores the entire document in Json format by default.
Network diagram:
Document Values (for sorting, aggregation)
It was born to solve sorting and aggregation. This structure is essentially a columnar storage.
Network diagram:
In order to improve efficiency, ES can read all the Document Vlaue under the index into the memory for operation, so as to improve the access speed, but it will consume memory space.
These data structures, Inverted Index, Sorted Fields, Document Values, and cache are all inside the Segment.
when searching
Lucene will search all segments, and then return the search results of each segment, and then return to the client.
cache
When ES searches for a document, it will create a corresponding cache for the document and refresh the cache every second.
segment merge
As time goes on, there will be more and more segments, es will merge these segments, and delete the original ones after merging, so there will be a situation where you add more documents and the space occupied by the index It may be getting smaller and smaller, because the merge is caused, so there is more compression.
Document Indexing Step Order
The sequence of steps required to create a new single document:
Network diagram:
1. The client sends a new index or delete request to Node1.
2. The node uses the document's _id to determine which shard the document belongs to. The request will be forwarded to the corresponding Node.
3. Execute the request on the shard. If the execution is successful, the request will be forwarded to the replica shards of each node, and if all succeed, a success message will be returned to the client.
The overall process is still a delusional (network) diagram. . .
1. After the request is sent, it will first come to the coordinating node. By default, the id of the document is used to calculate which shard will process the request.
shard=hash(_id)% number of shards
2. After the shard receives the request, it will first write the request to the memory buffer, and then write it to the Filesystem at regular intervals (1s by default). The process from memory buffer to Filesystem cache is called refresh.
3. Transaction Log is used to ensure data reliability, because data in the refresh process may be lost. After the shard receives the request, it will write the request into the translog at the same time. When the data in the Filesystem cache is written to the disk, it will be cleared. This process is called flush.
4. During the flush process, the cache in memory will be cleared, the content will be written into a new segment, the fsync of the segment will create a new commit point, and the content will be flushed to disk, and the old translog will be deleted and start a new trabslog. The flush trigger mechanism defaults to 30 minutes, or the translog is too large.