elasticsearch Column: https://www.cnblogs.com/hello-shf/category/1550315.html
First, prior knowledge
1.1, the index can not be changed
See this article I believe we all know is es inverted index, do not understand it does not matter, in another post in my blog a detailed analysis of inverted index mechanism es. During indexing es in order to satisfy the following characteristics, the index es off disc is immutable.
1 do not need to lock. If you never need to update an index, you do not have to worry about trying to modify multiple programs simultaneously. 2 Once the index is read into the file system cache (Translator: in memory) does not change, it has been there since. As long as the file system cache has enough space, most of the reading will be direct access to memory rather than disk. This helps performance. 3 in the lifecycle of the index, all other caches are available. They do not need every time data changes are rebuilt, so that the text can be searched because the data will not change. Write a single large inverted index, you can compress the data, and require less disk IO memory size of the cache index.
Of course, immutable index has its drawbacks, first of all it is immutable! You can not change it. If you want to search for a new document, you must rebuild the entire index. This not only severely limits the data that can be installed in an index, and an index can be updated frequency. So es introduces dynamic index.
1.2 Dynamic index
The next problem to be solved is how to update an inverted index while maintaining the immutable benefits. The answer is to use multiple indexes. Instead of rewriting the entire inverted index, but the recent increase in the index reflects changes in extra. Each inverted index can query sequence, beginning from the oldest, and finally the results of the polymerization. Elasticsearch underlying dependent Lucene, introduced the concept of per-segment search. A segment (segment) is a fully functional inverted index, but now Lucene index refers to a collection section, together with the commit point (commit point, including all segments of the file), as shown in FIG. New document before it is written to disk segment, first written in the index buffer memory area. Then through the fsync segment cache flush to the disk, the segments will be started after the opening can be retrieved i.e. paragraphs disc.
See here if the segment is still a little confusing, it does not matter, if you are familiar with the java language, ArrayList this collection we all know is a dynamic array, his underlying data structure is actually an array, we all know that the array is immutable, ArrayList is Touched dynamic array expansion achieved. Here we can commit point understood as ArrayList, segment is an array of small. Then combined into ArrayList. If you know of Java1.8 ConcurrentHashMap segmented lock trust you understand this segment is very easy.
1.3, several confusing concept
Es in the "Index" is the slice (Shard) set in lucene the "index" is from the macro level in a slice es, from the micro-segment is a collection of terms.
"The document indexing process", that is in the "Index", we can be understood as the process for the document es resume index.
Two, document indexing process
Document is indexed process as shown above, it can be roughly divided into a memory buffer buffer, translog, filesystem cache, disk systems these parts, let's sort out what this process.
Stage 1:
This stage is very simple, the first step in a document document will also be written into the buffer memory buffer and translog.
Phase 2:
refresh: documents every second memory buffer is refresh (refreshed) to the filesystem cache in a new segment, the segment is the index of the smallest unit, this time segment will be opened for retrieval. That Once the document is flushed to the file system cache, which can be retrieved for use. This is also the key to near-real time (NRT) is es. It will be described in detail later.
Stage 3:
merge: Every second new segment is generated, which would mean that the number does not take long segment will explode, each segment will be very consume file handles, memory, and cpu resources. This system would be unbearable, so this time, we urgently need to merge the fragmented segment. ES solve this problem by merging the background section. Section larger than small pieces are combined, and then combined into a larger segment. Then open a new segment for the search, delete the old segment.
Stage 4:
flush: After a consolidation phase 3 larger segment of the new generation will be flush to the system disk. So that the entire process is complete. But the opportunity to stay here is the flush of a burden. Translog behind the introduction of time will be introduced.
Do not worry, the next step we will split open more detailed analysis.
2.1, near real-time search (NRT)
In the early lucene, only when segement is written to disk, the segment will be open for the search, and we said above when doc is flushed to the filesystem cache will generate a new segment will be opened.
Because the mechanism between the per-segment search, index, and search for a document is delayed. The new document will be searched in a matter of minutes, but that's still not fast enough. Disk bottleneck. Submit a new segment to the disk needs fsync operation to ensure that segments are physically written to disk, instant power failure will not lose data. But fsync is expensive, it can not be triggered when each document is indexed.
So the need for a more lightweight way so that the new document can be searched, which means that the removal fsync.
Elasticsearch located between the disk and the file system cache. As mentioned earlier, the document in the index cache memory is written to a new segment, but the new segment first written to the file system cache, that price is very low, then will be synchronized to disk, this is a big price. But once a file is cached, it can also be opened and read, just like other file.
Es in the document is written every second memory buffer will be flushed to the new segment filesystem cache in, which means you can be searched. This is the ES NRT-- near real-time search.
A brief refresh API
If you come across you add a doc, but not retrieved, most likely because they have not automatically refresh, this is something you can try to manually refresh
POST /student/_refresh
Performance Optimization
Here we need to know a little refresh process is very consumption performance. If your system is less demanding real-time, you can control the refresh time interval through the API, but if your new system is real-time requirements, then you live with it now.
If you real-time requirements of the system is very low, we can adjust the refresh interval, turn up a little will improve system performance to a certain extent.
PUT /student { "settings": { "refresh_interval": "30s" } }
2.2, merge segment --merge
Automatically refresh every second by creating a new segment, the number will not be long segments exploded. There are too many segments is a problem. Each segment of consumer file handles, memory, cpu resources. More importantly, each search request requires checking each segment. The more segment, the slower the query.
ES solve this problem by merging the background section. Section larger than small pieces are combined, and then combined into a larger segment.
This was the old documents deleted from the file system. The old section will not be copied to a new, larger segments. This process is you do not have to do anything. When you ES automatically handles when indexing and searching.
Let us summarize the process segments combined.
1 Select some similar-sized segment, merge into one large segment 2 new segment flush to disk up 3 to write a new commit point, including a new segment, and exclude older those segment 4 new segment opens Search for 5 to delete the old segment
optimize API combined force can best be described as a segment API. It forces segments to achieve a combined fragment max_num_segments specified parameters. This is to reduce the number (usually 1) the purpose of improving search performance segment.
POST /logstash-2019-10-01/_optimize?max_num_segments=1
In general, try not to the scene performed manually, it automatically defaults to execute on it
2.3, reliable storage and disaster recovery
Useless fsync synchronize file system cache to disk, we can not be sure that the power failure, even after normal exit the application, data security. To the ES reliability, the need to ensure lasting changes to disk.
We have said that a full synchronization segment submitted to disk write commit point, which lists all of the known segment. Upon reboot, or re-open the index, ES submitted using the point to decide which segments belong to the current fragmentation.
When we get close to real-time search by refreshing per second, we still need to periodically perform a full submission to ensure recovery from failure. But the document submitted between how to do? We do not want to lose them.
Doc index above process stage 1, doc are written into a memory buffer and translog, then every second will refresh the memory buffer docs to filesystem cache in a new segment, then a number of segment will continue to be compressed, section larger than small pieces are combined, and then combined into a larger segment. After each refresh operation, docs memory buffer is flushed to the filesystem cache in segemnt, but it is still continuing increases tanslog increase. When translog to a certain extent, it will take place a commit operation is submitted to the full amount.
Detailed process is as follows:
. 1 , DOC buffer and written to memory translog log file 2 every one second, docs memory buffer is written to the new filesystem cache segment, segment was opened at this time for retrieval and 3 , the buffer memory region is emptied 4. repeat 1 to 3 , continually adding new segment, the memory buffer continues to be cleared and the data continues to accumulate in translog 5 , when translog length reaches a certain extent, commit operation occurs 5-1 , the buffer memory area docs are written to the filesystem cache in a new segment, open for retrieval 5-2 , the memory buffer is cleared 5-3 , a commit ponit is written to disk, indicating all of the segment index 5-4 , All index segment file filesystem cache data in the cache, are forced to brush fsync to disk 5-5, existing translog be cleared to create a new translog
In fact, here we find fsync still not been abandoned, but we have reduced the frequency of its use by dynamic indexing and translog technology, and realize near real-time search. Secondly, we found that the above steps to flush the segment including the filesystem cache refresh process to empty the two hard drives and translog by fsync. es default every 30 minutes to flush, but when translog to a certain extent will be flush operation.
Corresponding to the following process of FIG.
5-5步骤不难发现只有内存缓冲区中的docs全部刷新到filesystem cache中并fsync到硬盘，translog才会被清除，这样就保证了数据不会丢失，因为只要translog存在，我们就能根据translog进行数据的恢复。
简单介绍一下flush API
手动flush如下所示，但是并不建议使用。但是当要重启或关闭一个索引，flush该索引是很有用的。当ES尝试恢复或者重新打开一个索引时，它必须重放所有事务日志中的操作，所以日志越小，恢复速度越快。
POST /student/_flush
三、更新和删除
前面我们说过es的索引是不可变的，那么更新和删除是如何进行的呢？
参考文献：
《elasticsearch-权威指南》
如有错误的地方还请留言指正。
原创不易，转载请注明原文地址：https://www.cnblogs.com/hello-shf/p/11553317.html