In the running process of Elasticsearch, how to allocate and set the memory reasonably is a very important thing, otherwise it is very easy to have various problems.
1. Why does Elasticsearch eat memory:
Let's first look at the overall memory consumption of the ES server:
The introduction of Query Cache, Request Cache, FieldData Cache, Indexing Buffer and Segment has been introduced in the previous article and will not be repeated here:
Cache of Elasticsearch search engine: Request Cache, Query Cache, Fielddata Cache
To answer the question of why Elasticsearch consumes so much memory, we need to cut in from two angles:
(1) ES is a JAVA application, so it is closely related to JVM and GC. We won't discuss the JVM GC in depth here, just know that the application layer generates a large number of objects with long life cycle, which is the main reason for the pressure on the heap. For example, if a large amount of data is read and sorted in memory, or a large amount of data is cached in the heap, if the space released by GC is limited, and the application layer continues to apply for a large number of new objects, the frequency of GC will start to increase, which will not only consume a lot of CPU time, but also seriously A vicious cycle may occur, causing the entire cluster to shut down.
(2) The underlying storage engine of ES is based on Lucene. Lucene's Inverted Index is first generated in memory, and then periodically flushed to disk in the form of segment files. Each segment is actually a complete inverted index and is not modified once written to disk. Document updates and deletions at the API level are actually a special kind of document written incrementally and will be stored in new segments, so unchanged segment files are very easy to be cached by the operating system, and hot data is almost equivalent to memory access.
Second, the memory allocation of Elasticsearch:
Why can't the heap memory allocated to ES be more than half of the physical machine memory?
1. Reserve half of the memory for Lucene to use:
Why do you need to reserve half of the memory for Lucene, wouldn't it be better to allocate all the memory to Elasticsearch? It goes without saying that heap memory is absolutely important for ES, but there is another very important memory consumer - Lucene.
Before talking about Lucene, let's briefly introduce segments. Each segment is stored in a single file, the segment file. It is actually a complete index file containing forward row (90~95% of space) + backward row (5~10%), and once written to disk, it will not be modified. Document updates and deletions in ES are actually Is a special kind of incremental write, which will be saved in the new segment without modifying the old segment.
Back to Lucene, the actual purpose of Lucene is to cache the data in the underlying OS into memory. Since the segment segment of Lucene does not change, it is very beneficial to cache, and the operating system will cache these segment files for faster access. These segments include inverted indexes (for full-text search) and document values (for aggregation).
The performance of Lucene depends on this interaction with the OS. If all the memory is given to the heap memory of the ES, and not a little is left for Lucene, the performance of full-text retrieval will be very poor. So the official recommendation is to provide 50% of the available memory to the ES heap , and the other 50% of the remaining memory will not be idle, because Lucene will use them to cache the segment files that have been read.
The file storage type of ES uses mmap memory mapping by default, which maps Lucene index files into memory, so that the process can directly read Lucene data from memory. Due to the use of memory mapping, the data read when the ES process reads the Lucene file will occupy space in the off-heap memory.
2. The heap memory allocated to ES should not exceed 32G:
Why can't heap memory exceed 32GB? In fact, JVM will use a memory object pointer compression technology when the memory is less than 32G .
In Java, all objects are allocated on the heap, and then there is a pointer to it. The size of the pointers to these objects is usually the word size of the CPU, either 32 bits or 64 bits, depending on your processor, and the pointers point to the exact location of your value. For 32-bit systems, you can use up to 4 GB of memory. More memory can be used for 64 systems. But 64-bit pointers mean more waste, because your pointers themselves are bigger. And worse than wasted space, larger pointers take up more bandwidth when moving data between main memory and caches (eg LLC, L1, etc.).
Java uses a technique called memory pointer compression to solve this problem. Its pointer no longer represents the exact location of the object in memory, but an offset. This means that a 32-bit pointer can refer to 4 billion objects, not 4 billion bytes. Ultimately, that is to say, the heap memory grows to 32 G of physical memory, which can also be represented by a 32-bit pointer.
Once you cross that magic 30 - 32 G boundary, the pointer switches back to the normal object's pointer, and each object's pointer gets longer, using more CPU memory bandwidth, which means you effectively lose more memory. In fact, when the memory reaches 40 - 50 GB, the effective memory is equivalent to 32 GB of memory when using the memory object pointer compression technology.
So, even if you have enough memory, try not to exceed 32 G, because it wastes memory, reduces CPU performance, and also makes GC cope with large memory.
3. The heap memory of ElasticSearch is set:
The memory allocated to the Heap should not exceed half of the system's available physical memory to ensure enough physical memory is reserved for the Lucene system file cache, and should not exceed 32 GB. What about JVM arguments? Just set the minimum heap size (Xms) and the maximum heap size (Xmx) to the same size as the heap, and avoid dynamically allocating heap size. To ensure that the sizes of Xms and Xmx are the same, the purpose is to waste resources without re-splitting the size of the heap area after the Java garbage collection mechanism cleans up the heap area, and can reduce the pressure caused by the expansion and contraction of the heap size.
Although 32 GB is a memory setting limit for ES, what if the machine has a lot of memory? For example, the memory of the current machine is generally large, and the machine with 300-500 GB of memory is set. Of course, it would be great to have such a machine, and there are two options:
(1) If you mainly do full-text retrieval, you can consider giving Elasticsearch 32 G memory, and hand over the rest to Lucene as the file system cache of the operating system. All segments are cached, which will speed up full-text retrieval.
(2) If more sorting and aggregation are required, then larger heap memory is required. Consider creating two or more ES nodes on one machine instead of deploying one node with 32+GB of memory. Still sticking to the 50% rule, assuming you have a machine with 128 GB of memory, you can create two nodes with 32 GB of memory. That is to say, 64 G memory is given to the heap memory of ES, and the remaining 64 G is given to Lucene.
PS: If you choose the second option, you need to configure cluster.routing.allocation.same_shard.host: true to prevent the master copy of the same shard from existing on the same physical machine, because if it exists on one machine, the high availability of the copy will be lost.
Recommended article: ES memory problem analysis:
ElasticSearch optimization record with high CPU and memory usage