Since ES is built based on lucene, the strength of lucene design is that lucene can make good use of operating system memory to cache index data to provide fast query performance. Lucene's index file segements is stored in a single file and is immutable. For the OS, it is very friendly to keep the index file in the cache for quick access; therefore, it is necessary for us to reserve half of the physical memory For lucene; the other half of the physical memory is reserved for ES (JVM heap).
Therefore, in terms of ES memory settings, the following principles can be followed:
-
When the machine memory is less than 64G, follow the general principle, 50% for ES, 50% for lucene.
-
When the machine memory is greater than 64G, follow the following principles
- If the main usage scenario is full-text search, it is recommended to allocate 4~32G memory to ESHeap; the other memory is reserved for the operating system and used by lucene (segments cache) to provide faster query performance.
- If the main usage scenario is aggregation or sorting, and most of them are numerics, dates, geo_points and not_analyzed character types, it is recommended to allocate 4~32G memory to ES Heap, and reserve the other memory for the operating system for use by lucene ( doc values cache), providing fast document-based clustering and sorting performance.
- If the usage scenario is aggregation or sorting, and they are all based on analyzed character data, more heap size is required. It is recommended to run multiple ES instances on the machine, and each instance should keep no more than 50% of the ES heap settings (but no more than 32G , when the heap memory is set below 32G, the JVM uses the object index compression technique to save space), and more than 50% is reserved for lucene.
-
Disabling swap, once the memory and disk swap is allowed, will cause fatal performance problems.
By setting bootstrap.memory_lock: true in elasticsearch.yml, the JVM can keep the memory locked to ensure the performance of ES. -
GC setting principles:
- a. Keep the existing settings of GC, the default setting is: Concurrent-Mark and Sweep (CMS), do not change to G1GC, because there are still many bugs in G1.
- b. Keep the existing settings of the thread pool. At present, the ES thread pool has more optimized settings than 1.X, just keep the status quo; the default thread pool size is equal to the number of CPU cores. If you must change it, set it according to the formula ((CPU core number * 3) / 2) + 1; it cannot exceed 2 times the CPU core number; but it is not recommended to modify the default configuration, otherwise it will cause serious damage to the CPU.