Get a suitable Elasticsearch Docker base image
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.4.0
Linux operating system setup (Centos 8 )
Disable memory swapping (Disable Swapping)
The operating system expects to use as much memory as the system cache, so some memory that is not currently used by the application will be swapped to the disk through policy.
Allowing this configuration may swap the unused memory of the ES JVM and the running PageCache to the disk. Swapping is an extremely performance-consuming action and should be avoided.
1. Temporarily disable memory swapping (OS-level locking)
- No need to restart the operating system
sudo swapoff -a
- Requires an operating system restart
vi /etc/fstab
#注释包含`swap`的⾏
2. Reduced propensity for memory swapping (OS-level locking)
- No need to restart the operating system
sudo sysctl -w vm.swappiness=1
- Requires an operating system restart
sudo vi /proc/sys/vm/swappiness
3. Allow processes to lock memory
- No need to restart the operating system
ulimit -l unlimited
- Requires an operating system restart
vi /etc/security/limits.conf
es_user soft memlock -1
es_user hard memlock -1
- Verify that the file description settings are in effect
ulimit -l
4. Use the mlockall function on Linux (Centos) to lock the allocated memory address of the process to prevent
Elasticsearch's memory is swapped to disk (process-level locking)
- parameter configuration
vim $ES_HOME/conf/elasticsearch.yml
bootstrap.memory_lock: true
- Verify that memory locking is in effect
curl -XGET 'HOST:PORT/_nodes?pretty&human&filter_path=**.host,**.
Set file descriptors (File Descriptors)
Elasticsearch requires a large number of file descriptors (File Descriptors), and fewer file descriptors may result in the inability to establish more connections, which may eventually lead to data loss.
1. Temporarily set the maximum total number of virtual memory mappings (operating system level)
- No need to restart the operating system
ulimit -n 65536
- Requires an operating system restart
vi /etc/security/limits.conf
es_user soft nofile 65536
es_user hard nofile 65536
- Verify that the file description settings are in effect
ulimit -n
Max Map Count & Virtual Memory (Max Map Count & Virtual Memory)
Elasticsearch is a distributed search engine built on top of Lucene. Lucene uses MMapDirectory to store the underlying index by default in the 64-bit Linux operating system. In the operating system, the default mmap count setting is relatively small, which may lead to insufficient memory exception.
1. Temporarily set the maximum total number of virtual memory mappings (operating system level)
- No need to restart the operating system
sysctl -w vm.max_map_count=262144
- Requires an operating system restart
vi /etc/sysctl.conf
vm.max_map_count = 262144
- Verify that the maximum total number of virtual memory mappings is in effect
sysctl vm.max_map_count
2. Temporarily set virtual memory address limit (operating system level)
- No need to restart the operating system
ulimit -v unlimited
- Requires an operating system restart
vi /etc/security/limits.conf
es_user soft as unlimited
es_user hard as unlimited
ulimit -v 111211123
- Verify that virtual memory address restrictions are in effect
ulimit -v
Set the maximum number of threads that users are allowed to create (Max User processes)
Elasticsearch nodes internally use multiple thread libraries to process the work of different modules. If the setting is too small, it may cause requests or cluster exceptions because threads cannot be created.
1. Temporarily increase the limit on the maximum number of user threads (operating system level)
- No need to restart the operating system
ulimit -u 10240
- Requires an operating system restart
vi /etc/security/limits.conf
es_user soft nproc 10240
es_user hard nproc 10240
- Verify that the file description settings are in effect
ulimit -u
Set the maximum file size (Max File Size)
The index file of Elasticsearch (single segment file | single translog log file may occupy a large amount of disk), if the operating system does not support it, data may be lost
1. Temporarily remove the maximum file disk usage limit (operating system level)
- No need to restart the operating system
ulimit -f unlimited
- Requires an operating system restart
vi /etc/security/limits.conf
es_user soft fsize unlimited
es_user hard fsize unlimited
- Verify that the maximum file disk usage limit is in effect
ulimit -f