Author: Ming-yi world
Background
In today's world, all walks of life have massive data generated every day, in order to obtain the desired results from these huge amounts of data, the need for data extraction, transformation, storage, maintenance, management and analysis. This is already far beyond the ordinary ability to achieve processing tools, databases, and big data tools distributed and parallel processing architecture based on the only mechanism to achieve these functions. Elasticsearch response as described above is one of the most popular open source of most data storage engine use cases.
Elasticsearch is a distributed data storage and search engines, fault-tolerant and high-availability features. In order to take full advantage of its search function, you need to configure Elasticsearch correct.
Simple default configuration is not suitable for every real business scenarios. Combat development operation and maintenance, personalized fit cluster configuration to achieve their own business scenario is the only way to optimize cluster performance. This article collections actual business scenarios, highlighting improved performance intensive search Elasticsearch cluster of dry cargo configuration.
1, optimize the allocation of the index level
by default, versions 6.x and earlier in Elasticsearch there are five main index fragmentation and a copy, 7.X version 1 and later a master. This configuration does not apply to all business scenarios. Slice configuration must be set correctly in order to maintain stability and efficacy index.
1.1, fragment size
fragment size is very important for a search query.
On the one hand, if you assign too many fragments to the index, the Lucene segments will be very small, resulting in increased overhead. When multiple queries simultaneously, many small fragments query throughput also decreases.
On the other hand, too much fragmentation can cause performance degradation search and recovery time is longer.
Elasticsearch official suggested a fragment size should be around 20 Dao 40 GB.
For example, if you to calculate the index is stored 300 GB of data can be 9-15 primary slice allocated for the index.
The cluster size, the cluster is assumed that there are 10 nodes, you can choose to assign the index 10 for this master slice to slice evenly distributed between the cluster nodes.
1.2, dynamic persistence writing scene data
if there is a continuous stream of data is written to the cluster Elasticsearch, such as: Internet real-time data is written crawler ES cluster. Time-based index should be used in order to more easily maintain the index.
If the throughput varies with time into a data stream, it is necessary to appropriately change the configuration of the dynamic index extension data.
So, how to query all of the documents into different time-based index? The answer is an alias. More indexes can be placed in an alias, and the alias search query would just like on a single index.
Of course, the need to maintain a good balance. Note ponder: how much data is written to an alias? Too many small write performance index will have a negative impact on the alias.
For example, on a weekly or monthly basis in units of indexing is the need to combine business scenarios considered in balance?
If the index on a monthly basis is recommended optimal performance, then the same data on a weekly basis to index because the index is bound to lead to too much negative performance issues.
1.3, Index Sorting
Note: The index sorting mechanism is 6.X version only features.
When creating a new index Elasticsearch can configure each slice segment sorted. By default, Lucene does not apply any sort. index.sort. * define which fields should be used to sort documents within each Segment.
Example of use:
PUT /twitter
{
"settings" : {
"index" : {
"sort.field" : "date",
"sort.order" : "desc"
}
},
"mappings": {
"properties": {
"date": {
"type": "date"
}
}
}
}Objective: index sorting is one very important way to optimize Elasticsearch retrieval performance.
Vernacular: index sorting mechanism by the time of writing the sort designated one or more fields, will greatly improve the performance of retrieval.
2, slice level Optimization
fragmentation of the underlying basic reading and writing unit, the purpose of fragmentation index is divided great, so that reading and writing in parallel. Writing process to write the master slice, slice master copy write and then write successfully fragmented.
A copy of the minutes of the film appears to enhance the high-availability cluster and read throughput.
Fragmentation in the optimization, the size of the fragment, how many nodes fragmentation is a major consideration. A copy of the fragment is important to extend the search throughput, if the hardware conditions permit, you can increase the number of copies carefully slice.
A good starting point is to allocate capacity planning fragment, "" in-depth understanding elasticsearch "stressed: the ideal number of fragments should be dependent on the number of nodes," which is the node number of 1.5 to 3 times the number.
Copy sheet distribution fractal formula: max (max_failures, ceil (num_nodes /) num_primaries) - 1).
How it works: If you have num_nodes cluster nodes, a total of num_primaries main fragment, if you want to be able to handle up to max_failures node fails, then the number for your copy of the above formula value.
Overall: simple nodes and the number of formulas, the number of copies of fragments as follows:
the desired big number of fragments nodes * = (number of copies +1).
3, Elasticsearch overall level configuration
When configuring Elasticsearch cluster, one of the most important considerations is to ensure that at least half of the available memory into the file system cache, so Elasticsearch can be indexed hot regions remain in physical memory.
In the design of a cluster should also consider the physical space available heap. Elasticsearch recommended allocation should be at most 20 slices / GB based on the available heap space fragmentation, which is a good rule of thumb.
For example, a 30 GB stack node should be a maximum of 600 slices, to maintain a good state of the cluster.
Stored on one node can be expressed as follows: node may support disk space = 20 (heap size unit: GB) (GB units in fragment size), since the efficiency is usually seen in the cluster size of 20 to 40 GB between the slice, so that the maximum 16 GB of storage space may be available to a node heap space, up to 12 TB disk space (20 is 16 40 = to 12.8TB).
Border awareness helps prepare for better design and operation of future expansion.
Many settings can be configured at run-time as well as the initial stage.
When building Elasticsearch index and the cluster itself for better search performance, to understand what configuration can be modified at run time and can not be amended with what is essential.
3.1 Dynamic set
1, set the historical index data is read-only state.
Based on the execution phase of dynamic index of time, if the index does not store historical data write operation, the monthly index can be set to read-only mode, in order to improve the search performance of these indexes.
Read-only index after the actual arrangement of 6.X:
PUT /twitter/_settings
{
"index.blocks.read_only_allow_delete": null
}2, a read-only state index, a merge segment.
When the index is set to read-only, it can be forced through the segment merging operation to reduce the number of segments.
Combined segment optimization will lead to better search performance, the overhead since each fragment is dependent on the count and size of the segment.
Note 1: Do not merge segment index for reading and writing, because it will lead to a very large segment (per segment> 5Gb).
Note 2: This operation should be performed during off-peak time, because this is a very resource-intensive operation.
Segment merge operation combat mode:
curl -X POST "localhost:? 9200 / kimchy / _forcemerge only_expunge_deletes = false & max_num_segments = 100 = & flush to true"
3, using the preference optimize cache utilization
have multiple cache can help improve search performance, such as the file system cache, the cache request or query cache.
However, all these caches are maintained at the node level, which means that if you have one or more copies of the default routing algorithm and cluster continuous operation based on the same request twice, both requests will go to a different slice a copy of the node-level cache to help prevent.
Since the user of a search application run by one of similar requests are common, for example in order to retrieve the index portion narrower subset analysis, or the current user preference using preference value of the session identifier may be used to help optimize the cache.
preference practical example:
GET /_search?preference=xyzabc123
{
"query": {
"match": {
"title": "elasticsearch"
}
}4, prohibiting the exchange
can disable the exchange on each node to ensure stability, and should be avoided at all costs the exchange. It could lead to garbage collection for a few minutes rather than milliseconds, and may even lead to a slow response to a connection node disconnected from the cluster.
In Elasticsearch distributed system, so that the operating system more efficient termination node. Bootstrap.memory_lock can be set to True to disable it.
Linux system-level configuration:
`sudo swapoff -a``
Elasticsearch profile elasticsearch.yml configuration:bootstrap.memory_lock: true
5, increase the refresh interval refresh_interval
default refresh interval is 1 second. This forces Elasticsearch create one segment per second. In actual service, the refresh interval should be increased according to the use, for example: increased to 30 seconds.
After this, 30s produced a large segment, compared with the second refresh greatly reduce future segment consolidation pressure. Will ultimately enhance the search query write performance and more stable.
Update refresh interval combat:
PUT /twitter/_settings
{
"index" : {
"refresh_interval" : "1s"
}
}6, provided max_thread_count
index.merge.scheduler.max_thread_count default setting is
Math.max (1, Math.min (4, Runtime.getRuntime (). AvailableProcessors () / 2))
but this applies to SSD configuration. For the HDD, it should be set to 1.
Combat:
curl -XPUT 'localhost:9200/_settings' -d '{
"index.merge.scheduler.max_thread_count" : 1
}
7, dynamic allocation prohibit fragmentation
sometimes, elasticsearch rebalance the cluster fragments. This may reduce the performance of retrieval.
In production mode, if desired, can be provided by the rebalancing cluster.routing.rebalance.enable to none.
PUT / _cluster / settings
{
"transient" : {
"cluster.routing.allocation.enable" : "none"
}
}Typical application scenarios in which the comprises:
a cluster temporary restart, a node removed;
cluster-by upgrading nodes; When you turn off the node, the allocation process will immediately attempt to copy the slices on the node to other nodes in the cluster, resulting in wasting a lot of IO. disable allocation to avoid this situation before shutting down the node.
8, the full effect of using an approximate date cache
queries are usually not cached on the date field is now used, since the scope of the matching has been changed.
However, user experience, switch to the approximate date generally acceptable, and can make better use of the benefits brought by the query cache.
Combat is as follows:
GET index/_search
{
"query": {
"constant_score": {
"filter": {
"range": {
"my_date": {
"gte": "now-1h/m",
"lte": "now/m"
}
}
}
}
}
}3.2 Initial Set
1, with multiple fields to enhance the retrieval performance
query_string or multi_match query field the more targeted, the slower the retrieve.
A plurality of conventional techniques to improve search speed when the field is an index value to copy single field.
For some fields frequently queries, please use the copy-to Elasticsearch function.
For example, the car's brand name, the engine version, model name and color fields can be copied to the instruction merge. It will improve your search query performance in these fields.
PUT movies
{
"mappings": {
"properties": {
"cars_infos": {
"type": "text"
},
"brand_name": {
"type": "text",
"copy_to": "cars_infos"
},
"engine_version": {
"type": "text",
"copy_to": "cars_infos"
},
"model ": {
"type": "text",
"copy_to": "cars_infos"
},
"color": {
"type": "text",
"copy_to": "cars_infos"
}
}
}
}2, set the slice allocated to a specific node
business scenarios real business problems often encountered: how to set the fragmentation of non-equilibrium distribution, there is a new node configuration is extremely high, whether Multiply point in the past?
A shard distribution node on which, in general, is determined automatically by the ES. The following conditions trigger the dispensing action:
1) generating a new index
2) deleting the index
3) Add Cc slice
4) decrease initiator node equalized data
ES provides a list of parameters This section details the control logic, one of which They are: in the case of heterogeneous clusters having more slices hardware node allocating the allocated weights.
In order to assign weights
to set cluster.routing.allocation.balance.shard value, the default value is 0.45f.
The greater the value the more likely the node level slice equalizer.
Combat:
PUT _cluster/settings
{
“transient” : {
“cluster.routing.allocation.balance.shard” : 0.60
}
}3, adjusting the ratio of the size fuse memory
query itself will also have a significant impact on the response delay. In order not to trigger the fuse at the time of the query and result in Elasticsearch cluster in an unstable state,
depending on the complexity of the query will be indices.breaker.total.limit to fit your JVM heap size. The default value for this setting JVM heap is 70%.
PUT /_cluster/settings
{
"persistent" : {
"indices.breaker.fielddata.limit" : "60%"
}
}A circuit breaker is preferably provided a relatively conservative point value.
Author "Elastic source code analysis," Chao pointed out that:. "Elasticsearch 7.0 adds indices.breaker.total.use_real_memory configuration items, can be more precise analysis of the current memory conditions occur in time to prevent OOM While this configuration will add a little performance loss, but can increase the JVM memory usage, and enhance the protection of nodes. "
4, a particular search scene, increasing the search thread pool configuration
by default, elasticsearch the main use case is to search. In the case of the need to increase retrieve concurrency, it can increase the thread pool for search settings, at the same time, you can reduce how much discretion the thread pool for the index based on the number of cores on the CPU node.
Example: Change profile elasticsearch.yml add the following:
thread_pool.search.queue_size: 500
#queue_size允许控制没有线程执行它们的挂起请求队列的初始大小。5, open adaptive replica selection
should be opened adaptive replica selection. The request will be redirected to the fastest responding node.
When there are multiple copies of data, you can use a standard set of elasticsearch referred to as adaptive copy selection to select the best copy of the data in accordance with each node containing a copy of the fragment response time, service time, and the queue size.
This can improve throughput and reduce latency query search of large applications.
This configuration is disabled by default, open combat method:
PUT /_cluster/settings
{
"transient": {
"cluster.routing.use_adaptive_replica_selection": true
}
}4 Summary
Elasticsearch many cluster configuration settings response delay can be reduced to improve retrieval performance. These are just the tip of the iceberg.
Welcome to the bizarre exchange, like articles like this one point to remember, thanks for the support!