ElasticSearch 429 Too Many Requests circuit_breaking_exception

• Error message

{
  "statusCode": 429,
  "error": "Too Many Requests",
  "message": "[circuit_breaking_exception] 
  [parent] Data too large, data for [<http_request>] would be [2087772160/1.9gb], 
  which is larger than the limit of [1503238553/1.3gb], 
  real usage: [2087772160/1.9gb],
  new bytes reserved: [0/0b], 
  usages [request=0/0b, fielddata=1219/1.1kb, in_flight_requests=0/0b, accounting=605971/591.7kb], 
  with { bytes_wanted=2087772160 & bytes_limit=1503238553 & durability=\"PERMANENT\" }"
}

Try other queries, too. After investigation, it turned out to be the default ES cache settings allow not only into the buffer zone caused by specific analysis.

• ES buffer zone outlined
  ES in the query, the index data will be cached in memory (JVM) in which:
  

FIG JVM Heap ES is on the situation can see two limits: eviction line and circuit breakers. Eviction when the cache line data arrives, the partial data will be automatically expelled out, the cache is maintained within a safe range.
When the user is ready to execute a query operation, the circuit breaker will play a role, the current query cache data + the amount of data cache circuit breaker limit is reached, will return Data too large error, to prevent users from this query operation.

ES to cache data into two categories, FieldData and other data, we are going to see FieldData in detail, it is causing us this unusual "culprit."

• FieldData
  FieldData ES mentioned configuration refers to the field data. When sorted (sort), when the statistics (aggs), ES to all the data related to the fields read into memory (JVM Heap) in operation. It is equivalent to the data cache to improve query performance.

• Monitoring FieldData
  carefully monitored fielddata how much memory is being expelled and whether the data is very important.
  ielddata cache use can be monitored by the following way

# 对于单个索引使用 {ref}indices-stats.html[indices-stats API]

GET /_stats/fielddata?fields=*

# 对于单个节点使用 {ref}cluster-nodes-stats.html[nodes-stats API]

GET /_nodes/stats/indices/fielddata?fields=*

#或者甚至单个节点单个索引

GET /_nodes/stats/indices/fielddata?level=indices&fields=*

# 通过设置 ?fields=* 内存使用按照每个字段分解了

fielddata in memory_size_in_bytes indicate the amount of memory used, and evictions (expulsion) is 0. And after a period of observation, field length, memory size did not change. By inference, the moment of expulsion can not be effectively cached in the state.

• Cache Configuration

Cache size indices.fielddata.cache.size fieldData configuration may be equipped with an accurate percentage value. automatically clean up cache memory size to reach agreement, the expulsion of a part of FieldData data in order to accommodate the new data. The default is unbounded unlimited.
indices.fielddata.cache.expire conventions for how long do not have access to the data will be expelled, the default value is -1, i.e., infinite. expire configuration is not recommended, according to the data will consume a lot of time to expel performance. And this version is set in the near future will be discarded.

It seems, Data too large anomaly is due to the default value of unbounded fielddata.cache caused a.

• FieldData format
  in addition to the cache fetch size, we can also control the field data into the cache memory format.

In the mapping, we can set up this way:

{
    "tag": {
        "type":      "string",
        "fielddata": {
            "format": "fst"
        }
    }
}

For String type, format are the following:
paged_bytes (default): use a lot of memory to store and index terms in this field.
fst: with FSTthe form of storage terms. This case has internal memory can save more common prefixes used in terms, but slower than paged_bytes on the access speed.
doc_values: fieldData always stored in the disk, it is not loaded into memory. Access slowest and only in the index: the application of the no / not_analyzed of.

For digital geographic data and also have an optional format, but relatively String is more simple, concrete can be viewed in the api.
From the above we can see that a message: We are in addition to the configuration of the size of the cache, you can also choose the amount is not particularly important, but a lot of String type field use to compress the size of the cache type fst.

• Breaker
  cache configuration fieldData, there will be a point brought to our question: fielddata size is loaded only after the data is verified. If the next query ready to load coming fieldData let the cache size exceeds the available heap what will happen? Regrettably, it will generate a OOM exception.
  Breaker is used to control the cache loaded, it estimates the current query application memory usage and restrictions. A circuit breaker configured as follows:

indices.breaker.fielddata.limit: The circuit breaker limits fielddata fielddata size, default 60% of the size of the heap.
indices.breaker.request.limit: other parts of the size of the structure to complete the query request estimated circuit breaker requirements, limit their default case of 40% of the size of the heap.
indices.breaker.total.limit: The total circuit breaker enclosure of the circuit breaker to ensure that the request and fielddata default use these two parts does not exceed 70% of the total memory heap size.

Inquire

/_cluster/settings

Set up

PUT /_cluster/settings
{
  "persistent": {
    "indices.breaker.fielddata.limit": "60%"
  }
} 


PUT /_cluster/settings
{
  "persistent": {
    "indices.breaker.request.limit": "40%"
  }
} 


PUT /_cluster/settings
{
  "persistent": {
    "indices.breaker.total.limit": "70%"
  }
} 

Circuit breaker limit can be specified by the file config / elasticsearch.yml, can also be dynamically updated on the cluster:

PUT /_cluster/settings
{
  "persistent" : {
    "indices.breaker.fielddata.limit" : 40% 
  }
}

When the cache size reaches the size of the configured breaker what would happen? The answer is: will we say back to the beginning of the Data too large exception. This setting is to cause the user to reflect the ES services, our configuration problems? Is not a query of the form does not, a query cache you need to use so much?

Set the cache file config / elasticsearch.yml file using recycled

indices.fielddata.cache.size:  40%

• to sum up

1. The Data too large abnormal ES is the default configuration of a pit, we did not configure indices.fielddata.cache.size, it does not recover cache. Cache size limit is reached, you can not insert data inside. The default configuration unfriendly personal feeling, I do not know whether ES improvement in future versions.

1. FieldData current buffer size <indices.fielddata.cache.size
   current fieldData buffer size + load the next incoming query fieldData <indices.breaker.fielddata.limit
   fielddata.limit configuration requires slightly larger than fielddata.cache.size. And when fieldData reach fielddata.cache.size cache will automatically start cleanup mechanism. expire configuration is not recommended.
   Memory size limit 3.indices.breaker.request.limit other parts of the query need to use. indices.breaker.total.limit limit the total (fieldData + other parts) size.
   4. When creating a mapping, the cache may be provided fieldData format control data format.