ES processing log field exceeds 1000 caused by error

Original link of this article: https://blog.csdn.net/xzk9381/article/details/114630302

The following error messages are often found in the log records of the ES cluster:

[2021-02-04T22:40:06,673][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [m-7416] failed to put mappings on indices [[[nginx-server-log-000022/dQLTO5UGQKOUVRn2FnS3Zw]]], type [_doc]
java.lang.IllegalArgumentException: Limit of total fields [1000] in index [nginx-log-000022] has been exceeded

The key error message is Limit of total fields [1000] in index [nginx-server-log-000022] has been exceededthat the reason for triggering this error is that the log is of JSON type, and the maximum field length of ES default JSON type log index is 1000, but many of the current business logs exceed 1000 fields.

The default maximum of 1000 fields is because too many fields defined in the index may cause a surge in mapping, which may cause memory errors and difficult to recover. So there are two ways to solve this problem:

1. Need business developers to streamline the number of fields, control it below 1000
2. Adjust the default field limit

The second method is used here to solve this problem. In Kibana, execute the following command to modify the limit for the specified index:

PUT nginx-server-log/_settings
{
    
    
  "index.mapping.total_fields.limit": 2000
}

You can also modify the configuration for all indexes:

PUT _all/_settings
{
    
    
  "index.mapping.total_fields.limit": 2000
}

You can also specify the configuration directly when creating the index to avoid subsequent modifications:

PUT _template/ngx-server
{
  "index_patterns": ["ngx-server-*"],
  "settings": {
    "number_of_shards": 15,
    "number_of_replicas": 1,
    "index.lifecycle.name": "ngx-api-normal",
    "index.lifecycle.rollover_alias": "ngx-server",
    "index.mapping.total_fields.limit": 2000
  }
}

Original link of this article: https://blog.csdn.net/xzk9381/article/details/114630302