Dynamic mapping and dynamic templates in ES

To facilitate the presentation and switch ES address, in ~/.bashrcadding the following variables and scripts:
ES=localhost:9200
escurl () { 
   curl -H 'Content-Type: application/json' "$@";
}
复制代码

Elasticsearch is very dynamic and flexible. For example, when adding documents to a non-existent index, it will automatically create the index, for example:

$ escurl -XPUT $ES/my_article/doc/1?pretty -d '
{
  "title": "标题",
  "createdAt": "2020-02-02T02:02:02.020Z",
  "wordCount": 100,
  "extra": {
    "deleted": false,
    "score": 8.5
  }
}'
复制代码

If the my_article index does not exist, it will automatically create the my_article index and add data to it. The data structure is:

{
  "mappings": {
    "doc": {
      "properties": {
        "createdAt": { "type": "date" },
        "wordCount": { "type": "long" },
        "extra": {
          "properties": {
            "score": { "type": "float" },
            "deleted": { "type": "boolean" }
          }
        },
        "title": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}
复制代码

It can be found that ES automatically does the following field type mapping

The title field is mapped to the text type
The createdAt field is mapped to the date type
The wordCount field is mapped to the long type
extra.score field is mapped to float type
The extra.deleted field is mapped to boolean type

The guess is very accurate, and the subsequent addition of new fields can also dynamically guess and update the mapping.

Dynamically update the mapping

If you continue to add a new document, there are a few more fields:

$ escurl -XPUT $ES/my_article/doc/2?pretty -d '
{
  "title": "新文章",
  "createdAt": "2020-02-02T02:02:02.020Z",
  "wordCount": 100,
  "likes": 0,
  "author": "作者",
  "extra": {
    "deleted": false,
    "score": 8.5,
    "remark": "备注"
  }
}'
复制代码

Found that the mapping was automatically updated:

{
  "mappings": {
    "doc": {
      "properties": {
        "createdAt": { "type": "date" },
        "wordCount": { "type": "long" },
        "author": {
          "type": "text",
          "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }
        },
        "extra": {
          "properties": {
            "score": { "type": "float" },
            "deleted": { "type": "boolean" },
            "remark": {
              "type": "text",
              "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }
            }
          }
        },
        "title": {
          "type": "text",
          "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }
        },
        "likes": { "type": "long" }
      }
    }
  }
}
复制代码

In other words, ES will guess its data type based on the value of the field and dynamically add it to the type mapping.

Manual intervention dynamic mapping

Although dynamic mapping is flexible, sometimes you want to clarify the data structure because not all fields need to be stored. At this time, you can configure the dynamic option of the index. There are three optional values:

true: dynamically add new fields (default value)
false: ignore the new field
strict: throw an exception if a new field is encountered

$ escurl -XPUT $ES/my_article?pretty -d '
{
  "mappings": {
    "doc": {
      "dynamic": "false",
      "properties": {
        "title":  { "type": "keyword"},
        "wordCount": { "type": "long" },
        "createdAt": { "type": "date" },
        "extra":  {
          "type": "object",
          "dynamic": true
        }
      }
    }
  }
}'
复制代码

The above index means: if a new field is encountered, it will be automatically ignored, and the internal object extra will dynamically create a new field when it encounters a new field. Note that if the index already exists, an error will occur, and it must be deleted and rebuilt. At this time, if you add the following data:

$ escurl -XPUT $ES/your_article/doc/2?pretty -d '
{
  "title": "新文章",
  "createdAt": "2020-02-02T02:02:02.020Z",
  "wordCount": 100,
  "likes": 0,
  "author": "作者",
  "extra": {
    "deleted": false,
    "score": 8.5,
    "remark": "备注"
  }
}'
复制代码

The likes and author fields will be ignored, and the extra.remark field will be added.

Constrain dynamic mapping with dynamic templates

The automatic inference function of dynamic mapping is very powerful, but sometimes it does not fully meet business needs. For example, I hope that all string types are mapped to keywords instead of text, and all numbers are mapped to doubles instead of longs. At this time, dynamic templates are needed ( dynamic_templates), you can fully control the mapping type of the newly generated field. E.g:

$ escurl -XPUT $ES/my_article?pretty -d '
{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  },
  "mappings": {
    "doc": {
      "dynamic_templates": [
        {
          "string_fields": {
            "match": "*",
            "match_mapping_type": "string",
            "mapping": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        {
          "number_fields": {
            "match": "*",
            "match_mapping_type": "long",
            "mapping": {
              "type": "double"
            }
          }
        }
      ]
    }
  }
}'
复制代码

dynamic_templates is an array, which means that multiple templates can be added, and ES will check in order and enable the first matching template.

The above approach is to specify a dynamic template for the current index. In fact, you can also reverse it. Create a dynamic template first and let the template specify a matching index.

$ escurl -XPUT $ES/_template/my-template?pretty -d '
{
  "index_patterns":[ "my_*" ],
  "mappings": {
    "doc": {
      "dynamic_templates": [
        {
          "string_fields": {
            "match": "*",
            "match_mapping_type": "string",
            "mapping": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        {
          "number_fields": {
            "match": "*",
            "match_mapping_type": "long",
            "mapping": {
              "type": "double"
            }
          }
        }
      ]
    }
  }
}'
复制代码

This time to create a my-xxxtime index, dynamic field mapping will be mapped on the basis of inside my-template rules. Since the dynamic template is very practical, the syntax of its addition, deletion and modification is recorded below:

View template

escurl -XGET $ES/_template?pretty # 查看所有模板
escurl -XGET $ES/_template/tpl_1?pretty # 查看指定模板 tpl_1
escurl -XGET $ES/_template/tpl_1,tpl_2?pretty # 批量查看模板 tpl_1 和 tpl_2
复制代码

The return result is an object, the key is the template name, and the value is the template definition.

Check if the template exists

$ escurl --head $ES/_template/tpl_1
复制代码

Return if it exists:

HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 488
复制代码

Otherwise return:

HTTP/1.1 404 Not Found
content-type: application/json; charset=UTF-8
content-length: 2
复制代码

Create template

$ escurl -XPUT $ES/_template/my-prefix-template
{
  "order": 0,
  "index_patterns": [
    "prefix-*"
  ],
  "settings": {
    "index": {
      "number_of_shards": "5",
      "number_of_replicas": "0"
    }
  },
  "mappings": {
    "doc": {
      "dynamic_templates": [
        {
          "string_fields": {
            "match": "*",
            "match_mapping_type": "string",
            "mapping": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      ],
      "properties": {
        "discount": {
          "type": "double"
        },
        "pay": {
          "type": "double"
        }
      }
    }
  }
}
复制代码

Delete template

$ escurl -XDELETE $ES/_template/template_1