Elastic: using the index lifecycle management architecture to achieve hot warm cold
Index Lifecycle Management (ILM) was first introduced in Elasticsearch 6.6 (Beta) and officially launched the 6.7 version of a feature. ILM is part Elasticsearch mainly used to help you manage indexes.
In this blog, we will discuss how to implement ILM hot warm cold architecture. Warm and cold thermal architecture commonly used in logging sequential data or metrics class. For example, suppose you are using a polymerization Elasticsearch log files from multiple systems. Today's log is being indexed frequently, and this week's log search the largest amount (heat). Last week's log may be searched frequently, but this week the log frequency is not so high (temperature). Last month, the log search frequency may be higher, it may be low, but the best retained for a period of time just in case (cold).
Elasticsearch allows you to define which nodes are the nodes hot, warm or cold node node. ILM allows you to define when moving between the two phases, and how to handle an index when entering that stage.
For hot warm cold architecture, there is no hard and fast setup. However, in general, the hot node requires more CPU resources and faster IO. For cold and warm node for node, each node will usually require more disk space, but even with less CPU resources and slow IO device, can barely cope.
Distribution sensing configuration fragment
Since the cold heat depends on the temperature distribution sensing fragmentation, therefore, we first node flag which nodes are hot, warm node, and (optionally) a cold node. This operation can be done or by starting elasticsearch.yml profile parameters. E.g:
bin/elasticsearch -Enode.attr.data=hot
bin/elasticsearch -Enode.attr.data=warm
bin/elasticsearch -Enode.attr.data=coldConfiguring an ILM strategy
Next, we need to define an ILM strategy. ILM strategy can be reused in any number of indices of your choice.
ILM strategy is divided into four main phases - hot, warm, cold and removed. You do not need to define each stage, ILM will always be in that order to perform the various stages (skip any undefined stage) in a policy. For each stage, you need to define the time to enter the stage, but also need to define a set of actions as you see fit to manage the index. For the hot temperature cold architecture, you can configure the dispensing operation, the data from the mobile node to the temperature of the hot node, then again to the mobile node from the cold junction temperature.
In addition to moving data between the warm and cold heat node, but you can also configure a number of additional operations. Scroll update size or longevity for managing each index. The combined operation may be used to optimize the forced index. Freezing operation can be used to reduce the memory pressure in the cluster.
Basic ILM strategy
Let's look at a very basic ILM strategy:
PUT /_ilm/policy/my_policy
{
"policy":{
"phases":{
"hot":{
"actions":{
"rollover":{
"max_size":"50gb",
"max_age":"30d"
}
}
}
}
}
}This policy states that the index storage time of 30 days or when the size of the index reaches 50GB (master slice based), the index will rollover and start writing a new index.
ILM and Index Template
Next, we need to associate with this ILM strategy index template:
PUT _template/my_template
{
"index_patterns": ["test-*"],
"settings": {
"index.lifecycle.name": "my_policy",
"index.lifecycle.rollover_alias": "test-alias"
}
}Note: When using a rolling update the index template (rather than directly on the index) when specifying an ILM strategy, it is necessary to make this association.
For the operation, including rolling update policy, you must also use an alias index started writing after creating an index template.
PUT test-000001
{
"aliases": {
"test-alias":{
"is_write_index": true
}
}
} Assumptions rolling updates all the requirements are properly met, any updates to test- * at the beginning of the new index will scroll automatically after 30 days or up to 50GB. By using a rolling update management to the index at the beginning of max_size, you can greatly reduce the amount of fragmentation index, thereby reducing overhead.
Configuring an ILM strategy for acquisition
Beats and Logstash support ILM, and will enable the default policy settings on a similar example shown. In addition, Beats and Logstash will also handle all the requirements of the rolling update operations. This means that, when enabled ILM for the Beats and Logstash, unless you have a large amount of daily index (greater than 50GB / day), or the size of the index will likely be a major factor when you create a new index is determined (which is a good thing !). 7.0.0 From the start, with the rollover of ILM will be the Beats and Logstash default configuration.
However, because there is no hard and fast heat settings for warm and cold architecture, therefore, Beats and Logstash will not be included with the hot warm and cold strategy. We can develop a cold temperature suitable for hot new strategy, and some optimization in the process.
Although we can update or Logstash Beats default policy, but this will blur the boundaries between the default and custom values. In addition, the updated version of the default policy will increase in the future can not apply the correct risk strategy (Beats template defaults 7.0+ will have to change). We can use the Beats and Logstash configured to define custom policies through their respective configuration. This method is also not a bad idea, but you may need to change hundreds (or thousands) Beats a configuration to change ILM strategy.
The third method described herein, by using a multi-template matching to allow ILM Elasticsearch maintain complete control strategy.
ILM strategies for optimizing heat warm cold
First, let's create an optimized architecture for warm and cold hot ILM strategy. Again, this is not a one size fits all set, your request will be different.
PUT _ilm/policy/hot-warm-cold-delete-60days
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size":"50gb",
"max_age":"30d"
},
"set_priority": {
"priority":50
}
}
},
"warm": {
"min_age":"7d",
"actions": {
"forcemerge": {
"max_num_segments":1
},
"shrink": {
"number_of_shards":1
},
"allocate": {
"require": {
"data": "warm"
}
},
"set_priority": {
"priority":25
}
}
},
"cold": {
"min_age":"30d",
"actions": {
"set_priority": {
"priority":0
},
"freeze": {},
"allocate": {
"require": {
"data": "cold"
}
}
}
},
"delete": {
"min_age":"60d",
"actions": {
"delete": {}
}
}
}
}
}Hot
this index ILM strategy will first priority is set to a higher value, so that the heat index to recover before other indexes. 30 days or up to 50GB when (to meet any a), the index will rollover, the system creates a new index. The new index will restart strategy, and the current index (just rollover the index) will wait seven days after the rollover re-entering the warm phase.Temperature
after the index into the temperature stage, the ILM index will shrink to a slice, the combined force of an index segment, and the index is set to low priority (but higher than the cold phase) phase of specific heat values, by assigning temperature operation to move the index node. After completion of the operation, the index will wait for 30 days (counting from the time of rollover) into the cooling stage.Cold
after the index into the cold phase, ILM will again lower the index priority to ensure that the heat index and temperature index get undeleted. Then, ILM freeze index node and move it to cool. After the operation is completed, the index will wait 60 days (counting from the time of the rollover) After entering the delete phase.delete
We have not discussed this stage deleted. Simply put, delete phase has a delete operation for deleting the index. In the delete phase, you will always need to have a min_age conditions to allow the index to stay hot, warm or cold stage in a given period of time.
Creating an ILM strategy in the Kibana
现在，我们需要将这个新的 hot-warm-cold-delete-60days 策略与 Beats 和 Logstash 索引关联起来，确保它们写入 hot 数据节点。由于 Beats 和 Logstash 都会默认管理其自己的模板，因此，我们将使用多模板匹配，为您要应用 ILM 策略的索引模式添加策略和分配规则。因为这个模板匹配 Beats 和 Logstash 索引模式，所以您需要知道想要匹配的索引模式。
在这里，我们使用 logstash-、metricbeat- 和 filebeat-*，假设 Beats 和 Logstash 在其配置中启用了 ILM 支持，您可以在此添加任意数量的索引模式。如果在此处为不支持 ILM 的数据生产者添加索引模式，则需要手动满足此策略中针对滚动更新的要求。
PUT _template/hot-warm-cold-delete-60days-template
{
"order":10,
"index_patterns": ["logstash-*", "metricbeat-*", "filebeat-*"],
"settings": {
"index.routing.allocation.require.data": "hot",
"index.lifecycle.name": "hot-warm-cold-delete-60days"
}
}Enabling ILM in Beats and/or Logstash
最后，让我们为 Beats 和 Logstash 启用 ILM。
对于 6.7 Beats：
output.elasticsearch:
ilm.enabled: true对于 6.7 Logstash：
output {
elasticsearch {
ilm_enabled => true
}
}由于在更新的版本中可能会发生更改，因此，请参考相应版本的文档，了解如何在 Beats 和 Logstash 中启用。
现在，任何与索引模式匹配的新索引都将在热节点上创建新索引，ILM 将应用 hot-warm-cold-delete-60days 策略。
更新 ILM 策略
您可以随时更新 ILM 策略。但是，您对策略所做的更改将仅在阶段更改时应用。例如，如果您的索引当前处于热阶段（并等待进入温阶段），则您对热阶段所做的任何更改都不会对该索引生效，但一旦索引进入温阶段，对温阶段的任何更改都会被获取到。这样做是为了避免对给定的阶段重复操作。您可以通过解释 API 查看索引的 ILM 状态。
由于使用了相同的底层机制，因此，关于如何实现热温架构预 ILM 的许多先前信息仍然适用。但是，现在有了 ILM，就不再需要 Curator工具来实现这个模式了。
展望
从 7.0 版开始，当 Beats 和 Logstash 连接到支持生命周期管理的集群时，它们会默认使用索引生命周期管理。此外，Beats 已将大多数 ILM 设置从 output.elasticsearch.ilm 命名空间移动到 setup.ilm 命名空间。例如，请参阅 7.0 Filebeat 文档。而且，从 7.0 开始，系统索引（如 .watcher-history-*）可以由 ILM 管理。
ILM 可支持您为时序索引轻松实现类似热温冷这样的成本节约型架构。