ElasticSearch-- custom template

 

configure output

elasticsearch{
　　action => "index"
　　hosts => ["xxx"]
　　index => "http-log-logstash"
　　document_type => "logs"
　　template => "opt/http-logstash.json"
　　template_name => "http-log-logstash"
　　template_overwrite => true
}

 

Custom template example

{ 
    "template" : "logstash-*", 
    "order":1,
    "settings" : { "index.refresh_interval" : "60s" }, 
    "mappings" : { 
        "_default_" : { 
            "_all" : { "enabled" : false }, 
            "dynamic_templates" : [{ 
              "message_field" : { 
                "match" : "message", 
                "match_mapping_type" : "string", 
                "mapping" : { "type" : "string", "index" : "not_analyzed" } 
              } 
            }, { 
              "string_fields" : { 
                "match" : "*", 
                "match_mapping_type" : "string", 
                "mapping" : { "type" : "string", "index" : "not_analyzed" } 
              } 
            }], 
            "properties" : { 
                "@timestamp" : { "type" : "date"}, 
                "@version" : { "type" : "integer", "index" : "not_analyzed" }, 
                "path" : { "type" : "string", "index" : "not_analyzed" }, 
                "host" : { "type" : "string", "index" : "not_analyzed" },
                "record_time":{"type":"date","format": "yyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"}, 
                "method":{"type":"string","index" : "not_analyzed"},
                "unionid":{"type":"string","index" : "not_analyzed"},
                "user_name":{"type":"string","index" : "not_analyzed"},
                "query":{"type":"string","index" : "not_analyzed"},
                "ip":{ "type" : "ip"}, 
                "webbrower":{"type":"string","index" : "not_analyzed"},
                "os":{"type":"string","index" : "not_analyzed"},
                "device":{"type":"string","index" : "not_analyzed"},
                "ptype":{"type":"string","index" : "not_analyzed"},
                "serarch_time":{"type":"date","format": "yyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"},
                "have_ok":{"type":"string","index" : "not_analyzed"},
                "legal":{"type":"string","index" : "not_analyzed"}
            } 
        } 
    } 
}

 

Key settings

• template for index-pattern

Only match  logstash-* the index will use this template. Sometimes we change Logstash default index name, remember that you have to upload your custom template to match the name of the index by the PUT method. Of course, I recommended practice is to put your name on the back of a custom "logstash-", to become  index => "logstash-custom-%{+yyyy.MM.dd}" so.

 

• refresh_interval for indexing

Elasticsearch is a near- real-time search engine. It is actually a data refresh every 1 second. For log analysis application, so we do not need real-time, so logstash modified into template comes with 5 seconds. You can also refresh interval based on the need to continue to improve data write performance amplification.

 

• multi-field with not_analyzed

Elasticsearch will automatically use their default tokenizer (space, dot, slash equally divided) to analyze the field. Word breaker for search and ratings are very important, but greatly reduces the performance of the index and write polymerization request. So logstash defines a template called "multi-field" (multi-field) type of field. This type of field is automatically added at the end of a ".raw", and not to set this field to enable word breaker. Simply put, you want to get the url field of an aggregate result, do not directly use "url", but with "url.raw" as the field name.

 

• geo_point

Elasticsearch support  geo_point  type,  GEO Distance  polymerization and the like. For example, you can request a  geo_point  the total number of points within a radius of 10 km of data points. In Kibana bettermap type of panel, we will use this type of data.

 

• order

If you have your own individually customized template idea, very good. At this time there are several options:

1. Open in logstash / outputs / elasticsearch configuration  manage_template => false options, and then everything yourself;
2. Open in logstash / outputs / elasticsearch configuration  template => "/path/to/your/tmpl.json" options, let logstash to send write your own template files;
3. Avoid logstash change in the configuration, but another to send a template, use templates order function of elasticsearch.

The order function is elasticsearch creating an index when the index if it is found that while more than one template matches, it will first apply a small order value template set, and then apply again the high value of the order as a cover, to reach the final one effect of the merge.

For example, it has been very satisfied with the above template, just want to change it  refresh_interval , you only need to write a new:

{
  "order" : 1,
  "template" : "logstash-*",
  "settings" : {
    "index.refresh_interval" : "20s"
  }
}

Then run the following command:

curl -XPUT http://localhost:9200/_template/template_newid -d '@/path/to/your/tmpl.json'