2019-05-29 ES stored data amount Optimization

386 standard csv file, 9, 912M, 6839927 row,

If the index is the default configuration enter es, namely five shards, 1 Ge replica, then the space of 3.2G.

Then, by the following manner, provided 0 replica, 1 th shard

PUT _template/<template_name>
{
  "index_patterns": "*",
  "settings" : {
    "number_of_replicas" : 0,
    "number_of_shards" : 1
  },
  "version"  : 1
}

Then, in there analytic grok, it is discarding the original message (if not full-text index words). Then the final space is 1.6G. On kibana GET _cat/indices?vshow is 1gb
can be found in the import process, this space will grow to 2.3G or more. Then fall, rise, then fall.
Here is a data compression process, introduced carefully next time.

About 8 minutes and 30 seconds, to enter this data 684w es. The average speed is 13411 TPS.
Using a virtual machine, 16G memory, 600G SATA, 4-core, Intel (R) Xeon (R ) CPU E5-2650 v4 @ 2.20GHz. ES memory is open to

-Xms1g
-Xmx1g

Once a child is eight by three physical machine cluster, SATA disk, then data from kafka, up to 25w TPS.