ELK problem

Basic information
1. elasticsearch-5.2.2
2. 600G per day with more than 30T
storage 3. Storage node memory is 128G, and 32G is allocated for ES
4. Configuration details
cluster.name: "prod-elasticsearch-cluster"

node.name: " node1"
node.master: false
node.data: true
node.ingest: true # Whether it is a preprocessing node

#index.number_of_shards: 5
#index.number_of_replicas: 1
path.data: /opt/C3982/data
path.logs: / opt/C3982/elasticsearch-5.2.2/logs

bootstrap.memory_lock: true
bootstrap.system_call_filter: false

network.host: 10.17.4.247
transport.tcp.port: 19300
http.port: 19200

discovery.zen.minimum_master_nodes: 2
discovery.zen .ping_timeout: 30s
#discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.17.4.248:19300","10.17.4.249:19300","10.17.4.140:19300","10.17.4.141:19300","10.20.3.20:19300","10.20.3.21:19300","10.20.3.22:19300","10.20.3.23:19300","10.20.3.24:19300","10.20.3.25:19300"]
#discovery.zen.fd.connect_on_network_disconnect : true 
#discovery.zen.initial_ping_timeout : 10s 
#discovery.zen.fd.ping_interval : 2s 
#discovery.zen.fd.ping_retries  : 10 

cluster.routing.allocation.node_initial_primaries_recoveries: 8
cluster.routing.allocation.node_concurrent_recoveries: 4
cluster.routing.allocation.disk.threshold_enabled: true
indices.recovery.max_bytes_per_sec: 0

#index.cache.field.type: soft
#index.cache.field.max_size: 50000
#index.cache.field.expire: 10m

#cluster.routing.allocation.disk.watermark.low: 100GB
#cluster.routing.allocation.disk.watermark.high: 60GB

5.
Problems:
1. For various reasons, the old storage nodes in ES will exit the cluster, and then the whole storage node will not work.
Error log:
[2017-08-02T11:55:00,425][WARN ][oemjJvmGcMonitorService] [node1 ] [gc][old][17753][388] duration [11.4s], collections [1]/[11.5s], total [11.4s]/[3.5m], memory [31.8gb]->[31.8gb ]/[31.8gb], all_pools {[young] [1.1gb]->[1.1gb]/[1.1gb]}{[survivor] [136.5mb]->[135.6mb]/[149.7mb]}{[ old] [30.5gb]->[30.5gb]/[30.5gb]}