ElasticSearch CircuitBreakingException相关问题排查

现象

logstash向ES写入数据，出现错误

[2018-10-22T14:23:07,406][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"circuit_breaking_exception", "reason"=>"[parent] Data too large, data for [<transport_request>] would be [24153861037/22.4gb], which is larger than the limit of [22408154316/20.8gb]", "bytes_wanted"=>24153861037, "bytes_limit"=>22408154316})

ES刚开始的事后并没有什么事，一段事件后无法正常工作。
后台日志显示

[2018-10-22T10:00:43,603][DEBUG][o.e.a.a.c.n.i.TransportNodesInfoAction] [node-101] failed to execute on node [FnXP-863QFC0FnrHOBVycQ]
org.elasticsearch.transport.RemoteTransportException: [node-102][10.5.145.102:9300][cluster:monitor/nodes/info[n]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<transport_request>] would be [22842300539/21.2gb], which is larger than the limit of [22408154316/20.8gb]
	at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:230) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1540) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1414) [elasticsearch-6.3.2.jar:6.3.2]

排查

首先注意到了这个bug-fix，不过很可惜，这个bug-fix是2017年时针对5.3.x以下版本的。摔锅失败。
https://github.com/elastic/elasticsearch/pull/23310