Project features : netty-based data transmission services, the client receives the data, there is a local disk; regular tasks, timer processing local file and upload it to the cloud OSS Ali
online problem : over the past half-hour operation and maintenance monitoring found no written record into the DB (because each of us will remember once uploaded to the OSS record DB)
troubleshooting process :
1, operation and maintenance to copy out the dump file real-time process
2, analysis of the dump file (mat Description: Use the mat tool https://www.cnblogs.com/trust-freedom/p/6744948.html )
dump file analysis:
3, memory for up to analysis, most of the objects to create the object ......... analysis for a long time, and did not locate the problem, styled: Because we suspect that the default project performance problem, from gc, cpu and so on to the investigation, in fact, depending on the size of the dump file only 1.5G, and configure the line is 4G, the performance of the project description is not a problem, due to lack of experience, I hope to have the thread stack information in real time, but the project has been restarted operation and maintenance Can not get to the thread stack information, troubleshooting trouble. Later doubt whether it is outside the heap memory problems, because we opened the DisableExplictGC this parameter, this parameter is disabled to show the call System.gc (), because only through System.gc heap objects outside the recovery, the disabled, then may result in a heap foreign object can not be recovered, and eventually found this not so much because the monitor is not found outside the heap memory is over, there is no log OOM exception
4, and later a colleague said, dump files, and now will be the default thread stack information printed out, with Mat tools can directly see, shame ah, lack of experience, the tool will not be used, as the button to point out :
5, analysis and processing operations OSS-related thread, I found eight threads are part of, to find the cause of the problem, then why part of the
6、找到问题，就好解决了，有大神已经遇到过这个问题：https://github.com/aliyun/aliyun-oss-java-sdk/issues/10
从堆栈上是httpclient从连接池中获取连接卡住：
可能有以下原因：
1. 连接泄漏，使用了getObject返回的输入流，使用后一定要close；
2. 使用了getSimplifiedObjectMeta，2.2.3版本前有bug，没有关闭连接；
3. 使用了2.1.2前的sdk版本， 2.1.2前的sdk使用http client 4.4，http client 4.4有bug，请升级sdk到2.1.2以上。
如果还没有解决，请旺旺联系oss_support。
正好我们触发了第一条：“使用了getObject返回的输入流，使用后一定要close”，我们未关闭，导致http连接一直没有释放，前前后后排查问题耗时良久，主要还是经验不足，本次记录作为一次积累吧。
