1, the problems found
Prometheus alarm user-center services Old excessive GC, needs investigation
2, problem analysis
user-center deployment of four production services node, the entire heap size to 2g, the size of the new generation is set to 1g. The alarms are reported four nodes, wherein a node view find the node, from about 10:30 to about 11:20, in less than an hour, actually had 5 Full GC, this is extremely unusual.
3, GC log analysis using GCViewer
As can be seen, the entire memory 1g years old, almost all occupied, and after Full GC, did not have much memory recovery, it is clear that a memory leak. Memory leak causes insufficient memory space years old, found old's memory is not enough, the new generation of Full GC will be subject to a certain age, you need to upgrade, upgrade time, but each time Full GC after, how much memory did not recover, and although It did not lead to OOM, but soon next time they need to upgrade to a Full GC.
4, dump heap snapshot file
Use the command jinfo -flag +HeapDumpBeforeFullGC 11752and jinfo -flag +HeapDumpAfterFullGC 11752, before the Full GC heap dump memory snapshots after and Full GC (note: after the dump is complete, be sure to close these two parameters, or Full GC frequent cause frequent dump, will take up a lot of disk space); use the command jinfo -flag -HeapDumpBeforeFullGC 11752and jinfo -flag -HeapDumpAfterFullGC 11752shut down two parameters. In fact, you can not have to add these two parameters and then wait for the emergence of Full GC, because it is a memory leak, memory years old can not be recycled, so directly on the command jmap -dump:format=b,file=before.hprof 11752and
jmap -dump:live,format=b,file=after.hprof 11752to dump heap snapshot files is also possible, plus live means storing only live objects.
5, Eclipse MAT analysis of the snapshot
Obviously you can see the object heap memory is DoctorInvitePictureManagerImploccupied 811.8M, it is that it causes a memory leak. The rest is business students need to troubleshoot code.