Problem Description
A service is a service to detect whether changes MGR master node cluster, using python language.
For each cluster, the main thread creates a child thread by thread to detect child. Frequently the child thread creation and destruction.
After on-line, as there is often a feature release, thus restart the service, beginning a period of time and found no problems.
After two weeks before the service Tuesday released about a week, no longer published. On weekends, when suddenly the alarm system load is high, after investigation, it found that memory is almost exhausted, and found A service is occupied by a huge memory is not freed.
Investigation process
It has been determined, A service is a memory leak, in the end is where memory used up, but did not release it?
This is a very troublesome problem, I do not ever meet the Python memory leak.
First, the issue of online search python memory leak. Generally learned, Python's garbage collection is based on reference counting, that is, if an object is used once, the reference count is incremented by 1. When the object's reference count is 0, the memory will be recovered off.
Common situation leads to memory leaks in two ways:
- (1) the object has been using global variables, global variables relatively long life cycle, so the memory has not released.
- (2) circular reference object defines how the __del__.
Internet provides a variety of tools for troubleshooting memory leaks, such as the link after objgraph, guppy, pympler, with its specific use reference text.
Read a long time to use these tools, feel or should look at their code is not finished using the object exists, but the case has been cited.
First, the position troubleshoot memory leaks is the main thread or sub-thread. By looking and found "frequent creation and exit the sub-thread" in both cases, "child thread has been implementing" with no difference in memory consumption. In this way, you can navigate to a memory leak location is in the main thread.
Then, view the main thread code, found each time you create a child thread, the thread will be sub-manager object, into a dictionary, but when the child thread exits, but not the child thread manager object from the dictionary deleted. Due to frequently create and destroy child thread, so the dictionary will become increasingly large, resulting in a memory leak.
Solution
To find the cause of the problem, then the solution easier to handle. When the child thread exits, timely child thread manager object is removed from the dictionary, release the memory occupied.
reference
A python debug memory leak problems
using gc, objgraph kill python memory leak with circular references!
Python memory optimization: Profile, slots, compact dict