Python Interview【10】-Special Topic on Python Memory Management Mechanism

Python’s memory management mechanism and tuning methods

Python's memory management is automatic and mainly consists of a garbage collector and a memory allocator.
Python (especially CPython implementation) mainly uses the following memory management mechanisms, including reference counting, garbage collection and memory pool technologies.

1) Reference counting

Reference counting is a very efficient memory management method. When a Python object is referenced, its reference count is increased by 1, and when it is no longer referenced by a variable, the count is decremented by 1. The object is recycled when the reference count equals 0.

2) Garbage collection

(1) Reference counting

Reference counting is also a garbage collection mechanism, and it is also the most intuitive and simple garbage collection technology. When the reference count of an object in Python drops to 0, it means that there are no references pointing to the object, and the object becomes garbage to be recycled. For example, if a newly created object is assigned to a reference, the reference count of the object becomes 1. If the reference is deleted and the object's reference count reaches 0, the object can be garbage collected. However, if a circular reference occurs, the reference counting mechanism will no longer play an effective role.

(2) Mark clearing

If the reference counts of two objects are both 1, but there is only a circular reference between them, then both objects need to be recycled. That is to say, although their reference counts appear to be non-0, they are actually The effective reference count is 0. So first remove the circular reference, and you will get the effective count of these two objects.

(3) Generational recycling

Because every time the garbage collection mechanism recycles memory, it needs to traverse all object reference counts, which is very time-consuming. Therefore, after multiple scans, there are no variables that have been recycled, and the garbage collection mechanism will By dividing them by level, the garbage collection mechanism will think that the variable is a commonly used variable, and its scanning frequency will be reduced. This makes the garbage collection mechanism need to process less memory, and the efficiency is naturally improved.

3) Memory pool

When creating a large number of objects that consume small memory, frequent calls to new/malloc will cause a large amount of memory fragmentation, resulting in reduced efficiency. The function of the memory pool is to apply for a certain number of equal-sized memory blocks in the memory in advance as backup. When there is a new memory demand, it will be allocated to this demand from the memory pool. If it is not enough, new memory will be applied for. The most significant advantage of doing this is that it can reduce memory fragmentation and improve efficiency.
(1) Python’s memory pool is a cache area used to manage the memory allocation of objects smaller than 256 bytes, such as integer values, strings, tuples and other small memory objects. .
(2) When the program needs to create these small objects, python will allocate a memory space from the memory pool, divide it into blocks of multiple sizes, and save them in the memory pool< /span>
(3) When the program needs to destroy these objects, Python will mark them as unused and will not release the memory immediately. If this is the case, frequent application and release of memory space will make Python The execution efficiency is greatly reduced. The memory requested from the memory pool will be returned to the memory pool when it is used up, avoiding frequent memory allocation and release operations.

Regarding the running of python programs, are there any means to improve performance?

1. Use multiple processes to take full advantage of your machine's multi-core performance.
2. For some codes that have a greater impact on performance, you can use C or C++ code
3. Try to use python’s built-in functions
4. Use local variables whenever possible
5. For performance performance caused by IO blocking, IO multiplexing can be used to solve it.

The principle of garbage collection (Garbage Collection, GC) mechanism

Python's garbage collection mechanism uses reference counting as the main strategy, supplemented by mark-sweep and generational collection mechanisms. Among them, the mark-clear mechanism is used to solve the problem of circular references caused by counting references that cannot release memory. The generational recycling mechanism is to improve the efficiency of recycling.

• What is a memory leak?
  A memory leak occurs when an application allocates a certain segment of memory and loses control of the segment of memory due to design errors, resulting in a waste of memory. Serious consequences such as program slowdown or even system crash.
  

An object contains address id, type type, and stored value; each object contains two header information, one is the type identifier (that is, the type of the object), and the other is the counter. The counter reflects the number of times the object has been referenced. Once the counter reaches 0, it means that no variables refer to the object, and the memory space of the object will be automatically reclaimed. One advantage of this is that you can use objects arbitrarily when programming in python without considering freeing memory space. Once the object has no variable references, it will be automatically cleaned and released.