Memray is an open source memory profiler developed by Bloomberg; it has been open source for more than a month, and has already harvested over 8.4k stars. It is a veritable star project. Today we will recommend this python memory analysis artifact to everyone.
Memray can track memory allocation in python code, native extension modules and the python interpreter itself, and can generate many different types of reports to help you analyze the memory usage of python code.
-
The main features of the tool: Track the call of each function, can accurately track the call stack can track the call of c/c++ library, analyze the speed quickly, collect memory data, output various icons, use python thread to work with local thread
-
Problems it can help solve: Analyze memory allocations in your application to find the cause of high memory usage Find the cause of memory leaks Find code hotspots that cause large memory allocations
memray installation
-
Environmental requirements: python3.7+ or above, linux system (only linux system is supported)
-
Installation: pip3 install memray
memray use
memray usage help
python3 -m memray --help
| parameter |
effect |
| run |
Run the specified application and track memory usage |
| flamegraph |
In the html report, use the flame graph to display the memory usage |
| table |
In the html report file, the memory analysis is displayed in the form of a table |
| live |
Use real-time screen display to display various memory usage |
| tree |
In the terminal, display memory usage in a tree structure |
| parse |
Use debug mode to display the memory usage of each line |
| summary |
Summarizes the memory usage profile during a terminal run |
| stats |
Very detailed display of memory usage in the terminal |
The run command uses
-
python3 -m memray run --help to get help
| parameter |
effect |
| -o OUTPU,--output OUTPUT |
Specify where to output the result |
| --live |
Start live trace session mode |
| --live-remote |
Start a live trace session and wait for a client to connect |
| --live-port LIVE_PORT, -p LIVE_PORT |
Port to use when starting live tracing |
| --native |
Track C/C++ stack |
| --follow-fork |
Track allocations in child processes forked by script |
| --trace-python-allocators |
Record the allocation of the pymalloc allocator |
| -q, --quiet |
Does not display any trace-specific output when run |
| -f, --force |
Mandatory repurchase of existing files |
| --compress-on-exit |
Use lz4 to compress the generated file after tracking is complete |
| --no-compress |
Generated files without lz4 compression |
| -c |
program passed in as a string |
| -m |
Run library modules as scripts |
-
python3 -m memray run xxx.py Directly analyze the memory usage of a py file, and a memory usage record file of 'memray-py file name.process id.bin' will be generated in the current path. Of course, you can also follow -o outFiPath to specify the output path. If the running py file is a module code, you can also use -m xxx.py to run it.
The 'memray-py file name.process id.bin' file can be converted into an html-flame graph report by python3 -m memray flamegraph memray-py file name.process id.bin
As shown in the figure above, from top to bottom, it shows the calling process of the program, and the width represents how much memory the function occupies.
-
python3 -m memray run --native xxxx.py will track and analyze the memory consumed by calling the underlying C/C++ functions in the python code
-
python3 -m memray run --trace-python-allocators xxx.py trace and analyze the situation of python program memory allocator pymalloc
This looks like the effect of not adding parameters, but in fact it is completely different. In this way, memory allocation will be tracked in depth. There are four common memory allocators in python (malloc, free, realloc, pymalloc). This parameter is very useful when python has memory overflow. However, with this parameter added, the shipping speed will be slower, and the files generated from the collected data will be larger.
-
python3 -m memray run --live xxx.py displays traced memory data in live screen mode.
By default, according to the data of Total memory, it is sorted from large to small; press "O", you can sort and display memory objects according to private memory from large to small; press "A", according to the number of memory allocations, from high to low Sort.
With this statistical data, you can quickly locate which objects occupy a large amount of memory, and which objects are frequently allocated memory. These objects are the key analysis objects.
flamegraph command --- generate flame graph report
-
python3 -m memray flamegraph --help Get help
-
python3 -m memray flamegraph xxx.bin Generate flame graph
table command -- generate a table report
-
python3 -m memray table --help to get help
-
python3 -m memray table xxxx.bin converts the bin file into a table report
tree command -- generate a tree report
-
python3 -m memray tree --help to get help
-
python3 -m memray tree xxxx.bin converts the bin file into a tree report
summary command -- generate a summary report
-
python3 -m memray summary --help Get help
-
python3 -m memray summary xxxx.bin Analyze the bin file and generate a summary report
stats command --- generate detailed statistics report
-
python3 -m memray stats --help for help
-
python3 -m memray stats xxxx.bin Analyze the bin file and generate a detailed report
