Notes｜Python performance development｜cProfile usage skills

cProfile provides stability ; that is, by monitoring all function calls, function returns, and exception events, and precisely timing the intervals between these events. Implements statistics on how often and when various parts of the program execute. cPorfile's profiling can often provide extensive runtime statistics about a Python program's execution with little added processing overhead.

Documentation of cProfile: https://docs.python.org/zh-cn/3/library/profile.html

Simple usage example

When we need to analyze the running performance of a Python script, we can simply wrap the script entry function, and then use cProfile.run()the method to execute this function. For example:

import cProfile

def main():
    """封装的脚本入口函数"""

if __name__ == "__main__":
   cProfile.run("main()")

runMethod documentation of cProfile : https://docs.python.org/zh-cn/3/library/profile.html?#profile.run

To add a global namespace and a local namespace call runmethod , you can use the methodcProfile of .runctx

runctxMethod documentation of cProfile : https://docs.python.org/zh-cn/3/library/profile.html?#profile.runctx

After the above script finishes running, it will print similar information on the console:

         4140186 function calls (4071373 primitive calls) in 10.881 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      661    0.001    0.000    0.011    0.000 <__array_function__ internals>:177(all)
     8564    0.007    0.000    0.062    0.000 <__array_function__ internals>:177(amax)
     8562    0.006    0.000    0.065    0.000 <__array_function__ internals>:177(amin)
      656    0.001    0.000    0.011    0.000 <__array_function__ internals>:177(any)
......

in:

column number	column name	meaning
1	ncalls	Number of calls
2	tottime	The total time spent in the specified function (excluding the time spent calling sub-functions)
3	percall	tottimeis ncallsthe quotient divided by
4	cumtime	Cumulative time consumed by the specified function and all its subfunctions (from call to exit). This number is accurate for recursive functions.
5	percall	is the quotient cumtimedivided by the original call (number of times) (ie: the average time for the function to run once)
6	filename:lineno(function)	Each function that provides the corresponding data

If you need to sort the above results, you can runuse sortthe parameter of the method, which needs to pass in a string or an SortKeyenumeration class as an actual parameter. Sorting parameters include:

valid string parameters	Valid enumeration parameters	meaning
"calls"	SortKey.CALLS	Number of calls
"cumulative"	SortKey.CUMULATIVE	Cumulative time
"cumtime"	-	Cumulative time
"file"	-	file name
"filename"	SortKey.FILENAME	file name
"module"	-	file name
"ncalls"	-	Number of calls
"pcalls"	SortKey.PCALLS	raw call count
"line"	SortKey.LINE	line number
"name"	SortKey.NAME	function name
"nfl"	SortKey.NFL	name/file/line
"stdname"	SortKey.STDNAME	standard name
"time"	SortKey.TIME	internal time
"tottime"	-	internal time

Documentation for sorting parameters: https://docs.python.org/zh-cn/3/library/profile.html?#pstats.Stats.sort_stats

Profiling with the Profile class

Documentation for the cProfile.Profile class: https://docs.python.org/en-us/3/library/profile.html?#profile.Profile

If more precise performance analysis is required, we can implement it through the Profile class. This class supports direct formatting of profiling results without writing profiling data to a file. This class has the following common methods:

method name	use
enable()	Start collecting analytics data. (for precise control of the analysis range)
disable()	Stop collecting analytics data. (for precise control of the analysis range)
create_stats()	Stops collecting profiling data and internally records the results as the current profile.
print_stats(sort=-1)	Create a Stats object from the current profiling data and print the result to stdout.
dump_stats(filename)	Write the current profile results to filename.
run(cmd)	Perform performance analysis on exec()this command.
runctx(cmd, globals, locals)	exec()Profile the command with the specified global and environment variables.
runcall(func, /, *args, **kwargs)	func(*args, **kwargs)Perform performance analysis on .

It should be noted that the instance can be used to construct the class statistics object only after any one of the methods , , , is executed , enable()otherwise an error will be reported:run()runctx()runcall()profileStats

TypeError: Cannot create or construct a <class 'pstats.Stats'> object from <cProfile.Profile object at 0x000001BCE9C332E0>

An example of performance analysis using cProfile.Profile+ is as follows:pstat.Stats

import cProfile
import pstats

def main():
    """封装的脚本入口函数"""
    preprocessing()  # 待分析函数执行前的初始化（不需要分析性能的部分）
    profiler.enable()
    processing()  # 待分析函数（需要被性能分析的部分）
    profiler.disable()
    reprocessing()  # 待分析函数执行后的后处理（不需要分析性能的部分）

if __name__ == "__main__":
    profiler = cProfile.Profile()  # 实例化 Profile 类
    main()  # 调用封装的脚本入口函数
    stats = pstats.Stats(profiler).sort_stats(pstats.SortKey.CUMULATIVE)  # 根据 profile 实例构造 Stats 类，并执行排序
    stats.print_stats()  # 将排序后的汇总统计结果打印到 stdout

In the above example, we can also use run(), runctx(), runcall()and other methods to call the analyzed function.

Analysis with pstat

pstats.StatsClass documentation: https://docs.python.org/zh-cn/3/library/profile.html?#pstats.Stats

StatsClasses are constructed based on filenamefiles or Profileinstances. Output is printed to the stream specified by the stream parameter. This class has the following common methods:

method name	meaning
strip_dirs()	Remove all leading path information from the file
add(*fileanmes)	Add additional profiling information to the current profiling object
dump_stats(filename)	Write information from the current profiler object to a file
sort_stats(*keys)	Sort the records in the current performance analysis object (for parameters, see the sorting table above)
reverse_order()	Reverse the records in the current profiling object
print_callers(*restrictions)	print which functions call restrictionsthe function
print_callees(*restrictions)	Which functions are called in the print restrictionsfunction
get_stats_profile()	returns an StatsProfileinstance

Among them, it should be noted that: through get_stats_profile(), we can get StatsProfilethe instance , and the instance's func_profilesis the performance analysis information in dictionary format. This dictionary can be used if further processing of the analysis results is required; however, no per-call callersinformation .