Fall in love with the python series-python performance (2): line_profiler performance analysis

Python performance analysis is very important, because it can provide us with a reliable numerical reference for optimizing performance

Python performance analysis has two mainstream tools, cProfile and line_profiler . The former is a built-in tool of Python , but it is not particularly detailed in the analysis report.

Let's try cProfile:

import profile

class cached:
    def __init__(self, fn):
        self.fn = fn
        self.cache = {}
    def __call__(self, *args):
        try:
            return self.cache[args]
        except KeyError:
            self.cache[args] = self.fn(*args)
            return self.cache[args]


@cached
def fib(n):
    if n <= 1:return n
    else:return fib(n-1) + fib(n-2)
def fib_seq(n):
    seq = []
    if n > 0:
        seq.extend(fib_seq(n-1))
    seq.append(fib(n))
    return seq

if __name__ == '__main__':
    #print(fib_seq(20))
    profile.run('print(fib_seq(200)); print')

The results are as follows:

        11521 function calls (8023 primitive calls) in 0.009 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   2505/5    0.005    0.000    0.008    0.002 78797.py:19(fib_seq)
     2500    0.001    0.000    0.001    0.000 {method 'extend' of 'list' objects}
3503/2505    0.001    0.000    0.001    0.000 78797.py:7(__call__)
        5    0.001    0.000    0.001    0.000 {built-in method builtins.print}
      501    0.000    0.000    0.001    0.000 78797.py:15(fib)
     2505    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 cProfile.py:50(create_stats)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}


   Ordered by: cumulative time

Function                                          was called by...
                                                      ncalls  tottime  cumtime
78797.py:19(fib_seq)                              <-  2500/5    0.005    0.008  78797.py:19(fib_seq)
{method 'extend' of 'list' objects}               <-    2500    0.001    0.001  78797.py:19(fib_seq)
78797.py:7(__call__)                              <-     998    0.000    0.000  78797.py:15(fib)
                                                        2505    0.001    0.001  78797.py:19(fib_seq)
{built-in method builtins.print}                  <- 
78797.py:15(fib)                                  <-     501    0.000    0.001  78797.py:7(__call__)
{method 'append' of 'list' objects}               <-    2505    0.000    0.000  78797.py:19(fib_seq)
cProfile.py:50(create_stats)                      <- 
{method 'disable' of '_lsprof.Profiler' objects}  <-       1    0.000    0.000  cProfile.py:50(create_stats)

Note: Only the content of the report is written, the content of the print is not posted

We can see the summary of the function call from the result

But there is no way like line_profiler can give analysis report by line

Before using line_profiler, you need to install:

pip3 install line_profiler

It can be used after the installation is successful. Before using it, it is worth mentioning that kernprof is the line_profiler tool, which has been installed after pip3 install line_profiler. If you don’t believe it, we can try it:

root@root:/opt/pypy3.7-v7.3.2-linux64# kernprof
Usage: kernprof [-s setupfile] [-o output_file_path] scriptfile [arg] ...

Next, use kernprof for analysis. Before that, you need to create a python file named tt.py with the content such as:

@profile 
def run(): 
    a = [1]*100 
    b = [x**3 for x in a ] 
    c = [x for x in b] 
    d = c*2 
                     
run()

 Next is the time to witness the miracle:

kernprof -l -v tt.py

result:

Timer unit: 1e-06 s

Total time: 4e-05 s
File: tt.py
Function: run at line 1

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           @profile 
     2                                           def run(): 
     3         1          3.0      3.0      7.5      a = [1]*100 
     4         1         28.0     28.0     70.0      b = [x**3 for x in a ] 
     5         1          8.0      8.0     20.0      c = [x for x in b] 
     6         1          1.0      1.0      2.5      d = c*2 

At the same time, a file named .lprof will be generated under the current path

 This file can be understood as a persistent file of the report just run, which can be read out:

root@root:/opt/pypy3.7-v7.3.2-linux64# python3 -m line_profiler tt.py.lprof
Timer unit: 1e-06 s

Total time: 4e-05 s
File: tt.py
Function: run at line 1

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           @profile 
     2                                           def run(): 
     3         1          3.0      3.0      7.5      a = [1]*100 
     4         1         28.0     28.0     70.0      b = [x**3 for x in a ] 
     5         1          8.0      8.0     20.0      c = [x for x in b] 
     6         1          1.0      1.0      2.5      d = c*2 

We can see that it is the same as when running, we found that the decorator @ can perform performance analysis, is it very convenient?