Article source: Learn Python for rookies
Hello everyone, I am Brother Tao. Today I will share with you 8 cool operations to speed up the running of Python. The full text has a total of 7,000 words and takes about 10 minutes to read.
Python is a scripting language. Compared with compiled languages such as C/C++, it has some shortcomings in efficiency and performance. However, there are many times when Python's efficiency is not as exaggerated as imagined. This article summarizes some tips for speeding up the running of Python code.
0
Code optimization rules
This article will introduce many techniques to speed up the running of Python code. Before going into the details of code optimization, you need to understand some basic principles of code optimization.
The first basic principle: don’t optimize prematurely
Many people start writing code with the goal of performance optimization. "It is much easier to make a correct program faster than to make a fast program correct." Therefore, the prerequisite for optimization is that the code can work properly. Optimizing prematurely may neglect grasping the overall performance indicators. Don't reverse priorities before getting global results.
The second basic principle: weigh the cost of optimization
Optimization comes at a cost, and it is almost impossible to solve all performance problems. The choice usually faced is time for space or space for time. In addition, development costs also need to be considered.
The third principle: don’t optimize the parts that don’t matter
If every part of the code were to be optimized, these changes would make the code difficult to read and understand. If your code is running slowly, first find where the code is slow, usually the inner loop, and focus on optimizing where it is slow. Elsewhere, a little loss of time makes little difference.
1
Avoid global variables
# 不推荐写法。代码耗时：26.8秒
import math
size = 10000
for x in range(size):
for y in range(size):
z = math.sqrt(x) + math.sqrt(y)Many programmers will first write some simple scripts in Python language. When writing scripts, they are usually accustomed to writing them directly as global variables, such as the above code. However, due to the different implementations of global variables and local variables, code defined in the global scope will run much slower than code defined in a function. By putting script statements into functions, you can typically achieve speed improvements of 15% - 30%.
# 推荐写法。代码耗时：20.6秒
import math
def main(): # 定义到函数中，以减少全部变量使用
size = 10000
for x in range(size):
for y in range(size):
z = math.sqrt(x) + math.sqrt(y)
main()2
avoid partial access
2.1
Avoid module and function attribute access
# 不推荐写法。代码耗时：14.5秒
import math
def computeSqrt(size: int):
result = []
for i in range(size):
result.append(math.sqrt(i))
return result
def main():
size = 10000
for _ in range(size):
result = computeSqrt(size)
main()Each use .(attribute access operator) will trigger specific methods, such as __getattribute__()and __getattr__(), these methods will perform dictionary operations, so they will bring additional time overhead. With from importstatements, property access can be eliminated.
# 第一次优化写法。代码耗时：10.9秒
from math import sqrt
def computeSqrt(size: int):
result = []
for i in range(size):
result.append(sqrt(i)) # 避免math.sqrt的使用
return result
def main():
size = 10000
for _ in range(size):
result = computeSqrt(size)
main()In Section 1, we mentioned that local variables are searched faster than global variables, so for frequently accessed variables sqrt, changing them to local variables can speed up the operation.
# 第二次优化写法。代码耗时：9.9秒
import math
def computeSqrt(size: int):
result = []
sqrt = math.sqrt # 赋值给局部变量
for i in range(size):
result.append(sqrt(i)) # 避免math.sqrt的使用
return result
def main():
size = 10000
for _ in range(size):
result = computeSqrt(size)
main()In math.sqrtaddition, there is something else computeSqrtin the function ., that is, the calling listmethod append. By assigning the method to a local variable, the use inside the loop computeSqrtwithin the function can be completely eliminated .for.
# 推荐写法。代码耗时：7.9秒
import math
def computeSqrt(size: int):
result = []
append = result.append
sqrt = math.sqrt # 赋值给局部变量
for i in range(size):
append(sqrt(i)) # 避免 result.append 和 math.sqrt 的使用
return result
def main():
size = 10000
for _ in range(size):
result = computeSqrt(size)
main()2.2
Avoid intra-class property access
# 不推荐写法。代码耗时：10.4秒
import math
from typing import List
class DemoClass:
def __init__(self, value: int):
self._value = value
def computeSqrt(self, size: int) -> List[float]:
result = []
append = result.append
sqrt = math.sqrt
for _ in range(size):
append(sqrt(self._value))
return result
def main():
size = 10000
for _ in range(size):
demo_instance = DemoClass(size)
result = demo_instance.computeSqrt(size)
main()The principle of avoidance .also applies to in-class properties, where access self._valuewill be slower than accessing a local variable. By assigning frequently accessed intra-class properties to a local variable, you can improve code running speed.
# 推荐写法。代码耗时：8.0秒
import math
from typing import List
class DemoClass:
def __init__(self, value: int):
self._value = value
def computeSqrt(self, size: int) -> List[float]:
result = []
append = result.append
sqrt = math.sqrt
value = self._value
for _ in range(size):
append(sqrt(value)) # 避免 self._value 的使用
return result
def main():
size = 10000
for _ in range(size):
demo_instance = DemoClass(size)
demo_instance.computeSqrt(size)
main()3
Avoid unnecessary abstractions
# 不推荐写法，代码耗时：0.55秒
class DemoClass:
def __init__(self, value: int):
self.value = value
@property
def value(self) -> int:
return self._value
@value.setter
def value(self, x: int):
self._value = x
def main():
size = 1000000
for i in range(size):
demo_instance = DemoClass(size)
value = demo_instance.value
demo_instance.value = i
main()Any time you wrap code with additional layers of processing (such as decorators, property access, descriptors), you're going to make the code slower. In most cases, it is necessary to re-examine the definition of using property accessors. Using getter/setterfunctions to access properties is usually a legacy coding style of C/C++ programmers. If it's really not necessary, use simple attributes.
# 推荐写法，代码耗时：0.33秒
class DemoClass:
def __init__(self, value: int):
self.value = value # 避免不必要的属性访问器
def main():
size = 1000000
for i in range(size):
demo_instance = DemoClass(size)
value = demo_instance.value
demo_instance.value = i
main()4
Avoid data duplication
4.1
Avoid meaningless data duplication
# 不推荐写法，代码耗时：6.5秒
def main():
size = 10000
for _ in range(size):
value = range(size)
value_list = [x for x in value]
square_list = [x * x for x in value_list]
main()value_listIt's completely unnecessary in the code above and would create unnecessary data structures or copies.
# 推荐写法，代码耗时：4.8秒
def main():
size = 10000
for _ in range(size):
value = range(size)
square_list = [x * x for x in value] # 避免无意义的复制
main()Another situation is that you are too paranoid about Python's data sharing mechanism, do not understand or trust Python's memory model well, and abuse copy.deepcopy()such functions. Usually the copy operation can be eliminated in these codes.
4.2
No intermediate variables are used when exchanging values
# 不推荐写法，代码耗时：0.07秒
def main():
size = 1000000
for _ in range(size):
a = 3
b = 5
temp = a
a = b
b = temp
main()The above code creates a temporary variable when exchanging values temp. Without the help of intermediate variables, the code is more concise and runs faster.
# 推荐写法，代码耗时：0.06秒
def main():
size = 1000000
for _ in range(size):
a = 3
b = 5
a, b = b, a # 不借助中间变量
main()4.3
Use join instead of + when concatenating strings
# 不推荐写法，代码耗时：2.6秒
import string
from typing import List
def concatString(string_list: List[str]) -> str:
result = ''
for str_i in string_list:
result += str_i
return result
def main():
string_list = list(string.ascii_letters * 100)
for _ in range(10000):
result = concatString(string_list)
main()When a + bconcatenating strings, since the string in Python is an immutable object, it will apply for a memory space, and copy the aand bto the newly allocated memory space respectively. Therefore, if you want to splice nstrings, an intermediate result will be generated n-1. Each intermediate result needs to apply for and copy memory, which seriously affects the operating efficiency. When using join()concatenated strings, the total memory space that needs to be applied for will be calculated first, then the required memory will be applied for all at once, and each string element will be copied to the memory.
# 推荐写法，代码耗时：0.3秒
import string
from typing import List
def concatString(string_list: List[str]) -> str:
return ''.join(string_list) # 使用 join 而不是 +
def main():
string_list = list(string.ascii_letters * 100)
for _ in range(10000):
result = concatString(string_list)
main()5
Taking advantage of the short-circuit characteristics of if conditions
# 不推荐写法，代码耗时：0.05秒
from typing import List
def concatString(string_list: List[str]) -> str:
abbreviations = {'cf.', 'e.g.', 'ex.', 'etc.', 'flg.', 'i.e.', 'Mr.', 'vs.'}
abbr_count = 0
result = ''
for str_i in string_list:
if str_i in abbreviations:
result += str_i
return result
def main():
for _ in range(10000):
string_list = ['Mr.', 'Hat', 'is', 'Chasing', 'the', 'black', 'cat', '.']
result = concatString(string_list)
main()ifThe short-circuit characteristic of the condition means that for if a and bsuch a statement, when aisFalse will be returned directly without calculation b; for if a or bsuch a statement, awhenTrue is will be returned directly without calculation b. Therefore, in order to save running time, for statements, variables with a higher probability of orvalue should be written first and should be postponed.Trueorand
# 推荐写法，代码耗时：0.03秒
from typing import List
def concatString(string_list: List[str]) -> str:
abbreviations = {'cf.', 'e.g.', 'ex.', 'etc.', 'flg.', 'i.e.', 'Mr.', 'vs.'}
abbr_count = 0
result = ''
for str_i in string_list:
if str_i[-1] == '.' and str_i in abbreviations: # 利用 if 条件的短路特性
result += str_i
return result
def main():
for _ in range(10000):
string_list = ['Mr.', 'Hat', 'is', 'Chasing', 'the', 'black', 'cat', '.']
result = concatString(string_list)
main()6
Loop optimization
6.1
Use for loop instead of while loop
# 不推荐写法。代码耗时：6.7秒
def computeSum(size: int) -> int:
sum_ = 0
i = 0
while i < size:
sum_ += i
i += 1
return sum_
def main():
size = 10000
for _ in range(size):
sum_ = computeSum(size)
main()Python's forloops are whilemuch faster than loops.
# 推荐写法。代码耗时：4.3秒
def computeSum(size: int) -> int:
sum_ = 0
for i in range(size): # for 循环代替 while 循环
sum_ += i
return sum_
def main():
size = 10000
for _ in range(size):
sum_ = computeSum(size)
main()6.2
Use implicit for loop instead of explicit for loop
For the above example, you can go one step further and use an implicit forloop instead of an explicit forloop .
# 推荐写法。代码耗时：1.7秒
def computeSum(size: int) -> int:
return sum(range(size)) # 隐式 for 循环代替显式 for 循环
def main():
size = 10000
for _ in range(size):
sum = computeSum(size)
main()6.3
Reduce the calculation of the inner for loop
# 不推荐写法。代码耗时：12.8秒
import math
def main():
size = 10000
sqrt = math.sqrt
for x in range(size):
for y in range(size):
z = sqrt(x) + sqrt(y)
main()The above code sqrt(x)is located in the inner forloop and is recalculated during each training process, which increases time overhead.
# 推荐写法。代码耗时：7.0秒
import math
def main():
size = 10000
sqrt = math.sqrt
for x in range(size):
sqrt_x = sqrt(x) # 减少内层 for 循环的计算
for y in range(size):
z = sqrt_x + sqrt(y)
main()7
Using numba.jit
We follow the example introduced above and use it on this basis numba.jit. numbaPython functions can be JIT compiled into machine code for execution, greatly improving code running speed. numbaSee the home page below for more information about :
http://numba.pydata.org/numba.pydata.org
# 推荐写法。代码耗时：0.62秒
import numba
@numba.jit
def computeSum(size: float) -> int:
sum = 0
for i in range(size):
sum += i
return sum
def main():
size = 10000
for _ in range(size):
sum = computeSum(size)
main()8
Choose the right data structure
Python's built-in data structures such as str, tuple, list, are all implemented in C at the bottom level and are very fast. It is almost impossible setto dictachieve the built-in speed in terms of performance by implementing new data structures by yourself.
listSimilar to C++ std::vector, it is a dynamic array. It will pre-allocate a certain amount of memory space. When the pre-allocated memory space is used up and elements are added to it, it will apply for a larger memory space, then copy all the original elements there, and then destroy the previous memory. space before inserting new elements. The operation is similar when deleting elements. When the used memory space is less than half of the pre-allocated memory space, an additional small memory will be applied for, an element copy will be made, and then the original large memory space will be destroyed. Therefore, if there are frequent addition and deletion operations, and the number of added and deleted elements is large, the efficiency of the list will not be high. At this point, you should consider using it collections.deque. It is a double-ended queue that has the characteristics of both stack and queue, and can perform complex insertion and deletion operations collections.dequeat both ends .O(1)
listThe search operation is also very time-consuming. When you need to listfrequently search for certain elements, or frequently access these elements in order, you can maintain bisectthe listorder of objects and perform a binary search in them to improve the efficiency of the search.
Another common requirement is to find the minimum or maximum value. At this time, you can use heapqthe module to listconvert it into a heap, so that the time complexity of obtaining the minimum value is O(1).
The following web page gives the time complexity of various operations on commonly used Python data structures:
TimeComplexity - Python Wikiwiki.python.org
9
References
https://zhuanlan.zhihu.com/p/143052860
David Beazley & Brian K. Jones. Python Cookbook, Third edition. O'Reilly Media, ISBN: 9781449340377, 2013.
Zhang Ying & Lai Yonghao. Writing high-quality code: 91 suggestions for improving Python programs. Machinery Industry Press, ISBN: 9787111467045, 2014.
干货笔记整理
100个爬虫常见问题.pdf ，太全了！
124个Python案例，完整源代码！
PYTHON 3.10中文版官方文档
耗时三个月整理的《Python之路2.0.pdf》开放下载
最经典的编程教材《Think Python》开源中文版.PDF下载
成就感爆棚