1 Introduction
When we write programs, especially programs related to network requests, such as calling web interfaces, running web crawlers and other tasks, we often encounter some occasional request failures. In this case, if we simply catch the error and jump It is definitely not rigorous to pass the corresponding task, especially in the web crawler, there will be the risk of losing valuable data.
In such cases, it is necessary for us to add some "error retry" strategies to our program logic. Mr. Fei, I wrote an article a few years ago to introduce Pythonthe retrylibrary, but its function is relatively simple, and it can only deal with basic functions. demand.
And the library I want to introduce to you today tenacitymay be the best error retry library in the current Pythonecosystem. Let's take a look at its main functions~
2 Common functions in tenacity
As a third-party Pythonlibrary, we can use pip install tenacityit to install it. After the installation is complete, let's learn tenacitythe main usage methods and features:
2.1 Basic use of tenacity
tenacityThe core function of error retry retryis implemented by its decorator. When no retryparameters are passed to the decorator by default, it will keep retrying when an error is thrown during the operation of the decorated function, such as the following simple example:
import random
from tenacity import retry
@retry
def demo_func1():
a = random.random()
print(a)
if a >= 0.1:
raise Exception
demo_func1()
It can be seen that the random number between 0 and 1 is generated each time in our function body. When the random number does not exceed 0.1, it will stop throwing errors, otherwise it will tenacitybe caught every time the error throwing behavior is thrown and immediately Retry.
2.2 Set the maximum number of retries
Sometimes our tolerance for retrying a certain function logic error is limited. For example, when we call a certain network interface, if the execution fails for n consecutive times, we may think that the task itself is defective, not through Try again and it will be normal someday.
At this point we can take advantage tenacityof the stop_after_attemptfunction in , passed in as retry()a stopparameter in , to add an endpoint to our "endless" error retry process, which stop_after_attempt()accepts an integer input as the "maximum number of retries" :
from tenacity import retry, stop_after_attempt
@retry(stop=stop_after_attempt(3))
def demo_func2():
print('函数执行')
raise Exception
demo_func2()
It can be seen that after limiting the maximum number of retries, our function has retried 3 times, and after the 4th execution still throws an error, the corresponding Exceptionerror in the function is officially thrown to end the retry process.
2.3 Set the maximum retry timeout period
In addition to setting the maximum number of error retries as in the previous section, tenacitywe also provide us with a stop_after_delay()function to set the maximum time spent on the entire retry process, beyond which the retry process will end:
import time
from tenacity import retry, stop_after_delay
# 设置重试最大超时时长为5秒
@retry(stop=stop_after_delay(5))
def demo_func3():
time.sleep(1)
print(f'已过去 {time.time() - start_time} 秒')
raise Exception
# 记录开始时间
start_time = time.time()
demo_func3()
2.4 Combined Retry Stop Conditions
If our task needs to add the maximum number of retries and the maximum timeout duration at the same time, tenacitywe only need to use |operators to combine different constraints and then pass retry()in the stopparameters. For example, in the following example, when our function executes retry more than You can end the retry after 3 seconds or more than 5 times:
import time
import random
from tenacity import retry, stop_after_delay, stop_after_attempt
@retry(stop=(stop_after_delay(3) | stop_after_attempt(5)))
def demo_func4():
time.sleep(random.random())
print(f'已过去 {time.time() - start_time} 秒')
raise Exception
# 记录开始时间
start_time = time.time()
demo_func4()
It can be seen that in the above demo, the limit of "maximum 5 retries" is reached first, thus ending the retry process.
2.5 Set the time interval between adjacent retries
In some cases, we do not want to start the next retry immediately after each retry throws an error. For example, in order to better disguise our program in the crawler task, tenacitya series of very practical functions are provided in the matching retry()parameters wait. , to help us properly deal with the time interval between adjacent retries, among which the more practical are mainly the following two ways:
2.5.1 Setting fixed time interval
We can set a fixed number of seconds to wait between adjacent retries by using , as tenacityin the following simple example:wait_fixed()
import time
from tenacity import retry, wait_fixed, stop_after_attempt
# 设置重试等待间隔为1秒
@retry(wait=wait_fixed(1), stop=stop_after_attempt(3))
def demo_func5():
print(f'已过去 {time.time() - start_time} 秒')
raise Exception
# 记录开始时间
start_time = time.time()
demo_func5()
2.5.2 Set random time interval
In addition to setting a fixed time interval, tenacityit can also wait_random()help us set a uniformly distributed random number for adjacent retries, just set the uniformly distributed range:
import time
from tenacity import retry, wait_random, stop_after_attempt
# 设置重试等待间隔为1到3之间的随机数
@retry(wait=wait_random(min=1, max=3), stop=stop_after_attempt(5))
def demo_func6():
print(f'已过去 {time.time() - start_time} 秒')
raise Exception
# 记录开始时间
start_time = time.time()
demo_func6()
It can be observed that the waiting time after each retry is random~
2.6 Customize whether to trigger a retry
tenacityThe retry()default strategy in is to retry when the execution of the function it decorates "throws any error", but in some cases we may need to catch/ignore specific types of errors, or calculate the results of exceptions capture.
tenacityAlso built-in related useful functions:
2.6.1 Catch or ignore specific error types
Using tenacitythe in- retry_if_exception_type()and - retry_if_not_exception_type()match parameters, we can catch or ignore specific error types:retry()retry
from tenacity import retry, retry_if_exception_type, retry_if_not_exception_type
@retry(retry=retry_if_exception_type(FileExistsError))
def demo_func7():
raise TimeoutError
@retry(retry=retry_if_not_exception_type(FileNotFoundError))
def demo_func8():
raise FileNotFoundError
2.6.2 User-defined function result condition judgment function
We can write additional conditional judgment functions. In cooperation tenacity, we can retry_if_result()implement custom conditional judgment on the return result of the function, and the Trueretry operation will only be triggered when it returns:
import random
from tenacity import retry, retry_if_result
@retry(retry=retry_if_result(lambda x: x >= 0.1))
def demo_func9():
a = random.random()
print(a)
return a
# 记录开始时间
demo_func9()
2.7 Count the error retries of the function
tenacityFor the decorated retry()function, we can print its retry.statisticsproperties to view the statistics and record results of its past errors and retry. For example, here we demo_func9()print the statistics of the previously executed example functions:
demo_func9.retry.statistics
In addition to the above functions, it tenacityalso has many special features, which can be combined with loggingother functions such as modules, asynchronous functions, and coroutines Pythonto achieve more advanced functions. Interested friends can go to https://github.com/jd/tenacitylearn more.
Python learning
Python learning partners, welcome to join the new exchange [Jun Yang]: 1020465983
to discuss programming knowledge together, become a god, there are software installation packages, practical cases, learning materials in the group