Python + request timeout and retry

First, what is overtime?

1, connection timeout

Connection timeout refers not connected on the inside than the specified time are not connected, this is the connection timeout. (Where time is httpclient connection request is sent to the start time of the target host is connected url addresses)

2, Read Timeout

Read timeout represents the connection, but over the time frame specified when reading data, which is read timeout (read HttpClient time is already connected to the target server, and then acquires content data of the time)

Second, why should set the retry?

Such as connection timeout, the program has been in an unresponsive state. Then we need to continue to retry the connection, but it can not be endless to retry, you need to set a timeout timeout. Within the timeout timeout If you can not connect to the target host url address, it returns an exception error (this connection timeout special url written log logs for administrators to view; the large amount of data read or destination server the problem itself (such as reading the database slow, complicated by a large amount, etc. ...) will also affect the reading time can also be set read timeout returns an error. facilitate business management and fault location)

1-1 Connection Timeout Case (original)

import requests
url = 'http://www.google.com.hk'
r = requests.get(url)
print(r.text)

operation result:

    The raise the ConnectionError (E, Request = Request) 
requests.exceptions.ConnectionError: HTTPConnectionPool (Host = ' www.google.com.hk ' , Port = 80): URL with Max retries exceeded Number: / (Caused by NewConnectionError ( ' <urllib3. Object aT 0x0404D230 connection.HTTPConnection>: the failed to the Establish 
a new new connection: [WinError 10060] Since the connection is unable to correctly respond after a period of time or the connected host does not respond, the connection attempt fails. ' ,))

1-1 can be seen from the case because the connection timed out, reported a ConnectionError connection exception. So in order to prevent the program because the connection timeout anomaly causes the program to stop running. We try ...... except ...... do use an exception handler.

1-2 Case connection timeout (plus requests exception handling)

import requests
import time
url = 'http://www.google.com.hk'
try:
    print(time.strftime('%Y-%m-%d %H:%M:%S'))
    r = requests.get(url)
except requests.exceptions.ConnectionError as e:
    print("连接超时")
print(time.strftime('%Y-%m-%d %H:%M:%S'))

operation result:

2019-09-24 16:54:50 
connection timed out
 2019-09-24 16:55:11

1-2 can be seen from the cases, try to use after the except ...... ...... exception handling, no longer being given. And it may be seen that the default timeout is 21 seconds

1-3 Connection timeout case (modify the default timeout)

import requests
import time
url = 'http://www.google.com.hk'
try:
    print(time.strftime('%Y-%m-%d %H:%M:%S'))
    r = requests.get(url,timeout=5)#timeout修改了超时时间，以秒为单位
except requests.exceptions.ConnectionError as e:
    print("连接超时")
print(time.strftime('%Y-%m-%d %H:%M:%S'))

运行结果：

2019-09-24 17:16:52
连接超时
2019-09-24 17:16:57

由案例1-3得出，设置timeout=5后，默认超时时间被修改成了5秒。所以当碰到一些页面请求时间相应时间过长的情况下，我们可以适当的延长超时时间来达到成功访问页面

1-4连接超时案例（设置多次请求次数）

import requests
import time
from requests.adapters import HTTPAdapter
url = 'http://www.google.com.hk'
s = requests.Session()
s.mount('http://',HTTPAdapter(max_retries=3))#设置重试次数为3次
s.mount('https://',HTTPAdapter(max_retries=3))
try:
    s.get(url,timeout=5)
except requests.exceptions.ConnectionError as e:
    print('连接失败，该URL可能被墙掉了')

运行结果：

连接失败，该URL可能被墙掉了

参考文档：https://www.jianshu.com/p/0a15c253a0d9（设置requests中的重复请求）、https://2.python-requests.org//zh_CN/latest/user/advanced.html#transport-adapters（requests 2.18.1中文文档中适配器和请求超时）

Python + request timeout and retry

Guess you like