Python uses grequests to send requests concurrently

Preface

requests is a very easy-to-use three-party library for Python to send interface requests. It is written by God, simple and easy to use. But requests send requests are serial, that is, blocking. Only after sending one request can another request be sent.

In order to improve test efficiency, we generally need to send requests in parallel. Multi-threading, or coroutine, gevent or aiohttp can be used here, but it is relatively troublesome to use.

Grequests is a library for concurrently sending requests written by God K based on gevent+requests. It is very simple to use.

安装方法： pip install gevent grequests
项目地址：https://github.com/spyoungtech/grequests

grequests is simple to use

First construct a request list, use grequests.map() to send in parallel, and get a response list. Examples are as follows.

import grequests

req_list = [   # 请求列表
    grequests.get('http://httpbin.org/get?a=1&b=2'),
    grequests.post('http://httpbin.org/post', data={
    
    'a':1,'b':2}),
    grequests.put('http://httpbin.org/post', json={
    
    'a': 1, 'b': 2}),
]

res_list = grequests.map(req_list)    # 并行发送，等最后一个运行完后返回
print(res_list[0].text)  # 打印第一个请求的响应文本

grequests supports HTTP request methods supported by requests such as get, post, put, and delete. The parameters used are the same as those of requests, and it is very simple to send requests.
The returned results of all requests can be obtained by traversing res_list.

Performance comparison between grequests and requests

We can compare the time between requests serial and grequests 100 parallel requests to github.com. The example is as follows.
Send requests using requests

import requests
import time

start = time.time()
res_list = [requests.get('https://github.com') for i in range(100)]
print(time.time()-start)

Actually takes about 100s+

Send using grequests

import grequests
import time

start = time.time()
req_list = [grequests.get('https://github.com') for i in range(100)]
res_list = grequests.map(req_list)
print(time.time()-start)

The actual time is about 3.58s

Exception handling

When sending requests in batches, it is inevitable that a certain request URL cannot be accessed or timed out. The grequests.map() method also supports custom exception handling functions. Examples are as follows.

import grequests

def err_handler(request, exception):
    print("请求出错")

req_list = [
    grequests.get('http://httpbin.org/delay/1', timeout=0.001),   # 超时异常
    grequests.get('http://fakedomain/'),   # 该域名不存在
    grequests.get('http://httpbin.org/status/500')    #  正常返回500的请求
]

res_list = grequests.map(reqs, exception_handler=err_handler)
print(res_list)

operation result:

请求出错
请求出错
[None, None, <Response [500]>]

I am an automation tester. The above are some video resources I collected, which helped me a lot in this process. If you don't want to experience the feeling that you can't find the information during self-study, no one answers your questions, and insists on giving up after a few days, you can join our deduction group [313782132], which has various software testing resources and technical discussions.

Software testing is the easiest subject in IT-related industries to get started~ It does not require the logical thinking of developers, and operations and maintenance personnel are not required to be on call 24 hours a day. What is needed is a careful attitude and a broad understanding of IT-related knowledge. The growth path of each tester from entering the industry to becoming a professional expert can be divided into three stages: software testing, automated testing, and test development engineers.

If you don't want to experience the feeling that you can't find the materials during your self-study, no one answers your questions, and insists on giving up for a few days, you can join our software testing exchange group, which contains various software testing materials and technical exchanges.