知识点-协程gvent爬虫

采用协程,挡在io阻塞时候自动切换会加快程序缩短时间也利用了等待时间
本案例注意gevent.monkey.patch_all() 放在requests库引用前,避免出现ssl等错误

在这里插入图片描述

import gevent
import gevent.monkey

gevent.monkey.patch_all()
import requests



def download(url):
    print("start",url)
    data=requests.get(url).text
    print("lengch",len(data))

gevent.joinall([
    gevent.spawn(download,"http://www.baidu.com"),
gevent.spawn(download,"http://www.qq.com"),
gevent.spawn(download,"http://www.163.com"),
gevent.spawn(download,"http://www.sina.com"),
])

猜你喜欢

转载自blog.csdn.net/weixin_42357472/article/details/84582104
今日推荐