利用协程爬网页,自动切换io 精典案例:

首先Python提供的协程库gevent好像并不能知道那些程序使用了io  所以要加一个补丁,mondey,以下同步和异步各爬一次的案例

,

from urllib import request
import gevent,time
from gevent import monkey
monkey.patch_all() #把当前程序的所有的io操作给我单独的做上标记

def f(url):
print('GET: %s' % url)
resp = request.urlopen(url)
data = resp.read()
print('%d bytes received from %s.' % (len(data), url))

urls = ['https://www.python.org/',
'https://www.yahoo.com/',
'https://github.com/' ]
time_start = time.time()
for url in urls:
f(url)
print("同步cost",time.time() - time_start)
async_time_start = time.time()
gevent.joinall([
gevent.spawn(f, 'https://www.python.org/'),
gevent.spawn(f, 'https://www.yahoo.com/'),
gevent.spawn(f, 'https://github.com/'),
])
print("异步cost",time.time() - async_time_start)

猜你喜欢

转载自www.cnblogs.com/fgxwan/p/9649793.html