Getting Started with Python Reptile [8]: hummingbird network picture crawling ter

Hummingbird net pictures - long-winded two

Course relatively large amount of previous write a relatively simple today, crawling or hummingbird, still uses aiohttphope you enjoy
crawling pages `https://tu.fengniao.com front or process-based learning purposes, why choose hummingbird, no way, I'm blind election.

After the meal familiar with the operation, I found the link below
https://tu.fengniao.com/ajax/ajaxTuPicList.php?page=2&tagsId=15&action=getPicLists

This link returns data in JSON format

1. page = 2 page number, then the cycle starts from a like
2. tags = 15 name tags, 15 children, 13 is beautiful, is 6391 Private photos, can only help you to it, after all, this is my 专业博客ヾ(◍ ° ∇ ° ◍) Techno゙
3. action = getPicLists interface address, the same place

With the data, open bar crawl

import aiohttp
import asyncio

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36",
           "X-Requested-With": "XMLHttpRequest",
           "Accept": "*/*"}

async def get_source(url):
    print("正在操作:{}".format(url))
    conn = aiohttp.TCPConnector(verify_ssl=False)  # 防止ssl报错,其中一种写法
    async with aiohttp.ClientSession(connector=conn) as session:  # 创建session
        async with session.get(url, headers=headers, timeout=10) as response:  # 获得网络请求
            if response.status == 200:  # 判断返回的请求码
                source = await response.text()  # 使用await关键字获取返回结果
                print(source)
            else:
                print("网页访问失败")

if __name__=="__main__":
        url_format = "https://tu.fengniao.com/ajax/ajaxTuPicList.php?page={}&tagsId=15&action=getPicLists"
        full_urllist= [url_format.format(i) for i in range(1,21)]
        event_loop = asyncio.get_event_loop()   #创建事件循环
        tasks = [get_source(url) for url in full_urllist]
        results = event_loop.run_until_complete(asyncio.wait(tasks))   #等待任务结束
Python资源分享qun 784758214 ,内有安装包，PDF，学习视频，这里是Python学习者的聚集地，零基础，进阶，都欢迎

The code found in the implementation process, Shun made 20 requests like this can easily be judged others as reptiles, may be closed or IP account, we need to conduct some concurrency control.
Make Semaphoreconcurrency control simultaneous

import aiohttp
import asyncio
# 代码在上面
sema = asyncio.Semaphore(3)
async def get_source(url):
    # 代码在上面
    #######################
# 为避免爬虫一次性请求次数太多，控制一下
async def x_get_source(url):
    with(await sema):
        await get_source(url)

if __name__=="__main__":
        url_format = "https://tu.fengniao.com/ajax/ajaxTuPicList.php?page={}&tagsId=15&action=getPicLists"
        full_urllist= [url_format.format(i) for i in range(1,21)]
        event_loop = asyncio.get_event_loop()   #创建事件循环
        tasks = [x_get_source(url) for url in full_urllist]
        results = event_loop.run_until_complete(asyncio.wait(tasks))   #等待任务结束

Go wave of the code, the following results appear to be all right!

In the supplementary code to download the picture

import aiohttp
import asyncio

import json

## 蜂鸟网图片--代码去上面找
async def get_source(url):
    print("正在操作:{}".format(url))
    conn = aiohttp.TCPConnector(verify_ssl=False)  # 防止ssl报错,其中一种写法
    async with aiohttp.ClientSession(connector=conn) as session:  # 创建session
        async with session.get(url, headers=headers, timeout=10) as response:  # 获得网络请求
            if response.status == 200:  # 判断返回的请求码
                source = await response.text()  # 使用await关键字获取返回结果
                ############################################################
                data = json.loads(source)
                photos = data["photos"]["photo"]
                for p in photos:
                    img = p["src"].split('?')[0]
                    try:
                        async with session.get(img, headers=headers) as img_res:
                            imgcode = await img_res.read()
                            with open("photos/{}".format(img.split('/')[-1]), 'wb') as f:
                                f.write(imgcode)
                                f.close()
                    except Exception as e:
                        print(e)
                ############################################################
            else:
                print("网页访问失败")

# 为避免爬虫一次性请求次数太多，控制一下
async def x_get_source(url):
    with(await sema):
        await get_source(url)

if __name__=="__main__":
        #### 代码去上面找
Python资源分享qun 784758214 ,内有安装包，PDF，学习视频，这里是Python学习者的聚集地，零基础，进阶，都欢迎

Pictures download is successful, a small reptile, we finished, flattered