抓取某网站妹子图

其他 2019-06-15 06:22:39 阅读次数: 0

第一次做爬虫小项目，代码很粗糙，而且我只抓取了第一页上的所有图片集
其实也可以翻页抓，只是觉得太麻烦，学精了之后再抓吧，再说了抓那么多浪费电脑存储空间，妹子图什么的哪里有实战好，有那个时间看黄图，不如撩妹，谈个女朋友天天实战。
废话到此截止：

爬取网址为：https://www.mzitu.com/tag/youhuo/
我们先打开网站主页：

16825884-2aca502125e19978.png

妹子很多，很好看，我不挑，给我哪个我都要。
按部就班，打开开发页面：


16825884-9e41c4aeda40dde6.png

蓝色部分就是我们要提取的第一个图集的地址，下面的href依次为第二个第三个.......，代码如下：

import requests
from requests.exceptions import RequestException

headers = {'If-None-Match': 'W/"5cc2cd8f-2c58"',
           "Referer": "http://www.mzitu.com/all/",
           'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 UBrowser/6.1.2107.204 SafarMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}
#请求头的这个Referer一定要加，妹子网有反爬，反正粘贴复制就行，多加几个信息无所谓
def get_page(url):
    try:
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            print(response.text)
            return response.text
        return None
    except RequestException:
        print('获取索引页失败')
        return None

def main():
    url = 'https://www.mzitu.com/tag/youhuo/'
    get_page(url)


if __name__ == '__main__':
    main()

很容易我们就获得了输出：


16825884-ab1afc90ecf43e59.png

分析一个得到的这段HTML，红框框显然就是我们要得到的每一个图集的地址。
我们引"靓汤"，开始提取图集地址：

import requests
from requests.exceptions import RequestException
from bs4 import BeautifulSoup

headers = {'If-None-Match': 'W/"5cc2cd8f-2c58"',
           "Referer": "http://www.mzitu.com/all/",
           'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 UBrowser/6.1.2107.204 SafarMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}

def get_page(url):
    try:
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            print(response.text)
            return response.text
        return None
    except RequestException:
        print('获取索引页失败')
        return None

def parse_page(html):
    soup = BeautifulSoup(html, 'lxml')
    items = soup.select('#pins li')
    for link in items:
        href = link.a['href']
        print(href)
    # print(items)


def main():
    url = 'https://www.mzitu.com/tag/youhuo/'
    html = get_page(url)
    parse_page(html)


if __name__ == '__main__':
    main()

BeautifulSoup库很强大，我决定多多学习，靓汤的输出书一个list,必须先先遍历才能输出*，输入如下：

16825884-8a4b129e1cf4f832.png

我们得到了每一个图集的地址，打开一个图集看看：
16825884-172ee74158656676.png

每个图集的详情页里只有一张图，不过下边有一个索引，这时就需要考虑一下详情页中的翻页问题：


16825884-cb7f2deffcd64e91.png

16825884-bd577ac4b3de7906.png

16825884-f4c24030f3b7ff03.png
图集中第三幅图的地址

不同图集存在相同的地址排列方式，我们就可以创建一个循环列表来进行每个图集的同图片的详情页获取：

import requests
from requests.exceptions import RequestException
from bs4 import BeautifulSoup

headers = {'If-None-Match': 'W/"5cc2cd8f-2c58"',
           "Referer": "http://www.mzitu.com/all/",
           'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 UBrowser/6.1.2107.204 SafarMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}

def get_page(url):
    try:
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            print(response.text)
            return response.text
        return None
    except RequestException:
        print('获取索引页失败')
        return None

def parse_page(html):
    soup = BeautifulSoup(html, 'lxml')
    items = soup.select('#pins li')
    for link in items:
        href = link.a['href']
        get_detail_page(href)

    # print(items)

def get_detail_page(href):
    for i in range(1,100):
        detail_url = href + '/' + str(i)
        if requests.get(detail_url, headers=headers).status_code == 200:
            print(detail_url)
        else:
            print('已至末尾页')
            return None

    response = requests.get()

def main():
    url = 'https://www.mzitu.com/tag/youhuo/'
    html = get_page(url)
    parse_page(html)


if __name__ == '__main__':
    main()

我稍微解释一下，我设置的是循环到100，因为有的图片只有46页嘛，我就加了一个网页状态码判断，如果返回200,就是正常网址，不是200我就终止网址获取，这样我们就可以将每个图片集内，所有详情页的网址取出来了：


16825884-cecdcd5ff04449d6.png
16825884-d9a6b515851050f7.png

下一步就是通过网址获得网址对应的html：

import requests
from requests.exceptions import RequestException
from bs4 import BeautifulSoup

headers = {'If-None-Match': 'W/"5cc2cd8f-2c58"',
          "Referer": "http://www.mzitu.com/all/",
          'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 UBrowser/6.1.2107.204 SafarMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}

def get_page(url):
   try:
       response = requests.get(url, headers=headers)
       if response.status_code == 200:
           # print(response.text)
           return response.text
       return None
   except RequestException:
       print('获取索引页失败')
       return None

def parse_page(html):
   soup = BeautifulSoup(html, 'lxml')
   items = soup.select('#pins li')
   for link in items:
       href = link.a['href']
       get_detail_page(href)

   # print(items)

def get_detail_page(href):
   for i in range(1,100):
       detail_url = href + '/' + str(i)
       if requests.get(detail_url, headers=headers).status_code == 200:
           parse_detail_page(detail_url)
       else:
           print('已至末尾页')
           return None

def parse_detail_page(detail_url):
   try:
       response = requests.get(detail_url, headers=headers)
       if response.status_code == 200:
           print('获取详情页成功')
           detail_html = response.text
           print(detail_html)
           # get_image(detail_html)
       return None
   except RequestException:
       print('获取详情页失败')
       return None

# def get_image(detail_html):


def main():
   url = 'https://www.mzitu.com/tag/youhuo/'
   html = get_page(url)
   parse_page(html)




if __name__ == '__main__':
   main()

获得每个详情页的html:


16825884-8bc32ff25418b906.png

再用美丽汤解析出即可：

import requests
from requests.exceptions import RequestException
from bs4 import BeautifulSoup

headers = {'If-None-Match': 'W/"5cc2cd8f-2c58"',
           "Referer": "http://www.mzitu.com/all/",
           'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 UBrowser/6.1.2107.204 SafarMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}

def get_page(url):
    try:
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            # print(response.text)
            return response.text
        return None
    except RequestException:
        print('获取索引页失败')
        return None

def parse_page(html):
    soup = BeautifulSoup(html, 'lxml')
    items = soup.select('#pins li')
    for link in items:
        href = link.a['href']
        get_detail_page(href)

    # print(items)

def get_detail_page(href):
    for i in range(1,100):
        detail_url = href + '/' + str(i)
        if requests.get(detail_url, headers=headers).status_code == 200:
            parse_detail_page(detail_url)
        else:
            print('已至末尾页')
            return None

def parse_detail_page(detail_url):
    try:
        response = requests.get(detail_url, headers=headers)
        if response.status_code == 200:
            print('获取详情页成功')
            detail_html = response.text
            # print(detail_html)
            get_image(detail_html)
        return None
    except RequestException:
        print('获取详情页失败')
        return None

def get_image(detail_html):
    soup = BeautifulSoup(detail_html, 'lxml')
    items= soup.select('.main-image')
    # print(items)
    for item in items:
        return item.img['src']


def main():
    url = 'https://www.mzitu.com/tag/youhuo/'
    html = get_page(url)
    parse_page(html)




if __name__ == '__main__':
    main()
16825884-b762598d53af457c.png

下面就是保存图片了，下面是完整的代码，代码的模块化不高，后期再改进。

import requests
import os
from hashlib import md5
from requests.exceptions import RequestException
from bs4 import BeautifulSoup

headers = {'If-None-Match': 'W/"5cc2cd8f-2c58"',
          "Referer": "http://www.mzitu.com/all/",
          'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 UBrowser/6.1.2107.204 SafarMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}

def get_page(url):
   try:
       response = requests.get(url, headers=headers)
       if response.status_code == 200:
           # print(response.text)
           return response.text
       return None
   except RequestException:
       print('获取索引页失败')
       return None

def parse_page(html):
   soup = BeautifulSoup(html, 'lxml')
   items = soup.select('#pins li')
   for link in items:
       href = link.a['href']
       get_detail_page(href)

   # print(items)

def get_detail_page(href):
   for i in range(1,100):
       detail_url = href + '/' + str(i)
       if requests.get(detail_url, headers=headers).status_code == 200:
           parse_detail_page(detail_url)
       else:
           print('已至末尾页')
           return None

def parse_detail_page(detail_url):
   try:
       response = requests.get(detail_url, headers=headers)
       if response.status_code == 200:
           print('获取详情页成功')
           detail_html = response.text
           # print(detail_html)
           get_image(detail_html)
       return None
   except RequestException:
       print('获取详情页失败')
       return None

def get_image(detail_html):
   soup = BeautifulSoup(detail_html, 'lxml')
   items= soup.select('.main-image')
   # print(items)
   for item in items:
       image = item.img['src']
       save_image(image)

def save_image(image):
   response = requests.get(image,headers=headers)
   if response.status_code == 200:
       data = response.content
       file_path = '{0}/{1}.{2}'.format(os.getcwd(), md5(data).hexdigest(), 'jpg')
       print(file_path)
       if not os.path.exists(file_path):
           with open(file_path, 'wb') as f:
               f.write(data)
               f.close()
               print('保存成功')
   else:
       print('保存失败')
       return None


def main():
   url = 'https://www.mzitu.com/tag/youhuo/'
   html = get_page(url)
   parse_page(html)




if __name__ == '__main__':
   main()
16825884-87bd9db7bb25d633.png

16825884-ba5ca51f5ee6bbed.png

讲实话，我个人比较排斥YY，想实战，哈哈哈

猜你喜欢

转载自blog.csdn.net/weixin_33834628/article/details/90982319
今日推荐
美国拟限制 AI 大模型出口中国和俄罗斯
苹果将与 OpenAI 达成协议，将 ChatGPT 应用于 iPhone
openKylin 社区生态委员会第六次会议圆满召开
阿里云正式发布通义千问 2.5
Python 3.13 发布首个 Beta：实验性自由线程模式和 JIT、改进交互式解释器
Stack Overflow 拿我的代码去训练 AI 大模型，还封了我的账号
Pop!_OS 的 COSMIC 桌面完成 App Store 上架工作
报告：Django 仍然是 74% 开发者的首选
《2024 年一季度互联网投融资运行情况》研究报告
15 年前上了“FFmpeg 耻辱柱”，今天他还得谢谢咱——腾讯QQPlayer一雪前耻？
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
GCC 14.1 发布
周排行
curl的POST请求，封装方法
8.1.1. Integer Types
Java基础 Day05(个人复习整理)
Python - Django - 中间件 process_exception
小L的试卷
【Shell编程】 （函数）判断用户是否存在
python(css样式)
spring ant path 匹配原则 - 【笔记】
《JavaScript与JScript从入门到精通》(美)James.Jaworski.中译本.扫描版.pdf
Eclipse运行带参数的java程序
每日归档
更多
2024-05-12(0)
2024-05-11(38)
2024-05-10(38)
2024-05-09(35)
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022