day3 python reptile

A reptile principle
1. What is the Internet?
It refers to a stack of a network device, to the computer station to the Internet together with a called Internet.

2. The purpose of the establishment of the Internet?
The purpose is to establish the Internet transfer and data sharing data.

3. What is the data?
For example ... Taobao, Jingdong product information such as
number of securities investment information East Fortune, snowball network ...
the chain of home, such as availability of information freely ....
12306 ticket information ...

4. Internet whole process:
- Common User:
Open Browser -> sending a request to a target site -> the fetch response data -> renderer to the browser

- crawlers:
Analog Browser -> sending a request to a target site -> the fetch response data -> extract valuable data -> persisted to the data

5. What is the browser sends a request?
request http protocol.

- Client:
the browser is a software -> Client IP and port

- server
https://www.jd.com/
www.jd.com (Jingdong domain name) -> DNS parsing -> Jingdong server IP and port

Client ip and port ------> IP and port to send the request to the server can establish a link to obtain the corresponding data.

6. The crawler whole process
- the transmission request (request requires libraries: Requests database request, requesting the Selenium library)
- fetch response data (as long as the transmission request to the server, the request returns response data)
- parses and extracts data (requires parsing library : Re, BeautifulSoup4, Xpath ...)
- save to a local (file processing, database, MongoDB repository)

Two Requests library request

1. Installation and Use
- open cmd
- Input: pip3 install requests

2. crawling video

''''''
'''
视频选项:
    1.梨视频
'''
# import requests
#
# # 往视频源地址发送请求
# response = requests.get(
#     'https://video.pearvideo.com/mp4/adshort/20190625/cont-1570302-14057031_adpkg-ad_hd.mp4')
#
# # 打印二进制流，比如图片、视频等数据
# print(response.content)
#
# # 保存视频到本地
# with open('视频.mp4', 'wb') as f:
#     f.write(response.content)

'''
1、先往梨视频主页发送请求
    https://www.pearvideo.com/
    
    解析获取所有视频的id:
        video_1570302
        
        re.findall()
        

2、获取视频详情页url:
    惊险!男子抢上地铁滑倒,就脚进去了
    https://www.pearvideo.com/video_1570302
    揭秘坎儿井
    https://www.pearvideo.com/video_1570107
'''
import requests
import re  # 正则，用于解析文本数据
# 1、先往梨视频主页发送请求
response = requests.get('https://www.pearvideo.com/')
# print(response.text)

# re正则匹配获取所有视频id
# 参数1: 正则匹配规则
# 参数2: 解析文本
# 参数3: 匹配模式
res_list = re.findall('<a href="video_(.*?)"', response.text, re.S)
# print(res_list)

# 拼接每一个视频详情页url
for v_id in res_list:
    detail_url = 'https://www.pearvideo.com/video_' + v_id
    # print(detail_url)

    # 对每一个视频详情页发送请求获取视 频源url
    response = requests.get(url=detail_url)
    # print(response.text)

    # 解析并提取详情页视频url
    # 视频url
    video_url = re.findall('srcUrl="(.*?)"', response.text, re.S)[0]
    print(video_url)

    # 视频名称
    video_name = re.findall(
        '<h1 class="video-tt">(.*?)</h1>', response.text, re.S)[0]

    print(video_name)

    # 往视频url发送请求获取视频二进制流
    v_response = requests.get(video_url)

    with open('%s.mp4' % video_name, 'wb') as f:
        f.write(v_response.content)
        print(video_name, '视频爬取完成')

 

3.抓包分析
打开浏览器的开发者模式（检查）----> 选中network
找到访问的页面后缀 xxx.html（响应文本）

1)请求url（访问的网站地址）
2)请求方式：
GET:
直接发送请求获取数据
https://www.cnblogs.com/kermitjam/articles/9692597.html

POST:
需要携带用户信息往目标地址发送请求
https://www.cnblogs.com/login

3) 响应状态码:
2xx: 成功
3xx: 重定向
4xx: 找不到资源
5xx: 服务器错误

4) 请求头信息:
User-Agent: 用户代理（证明是通过电脑设备及浏览器发送的请求）
Cookies: 登录用户真实信息（证明你目标网站的用户）
Referer: 上一次访问的url （证明你是从目标网站跳转过来的）

5)请求体:
POST请求才会有请求体。
Form Data
{
'user': 'tank',
'pwd': '123'
}

四 爬取豆瓣电影
.: 从当前位置开始
*: 查找所有
?: 找到第一个不找

.*?: 非贪婪匹配
.*: 贪婪匹配

(.*?): 提取括号内的数据

电影排名、电影url、电影名称、导演-主演-类型、电影评分、评价人数、电影简介
<div class="item">.*?<em class="">(.*?)</em>
.*?<a href="(.*?)">.*?<span class="title">(.*?)</span>
.*?导演:(.*?)</p>.*?<span class="rating_num".*?>(.*?)</span>
.*?<span>(.*?)人评价</span>.*?<span class="inq">(.*?)</span>

 

<div class="item">
<div class="pic">
<em class="">226</em>
<a href="https://movie.douban.com/subject/1300374/">
<img width="100" alt="绿里奇迹" src="https://img3.doubanio.com/view/photo/s_ratio_poster/public/p767586451.webp" class="">
</a>
</div>
<div class="info">
<div class="hd">
<a href="https://movie.douban.com/subject/1300374/" class="">
<span class="title">绿里奇迹</span>
<span class="title">&nbsp;/&nbsp;The Green Mile</span>
<span class="other">&nbsp;/&nbsp;绿色奇迹(台) / 绿色英里</span>
</a>

<span class="playable">[可播放]</span>
</div>
<div class="bd">
<p class="">
导演: Frank Darabont&nbsp;&nbsp;&nbsp;主演: 汤姆·汉克斯 Tom Hanks / 大卫·摩斯 David M...<br>
1999&nbsp;/&nbsp;美国&nbsp;/&nbsp;犯罪 剧情 奇幻 悬疑
</p>

<div class="star">
<span class="rating45-t"></span>
<span class="rating_num" property="v:average">8.7</span>
<span property="v:best" content="10.0"></span>
<span>141370人评价</span>
</div>

<p class="quote">
<span class="inq">天使暂时离开。</span>
</p>
</div>
</div>
</div>

 

''''''
'''
https://movie.douban.com/top250?start=0&filter=
https://movie.douban.com/top250?start=25&filter=
https://movie.douban.com/top250?start=50&filter=

1.发送请求
2.解析数据
3.保存数据
'''
import requests
import re

# 爬虫三部曲
# 1.发送请求
def get_page(base_url):
    response = requests.get(base_url)
    return response

# 2.解析文本
def parse_index(text):

    res = re.findall('<div class="item">.*?<em class="">(.*?)</em>.*?<a href="(.*?)">.*?<span class="title">(.*?)</span>.*?导演:(.*?)</p>.*?<span class="rating_num".*?>(.*?)</span>.*?<span>(.*?)人评价</span>.*?<span class="inq">(.*?)</span>', text, re.S)
    # print(res)
    return res

# 3.保存数据
def save_data(data):
    with open('douban.txt', 'a', encoding='utf-8') as f:
        f.write(data)

# main + 回车键
if __name__ == '__main__':
    # num = 10
    # base_url = 'https://movie.douban.com/top250?start={}&filter='.format(num)

    num = 0
    for line in range(10):
        base_url = f'https://movie.douban.com/top250?start={num}&filter='
        num += 25
        print(base_url)

        # 1.发送请求，调用函数
        response = get_page(base_url)

        # 2.解析文本
        movie_list = parse_index(response.text)

        # 3.保存数据
        # 数据的格式化
        for movie in movie_list:
            # print(movie)

            # 解压赋值
            # 电影排名、电影url、电影名称、导演 - 主演 - 类型、电影评分、评价人数、电影简介
            v_top, v_url, v_name, v_daoyan, v_point, v_num, v_desc = movie
            # v_top = movie[0]
            # v_url = movie[1]
            moive_content = f'''
            电影排名: {v_top}
            电影url: {v_url}
            电影名称: {v_name}
            导演主演: {v_daoyan}
            电影评分: {v_point}
            评价人数: {v_num}
            电影简介: {v_desc}
            \n
            '''

            print(moive_content)

            # 保存数据
            save_data(moive_content)