豆瓣影评爬虫
今天要爬取https://movie.douban.com/review/best/?start=0该网址的30条最受欢迎影评。
我们可以看到影评比较长,需要展开才能完整显示。但是在网页源码中是没有显示完整影评的。所以我们考虑到这部分应该是异步加载的方式显示。所以打开网页的开发者选项,可以看到点击展开之后,多了一条full的网页。这个网页就包含了完整评论。
我们观察到全部评论所在的网址为https://movie.douban.com/j/review/9593388/full,其中9593388是对应的评论ID,评论ID在第一张图片所在的网页源码里面有。所以我们的流程大题分为以下几步:
- 获取索引页https://movie.douban.com/review/best/?start=0,索引页只需要改变start=0,20,40即可访问下一页
- 解析索引页,把索引页中的10个评论的ID获取到
- 通过获取到的ID构建完整评论网址https://movie.douban.com/j/review/' + str(id) + '/full
- 解析评论页,获取评论者姓名和评论内容,写入txt文件
由于比较简单我直接附上源码。
import requests
import re
from requests.exceptions import RequestException
import time
headers = {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "zh-CN,zh;q=0.9",
"Connection": "keep-alive",
"Host": "movie.douban.com",
"Referer": "https://movie.douban.com/explore",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
}
# 获取索引页
def get_index_page(html):
try:
response = requests.get(url=html, headers=headers)
response.encoding = 'utf-8'
if response.status_code == 200:
# print(response.text)
return response.text
return None
except RequestException:
print('获取索引页错误')
time.sleep(3)
return get_index_page(html)
# 解析索引页
def parse_index_page(html):
html = get_index_page(html)
id_pattern = re.compile('<h2><a href="https://movie.douban.com/review/(.*?)/', re.S)
id = re.findall(id_pattern, html)
return id
# 获取详情页
def get_detail_page(id):
html = 'https://movie.douban.com/j/review/' + str(id) + '/full'
html = get_index_page(html)
content_pattern = re.compile('data-original(.*?)main-author', re.S)
content = re.findall(content_pattern, html)
text_pattern = re.compile('[\u4e00-\u9fa5|,、“”‘’:!~@#¥【】*()——+。;?]+', re.S)
text = re.findall(text_pattern, content[0])
text = ''.join(text)
name_pattern = re.compile('data-author=.*?"(.*?)"', re.S)
name = re.findall(name_pattern, html)
# print(name[0].strip('\\'))
return name, text
def write2txt(name, text):
with open('comment.txt', 'a', encoding='utf-8-sig') as f:
f.write(name[0].strip('\\') + '\n')
f.write(str(text) + '\n')
f.write('\n')
f.close()
if __name__ == '__main__':
for i in range(3):
print(i)
i = i * 20
html = 'https://movie.douban.com/review/best/?start=' + str(i)
for id in parse_index_page(html):
name, text = get_detail_page(id)
write2txt(name, text)