豆瓣书评爬虫

版权声明：未经允许，随意转载，请附上本文链接谢谢（づ￣3￣）づ╭❤～
https://blog.csdn.net/xiaoduan_/article/details/80835248

豆瓣书评爬虫

一年前学的爬虫，忘得差不多了，现在复习一下
最近读了《白夜行》，就拿它下手喽

import requests
from lxml import etree
import pandas as pd
import time

data = []
def get_userdata(page):
    for i in range(1, page + 1):
        url = 'https://book.douban.com/subject/3259440/comments/hot?p={}'.format(i)
        print("正在爬取第{}页\n".format(i))
        print(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()))
        time.sleep(1)
        r = requests.get(url)
        s = etree.HTML(r.text)

        xiaodata = s.xpath('//div[@id="comments"]//div/p/text()')

        data.extend(xiaodata)

if __name__ == '__main__':
    get_userdata(20)
    df = pd.DataFrame(data)
    df.to_csv('白夜行.csv', encoding='utf-8-sig')
    print("任务完成")

最后结果，爬20页意思意思就好啦