糗事百科爬虫

版权声明：未经允许，随意转载，请附上本文链接谢谢（づ￣3￣）づ╭❤～
https://blog.csdn.net/xiaoduan_/article/details/80835238

糗事百科爬虫

不怎么看糗事百科，用来试一下PyQuery选择器还是好的

import requests
from lxml import etree
import time
from pyquery import PyQuery
import pandas as pd

def get_data(page):
    text_data = []
    for i in range(1, page + 1):
        url = 'https://www.qiushibaike.com/text/page/{}/'.format(i)
        print("正在爬去第{}页\n".format(i))
        print(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()))
        r = requests.get(url)
        s = etree.HTML(r.text)
        jpy = PyQuery(s)
        data = jpy('#content-left > div > a >div > span').text()
        text_data.append(data)
        time.sleep(1)
    df = pd.DataFrame(text_data)
    df.to_csv('糗事百科.csv', encoding='utf-8-sig')


if __name__ == '__main__':
    get_data(2)
    print("爬去完成")
    print(time.strftime("%Y-%m-%d %H:%M:%S",time.localtime()))

这个样子的结果