爬取三国演义全本内容，保存到TXT文件夹下（requests+bs4）

一、爬取界面分析

爬取网站：https://www.shicimingju.com/book/sanguoyanyi.html

通过分析，该界面采用静态加载的方式呈现数据，即获取网页源代码可获取相应的数据，在本界面中获取的数据有章节的标题，以及章节内容的链接。

在章节内容界面，同样时采用静态加载数据的方式。

二、程序源代码

import requests
from bs4 import BeautifulSoup

fp = open('./sanguo.txt','w',encoding='utf-8')
headers = {
    
    
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
    }
main_url = "https://www.shicimingju.com/book/sanguoyanyi.html"
page_text = requests.get(url = main_url,headers = headers)
page_text.encoding = page_text.apparent_encoding
page_text = page_text.text
soup = BeautifulSoup(page_text,'lxml')
a_list = soup.select('.book-mulu > ul > li > a')
for a in a_list:
    title = a.string
    detail_url = 'https://www.shicimingju.com' + a['href']
    page_text_detail = requests.get(detail_url,headers = headers)
    page_text_detail.encoding = page_text_detail.apparent_encoding
    page_text_detail = page_text_detail.text
    soup = BeautifulSoup(page_text_detail,'lxml')
    div_tag = soup.find('div',class_ = 'chapter_content')
    content = div_tag.text
    fp.write(title + ':' + content + '\n')
    print(title,'保存成功!!!')
fp.close()

三、程序运行结果