The 7 small python crawler cases this time involve knowledge points such as re regularization, xpath, beautiful soup, selenium, etc., which are very suitable for reference learning by friends who are just getting started with python crawlers. Note: If copyright or privacy issues are involved, please contact me in time to delete.
1. Use regular expressions and file operations to crawl and save the entire content of a post in "Xuba" (the post should not be less than 5 pages.
This time I selected a post in the NBA bar in a certain bar. The title of the post is "Klay or Harden, who has a higher historical status". The target of crawling is the reply content in the post.
Screenshots of the source program and key results:
import csv
import requests
import re
import time
def main(page):
url = f'https://tieba.baidu.com/p/7882177660?pn={page}'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36'
}
resp = requests.get(url,headers=headers)
html = resp.text
# 评论内容
comments = re.findall('style="display:;"> (.*?)</div>',html)
# 评论用户
users = re.findall('class="p_author_name j_user_card" href=".*?" target="_blank">(.*?)</a>',html)
# 评论时间
comment_times = re.findall('楼</span><span class="tail-info">(.*?)</span><div',html)
for u,c,t in zip(users,comments,comment_times):
# 筛选数据,过滤掉异常数据
if 'img' in c or 'div' in c or len(u)>50:
continue
csvwriter.writerow((u,t,c))
print(u,t,c)
print(f'第{page}页爬取完毕')
if __name__ == '__main__':
with open('01.csv','a',encoding='utf-8')as f:
csvwriter = csv.writer(f)
csvwriter.writerow(('评论用户','评论时间','评论内容'))
for page in range(1,8): # 爬取前7页的内容
main(page)
time.sleep(2)
2. Implement a multi-threaded crawler to crawl some chapters of a novel and store them in a database (not less than 10 chapters.
The novel website selected this time is a novel website, here we select the first novel to crawl
Then analyze the link of each chapter of the novel by analyzing the source code of the web page
After finding the location of the link, we use XPath to extract the link and each chapter title
Here, because it involves using requests to send requests multiple times, here we encapsulate it into a function for later use
After the link of each chapter is obtained, we start to enter the chapter content page of the novel for analysis
Through webpage analysis, the content of the novel is in the source code of the webpage, which is static data
Here we use re regular expressions for data extraction and clean the final results
Then we need to save the data to the database, here I will store the crawled data in the mysql database, first seal the operation of the database
Then save the crawled data
The last step is to use multithreading to improve crawler efficiency. Here we create a thread pool of 5 threads
Screenshots of source code and results:
import requests
from lxml import etree
import re
import pymysql
from time import sleep
from concurrent.futures import ThreadPoolExecutor
def get_conn():
# 创建连接
conn = pymysql.connect(host="127.0.0.1",
user="root",
password="root",
db="novels",
charset="utf8")
# 创建游标
cursor = conn.cursor()
return conn, cursor
def close_conn(conn, cursor):
cursor.close()
conn.close()
def get_xpath_resp(url):
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36'}
resp = requests.get(url, headers=headers)
tree = etree.HTML(resp.text) # 用etree解析html
return tree,resp
def get_chapters(url):
tree,_ = get_xpath_resp(url)
# 获取小说名字
novel_name = tree.xpath('//*[@id="info"]/h1/text()')[0]
# 获取小说数据节点
dds = tree.xpath('/html/body/div[4]/dl/dd')
title_list = []
link_list = []
for d in dds[:15]:
title = d.xpath('./a/text()')[0] # 章节标题
title_list.append(title)
link = d.xpath('./a/@href')[0] # 章节链接
chapter_url = url +link # 构造完整链接
link_list.append(chapter_url)
return title_list,link_list,novel_name
def get_content(novel_name,title,url):
try:
cursor = None
conn = None
conn, cursor = get_conn()
# 插入数据的sql
sql = 'INSERT INTO novel(novel_name,chapter_name,content) VALUES(%s,%s,%s)'
tree,resp = get_xpath_resp(url)
# 获取内容
content = re.findall('<div id="content">(.*?)</div>',resp.text)[0]
# 对内容进行清洗
content = content.replace('<br />','\n').replace(' ',' ').replace('全本小说网 www.qb5.tw，最快更新<a href="https://www.qb5.tw/book_116659/">宇宙职业选手</a>最新章节！<br><br>','')
print(title,content)
cursor.execute(sql,[novel_name,title,content]) # 插入数据
conn.commit() # 提交事务保存数据
except:
pass
finally:
sleep(2)
close_conn(conn, cursor) # 关闭数据库
if __name__ == '__main__':
# 获取小说名字，标题链接，章节名称
title_list, link_list, novel_name = get_chapters('https://www.qb5.tw/book_116659/')
with ThreadPoolExecutor(5) as t: # 创建5个线程
for title,link in zip(title_list,link_list):
t.submit(get_content, novel_name,title,link) # 启动线程
3. Use XPath and Beautiful Soup4 respectively to crawl and save data such as the name, description, rating, and number of reviewers of "a certain leaderboard" that is not loaded asynchronously, such as https://movie.douban.com/top250.
Preliminary analysis:
First of all, come to a certain Top250 page, first use the XPath version to grab the data, first analyze the data structure of the movie list page, and post it in the source code of the web page, which belongs to static data
Then we find the law of the data and use xpath to extract the link and movie name of each movie
Then follow the link to enter its details page
Analyze the data on the details page and find that it is also static data, continue to use xpath to extract data
Finally, we store the crawled data, here we use csv files for storage
Next is the version of Beautiful Soup4. Here, we directly use etree in bs4 for data extraction on the movie list page
Finally, also use csv files for data storage
The source code is the screenshot of the result:
XPath version:
import re
from time import sleep
import requests
from lxml import etree
import random
import csv
def main(page,f):
url = f'https://movie.douban.com/top250?start={page*25}&filter='
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.35 Safari/537.36',}
resp = requests.get(url,headers=headers)
tree = etree.HTML(resp.text)
# 获取详情页的链接列表
href_list = tree.xpath('//*[@id="content"]/div/div[1]/ol/li/div/div[1]/a/@href')
# 获取电影名称列表
name_list = tree.xpath('//*[@id="content"]/div/div[1]/ol/li/div/div[2]/div[1]/a/span[1]/text()')
for url,name in zip(href_list,name_list):
f.flush() # 刷新文件
try:
get_info(url,name) # 获取详情页的信息
except:
pass
sleep(1 + random.random()) # 休息
print(f'第{i+1}页爬取完毕')
def get_info(url,name):
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.35 Safari/537.36',
'Host': 'movie.douban.com',
}
resp = requests.get(url,headers=headers)
html = resp.text
tree = etree.HTML(html)
# 导演
dir = tree.xpath('//*[@id="info"]/span[1]/span[2]/a/text()')[0]
# 电影类型
type_ = re.findall(r'property="v:genre">(.*?)</span>',html)
type_ = '/'.join(type_)
# 国家
country = re.findall(r'地区:</span> (.*?)<br',html)[0]
# 上映时间
time = tree.xpath('//*[@id="content"]/h1/span[2]/text()')[0]
time = time[1:5]
# 评分
rate = tree.xpath('//*[@id="interest_sectl"]/div[1]/div[2]/strong/text()')[0]
# 评论人数
people = tree.xpath('//*[@id="interest_sectl"]/div[1]/div[2]/div/div[2]/a/span/text()')[0]
print(name,dir,type_,country,time,rate,people) # 打印结果
csvwriter.writerow((name,dir,type_,country,time,rate,people)) # 保存到文件中
if __name__ == '__main__':
# 创建文件用于保存数据
with open('03-movie-xpath.csv','a',encoding='utf-8',newline='')as f:
csvwriter = csv.writer(f)
# 写入表头标题
csvwriter.writerow(('电影名称','导演','电影类型','国家','上映年份','评分','评论人数'))
for i in range(10): # 爬取10页
main(i,f) # 调用主函数
sleep(3 + random.random())
Beautiful Soup 4**** Edition:
import random
import urllib.request
from bs4 import BeautifulSoup
import codecs
from time import sleep
def main(url, headers):
# 发送请求
page = urllib.request.Request(url, headers=headers)
page = urllib.request.urlopen(page)
contents = page.read()
# 用BeautifulSoup解析网页
soup = BeautifulSoup(contents, "html.parser")
infofile.write("")
print('爬取豆瓣电影250: \n')
for tag in soup.find_all(attrs={"class": "item"}):
# 爬取序号
num = tag.find('em').get_text()
print(num)
infofile.write(num + "\r\n")
# 电影名称
name = tag.find_all(attrs={"class": "title"})
zwname = name[0].get_text()
print('[中文名称]', zwname)
infofile.write("[中文名称]" + zwname + "\r\n")
# 网页链接
url_movie = tag.find(attrs={"class": "hd"}).a
urls = url_movie.attrs['href']
print('[网页链接]', urls)
infofile.write("[网页链接]" + urls + "\r\n")
# 爬取评分和评论数
info = tag.find(attrs={"class": "star"}).get_text()
info = info.replace('\n', ' ')
info = info.lstrip()
print('[评分评论]', info)
# 获取评语
info = tag.find(attrs={"class": "inq"})
if (info): # 避免没有影评调用get_text()报错
content = info.get_text()
print('[影评]', content)
infofile.write(u"[影评]" + content + "\r\n")
print('')
if __name__ == '__main__':
# 存储文件
infofile = codecs.open("03-movie-bs4.txt", 'a', 'utf-8')
# 消息头
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'}
# 翻页
i = 0
while i < 10:
print('页码', (i + 1))
num = i * 25 # 每次显示25部 URL序号按25增加
url = 'https://movie.douban.com/top250?start=' + str(num) + '&filter='
main(url, headers)
sleep(5 + random.random())
infofile.write("\r\n\r\n")
i = i + 1
infofile.close()
4. Crawl the review data of a product in a certain East Mall (the review data is not less than 100, including review content, time and rating).
Preliminary analysis:
This time, I selected a Lenovo laptop on the official website of a certain website. The data is loaded dynamically, and the data can be captured and analyzed through the developer tool.
Screenshots of source code and results:
import requests
import csv
from time import sleep
import random
def main(page,f):
url = 'https://club.jd.com/comment/productPageComments.action'
params = {
'productId': 100011483893,
'score': 0,
'sortType': 5,
'page': page,
'pageSize': 10,
'isShadowSku': 0,
'fold': 1
}
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.35 Safari/537.36',
'referer': 'https://item.jd.com/'
}
resp = requests.get(url,params=params,headers=headers).json()
comments = resp['comments']
for comment in comments:
content = comment['content']
content = content.replace('\n','')
comment_time = comment['creationTime']
score = comment['score']
print(score,comment_time,content)
csvwriter.writerow((score,comment_time,content))
print(f'第{page+1}页爬取完毕')
if __name__ == '__main__':
with open('04.csv','a',encoding='utf-8',newline='')as f:
csvwriter = csv.writer(f)
csvwriter.writerow(('评分','评论时间','评论内容'))
for page in range(15):
main(page,f)
sleep(5+random.random())
5. Implement multiple methods to simulate logging in to Huhu, and crawl a question and answer related to Jianghan University.
First use selenium to open a certain login page, and then use the mobile phone to scan the QR code to log in
After entering the page, open the developer tool, find the element, locate the input box, enter Hanjiang University, and click the search button
Take the second post as an example for elemental analysis.
Screenshots of source code and results:
from time import sleep
from selenium.webdriver.chrome.service import Service
from selenium.webdriver import Chrome,ChromeOptions
from selenium.webdriver.common.by import By
import warnings
def main():
#忽略警告
warnings.filterwarnings("ignore")
# 创建一个驱动
service = Service('chromedriver.exe')
options = ChromeOptions()
# 伪造浏览器
options.add_experimental_option('excludeSwitches', ['enable-automation','enable-logging'])
options.add_experimental_option('useAutomationExtension', False)
# 创建一个浏览器
driver = Chrome(service=service,options=options)
# 绕过检测
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
"source": """
Object.defineProperty(navigator, 'webdriver', {
get: () => false
})
"""
})
# 打开知乎登录页面
driver.get('https://www.zhihu.com/')
sleep(30)
# 点击搜索框
driver.find_element(By.ID,'Popover1-toggle').click()
# 输入内容
driver.find_element(By.ID,'Popover1-toggle').send_keys('汉江大学')
sleep(2)
# 点击搜索图标
driver.find_element(By.XPATH,'//*[@id="root"]/div/div[2]/header/div[2]/div[1]/div/form/div/div/label/button').click()
# 等待页面加载完
driver.implicitly_wait(20)
# 获取标题
title = driver.find_element(By.XPATH,'//*[@id="SearchMain"]/div/div/div/div/div[2]/div/div/div/h2/div/a/span').text
# 点击阅读全文
driver.find_element(By.XPATH,'//*[@id="SearchMain"]/div/div/div/div/div[2]/div/div/div/div/span/div/button').click()
sleep(2)
# 获取帖子内容
content = driver.find_element(By.XPATH,'//*[@id="SearchMain"]/div/div/div/div/div[2]/div/div/div/div/span[1]/div/span/p').text
# 点击评论
driver.find_element(By.XPATH,'//*[@id="SearchMain"]/div/div/div/div/div[2]/div/div/div/div/div[3]/div/div/button[1]').click()
sleep(2)
# 点击获取更多评论
driver.find_element(By.XPATH,'//*[@id="SearchMain"]/div/div/div/div/div[2]/div/div/div/div[2]/div/div/div[2]/div[2]/div/div[3]/button').click()
sleep(2)
# 获取评论数据的节点
divs = driver.find_elements(By.XPATH,'/html/body/div[6]/div/div/div[2]/div/div/div/div[2]/div[3]/div')
try:
for div in divs:
# 评论内容
comment = div.find_element(By.XPATH,'./div/div/div[2]').text
f.write(comment) # 写入文件
f.write('\n')
print(comment)
except:
driver.close()
if __name__ == '__main__':
# 创建文件存储数据
with open('05.txt','a',encoding='utf-8')as f:
main()
6. Comprehensively use the knowledge learned to crawl the first 5 pages of Weibo content of a certain blog user.
Here we have selected the Weibo content of the People's Daily for crawling. I will not post the specific pages here, for fear of violating regulations.
Screenshots of source code and results:
import requests
import csv
from time import sleep
import random
def main(page):
url = f'https://weibo.com/ajax/statuses/mymblog?uid=2803301701&page={page}&feature=0&since_id=4824543023860882kp{page}'
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36',
'cookie':'SINAGLOBAL=6330339198688.262.1661412257300; ULV=1661412257303:1:1:1:6330339198688.262.1661412257300:; PC_TOKEN=8b935a3a6e; SUBP=0033WrSXqPxfM725Ws9jqgMF55529P9D9WWoQDW1G.Vsux_WIbm9NsCq5JpX5KMhUgL.FoMNShMN1K5ESKq2dJLoIpjLxKnL1h.LB.-LxKqLBoBLB.-LxKqLBKeLB--t; ALF=1697345086; SSOLoginState=1665809086; SCF=Auy-TaGDNaCT06C4RU3M3kQ0-QgmTXuo9D79pM7HVAjce1K3W92R1-fHAP3gXR6orrHK_FSwDsodoGTj7nX_1Hw.; SUB=_2A25OTkruDeRhGeFJ71UW-S7OzjqIHXVtOjsmrDV8PUNbmtANLVKmkW9Nf9yGtaKedmyOsDKGh84ivtfHMGwvRNtZ; XSRF-TOKEN=LK4bhZJ7sEohF6dtSwhZnTS4; WBPSESS=PfYjpkhjwcpEXrS7xtxJwmpyQoHWuGAMhQkKHvr_seQNjwPPx0HJgSgqWTZiNRgDxypgeqzSMsbVyaDvo7ng6uTdC9Brt07zYoh6wXXhQjMtzAXot-tZzLRlW_69Am82CXWOFfcvM4AzsWlAI-6ZNA=='
}
resp = requests.get(url,headers=headers)
data_list = resp.json()['data']['list']
for item in data_list:
created_time = item['created_at'] # 发布时间
author = item['user']['screen_name'] # 作者
title = item['text_raw'] # 帖子标题
reposts_count = item['reposts_count'] # 转发数
comments_count = item['comments_count'] # 评论数
attitudes_count = item['attitudes_count'] # 点赞数
csvwriter.writerow((created_time,author,title,reposts_count,comments_count,attitudes_count))
print(created_time,author,title,reposts_count,comments_count,attitudes_count)
print(f'第{page}页爬取完毕')
if __name__ == '__main__':
# 创建保存数据的csv文件
with open('06-2.csv','a',encoding='utf-8',newline='')as f:
csvwriter = csv.writer(f)
# 添加文件表头
csvwriter.writerow(('发布时间','发布作者','帖子标题','转发数','评论数','点赞数'))
for page in range(1,6): # 爬取前5页数据
main(page)
sleep(5+random.random())
7. Choose a hot topic or a topic you are interested in, crawl data and conduct a brief data analysis (for example, by crawling the name, type, total box office and other data of the movie to statistically analyze the average box office of different types of movies, the annual box office champion in ten years Box office trends, etc.; by crawling the population of each province and region in China, statistical analysis of my country's population distribution, etc.).
The URL selected this time is Yienyushu. The goal is to crawl the box office chart data inside, find the data interface through the packet capture analysis of the developer tool, and then start writing code to capture the data.
Screenshots of source code and results:
import requests
import csv
import pandas as pd
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings('ignore')
plt.rcParams['font.sans-serif'] = ['SimHei'] #解决中文显示
plt.rcParams['axes.unicode_minus'] = False #解决符号无法显示
def main():
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36',}
data = {
'r': '0.9936776079863086',
'top': '50',
'type': '0',
}
resp = requests.post('https://ys.endata.cn/enlib-api/api/home/getrank_mainland.do', headers=headers, data=data)
data_list = resp.json()['data']['table0']
for item in data_list:
rank = item['Irank'] # 排名
MovieName = item['MovieName'] # 电影名称
ReleaseTime = item['ReleaseTime'] # 上映时间
TotalPrice = item['BoxOffice'] # 总票房(万)
AvgPrice = item['AvgBoxOffice'] # 平均票价
AvgAudienceCount = item['AvgAudienceCount'] # 平均场次
# 写入csv文件
csvwriter.writerow((rank,MovieName,ReleaseTime,TotalPrice,AvgPrice,AvgAudienceCount))
print(rank,MovieName,ReleaseTime,TotalPrice,AvgPrice,AvgAudienceCount)
def data_analyze():
# 读取数据
data = pd.read_csv('07.csv')
# 从上映时间中提取出年份
data['年份'] = data['上映时间'].apply(lambda x: x.split('-')[0])
# 各年度上榜电影总票房占比
df1 = data.groupby('年份')['总票房(万)'].sum()
plt.figure(figsize=(6, 6))
plt.pie(df1, labels=df1.index.to_list(), autopct='%1.2f%%')
plt.title('各年度上榜电影总票房占比')
plt.show()
# 各个年份总票房趋势
df1 = data.groupby('年份')['总票房(万)'].sum()
plt.figure(figsize=(6, 6))
plt.plot(df1.index.to_list(), df1.values.tolist())
plt.title('各年度上榜电影总票房趋势')
plt.show()
# 平均票价最贵的前十名电影
print(data.sort_values(by='平均票价', ascending=False)[['年份', '电影名称', '平均票价']].head(10))
# 平均场次最高的前十名电影
print(data.sort_values(by='平均场次', ascending=False)[['年份', '电影名称', '平均场次']].head(10))
if __name__ == '__main__':
# 创建保存数据的csv文件
with open('07.csv', 'w', encoding='utf-8',newline='') as f:
csvwriter = csv.writer(f)
# 添加文件表头
csvwriter.writerow(('排名', '电影名称', '上映时间', '总票房(万)', '平均票价', '平均场次'))
main()
# 数据分析
data_analyze()
Judging from the proportion of the box office of the movies on the annual list, 2019 has the highest proportion, which shows that the quality of the movies in 2019 is very good, there are many movies on the list and the box office is high.
From the perspective of the trend, from 2016 to 2019, the total box office of the movies on the list has been increasing, and reached its peak in 2019, which shows that the movies this year are very popular, but the sharp decline from 2020 should be the biggest reason. The outbreak of the epidemic began at the beginning of the year, which led to the fact that the Lunar New Year file was not released in the first stage, and due to the impact of the epidemic, the cinema has been closed, so the box office this year was dismal.
Well, this is the end of this case sharing, I hope it will be helpful to those who are new to reptiles.
Finally, I would like to thank everyone who has read my article carefully. Reciprocity is always necessary. Although the following information is not very valuable, you can take it away if you need it:
