开发环境：win7,8,10，python3+
python模块：requestes，bs4，matplotlib，jieba，wordcloud，PIL，numpy，random
实现的功能和思路：
（1）打开豆瓣《狂暴巨兽》评论区，根据html结构捕获三个信息：
一，每账号的评分等级为5星、4星、3星、2星、1星；
二，每个账号的评论留言；
三，跳转到下个评论页面的http链接
（2）获取所有的信息后对信息进行处理：
一，计算出每个星级的总数和一共多少账户进行了评级
二、将所有的评论内容放在一起，处理评论中的空格和其他不规范形式
（3）用matplotlib绘制评分等级占比的饼图，用jieba进行分词处理，用wordcloud生成词云图
最终结果：
实现步骤和代码：
import requests from bs4 import BeautifulSoup import random import matplotlib.pyplot as plt import jieba from wordcloud import WordCloud import PIL import numpy as np agents = [ "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0; Baiduspider-ads) Gecko/17.0 Firefox/17.0", "Mozilla/5.0 (Linux; U; Android 2.3.6; en-us; Nexus S Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1", "Avant Browser/1.2.789rel1 (http://www.avantbrowser.com)", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.0 Safari/532.5", "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/532.9 (KHTML, like Gecko) Chrome/5.0.310.0 Safari/532.9", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.514.0 Safari/534.7", "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/9.0.601.0 Safari/534.14", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/10.0.601.0 Safari/534.14", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.20 (KHTML, like Gecko) Chrome/11.0.672.2 Safari/534.20", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.27 (KHTML, like Gecko) Chrome/12.0.712.0 Safari/534.27", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.24 Safari/535.1", "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.36 Safari/535.7", "Mozilla/5.0 (Windows; U; Windows NT 6.0 x64; en-US; rv:1.9pre) Gecko/2008072421 Minefield/3.0.2pre", "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9b4) Gecko/2008030317 Firefox/3.0b4", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10", "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)", "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 GTB5", "Mozilla/5.0 (Windows; U; Windows NT 5.1; tr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729; .NET4.0E)", "Mozilla/5.0 (Windows; U; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; BIDUBrowser 7.6)", "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko", "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.99 Safari/537.36", "Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; Touch; LCJB; rv:11.0) like Gecko", "Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1", "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0a2) Gecko/20110622 Firefox/6.0a2", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b4pre) Gecko/20100815 Minefield/4.0b4pre", "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0 )", "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)", "Mozilla/5.0 (Windows; U; Windows XP) Gecko MultiZilla/1.6.1.0a", "Mozilla/2.02E (Win95; U)", "Mozilla/3.01Gold (Win95; I)", "Mozilla/4.8 [en] (Windows NT 5.1; U)", "Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.4) Gecko Netscape/7.1 (ax)", "HTC_Dream Mozilla/5.0 (Linux; U; Android 1.5; en-ca; Build/CUPCAKE) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1", "Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.2; U; de-DE) AppleWebKit/534.6 (KHTML, like Gecko) wOSBrowser/234.40.1 Safari/534.6 TouchPad/1.0", "Mozilla/5.0 (Linux; U; Android 1.5; en-us; sdk Build/CUPCAKE) AppleWebkit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1", "Mozilla/5.0 (Linux; U; Android 2.1; en-us; Nexus One Build/ERD62) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17", "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1", "Mozilla/5.0 (Linux; U; Android 1.5; en-us; htc_bahamas Build/CRB17) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1", "Mozilla/5.0 (Linux; U; Android 2.1-update1; de-de; HTC Desire 1.19.161.5 Build/ERE27) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17", "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Sprint APA9292KT Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1", "Mozilla/5.0 (Linux; U; Android 1.5; de-ch; HTC Hero Build/CUPCAKE) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1", "Mozilla/5.0 (Linux; U; Android 2.2; en-us; ADR6300 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1", "Mozilla/5.0 (Linux; U; Android 2.1; en-us; HTC Legend Build/cupcake) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17", "Mozilla/5.0 (Linux; U; Android 1.5; de-de; HTC Magic Build/PLAT-RC33) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 FirePHP/0.3", "Mozilla/5.0 (Linux; U; Android 1.6; en-us; HTC_TATTOO_A3288 Build/DRC79) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1", "Mozilla/5.0 (Linux; U; Android 1.0; en-us; dream) AppleWebKit/525.10 (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2", "Mozilla/5.0 (Linux; U; Android 1.5; en-us; T-Mobile G1 Build/CRB43) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari 525.20.1", "Mozilla/5.0 (Linux; U; Android 1.5; en-gb; T-Mobile_G2_Touch Build/CUPCAKE) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1", "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Droid Build/ESD20) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17", "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Droid Build/FRG22D) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1", "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Milestone Build/ SHOLS_U2_01.03.1) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17", "Mozilla/5.0 (Linux; U; Android 2.0.1; de-de; Milestone Build/SHOLS_U2_01.14.0) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17", "Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/525.10 (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2", "Mozilla/5.0 (Linux; U; Android 0.5; en-us) AppleWebKit/522 (KHTML, like Gecko) Safari/419.3", "Mozilla/5.0 (Linux; U; Android 1.1; en-gb; dream) AppleWebKit/525.10 (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2", "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Droid Build/ESD20) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17", "Mozilla/5.0 (Linux; U; Android 2.1; en-us; Nexus One Build/ERD62) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17", "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Sprint APA9292KT Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1", "Mozilla/5.0 (Linux; U; Android 2.2; en-us; ADR6300 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1", "Mozilla/5.0 (Linux; U; Android 2.2; en-ca; GT-P1000M Build/FROYO) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1", "Mozilla/5.0 (Linux; U; Android 3.0.1; fr-fr; A500 Build/HRI66) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13", "Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/525.10 (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2", "Mozilla/5.0 (Linux; U; Android 1.6; es-es; SonyEricssonX10i Build/R1FA016) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1", "Mozilla/5.0 (Linux; U; Android 1.6; en-us; SonyEricssonX10i Build/R1AA056) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1", ] # 反爬虫对策——浏览器列表 heads = { # ip代理池获取时用的url的headers 'User-Agent': random.choice(agents), 'ue': 'utf-8' # 设置翻译支持中文 } list_all = [] # 存放每个评论页面所有的信息的列表 # 获取代理ip池 def get_ip_list(): urlip = 'http://www.xicidaili.com/nn/' html = requests.get(urlip, headers=heads).text soup = BeautifulSoup(html, 'html.parser') ips = soup.find_all('tr') ip_list = [] for i in range(1, len(ips)): ip_info = ips[i] tds = ip_info.find_all('td') ip_list.append(tds[1].text + ':' + tds[2].text) return ip_list # 从ip代理池随机选取一个ip返回 def get_random_ip(): ip_list = get_ip_list() proxy_list = [] for ip in ip_list: proxy_list.append('http://' + ip) proxy_ip = random.choice(proxy_list) proxies = {'http': proxy_ip} return proxies # 用于评论页面html获取的url的headers heade = { 'User-Agent': random.choice(agents), 'proxies': get_random_ip(), # 反爬虫策略——每次从ip池中随机调取一个ip 'ue': 'utf-8' # 设置翻译支持中文 } # 获取一个评论页面的所有html文本信息 def get_html(url): response = requests.get(url, params=heade) response.encoding = 'utf-8' html = response.text return html # 获取下一页评论区信息的url def get_url(html): bs = BeautifulSoup(html, 'html.parser') url_list = bs.find_all('div', attrs={'id': "paginator"}) if len(url_list) > 0: url_next_part = url_list[0].find('a', attrs={'class': "next"})['href'] url_next = 'https://movie.douban.com/subject/26430636/comments' + url_next_part return url_next # 获取评论每种等级和影评信息 def get_star_and_conments(url_comment): comment_html = get_html(url_comment) bs = BeautifulSoup(comment_html, 'html.parser') comment_list = bs.find_all('div', attrs={'class': "comment-item"}) comment_all ='' five_star = 0 four_star = 0 three_star = 0 two_star = 0 one_star = 0 for comment in comment_list: comments = (comment.find('p')).text comment_all += comments span = (comment.find_all('span')[4])['class'] if span[0] == 'allstar50': five_star += 1 elif span[0] == 'allstar40': four_star += 1 elif span[0] == 'allstar30': three_star += 1 elif span[0] == 'allstar20': two_star += 1 elif span[0] == 'allstar10': one_star += 1 return [comment_all.strip().replace('\n', '').replace(' ', ';'), [five_star, four_star, three_star, two_star, one_star]] # 创建循环调用实现完成一个页面信息获取后，自动获取下个页面信息的函数 def get_all(url): all_list = get_star_and_conments(url) comment_url = get_url(get_html(url)) list_all.append(all_list) if comment_url != None: get_all(comment_url) # 主函数运行程序 if __name__ == '__main__': url = 'https://movie.douban.com/subject/26430636/comments?start=0&limit=20&sort=new_score&status=P&percent_type=' get_all(url) comment = '' five_star = 0 four_star = 0 three_star = 0 two_star = 0 one_star = 0 for i in list_all: comment += i[0] five_star += i[1][0] four_star += i[1][1] three_star += i[1][2] two_star += i[1][3] one_star += i[1][4] all = five_star + four_star + three_star + two_star + one_star # print(comment, one_star, two_star, three_star, four_star, five_star) with open('狂暴巨兽影评.txt', 'w', encoding='utf-8') as f_obj: # 生成影评文本 f_obj.write(comment) # 调用matplotlib绘制饼图，并生成png图片 labels = 'one star', 'two star', 'three star', 'four star', 'five star' faces = [one_star*1.00/all, two_star*1.00/all, three_star*1.00/all, four_star*1.00/all, five_star*1.00/all] explore = [0, 0, 0, 0, 0.1] colors = ['red', 'yellow', 'blue', 'green', 'orange'] plt.axis(aspect=1) patches, l_text, p_text = plt.pie(x=faces, labels=labels, explode=explore, colors=colors, autopct='%3.1f%%', shadow=True, labeldistance=1.1, startangle=90, pctdistance=0.6) for t in l_text: t.set_size = (10) for t in p_text: t.set_size = (30) plt.axis('equal') plt.title('the comment', fontsize=10) plt.savefig('狂暴巨兽-pie-comment.png') plt.show() # 调用wordcloud生成词云图并保存为ipg path = r'F:\文档类\python36\图像处理\STXINWEI.TTF' alien_mask = np.array(PIL.Image.open(r'F:\meizitu\54552016a-08-10\03.jpg')) wc = WordCloud(font_path=path, background_color='white', margin=5, width=1800, height=800, mask=alien_mask, max_words=2000, max_font_size=60, random_state=42) a = [] words = list(jieba.cut(comment)) for word in words: if len(word) > 1: a.append(word) txt = r' '.join(a) wc = wc.generate(txt) wc.to_file('狂暴巨兽词云.jpg')
总结：
我们分析url=https://movie.douban.com/subject/26430636/comments?start=0&limit=20&sort=new_score&status=P&percent_type=
之中“26430636”为电影的代表，将其换做其他的编号就可以读取和生成其他电影的matplotlib和wordcloud制作评分图和词云图