今天无意间搜索问题的时候跳转到了百度指数这里，索性就打开来看看，下面是首页截图：
这里你可以自己输入自己想要查询的人物、事件等等，anything，只要是你感兴趣的都可以，有一种感觉就是你认为是热点的就是热点。。。。
闲话不多说了，这里直接进入实践，先看代码，完整的实现如下：
#!usr/bin/env python
#encoding:utf-8
'''
__Author__:沂水寒城
功能： 爬取百度指数结果
搜索入口地址：
http://index.baidu.com/v2/index.html#/
'''
import os
import sys
import json
import time
import random
import requests
import datetime
if sys.version_info < (3, 0):
reload(sys)
sys.setdefaultencoding("utf-8")
month_day_dict = {
"01": 31,
"02": 28,
"03": 31,
"04": 30,
"05": 31,
"06": 30,
"07": 31,
"08": 31,
"09": 30,
"10": 31,
"11": 30,
"12": 31,
}
def generateMonthDays(month_day_dict,year='2017',month='03'):
'''
生成指定年份、月份中的所有日期
'''
day_num=month_day_dict[month]
day_date_list=[]
for i in range(1,day_num+1):
one=str(i)
if len(one)==1:
one='0'+one
day_date_list.append(year+'-'+month+'-'+one)
return day_date_list
def generatePeriodDays(start="2019-11-01", end="2019-11-31"):
"""
生成指定时间段内所有的 天 日期
"""
start_year, start_mon = start.split("-")[0].strip(), start.split("-")[1].strip()
all_day_list = generateMonthDays(month_day_dict, year=start_year, month=start_mon)
end_year, end_mon = start.split("-")[0].strip(), start.split("-")[1].strip()
all_day_list += generateMonthDays(month_day_dict, year=end_year, month=end_mon)
day_list = [one for one in all_day_list if one >= start and one <= end]
return sorted(list(set(day_list)))
def reqData(start="2019-11-01", end="2019-12-03", kw="海康威视"):
'''
网络请求数据
'''
date_list = generatePeriodDays(start = start, end = end)
date_str = ','.join(date_list)
search_url = urlTemplate.format(date_str, kw)
data = requests.get(search_url, timeout=5)
content = data.json()
return content
def dataParser(content):
'''
数据解析
'''
data_list= content['data']["追我吧"]
result = []
key_list = ['date', 'title', 'url', 'source', 'same_news']
for one_dict in data_list:
T = one_dict['date']
news_list = one_dict['news']
for one_son_dict in news_list:
one_tmp_list = []
for i in range(len(key_list)):
try:
one_tmp_list.append(one_son_dict[key_list[i]])
except:
one_tmp_list.append(None)
result.append(one_tmp_list)
result = sorted(result, key = lambda e:e[0])
result.insert(0, key_list)
for one in result:
print one
if __name__ == '__main__':
content = reqData(start="2019-11-01", end="2019-12-03", kw="追我吧")
dataParser(content)最近《追我吧》这个节目组还是上了好久的热点头条的，这里我就搜索“追我吧”，结果如下：
{"status":0,"data":{"追我吧":[{"date":"2019-11-24","news":[{"date":"2019-11-24","title":"《追我吧<\/em>》这期大咖云集,跑男家族相聚一起,阵容再变引起热议","url":"https:\/\/new.qq.com\/omn\/20191124\/20191124A00PJP.html?pc","source":"腾讯新闻","same_news":2}]},{"date":"2019-11-25","news":[{"date":"2019-11-25","title":"《追我吧<\/em>》VS跑男家族团魂"开战" 喜提收视三连冠","url":"http:\/\/news.gxnews.com.cn\/staticpages\/20191125\/newgx5ddb716d-19060415.shtml","source":"广西新闻网","same_news":5},{"date":"2019-11-25","title":"《追我吧<\/em>》VS跑男家族团魂“开战”引爆多平台喜提收视三连冠","url":"http:\/\/www.dzwww.com\/yule\/zy\/201911\/t20191125_19393325.htm","source":"大众网","same_news":3},{"date":"2019-11-25","title":"《追我吧<\/em>》下期放大招!请来两大流量男团,粉丝都开心坏了","url":"https:\/\/new.qq.com\/omn\/20191125\/20191125A057NX.html?pc","source":"腾讯新闻","same_news":1}]},{"date":"2019-11-26","news":[{"date":"2019-11-26","title":"跑男团神武3战队双双踢馆追我吧<\/em> “踢”出周末最高收视","url":"http:\/\/www.cet.com.cn\/xwsd\/2426554.shtml","source":"中国经济新闻网","same_news":6},{"date":"2019-11-26","title":"《追我吧<\/em>》VS跑男家族团魂“开战” 引爆多平台喜提收视三连冠","url":"http:\/\/ent.ynet.com\/2019\/11\/26\/2226817t1254.html","source":"北青网","same_news":2},{"date":"2019-11-26","title":"《追我吧<\/em>》190长腿追跑者 中国山地车攀爬第一人张京坤","url":"http:\/\/www.cnr.cn\/rdzx\/cxxhl\/zxxx\/20191126\/t20191126_524873496.shtml","source":"中国广播网","same_news":1}]},{"date":"2019-11-27","news":[{"date":"2019-11-27","title":"高以翔去世是什么原因?追我吧<\/em>凌晨录制极限项目被质疑安全性","url":"http:\/\/www.mnw.cn\/news\/ent\/2224569.html","source":"闽南网","same_news":5},{"date":"2019-11-27","title":"钟楚曦回应追我吧<\/em>难度和强度:连吃3天速效救心丸","url":"http:\/\/ent.sina.com.cn\/tv\/zy\/2019-11-27\/doc-iihnzahi3716875.shtml","source":"新浪","same_news":4},{"date":"2019-11-27","title":"高以翔出事的《追我吧<\/em>》,李小鹏喊累邹市明要吸氧 高强度节目如此...","url":"http:\/\/www.dzwww.com\/yule\/nd\/201911\/t20191127_19400574.htm","source":"大众网","same_news":4}]},{"date":"2019-11-28","news":[{"date":"2019-11-28","title":"追我吧<\/em>结束录制 陈伟霆张继科离开剧组","url":"https:\/\/news.china.com\/socialgd\/10000169\/20191128\/37471028.html","source":"中华网","same_news":10},{"date":"2019-11-28","title":"探访高以翔节目录制地:《追我吧<\/em>》停录,部分设施被拆除","url":"http:\/\/news.sina.com.cn\/c\/2019-11-28\/doc-iihnzhfz2240506.shtml","source":"新浪新闻","same_news":8},{"date":"2019-11-28","title":"浙江卫视终于发声:为高以翔猝死负责!徐峥痛斥!《追我吧<\/em>》节目组有...","url":"http:\/\/finance.ifeng.com\/c\/7ryAnVtT1A7","source":"凤凰网","same_news":6}]},{"date":"2019-11-29","news":[{"date":"2019-11-29","title":"网曝《追我吧<\/em>》负责人因高以翔事件被全部开除","url":"http:\/\/ent.sina.com.cn\/tv\/zy\/2019-11-29\/doc-iihnzhfz2423924.shtml","source":"新浪","same_news":14},{"date":"2019-11-29","title":"浙江卫视宣布高以翔参录综艺《追我吧<\/em>》今晚停播","url":"http:\/\/ent.163.com\/19\/1129\/15\/EV5K41ET00038FO9.html","source":"网易","same_news":6},{"date":"2019-11-29","title":"《追我吧<\/em>》正常播出?知情人士:非新一期节目","url":"http:\/\/ent.sina.com.cn\/tv\/zy\/2019-11-29\/doc-iihnzahi4134718.shtml","source":"新浪","same_news":4}]},{"date":"2019-11-30","news":[{"date":"2019-11-30","title":"疑《追我吧<\/em>》工作人员发声:带节奏的请消停一点","url":"http:\/\/ent.163.com\/19\/1130\/07\/EV7CIOE600038FO9.html","source":"网易","same_news":3},{"date":"2019-11-30","title":"邹市明参加追我吧<\/em>腿部失去知觉,粉丝:轩轩爸爸命真大","url":"http:\/\/m.sohu.com\/a\/357415896_403336","source":"搜狐","same_news":2},{"date":"2019-11-30","title":"凭借《奔跑吧》走红的宋雨琦人气不再,参加《追我吧<\/em>》如今遭停播","url":"https:\/\/new.qq.com\/omn\/20191130\/20191130A0JQF900","source":"腾讯新闻","same_news":1}]}]},"message":""}接下来搜索一下“海康威视”，结果如下：
{
"status": 0,
"data": {
"海康威视": [
{
"date": "2019-11-20",
"news": [
{
"date": "2019-11-20",
"title": "海康威视</em>高级副总经理蒋海青辞职 年薪为248.62万元",
"url": "http://finance.eastmoney.com/a/201911201296670767.html",
"source": "东方财富网",
"same_news": 3
},
{
"date": "2019-11-20",
"title": "...高新技术企业创新能力百强榜发布,士兰微、新华三、海康威视</em>等...",
"url": "https://laoyaoba.com/html/share/news?source",
"source": "集微网",
"same_news": 1
},
{
"date": "2019-11-20",
"title": "海康威视</em>与国网杭州供电公司签订战略合作协议",
"url": "http://www.afzhan.com/news/detail/78811.html",
"source": "中国安防展览网",
"same_news": 1
}
]
},
{
"date": "2019-11-22",
"news": [
{
"date": "2019-11-22",
"title": "海康威视</em>:融资净偿还8784.07万元,两市排名第三(11-21)",
"url": "http://stock.eastmoney.com/a/201911221299836442.html",
"source": "东方财富网",
"same_news": 2
},
{
"date": "2019-11-22",
"title": "后安防时代,海康威视</em>的信心与底气在哪? ",
"url": "http://www.techsir.com/a/201911/59129.html",
"source": "Techsir",
"same_news": 2
}
]
},
{
"date": "2019-11-25",
"news": [
{
"date": "2019-11-25",
"title": "海康威视</em>闪耀2019首届四川教育装备博览会 ",
"url": "http://www.ceiea.com/html/201911/201911251324011299.shtml",
"source": "中国教育..",
"same_news": 1
},
{
"date": "2019-11-25",
"title": "湖南广电、网易、海康威视</em>都来了!浙传这场招聘会让2000多人达成...",
"url": "http://edu.zjol.com.cn/jyjsb/gx/201911/t20191125_11397473.shtml",
"source": "浙江在线",
"same_news": 1
},
{
"date": "2019-11-25",
"title": "海康威视</em>龚虹嘉的财富密码:十年暴赚两万倍后,为何被查?",
"url": "http://finance.sina.com.cn/stock/relnews/cn/2019-11-25/doc-iihnzahi3264395.shtml",
"source": "新浪",
"same_news": 1
}
]
},
{
"date": "2019-11-29",
"news": [
{
"date": "2019-11-29",
"title": "龚虹嘉退胡扬忠进 海康威视</em>玩“对敲”?",
"url": "http://finance.sina.com.cn/stock/s/2019-11-29/doc-iihnzhfz2363840.shtml",
"source": "新浪",
"same_news": 3
},
{
"date": "2019-11-29",
"title": "海康威视</em>:融资净偿还110.94万元,融资余额16.31亿元(11-28)",
"url": "http://stock.eastmoney.com/a/201911291307749335.html",
"source": "东方财富网",
"same_news": 1
},
{
"date": "2019-11-29",
"title": "投资者提问:标普:将海康威视</em>移出观察名单,因供应链恢复。海康是被...",
"url": "http://finance.sina.com.cn/stock/relnews/dongmiqa/2019-11-29/doc-iihnzhfz2562007.shtml?source",
"source": "新浪",
"same_news": 1
}
]
},
{
"date": "2019-12-02",
"news": [
{
"date": "2019-12-02",
"title": "海康威视</em>:关于部分国有股权无偿划转完成的公告",
"url": "http://finance.sina.com.cn/roll/2019-12-02/doc-iihnzhfz3131565.shtml",
"source": "新浪",
"same_news": 3
},
{
"date": "2019-12-02",
"title": "海康威视</em>如何构建智慧农业的基石?",
"url": "http://www.afzhan.com/news/detail/79064.html",
"source": "中国安防展览网",
"same_news": 1
},
{
"date": "2019-12-02",
"title": "[公司]海康威视</em>:实控人完成划转公司0.22%国有股权",
"url": "http://www.p5w.net/kuaixun/201912/t20191202_2358966.htm",
"source": "全景网",
"same_news": 1
}
]
}
]
},
"message": ""
}格式化解析结果如下：
['date', 'title', 'url', 'source', 'same_news']
[u'2019-11-20', u'海康威视</em>高级副总经理蒋海青辞职 年薪为248.62万元', u'http://finance.eastmoney.com/a/201911201296670767.html', u'东方财富网', 3]
[u'2019-11-20', u'...高新技术企业创新能力百强榜发布,士兰微、新华三、海康威视</em>等...', u'https://laoyaoba.com/html/share/news?source', u'集微网', 1]
[u'2019-11-20', u'海康威视</em>与国网杭州供电公司签订战略合作协议', u'http://www.afzhan.com/news/detail/78811.html', u'中国安防展览网', 1]
[u'2019-11-22', u'海康威视</em>:融资净偿还8784.07万元,两市排名第三(11-21)', u'http://stock.eastmoney.com/a/201911221299836442.html', u'东方财富网', 2]
[u'2019-11-22', u'后安防时代,海康威视</em>的信心与底气在哪? ', u'http://www.techsir.com/a/201911/59129.html', u'techsir', 2]
[u'2019-11-25', u'海康威视</em>闪耀2019首届四川教育装备博览会 ', u'http://www.ceiea.com/html/201911/201911251324011299.shtml', u'中国教育..', 1]
[u'2019-11-25', u'湖南广电、网易、海康威视</em>都来了!浙传这场招聘会让2000多人达成...', u'http://edu.zjol.com.cn/jyjsb/gx/201911/t20191125_11397473.shtml', u'浙江在线', 1]
[u'2019-11-25', u'海康威视</em>龚虹嘉的财富密码:十年暴赚两万倍后,为何被查?', u'http://finance.sina.com.cn/stock/relnews/cn/2019-11-25/doc-iihnzahi3264395.shtml', u'新浪', 1]
[u'2019-11-29', u'龚虹嘉退胡扬忠进 海康威视</em>玩“对敲”?', u'http://finance.sina.com.cn/stock/s/2019-11-29/doc-iihnzhfz2363840.shtml', u'新浪', 3]
[u'2019-11-29', u'海康威视</em>:融资净偿还110.94万元,融资余额16.31亿元(11-28)', u'http://stock.eastmoney.com/a/201911291307749335.html', u'东方财富网', 1]
[u'2019-11-29', u'投资者提问:标普:将海康威视</em>移出观察名单,因供应链恢复。海康是被...', u'http://finance.sina.com.cn/stock/relnews/dongmiqa/2019-11-29/doc-iihnzhfz2562007.shtml?source', u'新浪', 1]
[u'2019-12-02', u'海康威视</em>:关于部分国有股权无偿划转完成的公告', u'http://finance.sina.com.cn/roll/2019-12-02/doc-iihnzhfz3131565.shtml', u'新浪', 3]
[u'2019-12-02', u'海康威视</em>如何构建智慧农业的基石?', u'http://www.afzhan.com/news/detail/79064.html', u'中国安防展览网', 1]
[u'2019-12-02', u'[公司]海康威视</em>:实控人完成划转公司0.22%国有股权', u'http://www.p5w.net/kuaixun/201912/t20191202_2358966.htm', u'全景网', 1]简单的小实践。
