什么是爬虫
按照一定规则自动的获取互联网上的信息(如何快速有效的利用互联网上的大量信息)
爬虫的应用
- 搜索引擎(Google、百度、Bing等搜索引擎,辅助人们检索信息)
- 股票软件(爬取股票数据,帮助人们分析决策,进行金融交易)
- Web扫描(需要对网站所有的网页进行漏洞扫描)
- 获取某网站最新文章收藏
- 爬取天气预报
- 爬取漂亮mm照片
基础知识
1.HTTP 协议
客户端发起请求,服务器接收到请求后返回格式化的数据,客户端接收数据,并进行解析和处理
2.HTML(超文本标记语言)
- 基础语法&常用系统模块
- 第三方模块requests,pyquery使用
安装:
pip install requests
pip install pyquery
requests模块使用:
#requests(发起HTTP请求,并获取结果) response = requests.get('http://localhost:9999/index.html') response = requests.post() print response.content
pyquery模块使用:
page = PyQuery(html) 选择器 tag: page('title') id: page('#job_1') class: page('.job') 复合选择器 page('div#job_1') page('div.job') 子选择器 page('div#job_1 li') page('div#job_1 > li') page('div#job_1').find('li') page('div#job_1').children('li') 获取标签内的html page('div#job_1').html() 获取标签内的文本 page('div#job_1').text() 获取标签属性 page('div#job_1').attr['id']
csv模块使用:
writer = csv.writer()
writer.writerow()
writer.writerows()
程序运行
1.程序启动
2.运行结果
1.获取电影列表
2.获取电影详情超链接
3.获取电影详情
1.程序启动
2.查找电影列表
3.查找电影详情
4.写入csv文件
#encoding: utf-8 import requests from pyquery import PyQuery as pq import csv attrs = [u'超链接', u'名称', u'评分', u'导演', u'编剧', u'主演', u'类型', u'制片国家/地区', u'语言', u'上映日期', u'片长', u'又名', u'IMDb链接'] ''' 获取电影详情 ''' def attch_info(info, text, key, value): text = text.strip(' ') if text: if text in attrs: if key and value: info[key] = ' '.join(value) key = text value = [] else: value.append(text) return info, key, value ''' 解析电影信息 ''' def parse_movie_info(text, info): key = None value = [] for e in text.split(':'): e = e.strip() pos = e.rfind(' ') if -1 == pos: info, key, value = attch_info(info, e, key, value) else: info, key, value = attch_info(info, e[:pos], key, value) info, key, value = attch_info(info, e[pos:], key, value) if key not in info: info[key] = ' '.join(value) ''' 解析电影页面 ''' def crawl_info(url): info = {} print url response = requests.get(url) page = pq(response.content) content = page('div#content').eq(0) info[u'超链接'] = url info[u'名称'] = content('h1 span').eq(0).text() info[u'评分'] = content('div.rating_wrap strong.rating_num').text() info_text = content('div#info').text() parse_movie_info(info_text, info) return info ''' 获取电影列表 ''' def crawl(query_text, count): start = 0 rt_list = [] isStop = False url = 'https://movie.douban.com/subject_search?start={start}&search_text={query_text}&cat=1002' while True: response = requests.get(url.format(query_text=query_text.encode('utf-8', 'ignore'), start=start)) page = pq(response.content) links = page('div#content table a').not_('.nbg') if len(links) == 0: isStop = True for link in links: href = pq(link).attr['href'] rt_list.append(crawl_info(href)) start += 1 if len(rt_list) >= count: isStop = True break if isStop: break return rt_list ''' 写入文件 ''' def write_to_file(lines, path): with open(path, 'wb') as fhandler: writer = csv.writer(fhandler) writer.writerow(map(lambda x: x.encode('gbk', 'ignore'), attrs)) for line in lines: row = [] for key in attrs: row.append(line.get(key, '').encode('gbk', 'ignore')) writer.writerow(row) if __name__ == '__main__': query_text = raw_input(u"请输入关键字:".encode('utf-8', 'ignore')) count = raw_input(u"请输入爬取得数据量:".encode('utf-8', 'ignore')) query_text = query_text.strip().decode('utf-8') if query_text.strip() else u'长城' count = int(count) if count.isdigit() else 10 print u'关键字:{query_text}, 数量:{count}'.format(query_text=query_text, count=count) rt_list = crawl(query_text, count) write_to_file(rt_list, 'result.csv')
作者:imsilence
链接:https://www.jianshu.com/p/7eceedb39f3b