爬取网站所有Python书籍到数据库
如果大家没有看第一天的课程,必须去看一下,不然看不到今天的课程的!
点我查看第一天内容:Scrapy框架的安装与创建 Scrapy从入门到精通第一天
一、课程介绍
今天我做的项目是利用Scrapy框架爬取当当网站Python相关书籍到数据库
今天的目标是:
一、获取当当网所有有关python书籍的名字
二、获取当当网所有有关python书籍的链接
三、获取当当网所有有关python书籍的评论数量
四、将获取到的数据存入到数据库中
五、最终结果
1、获取当当网域名
这个就比较容易啦,百度搜索就出来了
域名是“dangdang.com”注意域名是不带http://www.的
2、获取爬取内容的首页
我们只要是在上面粘贴就好啦,我们的起始网址是:
http://search.dangdang.com/?key=python&act=input
所有我们的爬虫文件应该这样写
这里我们用的匹配方式是XPath表达式,大家使用正则表达式也可以,
3、编写爬虫文件,这里我将爬虫命名为fst.py
上一章节有介绍生成爬虫项目和爬虫文件的介绍
# -*- coding: utf-8 -*-
import scrapy
from demo.items import DemoItem
from scrapy.http import Request
#scrapy crawl fst运行
class FstSpider(scrapy.Spider):
name = 'fst' #爬虫名称
allowed_domains = ['dangdang.com'] #可爬取的域名
start_urls = ['http://search.dangdang.com/?key=python&act=input'] #起始网址
def parse(self, response): #回调函数response相应信息
item = DemoItem()
item["title"] = response.xpath("//a[@name='itemlist-title']/@title").extract()
item["link"] = response.xpath("//a[@name='itemlist-title']/@href").extract()
item["comment"] = response.xpath("//a[@name='itemlist-review']/text()").extract()
yield item #跳转到pipelines.py文件在这个文件中将数据保存到数据库
for i in range(2,101):#获取100页的内容
url = "http://search.dangdang.com/?key=python&act=input&page_index=" + str(i) #实现翻页功能
yield Request(url,callback=self.parse)
4、编写:pipelines.py文件
将数据保存到数据库需要安装pymysql,python的一个模块。
控制台输入pip install pymysql 即可安装成功
安装成功还要解决编码的问题,因为源文件"charset="默认是空的,咱们找到他并加上utf8
文件修改方法位置:
修改python环境目录下lib/site-packages/pymysql/connections.py#不知道的环境在哪的小伙伴就去搜索pymysql就行,就是慢点
#进入connectioins.py搜索"charset="默认是空的,咱们加上utf8
# -*- coding: utf-8 -*-
import pymysql
class DemoPipeline(object):
def process_item(self, item, spider):
#创建数据库链接
conn = pymysql.connect(host="127.0.0.1",user="root",passwd="123",db="test")
for i in range(0,len(item["title"])):
#将爬取的信息数组分离,存储在单个文件中
title = item["title"][i]
link = item["link"][i]
comment = item["comment"][i]
#数据库插入语句
sql="insert into books(title,link,comment) values('" + title + " ',' " + link + " ',' " + comment + "')"
conn.query(sql)
conn.commit()
conn.close()
return item
5、最后一步更改配置文件settings.py
这里不用自己写代码,这就是框架的好处,
我们只需要将67,68,69三行代码的注释解开即可
当当网不需要服务器伪装,所以这里我就没写服务器伪装
服务器伪装的话主要修改下代码里面的USER_AGENT即可
# -*- coding: utf-8 -*-
# Scrapy settings for demo project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'demo'
SPIDER_MODULES = ['demo.spiders']
NEWSPIDER_MODULE = 'demo.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
# USER_AGENT = 'demo (+http://www.yourdomain.com)'
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
# 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
# 'Accept-Language': 'en',
#}
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'demo.middlewares.DemoSpiderMiddleware': 543,
#}
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# 'demo.middlewares.DemoDownloaderMiddleware': 543,
#}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
'demo.pipelines.DemoPipeline': 300,
}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
二、获取源代码
代码代写(实验报告、论文、小程序制作)服务请加微信:ppz2759
关注下方公众号,回复“爬虫130”即可获取源代码