最终要实现的是将当当网上面的书籍信息,书籍名字,网址和评论数爬取,存入到数据库中。(首先要做的是创建好数据库,创建的数据库名字为dd,创建的表为books,字段为title,link,comment)。
1、创建项目 scrapy startproject dangdang
2、进入项目文件夹创建爬虫文件
>scrapy genspider –t basic dd dangdang.com
3、用pycharm打开这个项目
编辑items.py文件
# -*- coding: utf-8 -*-
# Define here the models for your scraped items
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html
import scrapy
class DangdangItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
title=scrapy.Field()
link=scrapy.Field()
comment=scrapy.Field()
编辑dd.py
# -*- coding: utf-8 -*-
import scrapy
from dangdang.items import DangdangItem
from scrapy.http import Request
class DdSpider(scrapy.Spider):
name = 'dd'
allowed_domains = ['dangdang.com']
start_urls = ['http://dangdang.com/']
def parse(self, response):
item=DangdangItem()
item['title']=response.xpath('//a[@class="pic"]/@title').extract()
item['link'] = response.xpath('//a[@class="pic"]/@href').extract()
item['comment'] = response.xpath('//a[@class="search_comment_num"]/text()').extract()
yield item
for i in range(2,101):#循环爬多页的东西
url='http://category.dangdang.com/pg'+str(i)+'-cp01.54.06.00.00.00.html'
yield Request(url,callback=self.parse)
在seetings.py文件中打开pipelines
ITEM_PIPELINES = {
'dangdang.pipelines.DangdangPipeline': 300,
}
Pipelines.py文件,将数据写入数据库
# -*- coding: utf-8 -*-
# Define your item pipelines here
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
import pymysql
class DangdangPipeline(object):
def process_item(self, item, spider):
conn=pymysql.connect(host='localhost',port=3306,user='root',passwd='123456',db='dd')
for i in range(0,len(item['title'])):
title=item['title'][i]
link=item['link'][i]
comment=item['comment'][i]
sql="insert into books(title,link,comment)values('"+title+"','"+link+"','"+comment+"')"
conn.query(sql)
conn.commit()
conn.close()
return item