scrapy框架的安装配置

Scrapy框架安装配置

1.创建一个虚拟环境,在虚拟环境中进行操作

安装:
pip install virtualenv
​
创建:
到对应目录下:
virtualenv 虚拟环境名称

  进入虚拟环境:

D:\virtu\spiderenv\Scripts>activate

2.安装twiste依赖,

https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted

到该网站下载对应python版本对应位数的twisted,下载后

pip install 路径\twisted包名

3.安装pywin32

https://www.lfd.uci.edu/~gohlke/pythonlibs/#pywin32

下载对应python版本的安装包后安装:

pip install 路径\pywin32包名

或者直接安装

安装后在虚拟环境中运行python,输入import win32com ,如果没有报错,则说明安装成功

D:\virtu\spiderenv\Scripts>python
Python 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 13:35:33) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import win32com

4.安装lxml

pip install lxml

5.安装Scrapy

pip install scrapy

6.验证安装是否成功,如下情况则为安装成功

(spiderenv) D:\virtu\spiderenv\Scripts>scrapy
Scrapy 1.5.1 - no active project
​
Usage:
  scrapy <command> [options] [args]
​
Available commands:
  bench         Run quick benchmark test
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy
​
  [ more ]      More commands available when run from project directory
​
Use "scrapy <command> -h" to see more info about a command