Install scrapy on Ubuntu 14

Scrapy, a fast, high-level screen scraping and web scraping framework developed in Python for scraping web sites and extracting structured data from pages. Scrapy is versatile and can be used for data mining, monitoring, and automated testing .

Installation steps under Ubuntu14:

Environmental preparation

（1）Ubuntu 14.04

(2) Python 2.7.6 (included with the system)

 （3）OpenSSL

Check if it is installed

If not installed

Install by command  sudo apt-get install openssl

（4） lxml







If not installed

Install by command  sudo apt-get install xml2

（5）Installing libevent-dev  libeventis a high-performance network library from time to time, which is used at the bottom of many frameworkslibevent

sudo apt-get install libevent-dev

（6） 安装 scrapy

sudo pip install scrapy





The installation is successful, you can check the scrapy version with the command

Problems with installation

1. sudo pip install scrapy error

Solution

Add the file ~/.pip/pip.conf and add the content 

[global]  
index-url=http://pypi.douban.com/simple/

Operation command

cd ~/.pip

 sudo vim pip.conf

Open vim editor, type i

Enter the contents of the file, then press Esc, then Shift+: to enter wq.

Execute sudo pip install scrapy again and the operation is successful.

2. Error executing scrapy version

Because of the TWIST version problem, you need to install a lower version

Execute sudo pip install Twisted==10.0.0