Scrapy installation instructions
Scrapy install a variety of ways, and it supports Python2.7 version Python3.3 version or above and more. The installation process will be described in Python3 environment.
Scrapy dependent libraries are more dependent libraries need to have at least Twisted 14.0, lxml 3.4, pyOpenSSL 0.14. And it varies in different platform environment, so before installation to ensure the best library to install some basic good, especially Windows.
Tencent cloud has Windows, Linux multiple versions of the system presented here separately.
Each platform installation Introduction
Anaconda
This method is a relatively simple method of installation of Scrapy (especially for Windows), you can use this method to install, the installation method below can also use a dedicated platform.
Anaconda is the inclusion of common Python releases data library science, if not installed, you can download the package to https://www.continuum.io/downloads install the corresponding platform.
If you have installed, you can easily install by conda command Scrapy.
Installation command as follows:
conda install Scrapy
Windows
Installation lxml
The best way is to install the installation file through the wheel, http://www.lfd.uci.edu/~gohlke/pythonlibs/, lxml find the relevant documents from the site. If it is Python3.5 version, WIndows 64-bit system, then find lxml-3.7.2-cp35-cp35m- win_amd64.whl and download the file, and then install pip.
Once downloaded, run the following command to install:
Wheel install PIP3
PIP3 install lxml-3.7.2-cp35-cp35m-win_amd64.whl
to complete the installation lxml other files replace the file name.
Installation zope.interface
to the official website to download the corresponding version of the wheel https://pypi.python.org/pypi/zope.interface#downloads file, and then install pip. For example, Python 3.5 version, Windows 64-bit systems, download zope.interface-4.3.3-cp35-cp35m- win_amd64.whl.
Files can then install the wheel, the command is as follows:
pip3 install zope.interface 4.3.3-cp35-cp35m -win_amd64.whl-
other versions are replaced with the file name.
Installation pyOpenSSL
official website to download the file wheel, https://pypi.python.org/pypi/pyOpenSSL#downloads, such as the latest version of the current name is pyOpenSSL-16.2.0-py2.py3-none- any.whl, download and install that is can.
pip3 install pyOpenSSL-16.2.0-py2.py3- none-any.whl
installation Twisted
Similarly, http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted wheel download files, you can use pip install .
Such as Python 3.5 version, Windows 64-bit systems, download
Twisted-17.1.0-cp35-cp35m- win_amd64.whl, then pip install.
pip3 install Twisted-17.1.0-cp35- cp35m-win_amd64.whl
installation pywin32
from the official website https://sourceforge.net/projects/pywin32/files/pywin32/Build 220 / download the corresponding version of the installation package can be installed.
Installation Scrapy
final installation Scrapy can still use the pip, the command is as follows:
pip3 install Scrapy
CentOS、RedHat、Fedora
Dependent libraries are installed
to ensure that some of the necessary libraries installed, run the following command:
Development Tools yum groupinstall the sudo
the sudo yum the install python34 devel-EPEL the libxslt-devel-Release OpenSSL the libxml2-devel devel-
mounted Scrapy
using pip Scrapy to install, run the following command:
pip3 install Scrapy
Ubuntu、Debian、Deepin
Dependent libraries are installed
first make sure that some of the necessary libraries installed, run the following command:
sudo apt-get install build-essential python3-dev libssl-dev libffi-dev libxml2 libxml2-dev libxslt1-dev zlib1g-dev
mounted Scrapy
using pip Scrapy to install, run the following command:
pip3 install Scrapy
although Tencent cloud no Mac, but here or MacOS plus installation.
Mac OS
Dependent libraries installed
on the Mac build Scrapy dependent libraries need a C compiler and development header file, which is generally provided by Xcode, you can run the following command to install:
xcode-select --install
mounted Scrapy
using pip Scrapy to install, run the following command:
pip3 install Scrapy
verification
after installation, enter scrapy at the command line, if the result is similar to the bottom of, and certification Scrapy successful installation.
Common Errors
pkg_resources.VersionConflict: (Six 1.5.2 (/ usr / lib / to python3 / dist-Packages), Requirement.parse ( 'Six> = 1.6.0'))
Six package version is too low, six pack is to provide a compatible and Python2 Python3 library, you can upgrade six pack.
-U Six PIP3 install sudo
c / _cffi_backend.c: 15: 17: fatal error: ffi.h: No SUCH File or Directory
lack Libffi this library. What is libffi? "FFI" full name is the Foreign Function Interface, usually it refers to the code to allow the calling code written in one language in another language. The Libffi library provides only the lowest level, related to architecture, complete "FFI".
Install the appropriate library can be.
Ubuntu、Debian：
sudo apt-get install build-essential libssl-dev libffi-dev python3-dev
CentOS、RedHat:
GCC libffi-yum the install the sudo devel Python OpenSSL-devel-devel
ImportError: No Module1 the named 'Cryptography'
This is the lack of encryption related components, can be installed using a pip.
Cryptography PIP3 install sudo
ImportError: No Module named 'packaging'
lack of packaging this package, which provides the core functionality of Python packages, use pip install.
Packaging PIP3 the install the sudo
ImportError: No Module1 the named 'appdirs'
missing appdirs the package, which is used to determine the file directory by pip can be installed separately.
sudo pip3 install appdirs