System environment: Redhat6.5 system
1. Add scrapy environment dependencies
yum install -y python-devel libxml2 zlib-devel bzip2-devel xz-libs xz wget git tar gcc gcc-c++ openssl openssl-devel pcre-devel python-devel libevent automake autoconf libtool make git sqlite-devel
2、升级python
Redhat6.5 comes with python2.6.6, the latest version of scrapy is based on python2.7 and needs to be upgraded to python2.7.12
Available for download at https://www.python.org/ftp/python/2.7.12/Python-2.7.12.tgz
Unzip and install
wget https://www.python.org/ftp/python/2.7.12/Python-2.7.12.tgz tar -xvf Python-2.7.12.tar cd Python-2.7.12 ./configure --prefix=/usr/local/python2.7.12 make && make install
Modify the system default python to the latest installed version
mv /usr/bin/python /usr/bin/python2.6.6 ln -s /usr/local/python2.7.12/bin/python2.7 /usr/bin/python
Using the latest version of python may cause the system's yum function to malfunction, you need to restore the python point of yum
vim /usr/bin/yum Will #!/usr/bin/python change into #!/usr/bin/python2.6.6
3. Install setuptools
Search for the setuptools installation package on the https://pypi.python.org/ website. I downloaded and installed setuptools-26.0.0.tar.gz
Install after decompression
cd setuptools-26.0.0 python setup.py install
4. Install pip
Search for the pip installation package on the https://pypi.python.org/ website, I downloaded and installed pip-8.1.2.tar.gz
Install after decompression
cd pip-8.1.2 python setup.py install
5. Set python commands to use directly
vim /etc/profile add python path export PATH=$HOME/bin:/usr/local/python2.7.12/bin:/usr/local/bin:$PATH Reload after modification takes effect source /etc/profile
6. Install scrapy
pip install Scrapy
At this point scrapy can be used.