1: Install Centos system first:
Why choose CentOs system instead of Ubuntu?
I tried to install python and Scrapy three times on Ubuntu, and the results were unsuccessful. I used CentOs under the suggestion of the operation and maintenance old man;
(I can't help but complain, the python installation package dependency package is really hard to install! )
2: CentOs installation process
The overall installation process is relatively smooth, much simpler than Ubuntu; I chose a mini ios and there is no desktop environment after installation, which is very good and meets the needs.
Interface after installation: command line environment, very good.
3: Use MobaXterm SSH tool to link CentOs
Need to know the ip address of the CentOs system:
Then go to check ifconfig -a. The result is that there is no such command. It is understandable because the mini os is installed. Some of them are not understandable. If not, just install it by yourself;
CentOS 7 can't find 'ifconfig' command after minimal install - fix tip
CentOS7 uses yum to install software prompts cannot find a valid baseurl for repobase7x86_64
Both of the above are used and tested and can be used;
After the link is on, it will look like the following image:
4 Next, let's see if the installation of python goes well?
Before installing python, check if the system has its own python
It seems that he comes with python 2.7; what I need here is python 3.6
Then install python3.6 yourself and set it as the system default
The following method can only be used for centos7 or el7 kernel
yum -y update
yum -y install https://centos7.iuscommunity.org/ius-release.rpm
to install the latest version Python3.6.x
yum -y install ncurses-devel gdbm- devel xz-devel tk-devel
yum -y install gcc openssl-devl mysql-devl zlib-devl
yum -y install python36u python36u-libs python36u-pip python36u-devel
Create soft link
ln -s /usr/bin/python3.6 /usr/bin/python3
ln -s /usr/bin/pip3.6 /usr/bin/pip3
pip3 install --upgrade pip
In addition, Python 3.6 can directly use python3 -m venv venv_name to create a virtual environment
Next, create a virtual environment
python3 -m venv scrapy_venv
Activate the virtual environment
source /opt/python_venv/scrapy_venv/bin/activate
pip install -r requirement.txt
File below: requirement.txt
asn1crypto==0.24.0 attrs==17.4.0 Automat==0.6.0 certifi==2018.1.18 cffi==1.11.5 chardet==3.0.4 constantly==15.1.0 cryptography==2.1.4 cssselect==1.0.3 hyperlink==18.0.0 idna==2.6 incremental==17.5.0 lxml==4.1.1 numpy==1.14.1 opencv-python==3.4.0.12 parsel==1.4.0 Pillow==5.0.0 pyasn1==0.4.2 pyasn1-modules==0.2.1 pycparser==2.18 PyDispatcher==2.0.5 PyMySQL==0.8.0 pyOpenSSL==17.5.0 pytesseract==0.2.0 python-logstash==0.4.6 qiniu==7.2.0 queuelib==1.4.2 redis==2.10.6 requests==2.18.4 Scrapy==1.5.0 scrapy-splash==0.7.2 scrapyd-client==1.1.0 service-identity==17.0.0 six==1.11.0 Twisted==17.9.0 urllib3==1.22 w3lib==1.19.0 zope.interface==4.4.3
So far basically completed .