Windows安装scrapy报错解决办法
在Windows系统中，如果我们直接使用pip install scrapy 进行安装会报一个错误
报错信息如下：
Using legacy 'setup.py install' for Twisted, since package 'wheel' is not installed.
Using legacy 'setup.py install' for protego, since package 'wheel' is not installed.
Using legacy 'setup.py install' for PyDispatcher, since package 'wheel' is not installed.
Installing collected packages: pycparser, cffi, cryptography, attrs, pyasn1, pyasn1-modules, service-identity, zope.interface, pyOpenSSL, constantly, increm
ental, Automat, hyperlink, PyHamcrest, Twisted, itemadapter, cssselect, w3lib, parsel, protego, queuelib, jmespath, itemloaders, PyDispatcher, scrapy
Running setup.py install for Twisted ... error
ERROR: Command errored out with exit status 1:
command: 'd:\python_projects\lemtest\venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\PC\\AppData\\Local
\\Temp\\pip-install-fex100n2\\twisted\\setup.py'"'"'; __file__='"'"'C:\\Users\\PC\\AppData\\Local\\Temp\\pip-install-fex100n2\\twisted\\setup.py'"'"';f=geta
ttr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' i
nstall --record 'C:\Users\PC\AppData\Local\Temp\pip-record-y7lsfj81\install-record.txt' --single-version-externally-managed --compile --install-headers 'd:\
python_projects\lemtest\venv\include\site\python3.8\Twisted'
cwd: C:\Users\PC\AppData\Local\Temp\pip-install-fex100n2\twisted\
......中间省略若干
running build_ext
building 'twisted.test.raiser' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-to
ols/
----------------------------------------
ERROR: Command errored out with exit status 1: 'd:\python_projects\lemtest\venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '
"'"'C:\\Users\\PC\\AppData\\Local\\Temp\\pip-install-fex100n2\\twisted\\setup.py'"'"'; __file__='"'"'C:\\Users\\PC\\AppData\\Local\\Temp\\pip-install-fex100
n2\\twisted\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(co
de, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\PC\AppData\Local\Temp\pip-record-y7lsfj81\install-record.txt' --single-version-externally-managed
--compile --install-headers 'd:\python_projects\lemtest\venv\include\site\python3.8\Twisted' Check the logs for full command output.
解决上面的错误，我们需要按照如下步骤：
-
在https://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml 中下载对应Python版本与系统的
Twisted包，然后用pip 进行安装
-
在https://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml 中下载对应Python版本与系统的
lxml包，然后用pip进行安装
-
使用
pip install scrapy进行安装scrapy,安装完成后在命令行输入scrapy检查安装是否成功，若未报错，则安装成功
# 安装lxml
(venv) D:\Python_Projects>pip install d:\download\lxml-4.6.1-cp38-cp38-win_amd64.whl
Looking in indexes: https://mirrors.aliyun.com/pypi/simple
Processing d:\python_projects\lemtest\lxml-4.6.1-cp38-cp38-win_amd64.whl
Installing collected packages: lxml
Successfully installed lxml-4.6.1
# 安装Twisted
(venv) D:\Python_Projects>pip install d:\download\Twisted-20.3.0-cp38-cp38-win_amd64.whl
Looking in indexes: https://mirrors.aliyun.com/pypi/simple
Processing d:\python_projects\lemtest\twisted-20.3.0-cp38-cp38-win_amd64.whl
Requirement already satisfied: zope.interface>=4.4.2 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (5.1.2)
Requirement already satisfied: Automat>=0.3.0 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (20.2.0)
Requirement already satisfied: constantly>=15.1 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (15.1.0)
Requirement already satisfied: hyperlink>=17.1.1 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (20.0.1)
Requirement already satisfied: attrs>=19.2.0 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (20.2.0)
Requirement already satisfied: PyHamcrest!=1.10.0,>=1.9.0 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (2.0.2)
Requirement already satisfied: incremental>=16.10.1 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (17.5.0)
Requirement already satisfied: setuptools in d:\python_projects\lemtest\venv\lib\site-packages (from zope.interface>=4.4.2->Twisted==20.3.0) (50.3.2)
Requirement already satisfied: six in d:\python_projects\lemtest\venv\lib\site-packages (from Automat>=0.3.0->Twisted==20.3.0) (1.15.0)
Requirement already satisfied: idna>=2.5 in d:\python_projects\lemtest\venv\lib\site-packages (from hyperlink>=17.1.1->Twisted==20.3.0) (2.10)
Installing collected packages: Twisted
Successfully installed Twisted-20.3.0
# 安装scrapy
(venv) D:\Python_Projects>pip install scrapy
Looking in indexes: https://mirrors.aliyun.com/pypi/simple
Collecting scrapy
Using cached https://mirrors.aliyun.com/pypi/packages/fd/49/425c39549e277510d5cbc9f8f6ae36e1d4faf9fd768b613af1dd6b786c60/Scrapy-2.4.0-py2.py3-none-any.whl
(239 kB)
Requirement already satisfied: protego>=0.1.15 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (0.1.16)
Requirement already satisfied: pyOpenSSL>=16.2.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (19.1.0)
Requirement already satisfied: service-identity>=16.0.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (18.1.0)
Requirement already satisfied: cssselect>=0.9.1 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (1.1.0)
Requirement already satisfied: Twisted>=17.9.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (20.3.0)
Requirement already satisfied: parsel>=1.5.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (1.6.0)
Requirement already satisfied: queuelib>=1.4.2 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (1.5.0)
Requirement already satisfied: w3lib>=1.17.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (1.22.0)
Requirement already satisfied: itemadapter>=0.1.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (0.1.1)
Requirement already satisfied: lxml>=3.5.0; platform_python_implementation == "CPython" in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (
4.6.1)
Requirement already satisfied: itemloaders>=1.0.1 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (1.0.3)
Requirement already satisfied: PyDispatcher>=2.0.5 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (2.0.5)
Requirement already satisfied: zope.interface>=4.1.3 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (5.1.2)
Requirement already satisfied: cryptography>=2.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (3.2)
Requirement already satisfied: six in d:\python_projects\lemtest\venv\lib\site-packages (from protego>=0.1.15->scrapy) (1.15.0)
Requirement already satisfied: pyasn1 in d:\python_projects\lemtest\venv\lib\site-packages (from service-identity>=16.0.0->scrapy) (0.4.8)
Requirement already satisfied: attrs>=16.0.0 in d:\python_projects\lemtest\venv\lib\site-packages (from service-identity>=16.0.0->scrapy) (20.2.0)
Requirement already satisfied: pyasn1-modules in d:\python_projects\lemtest\venv\lib\site-packages (from service-identity>=16.0.0->scrapy) (0.2.8)
Requirement already satisfied: hyperlink>=17.1.1 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted>=17.9.0->scrapy) (20.0.1)
Requirement already satisfied: incremental>=16.10.1 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted>=17.9.0->scrapy) (17.5.0)
Requirement already satisfied: constantly>=15.1 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted>=17.9.0->scrapy) (15.1.0)
Requirement already satisfied: Automat>=0.3.0 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted>=17.9.0->scrapy) (20.2.0)
Requirement already satisfied: PyHamcrest!=1.10.0,>=1.9.0 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted>=17.9.0->scrapy) (2.0.2)
Requirement already satisfied: jmespath>=0.9.5 in d:\python_projects\lemtest\venv\lib\site-packages (from itemloaders>=1.0.1->scrapy) (0.10.0)
Requirement already satisfied: setuptools in d:\python_projects\lemtest\venv\lib\site-packages (from zope.interface>=4.1.3->scrapy) (50.3.2)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in d:\python_projects\lemtest\venv\lib\site-packages (from cryptography>=2.0->scrapy) (1.14.3)
Requirement already satisfied: idna>=2.5 in d:\python_projects\lemtest\venv\lib\site-packages (from hyperlink>=17.1.1->Twisted>=17.9.0->scrapy) (2.10)
Requirement already satisfied: pycparser in d:\python_projects\lemtest\venv\lib\site-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.0->scrapy) (2.20)
Installing collected packages: scrapy
Successfully installed scrapy-2.4.0
# 验证是否安装成功
(venv) D:\Python_Projects>scrapy
Scrapy 2.4.0 - no active project
Usage:
scrapy <command> [options] [args]
Available commands:
bench Run quick benchmark test
commands
fetch Fetch a URL using the Scrapy downloader
genspider Generate new spider using pre-defined templates
runspider Run a self-contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version Print Scrapy version
view Open URL in browser, as seen by Scrapy
[ more ] More commands available when run from project directory
Use "scrapy <command> -h" to see more info about a command