mac系统安装：
1.先下载https://bootstrap.pypa.io/get-pip.py，然后直接sudo python get-pip.py，pip就安装好了。
cd到下载目录
sudo python get-pip.py2.pyspider安装
sudo pip install pyspider＃安装 ＃卸载pip uninstall pyspider
安装报错：
Collecting pycurl (from pyspider)
Downloading https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz (214kB)
100% |████████████████████████████████| 215kB 12kB/s
Complete output from command python setup.py egg_info:
Using curl-config (libcurl 7.30.0)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/tmp/pip-install-uJlOEi/pycurl/setup.py", line 913, in <module>
ext = get_extension(sys.argv, split_extension_source=split_extension_source)
File "/private/tmp/pip-install-uJlOEi/pycurl/setup.py", line 582, in get_extension
ext_config = ExtensionConfiguration(argv)
File "/private/tmp/pip-install-uJlOEi/pycurl/setup.py", line 99, in __init__
self.configure()
File "/private/tmp/pip-install-uJlOEi/pycurl/setup.py", line 316, in configure_unix
specify the SSL backend manually.''')
__main__.ConfigurationError: Curl is configured to use SSL, but we have not been able to determine which SSL backend it is using. Please see PycURL documentation for how to specify the SSL backend manually.
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/tmp/pip-install-uJlOEi/pycurl/
解决：参考https://blog.csdn.net/mr_yang__/article/details/81109348
sudo python -m pip install --upgrade --force pip＃强制重新安装更新pip
sudo pip install pycurl==7.43.0＃安装的时候的报错可能因为下载的是7.43，但自动用7.30配置安装了所以报错（Using curl-config (libcurl 7.30.0)），所以指定下pycurl版本安装
＃问题解决后重新安装pyspider
sudo pip install pyspider
3.启动pyspider
zdjdeMacBook-Pro:~ zdj$ pyspider
Traceback (most recent call last):
File "/usr/local/bin/pyspider", line 5, in <module>
from pkg_resources import load_entry_point
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 2603, in <module>
working_set.require(__requires__)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 666, in require
needed = self.resolve(parse_requirements(requirements))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 565, in resolve
raise DistributionNotFound(req) # XXX put more info here
pkg_resources.DistributionNotFound: pyquery
zdjdeMacBook-Pro:~ zdj$ pyspider -all
Traceback (most recent call last):
File "/usr/local/bin/pyspider", line 5, in <module>
from pkg_resources import load_entry_point
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 2603, in <module>
working_set.require(__requires__)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 666, in require
needed = self.resolve(parse_requirements(requirements))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 565, in resolve
raise DistributionNotFound(req) # XXX put more info here
pkg_resources.DistributionNotFound: pyquery
报错解决：参考https://blog.csdn.net/patrickzheng/article/details/73478082
sudo pip install -U setuptools4.退出pyspider，mac下control＋c；linux下control＋z
linux系统安装：
python系统版本是2.6，（输入python查看系统python版本），网上说2.6版本无法打开pyspider的webui，但我还是想尝试下，开始了2.6下安装，结果还是以安装失败告终，最后还是在2.7版本下安装成功，具体看下面：
1.pip安装
yum install python-pip ＃安装pip时，没有setuptools会自动安装，安装后如果单独升级setuptools，会升级到高版本，高版本只支持python2.7以上版本，会造成pip命令无法使用，后续有详诉
===================================================================================================================================================================================
Installing:
python-pip noarch 7.1.0-1.el6 epel 1.5 M
Installing for dependencies:
python-setuptools noarch 0.6.10-4.el6_9 os 336 k
2.pyspider安装
＃pip install pyspider＃安装中报错，缺少libxslt开发环境
** make sure the development packages of libxml2 and libxslt are installed **
Using build configuration of libxslt
.
.
.
creating build/temp.linux-x86_64-2.6/src/lxml
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/usr/include/python2.6 -c src/lxml/etree.c -o build/temp.linux-x86_64-2.6/src/lxml/etree.o -w
src/lxml/etree.c:98:20: 错误：Python.h：没有那个文件或目录
src/lxml/etree.c:100:6: 错误：#error Python headers needed to compile C extensions, please install development version of Python.
Compile failed: command 'gcc' failed with exit status 1
creating tmp
cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitKTUiCW.c -o tmp/xmlXPathInitKTUiCW.o
cc tmp/xmlXPathInitKTUiCW.o -lxml2 -o a.out
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-s1dXin/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-opr2KI-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-s1dXin/lxml
＃pip install --upgrade pip ＃一开始以为是pip版本低，想升级下，升级后，pip直接无法使用报错，所以别乱升级
＃yum remove python-pip ＃卸载pip
＃yum install python-pip ＃重新安装pip后，可以使用
＃yum install gcc libffi-devel python-devel openssl-devel -y ＃不知道是否有影响，运行pip install pyspider结果又报编译的新错
＃yum install libxslt-devel ＃运行后，再安装pyspider没有报错
＃ pip install pyspider ＃安装pyspider成功，开心的太早了
＃pyspider ＃启动失败，报错，一开始以为pyquery版本低了，开始升级指路。。。
Traceback (most recent call last):
File "/usr/bin/pyspider", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in <module>
working_set.require(__requires__)
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: pyquery<1.3.0
# pip install pyquery==1.3.0 ＃装了1.3，1.4都没有用
# pip install -U setuptools ＃脑子傻掉了，去升级了下setuptools，结果升级到了40.6.3版本，啊，python2.6不支持的啊，2.7以上版本才支持。结果pip又无法使用了。由于pip无法使用了，想卸载setuptools（命令pip uninstall setuptools无法使用）
You are using pip version 7.1.0, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
/usr/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Collecting setuptools
/usr/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)",)': /packages/37/06/754589caf971b0d2d48f151c2586f62902d93dc908e2fd9b9b9f6aa3c9dd/setuptools-40.6.3-py2.py3-none-any.whl
/usr/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)",)': /packages/37/06/754589caf971b0d2d48f151c2586f62902d93dc908e2fd9b9b9f6aa3c9dd/setuptools-40.6.3-py2.py3-none-any.whl
/usr/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Downloading https://files.pythonhosted.org/packages/37/06/754589caf971b0d2d48f151c2586f62902d93dc908e2fd9b9b9f6aa3c9dd/setuptools-40.6.3-py2.py3-none-any.whl (573kB)
100% |████████████████████████████████| 573kB 669kB/s
Installing collected packages: setuptools
Found existing installation: setuptools 0.6rc11
DEPRECATION: Uninstalling a distutils installed project (setuptools) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling setuptools-0.6rc11:
Successfully uninstalled setuptools-0.6rc11
Successfully installed setuptools-40.6.3
# pyspider ＃允许pip也是报这个错
Traceback (most recent call last):
File "/usr/bin/pyspider", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python2.6/site-packages/pkg_resources/__init__.py", line 957, in <module>
class Environment:
File "/usr/lib/python2.6/site-packages/pkg_resources/__init__.py", line 961, in Environment
self, search_path=None, platform=get_supported_platform(),
File "/usr/lib/python2.6/site-packages/pkg_resources/__init__.py", line 188, in get_supported_platform
plat = get_build_platform()
File "/usr/lib/python2.6/site-packages/pkg_resources/__init__.py", line 391, in get_build_platform
from sysconfig import get_platform
ImportError: No module named sysconfig
# yum remove python-pip
＃yum install python-pip ＃尝试重新安装pip，安装后还是无法使用，崩溃，没找到如何降级setuptools，2.6环境就这样搞坏了，无法使用pip了。只好再装个2.7版本了。
3.由于在2中没能装成pyspider，所以装个python2.7环境，参考：https://www.cnblogs.com/Yiutto/p/5962906.html
# yum install git ＃安装git环境
# git clone git://github.com/yyuu/pyenv.git ~/.pyenv
# echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
# echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
# echo 'eval "$(pyenv init -)"' >> ~/.bashrc
# exec $SHELL -l
# pyenv install --list ＃查看可安装的版本
bash: pyenv: command not found
# source ~/.bashrc ＃如果pyenv命令无法找到，那使用pyenv前需要先运行这句
安装依赖包
# yum install readline readline-devel readline-static
# yum install openssl openssl-devel openssl-static
# yum install sqlite-devel
# yum install bzip2-devel bzip2-libs
安装python2.7.10
# pyenv install 2.7.10 -v
/tmp/python-build.20181228230316.15719 ~
Downloading Python-2.7.10.tar.xz...
-> https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tar.xz
curl: (35) SSL connect error
error: failed to download Python-2.7.10.tar.xz
BUILD FAILED (CentOS release 6.5 (Final) using python-build 1.2.8-12-g775a4b6)
ssl错误，安装
# yum update nss
# pyenv rehash ＃更新数据库，注意：使用 pip 安装模块后，可能需要执行 pyenv rehash 更新数据库
# pyenv versions ＃查看当前已安装的python版本
# pyenv local 2.7.10 ＃切换使用python2.7.10版本
# python 确认使用的python版本
# pip install pyspider ＃在python2.7.10版本中安装pyspider
# pyenv rehash ＃pyspider安装成功后需要更新数据库，不如无法使用
# pyspider ＃启动pyspider成功。
pymysql安装
sudo pip install PyMySQL参考：
https://blog.csdn.net/shaoyingchendsg/article/details/77897261
https://blog.csdn.net/weixin_37947156/article/details/76495144
http://www.runoob.com/python3/python3-mysql.html
https://blog.csdn.net/lmb1612977696/article/details/78166180