环境硬件配置及Hadoop，Hive版本

此博客置顶文章中有

安装步骤

pip install pure-sasl

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting pure-sasl
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/16/83/30eaf3765de898083
75a8358f9c15d45a3dd44ed26be991471abc0b4480b/pure_sasl-0.5.1-py2.py3-none-any.whl

Installing collected packages: pure-sasl
Successfully installed pure-sasl-0.5.1

pip install thrift_sasl==0.2.1 --no-deps

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting thrift_sasl==0.2.1
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/80/36/16dfe92d32df63cc2
b7b7be8d0e4a736617b7e52daaa7f83ae386a89d179/thrift_sasl-0.2.1.tar.gz
Installing collected packages: thrift-sasl
Running setup.py install for thrift-sasl … done
Successfully installed thrift-sasl-0.2.1

pip install thrift==0.9.3

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting thrift==0.9.3
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ae/58/35e3f0cd290039ff8
62c2c9d8ae8a76896665d70343d833bdc2f748b8e55/thrift-0.9.3.tar.gz
Installing collected packages: thrift
Running setup.py install for thrift … done
Successfully installed thrift-0.9.3

pip install impyla

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting impyla
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/80/93/f0d226061ee4679d5
b593c88c7b2e9e077a271c799d29facf31bf03666c1/impyla-0.14.1.tar.gz (151kB)
40% |█████████████ | 61kB 1.3MB/s eta 0:00:01
47% |███████████████ | 71kB 1.1MB/s eta 0:00:
53% |█████████████████ | 81kB 1.1MB/s eta 0:0
60% |███████████████████▌ | 92kB 1.2MB/s eta
67% |█████████████████████▌ | 102kB 1.1MB/s e
74% |████████████████████████ | 112kB 1.1MB/s
80% |██████████████████████████ | 122kB 759kB
87% |████████████████████████████ | 133kB 786
94% |██████████████████████████████ | 143kB 8
100% |████████████████████████████████| 153k
B 1.0MB/s
Requirement already satisfied: six in c:\program files\python36\lib\site-package
s (from impyla) (1.11.0)
Collecting bitarray (from impyla)
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e2/1e/b93636ae36d08d0ee
3aec40b08731cc97217c69db9422c0afef6ee32ebd2/bitarray-0.8.3.tar.gz
Collecting thriftpy>=0.3.5 (from impyla)
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f4/19/cca118cf7d2087310
dbc8bd70dc7df0c1320f2652873a93d06d7ba356d4a/thriftpy-0.3.9.tar.gz (208kB)
39% |████████████▌ | 81kB 875kB/s eta 0:00:01
44% |██████████████ | 92kB 984kB/s eta 0:00:0
49% |████████████████ | 102kB 820kB/s eta 0:0
54% |█████████████████ | 112kB 820kB/s eta 0:
59% |███████████████████ | 122kB 820kB/s eta
63% |████████████████████▌ | 133kB 729kB/s et
68% |██████████████████████ | 143kB 937kB/s e
73% |███████████████████████▌ | 153kB 937kB/s
78% |█████████████████████████ | 163kB 820kB/
83% |███████████████████████████ | 174kB 937k
88% |████████████████████████████▌ | 184kB 93
93% |██████████████████████████████ | 194kB 9
98% |███████████████████████████████▌| 204kB
100% |████████████████████████████████| 215k
B 1.1MB/s
Requirement already satisfied: ply<4.0,>=3.4 in c:\program files\python36\lib\si
te-packages (from thriftpy>=0.3.5->impyla) (3.11)
thrift-sasl 0.2.1 requires sasl>=0.2.1, which is not installed.
Installing collected packages: bitarray, thriftpy, impyla
Running setup.py install for bitarray … done
Running setup.py install for thriftpy … done
Running setup.py install for impyla … done
Successfully installed bitarray-0.8.3 impyla-0.14.1 thriftpy-0.3.9

执行数据库连接后，出现问题

ThriftParserError: ThriftPy does not support generating module with path in protocol ‘c’

定位到 \Lib\site-packages\thriftpy\parser\parser.py的

if url_scheme == '':
    with open(path) as fh:
        data = fh.read()
elif url_scheme in ('http', 'https'):
    data = urlopen(path).read()
else:
    raise ThriftParserError('ThriftPy does not support generating module '
                            'with path in protocol \'{}\''.format(
                                url_scheme))

更改为

if url_scheme == '':
    with open(path) as fh:
        data = fh.read()
elif url_scheme in ('c', 'd','e','f''):
    with open(path) as fh:
        data = fh.read()


elif url_scheme in ('http', 'https'):
    data = urlopen(path).read()
else:
    raise ThriftParserError('ThriftPy does not support generating module '
                            'with path in protocol \'{}\''.format(
                                url_scheme))

执行数据库连接后，再次出现问题

TypeError: can’t concat str to bytes

定位到错误的最后一条，在init.py第94行

...
header = struct.pack(">BI", status, len(body))
self._trans.write(header + body)
...

修改为

...
header = struct.pack(">BI", status, len(body))
if(type(body) is str):
    body = body.encode() 
self._trans.write(header + body)
...

执行连接成功

最终模块

模块	版本
pure_sasl	0.5.1
thrift_sasl	0.2.1
thrift	0.9.3
bitarray	0.8.3
thriftpy	0.3.9
impyla	0.14.1

连接代码

from impala.dbapi import connect
from impala.util import as_pandas

conn = connect(host='***', port=10000, auth_mechanism='PLAIN', user='***', password='***', database='***')
cursor = conn.cursor()
cursor.execute('show databases')
print(as_pandas(cursor))

遇到的问题

除去以上两个问题还有

‘TSocket’ object has no attribute ‘isOpen’
Can’t connect to unsecured hive. SASL error: TTransportException: Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no mechanism available: Unable to find a callback: 2
TTransportException: Could not start SASL: Error in sasl_client_start (-4) SASL( -4): no mechanism available: Unable to find a callback: 2
‘No protocol version header’) thriftpy.protocol.exc.TProtocolException: TProtocolException(type=4)

以上问题均可以

pip uninstall

卸载所有相关模块之后，使用本文的顺序和版本号安装来保证运行正确

参考文章

https://blog.csdn.net/a6822342/article/details/80844072
https://github.com/cloudera/impyla/issues/268
https://github.com/cloudera/impyla/issues/165
https://github.com/dropbox/PyHive/issues/161
https://github.com/cloudera/impyla/issues/149
https://github.com/dropbox/PyHive/issues/32
https://github.com/eleme/thriftpy/issues/234
https://github.com/cloudera/impyla/issues/267
https://blog.csdn.net/qq_24908345/article/details/80595948
https://blog.csdn.net/sinolover/article/details/77714648
https://github.com/eleme/thriftpy/issues/234
https://www.cnblogs.com/angellst/p/7510486.html
https://www.cnblogs.com/tomato0906/articles/6057335.html

Win7平台Python3使用impyla连接Hive遇到的坑

环境硬件配置及Hadoop，Hive版本

安装步骤

最终模块

连接代码

遇到的问题

参考文章

猜你喜欢