HDFS之我遇到的各种问题

Hostname问题

问题描述

Datanode denied communication with namenode because hostname cannot be resolved (ip=xx.xx.xx.xx, hostname=xx.xx.xx.xx)
    at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:876)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:5269)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:1178)
    at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:100)
    at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:29184)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)

解决方案

A. 在/etc/hosts中添加映射关系

IP  hostname
IP  hostname
...

B. 设置取消hostname检查

dfs.namenode.datanode.registratin.io-hostname-check: false

安装问题

deltarpm

A. 问题描述

Delta RPMs disabled because /usr/bin/applydeltarpm not installed

B. 解决方案

yum install deltarpm

SASL

A. 问题描述

Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no mechanism available: No worthy mechs found (code THRIFTTRANSPORT): TTransportException('Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no mechanism available: No worthy mechs found',)

B. 解决方案

yum -y install cyrus-sasl-plain cyrus-sasl-devel cyrus-sasl-gssapi

libxslt.so

A. 问题描述

 Couldn't import snappy. Support for snappy compression disabled.
File "/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hue/build/env/lib/python2.7/site-packages/lxml-3.3.6-py2.7-linux-x86_64.egg/lxml/html/__init__.py", line 42, in <module> from lxml import etree
ImportError: libxslt.so.1: cannot open shared object file: No such file or directory

B. 解决方案

yum install httpd -y

Hive

注释中文乱码问题

alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8; 
alter table TABLE_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8; 
alter table PARTITION_KEYS modify column PKEY_COMMENT varchar(4000) character set utf8; （分区表）