【转】hadoop 3.x 安装问题汇总

一、 伪分布式节点启动报错

./start-dfs.sh 
 
Starting namenodes on [10.1.4.57]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [10.1.4.57]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.

环境变量 hadoop-env.sh 中指定用户名:

export HDFS_DATANODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root

再次启动ok。***_USER设置错误,会报

cannot set priority of datanode process 32156

二、 namenode节点启动成功,web服务无法访问

    检查linux系统的防火墙设置

firewall-cmd    --state

 CentOS-7 防火墙默认使用的是firewall,与之前的版本使用iptables不一样。防火墙操作命令:

关闭防火墙:systemctl stop firewalld.service
开启防火墙:systemctl start firewalld.service
关闭开机启动:systemctl disable firewalld.service
开启开机启动:systemctl enable firewalld.service

关闭防火墙并设置禁用开机启动后,成功访问 http://ip:9870/

三、 datanode启动出错

2018-04-23 10:33:29,644 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /tmp/hadoop-root/dfs/data/in_use.lock acquired by nodename 3692@localhost
2018-04-23 10:33:29,647 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to add storage directory [DISK]file:/tmp/hadoop-root/dfs/data
java.io.IOException: Incompatible clusterIDs in /tmp/hadoop-root/dfs/data: namenode clusterID = CID-103c769e-5fff-427c-9913-1004480fce63; datanode clusterID = CID-659951c9-642c-4325-8dc3-79f5edbdf175
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:736)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:294)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:407)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:387)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:551)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1705)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1665)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:390)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:280)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
	at java.lang.Thread.run(Thread.java:748)

原因是bin/hdfs namenode -format只会格式化namenode,并不会影响到datanode,如果再次格式化会导致datanode和namenode的clusterID不一致。解决方法:先停掉hadoop,把slaves的dfs/data的内容删除,再次启动后,会创建新的clusterID,也可以复制master的clusterID到slaves中。

sbin/stop-all.sh
rm -Rf /tmp/hadoop-your-username/*
bin/hadoop namenode -format
sbin/start-dfs.sh

四 、 单节点配置YARN启动报错

    官方配置说明:

进行上述配置后,执行start-yarn.sh启动报错:

 "[26,2]"、“start tag in epilog”描述的应该是标签错误,合并<configuration>标签配置如下:

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
        <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>
</configuration>

    再次启动,成功访问http://{IP}:8088 -->

猜你喜欢

转载自blog.csdn.net/a8131357leo/article/details/81281088