最近在做hadoop多机分布式集群的时候,主要用了三台服务器,server1 ,server2,server3.
在linux下通过jps命令查看几台服务器的进程
正常情况下
server1 的进程中应该含有(NameNode、SecondaryNameNode、JobTracker、DataNode、TaskTracker)这5个进程.
Server2的进程中应该含有(DataNode、TaskTracker)这2个进程.
Server3的进程中也应该含有(DataNode、TaskTracker)这2个进程.
但是在Server2中的进程中DataNode节点不存在,只有一个TaskTracker节点.
通过ssh 链接到Server2的服务器,查看日志: cat logs/hadoop-hadoop-datanode-server2.log
经过博主研究,这个错误是由于,我用虚拟机从同一个系统中(原系统以配置好hadoop dfs目录)克隆了3个不同的系统,导致dfs中的data 中的DateNode Version ID重复导致的..
具体解决方案,先停止hadoop集群,直接把几个系统中的 dfs/filesystem/data中的内容给删掉,然后重启,问题即可
解决.
rm -rf dfs/filesystem/data/*
2015-10-23 11:23:59,581 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node 192.168.137.175:50010 is attempting to report storage ID DS-2033798183-127.0.1.1-50010-1445226283737. Node 192.168.137.110:50010 is expected to serve this storage.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDatanode(FSNamesystem.java:4776)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processBlocksBeingWrittenReport(FSNamesystem.java:3558)
at org.apache.hadoop.hdfs.server.namenode.NameNode.blocksBeingWrittenReport(NameNode.java:1055)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
at org.apache.hadoop.ipc.Client.call(Client.java:1107)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at com.sun.proxy.$Proxy3.blocksBeingWrittenReport(Unknown Source)
at org.apache.hadoop.hdfs.server.datanode.DataNode.register(DataNode.java:789)
at org.apache.hadoop.hdfs.server.datanode.DataNode.runDatanodeDaemon(DataNode.java:1549)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1609)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1734)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1751)