解决新手hadoop2.7.3在window10 提交工作至远程服务上失败的问题

错误1 Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control

修改hadoop源码在项目的当前路径的YARNRunner.java中

390 行将这行代码vargs.add(Environment.JAVA_HOME.$() + "/bin/java");

替换为 vargs.add("$JAVA_HOME/bin/java");

增加方法

private void replaceEnvironment(Map<String, String> environment) {
String tmpClassPath = environment.get("CLASSPATH");
tmpClassPath=tmpClassPath.replaceAll(";", ":");
tmpClassPath=tmpClassPath.replaceAll("%PWD%", "\\$PWD");
tmpClassPath=tmpClassPath.replaceAll("%HADOOP_MAPRED_HOME%", "\\$HADOOP_MAPRED_HOME");
tmpClassPath= tmpClassPath.replaceAll("\\\\", "/" );
environment.put("CLASSPATH",tmpClassPath);
}

在466行添加代码 replaceEnvironment(environment);

错误二用户权限检查问题

在当前提交任务的机器上配置环境变量 HADOOP_USER_NAME ＝ albert //hdfs有权限的用户

groupadd supergroup
usermod -a -G supergroup albert

把这个用户添加到hdfs的超级用户组中

错误三Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

在mapred-site.xml中

增加如下陪配置

<property>
    <name>mapreduce.application.classpath</name>
    <value>
        /opt/hadoop-2.6.0/etc/hadoop,
        /opt/hadoop-2.6.0/share/hadoop/common/*,
        /opt/hadoop-2.6.0/share/hadoop/common/lib/*,
        /opt/hadoop-2.6.0/share/hadoop/hdfs/*,
        /opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*,
        /opt/hadoop-2.6.0/share/hadoop/mapreduce/*,
        /opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,
        /opt/hadoop-2.6.0/share/hadoop/yarn/*,
        /opt/hadoop-2.6.0/share/hadoop/yarn/lib/*
    </value>
</property>    
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>master:10020</value>
    </property>
       <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>master:19888</value>
        </property>

修改yarn-site.xml

<property>
    <name>yarn.application.classpath</name>
    <value>
        /opt/hadoop-2.6.0/etc/hadoop,
        /opt/hadoop-2.6.0/share/hadoop/common/*,
        /opt/hadoop-2.6.0/share/hadoop/common/lib/*,
        /opt/hadoop-2.6.0/share/hadoop/hdfs/*,
        /opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*,
        /opt/hadoop-2.6.0/share/hadoop/mapreduce/*,
        /opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,
        /opt/hadoop-2.6.0/share/hadoop/yarn/*,
        /opt/hadoop-2.6.0/share/hadoop/yarn/lib/*
    </value>
  </property>

解决新手hadoop2.7.3在window10 提交工作至远程服务上失败的问题

猜你喜欢