运行hadoop任务报错org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource requ

yarn-site.xml文件中加上

<property>
        <name>yarn.app.mapreduce.am.resource.mb</name>
        <value>200</value>
</property>

原因:yarn默认是设置1.5G内存来运行任务,但是虚拟机内存太小,没有1.5G,要改小运行任务内存

[root@s100 lxz]# hadoop jar hadoopmaven-0.0.1-SNAPSHOT.jar com.lxz.hadoopmaven.mr.MyMaxTempApp /user/sample.txt /user/sample_out.txt
18/08/22 16:41:07 INFO client.RMProxy: Connecting to ResourceManager at s100/192.168.88.100:8032
18/08/22 16:41:14 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
18/08/22 16:41:18 INFO input.FileInputFormat: Total input paths to process : 1
18/08/22 16:41:18 INFO mapreduce.JobSubmitter: number of splits:1
18/08/22 16:41:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1534842402074_0001
18/08/22 16:41:24 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1534842402074_0001
Exception in thread "main" java.io.IOException: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=1536, maxMemory=1024
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:272)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:228)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:236)
    at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
    at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:330)
    at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:282)
    at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:580)
    at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218)
    at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)

    at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:316)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)

猜你喜欢

转载自blog.csdn.net/luxiangzhou/article/details/82012104
今日推荐