This problem is a pain in the ass. The yarn application was not written by myself (learning from others), so I used it to be compatible with mapreduce v1.
org.apache.hadoop.mapreduce.Job This class is used for client connection
If compatibility issues are not considered, you can use org.apache.hadoop.mapred.JobClient directly
Let's talk about this problem I encountered: After executing the yarn application many times, I found that there were successes and failures, and the probability was not necessarily.
This kind of problem is annoying. Fortunately, the log of hadoop is not bad.
http://192.168.146.13:8088/cluster This address is easy to find the desired log, remember to replace it with your own RM's ip.
Let me explain first, the RM address in my environment is 192.168.146.13, and the resources cannot be wasted. I started NM on the RM at the same time.
Check the log of the app, all reported are connectionException
Of course there will be
2012-12-26 18:14:21,218 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s).
I tested many times and found the problem inside:
If the container is created on the NM of the machine where the RM is located, there is no problem;
If not, the above paragraph will appear... A bunch of Retrying...
Check yarn-site.xml, the configuration is ok.
the problem is
yarn.resourcemanager.scheduler.address
This configuration is not the problem with yarn-site.xml, the problem is actually the client program.
paste code
Configuration conf = new Configuration();
conf.set("fs.default.name",hdfsUri);
conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem");
conf.set("mapreduce.framework.name", "yarn");
conf.set("fs.AbstractFileSystem.hdfs.impl", "org.apache.hadoop.fs.Hdfs");
conf.set("yarn.resourcemanager.address", yarnip+":"+8032); //Set RM access location
//conf.set("yarn.resourcemanager.scheduler.address", yarnip+":"+8030);//Set scheduling address
Job job = new Job(conf, "MR2SmokeTest");
The problem lies in the sentence in the comment. I really don't understand. Should I even tell the Cluster about the scheduling between RM and NM? ? ?
That's true, if no scheduling address is injected, NM will default to 0.0.0.0:8030...