After the Session-Cluster mode, need to start the cluster, and then mention homework, will then apply for a space to yarn, resources always remain the same. If the resource is full, the next job can not be submitted, only to wait until the yarn in which a job is executed after completion of the release of the resource, the next job will be properly submitted. All jobs share Dispatcher and ResourceManager; shared resources; suitable for small-scale short execution time job.
Hadoop Yarn first start and this is not to say yo ...
xxxx/app/hadoop-2.6.0-cdh5.15.1/sbin/start-all.sh 全部启动hdfs和yarn
jps可以查看的
Then go to the root directory execution Flink
./bin/yarn-session.sh -n 1 -jm 1024m -tm 4096m
What does it mean parameter everyone see that you, I will not say
-n, -container of YARN Number The Container to the allocate (= Number The Task Managers of)
JM, -jobManagerMemory for JobManager Container with optional Memory Unit (default: MB)
- tm, -taskManagerMemory Memory per TaskManager Container with optional unit (default: MB)
The question is, do you have an issue:
ClassNotFoundException: org.apache.hadoop.yarn.exceptions.
My solution is to find the following two jar package, put flink's lib to the following:
flink-shaded-hadoop-2-2.6.0-cdh5.15.1-7.0.jar
flink-shaded-hadoop-2-uber-2.6.0-cdh5.15.1-7.0.jar
Re-run: ./ bin / yarn-session.sh -n 1 -jm 1024m -tm 4096m
but unfortunately there is another problem:
org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder
This time for it, and then look for a jar package:
commons-cli-1.4.jar
Finally, the implementation of ./bin/yarn-session.sh -n 1 -jm 1024m -tm 4096m no problem, happy!
The jar package if you ask me where to go to find it, because I maven compile the source code has been downloaded Flink good time, I'm in the middle of the warehouse server inside. My Flink is 1.9.0 version, hadoop is 2-2.6.0-cdh5.15.1 version, version if you like me can leave a message, I sent you! Thanks for watching!
