Flink on Yarn执行参考

Flink on yarn setup refer link

bin/yarn-session.sh
bin/yarn-session.sh -n 4 -jm 1024m -tm 4096m
Start a long-running Flink cluster on YARN

This command will show you the following overview:

Usage:
   Required
     -n,--container <arg>   Number of YARN container to allocate (=Number of Task Managers)
   Optional
     -D <arg>                        Dynamic properties
     -d,--detached                   Start detached
     -jm,--jobManagerMemory <arg>    Memory for JobManager Container with optional unit (default: MB)
     -nm,--name                      Set a custom name for the application on YARN
     -q,--query                      Display available YARN resources (memory, cores)
     -qu,--queue <arg>               Specify YARN queue.
     -s,--slots <arg>                Number of slots per TaskManager
     -tm,--taskManagerMemory <arg>   Memory per TaskManager Container with optional unit (default: MB)
     -z,--zookeeperNamespace <arg>   Namespace to create the Zookeeper sub-paths for HA mode

---

附着到一个已经存在的Yarn Session

Use the following command to start a session

bin/yarn-session.sh

This command will show you the following overview:

Usage:
   Required
     -id,--applicationId <yarnAppId> YARN application Id

As already mentioned, YARN_CONF_DIR or HADOOP_CONF_DIR environment variable must be set to read the YARN and HDFS configuration.

Example: application_1463870264508_0029:

./bin/yarn-session.sh -id application_1463870264508_0029

Attaching to a running session uses YARN ResourceManager to determine Job Manager RPC port. Stop the YARN session by stopping the unix process (using CTRL+C) or by entering ‘stop’ into the client.

---

Submit Job to Flink

Use the following command to submit a Flink program to the YARN cluster:

bin/flink

./bin/flink run ./examples/batch/WordCount.jar \
        hdfs:///..../LICENSE-2.0.txt hdfs:///.../wordcount-result.txt

---

Run a single Flink job on YARN

The documentation above describes how to start a Flink cluster within a Hadoop YARN environment. It is also possible to launch Flink within YARN only for executing a single job.

Please note that the client then expects the -yn value to be set (number of TaskManagers).

Example:

./bin/flink run -m yarn-cluster -yn 2 ./examples/batch/WordCount.jar

The command line options of the YARN session are also available with the ./bin/flink tool. They are prefixed with a y or yarn (for the long argument options).