Flink on Yarn has two execution methods, one is Session-Cluster mode, the other is Pre-Job-Cluster mode
Flink On Yarn performs tasks in two ways
Flink Session （Session-Cluster）
- Centralized memory management mode: Initialize a Flink cluster in Yarn, open up designated resources, and then submit Flink Jon in this Flink yarn-session, which means that no matter how many jobs are submitted, these jobs will share the yarn at the beginning Resources requested in. This Flink cluster will reside in the Yarn cluster unless it is manually stopped.
- Start a daemon on yarn to start multiple jobs, that is, start an application master to manage multiple jobs
- This mode is divided into two steps: yarn-session.sh (resource development) + flink run (submit task)
./yarn-session.sh -n 4 -jm 1024 -tm 5120 -s 5 -nm yarn-session-jobs -d
Parameter Description:
- -n: Specify the number of task manager, specify the number of task manager
- -jm: the amount of memory occupied by jobmanager, in MB
- -tm: Specify the memory occupied by each taskmanager, in MB
- -s: Specify the number of CPU cores that each taskmanager can use
- -nm: Specify the name of the Application
- -d: Start in the background, after the session is started, the process is closed
Startup process description:
- After starting the session, yarn will first allocate a Container to start the APP master and jobmanager. The occupied memory is the memory size specified by -jm, and the cpu is 1 core
- Before starting the job, jobmanager will not start taskmanager (jobmanager will dynamically allocate taskmanager according to the degree of parallelism of the job, that is, the occupied slots)
- Submit task to APP master
./flink run -p 3 -yid application_id -d -c com.xxx ~/jar/xxx-1.0-SNAPSHOT.jarUsed to start a job to the specified APP master
Note: The job parameter should be written before -c, otherwise the specified parameter will not work
Parameter Description:
- -p: Specify the degree of parallelism of the task. If you specify the degree of parallelism in the program code, then the degree of parallelism parameter here has no effect
- -yid: Specify which application-id the task is submitted to, the default is the latest application submitted to this node
- -c: main entrance of job + jar path
Flink run （Per-Job-Cluster）
Start a single job and submit it to the yarn cluster, that is, single job and single session, to achieve complete resource isolation.
The command to start the job is different from that in yarn-session. By specifying -m yarn-cluster, the parameters have -y compared to session.
./flink run \
-m yarn-cluster \
-yn 2 \
-yjm 1024 \
-ytm 3076 \
-p 2 \
-ys 3 \
-yD name=hadoop \
-ynm RTC_KB_FLINK_TEST \
-yqu rt_constant \
-c com.kb.rt.Test02 ~/jar/kb-1.0-SNAPSHOT.jar
Parameter Description:
- -m: yarn-cluster, which means to start a single session and submit a single job
- -yn: number of taskmanager
- -yjm: memory usage of jobmanager
- -ytm: memory usage of each taskmanager
- -ys: The number of CPU cores that can be used by each taskmanager
- -ynm: application name
- -yqu: Specify the name of the job queue
- -c: program main entrance + jar path
- -p: Specify the degree of parallelism of the task
- -yD: dynamic parameter setting
Task submission process
- flink-client submits the task to yarn, and uploads all dependent jars and configs to hdfs
- To apply for resources from ResourceManager, ResourceManager mainly contains two parts: ApplicationManager and resourceScheduler
- ApplicationManager starts ApplicationMaster, and ResourceScheduler allocates the Containers needed to run ApplicationMaster.
- Start Flink's JobManager on ApplicationMaster, and apply for resources from ResourceManager through RPC in a polling manner
- After the ResourceManager receives the request, it will allocate the corresponding Container to start the TaskManager in the future and download the corresponding jar and config from hdfs.
- ApplicationMater requires the corresponding Container to start the task.
- The corresponding JobManager and TaskManager are started.
Flink resource management
operator chains
In the entire dataflow processing process, there are many operators, which can also be understood as operators. There is a certain relationship between adjacent operators. Connecting adjacent operators to form a chain is the key to flink's efficient distributed computing. It can reduce the switching between threads and the serialization and deserialization of messages.
ono-to-one streams: Between source and map, there is no partition change and no data exchange between partitions.
Redistributing streams: Between map and keyby/window and between keyby/window and sink, there is a change in data exchange between partitions or the number of partitions.
slot
-
Slots generally refer to a fixed resource subset of taskmanager, or threads. If there are 3 slots in a taskmanager, then each slot manages 1/3 of the memory hosting.
-
Memory resources will not have memory competition due to the existence of slots, and slots isolate the memory, but note that the cpu is not isolated
Slots and parallelism are different. Slot is a static concept, which means that taskmanager can have the ability to execute in parallel. Parallelism is only meaningful when the program is executed. It is a dynamic concept and refers to the actual use of the program when it is running. The concurrency ability, but the two are related. Parallelism cannot be greater than the number of slots.
Sharing Slot: SlotSharingGroup and CoLocationGroup
By default, Flink allows subtasks to share slots, provided that they all come from the same job and have subtasks of different tasks with the same SlotSharingGroup (the default name is default). Then it means that the subtasks of tasks with different names are shared in a slot, so that a slot has the opportunity to hold a completed pipeline. Therefore, in the default slotsharing case, the number of slots required for job startup is equal to the maximum parallelism of the operator in the job.
The figure above is a source-map-reduce model job, shown in the lower left corner of the figure, where the parallelism of source and map is 4, and the parallelism of reduce is 3. Each circle represents a subtask; and the maximum parallelism of the operator in the entire job is 4; if the task is submitted for execution, you can know that the job requires 4 slots; if the job is submitted to 2 taskManagers, each taskmanager There are 3 slots, as shown on the right in the figure, of which 3 slots have a pipeline with a complete source-map-reduce model. However, the process of data shuffle is not drawn.
The above figure contains the model job of source-map, keyBy/window/apply, sink, where the parallelism of source-map is 6, the parallelism of keyBy/window/apply is 6, and the parallelism of sink is 1, then the job is required for submission There are 6 slots. After allocation, the first slot on the left has a complete pipeline, running 3 subtasks, and the remaining 5 slots run these 2 subtasks respectively, and finally the data is transmitted to the sink for output.
Flink run parameters
参数说明
Action "run" compiles and runs a program.
Syntax: run [OPTIONS] <jar-file> <arguments>
"run" action options:
-c,--class <classname> Class with the program entry
point ("main" method or
"getPlan()" method. Only
needed if the JAR file does
not specify the class in its
manifest.
-C,--classpath <url> Adds a URL to each user code
classloader on all nodes in
the cluster. The paths must
specify a protocol (e.g.
file://) and be accessible
on all nodes (e.g. by means
of a NFS share). You can use
this option multiple times
for specifying more than one
URL. The protocol must be
supported by the {
@link
java.net.URLClassLoader}.
-d,--detached If present, runs the job in
detached mode
-m,--jobmanager <host:port> Address of the JobManager
(master) to which to
connect. Use this flag to
connect to a different
JobManager than the one
specified in the
configuration.
-p,--parallelism <parallelism> The parallelism with which
to run the program. Optional
flag to override the default
value specified in the
configuration.
-q,--sysoutLogging If present, suppress logging
output to standard out.
-s,--fromSavepoint <savepointPath> Path to a savepoint to reset
the job back to (for example
file:///flink/savepoint-1537
).
-z,--zookeeperNamespace <zookeeperNamespace> Namespace to create the
Zookeeper sub-paths for high
availability mode
Options for yarn-cluster mode:
-yD <arg> Dynamic properties
-yd,--yarndetached Start detached
-yid,--yarnapplicationId <arg> Attach to running YARN session
-yj,--yarnjar <arg> Path to Flink jar file
-yjm,--yarnjobManagerMemory <arg> Memory for JobManager Container [in
MB]
-yn,--yarncontainer <arg> Number of YARN container to allocate
(=Number of Task Managers)
-ynm,--yarnname <arg> Set a custom name for the application
on YARN
-yq,--yarnquery Display available YARN resources
(memory, cores)
-yqu,--yarnqueue <arg> Specify YARN queue.
-ys,--yarnslots <arg> Number of slots per TaskManager
-yst,--yarnstreaming Start Flink in streaming mode
-yt,--yarnship <arg> Ship files in the specified directory
(t for transfer)
-ytm,--yarntaskManagerMemory <arg> Memory per TaskManager Container [in
MB]
-yz,--yarnzookeeperNamespace <arg> Namespace to create the Zookeeper
sub-paths for high availability mode
-n 10 一共启动10个TaskManager节点
-jm 1024 JobManager的内存大小为1024M
-tm 2048 TaskManager的内存大小为2048M
-d 使用detached模式进行部署（部署完成后本地命令行可以退出）
-qu default 部署到YARN的default队列中