Transfer: https://www.jianshu.com/p/5b670d524fa5
Write your answer in the forefront: the maximum degree of parallelism Job groove divided by the number of tasks assigned to each TaskManager.
problem
In Flink 1.5 Release Notes , there is this passage, the direct theme.
This indicates the version 1.5, Flink on the number of containers YARN - i.e. the number TaskManager - parallelism automatically calculated by the program, i.e. -yn flink run scripts / - yarncontainer parameter does not work. So what is it automatically calculate the rules? To figure it out, let's review the degree of parallelism Flink (Parallelism) and task slot (Task Slot).
Parallelism (of Parallelism)
And Spark Similarly, a Flink Job in the execution plan is also divided into a plurality of Task. Task can be Source, Sink, operator or operator chain (chain operator a bit mean, then I will write another article in detail). Task may be performed concurrently by multiple threads, each thread processing Task input a subset of the data. And the number of concurrent called Parallelism, i.e., the degree of parallelism.
Flink set in the program has four levels of parallelism, respectively, from low to high: operator level, the execution environment (ExecutionEnvironment) level, the client (command line) level, profile (flink-conf.yaml) level. The actual implementation, the priority is, in turn, the operator at the highest level. The following simple example.
- Operator level
dataStream.flatMap(new SomeFlatMapFunction()).setParallelism(4); - Level execution environment
streamExecutionEnvironment.setParallelism(4); - Command Levels
bin/flink -run --parallelism 4 example-0.1.jar
- flink-conf.yaml level
parallelism.default: 4
Task groove (Task Slot)
Flink run consists of two components: JobManager and TaskManager, and the Master and Worker Spark Standalone mode under the same concept. Copied from the official website of FIG below, it is easy to understand.
JobManager和TaskManager本质上都是JVM进程。为了提高Flink程序的运行效率和资源利用率，Flink在TaskManager中实现了任务槽（Task Slot）。任务槽是Flink计算资源的基本单位，每个任务槽可以在同一时间执行一个Task，而TaskManager可以拥有一个或者多个任务槽。
任务槽可以实现TaskManager中不同Task的资源隔离，不过是逻辑隔离，并且只隔离内存，亦即在调度层面认为每个任务槽“应该”得到taskmanager.heap.size的N分之一大小的内存。CPU资源不算在内。
TaskManager的任务槽个数在使用flink run脚本提交on YARN作业时用-ys/--yarnslots参数来指定，另外在flink-conf.yaml文件中也有默认值taskManager.numberOfTaskSlots。一般来讲，我们设定该参数时可以将它理解成一个TaskManager可以利用的CPU核心数，因此也要根据实际情况（集群的CPU资源和作业的计算量）来确定。
确定TaskManager数
以Flink自带示例中简化的WordCount程序为例：
// 执行环境并行度设为6
env.setParallelism(6); // Source并行度为1 DataStream<String> text = env .readTextFile(params.get("input")) .setParallelism(1); DataStream<Tuple2<String, Integer>> counts = text .flatMap(new Tokenizer()) .keyBy(0) .sum(1); counts.print(); 用--yarnslots 3参数来执行，即每个TaskManager分配3个任务槽。TaskManager、任务槽和任务的分布将如下图所示，方括号内的数字为并行线程的编号。
由图中可以看出，由于算子链机制的存在，KeyAgg与Sink操作链接在了一起，作为一个Task来执行。
Flink允许任务槽共享，即来自同一个Job的不同Task的Sub-Task（理解为Task的子集就行）进入同一个槽位，因此在图中也可以见到任务槽X中同时存在FlatMap[X]与KeyAgg[X]+Sink[X]。任务槽共享有两点好处：
- 能够让每个Task的Sub-Task都均摊到不同的TaskManager，避免负载倾斜。
- 不需要再计算App一共需要起多少个Task，因为作业需要的任务槽数量肯定等于Job中最大的并行度。
所以，可以得出Flink on YARN时，TaskManager的数量就是：max(parallelism) / yarnslots（向上取整）。例如，一个最大并行度为10，每个TaskManager有两个任务槽的作业，就会启动5个TaskManager，如Web UI所示。
