2. Flink of data transmission
In a running application in its tasks in the continuous exchange of data. TaskManager is responsible for data transmission. TaskManager network component is first collected in the buffer from the buffer Records , and then transmitted. In other words, Records not send one by one, but first into the buffer, and then to batch send form. This technology allows efficient use of network resources, and achieve high throughput. Similar to the network or disk I / O buffering techniques used in the protocol.
It should be noted that: the recording buffer transmission buffer, the implicit representation, Flink processing model is based on a micro batch.
Each TaskManager has a set of network buffer pool (default value for each buffer is 32KB ), for transmitting and receiving data. The transmitter and receiver are located at different TaskManager process, they need to communicate through the network stack of the operating system. Streaming applications require data exchange mode of the conduit, that is, each pair TaskManager will remain a permanent TCP connection is used for data exchange. In shuffle connected mode (more sender to a plurality of Receiver ), each sender Task needs to each receiver task , at this time TaskManager required for each receiver task is assigned a buffer. The following figure shows this architecture:
In the figure, there are four sender task, for each sender , needs to have at least four network buffer used to each receiver to transmit data. Each receiver needs to have at least four buffer for receiving data. TaskManager between buffer is connected in a multiplexed manner using the same network. In order to provide a smooth data exchange data pipeline type, a TaskManager must provide sufficient buffer, to serve out all parallel connected. For shuffle or broadcast connection, between each transmit and each task given the task requires a Buffer . Flink default network configuration sufficient buffer applies small and medium-sized cluster tasks. For large cluster tasks that require this configuration tuning.
若sender与receiver任务都运行在同一个TaskManager进程，则sender任务会将发送的条目做序列化，并存入一个字节缓冲。然后将缓冲放入一个队列，直到队列被填满。Receiver任务从队列中获取缓冲，并反序列化输入的条目。所以，在同一个TaskManager内，任务之间的数据传输并不经过网络交互。
Flink采用了不同的技术用于减少tasks之间的沟通成本。在接下来的部分中，我们会讨论基于积分的（credit-based ）流控制与任务链（task chaining）。
基于积分的（Credit-Based ）流控制
通过网络发送单独的条目是一个并不高效的方式，并且会造成大量负载。使用缓冲技术可以更好的使用网络连接的带宽。在流处理场景中，缓冲的一个缺点是：它增加了延时，因为records需要先放入缓冲，而不是被立即传输。
Flink实现了一个credit-based 流控制机制，工作方式为：一个接收任务会授权给一个发送任务一些积分（credit），用于控制预留的缓冲区个数。当一个sender接收到了积分通知，它向会receiver发送 buffers（最多不超过被授权的数量）以及它的backlog大小（已经充满了并等待被发送的buffer数量）。Receiver使用预留的buffer处理接收到的数据，并使用sender的backlog大小作为下一次授权的积分数，提供给所有与它连接的senders。
Credit-based 流控制减少了延时，因为senders可以在receiver有足够的资源接受数据时，尽快向它发送数据。它在Flink中是一个重要的部分，助力Flink达到高吞吐与低延时。
任务链（task chaining）
Flink另一个优化技术称为任务链，用于（在某些情况下）减少本地通信的过载。为了满足任务链的条件，至少两个以上的operator必须配置为同一并行度，并且使用本地向前的（local forwad）方式连接。下图的operator管道即满足这些条件。它包含3个operators，全部被配置为并行度为2，并且以local-forward的方式连接：
下图描述了管道是如何以任务链的方式执行的。Operators的函数被融合成单个任务，并由一个单独的线程执行。一个function产生的records，通过使用一个简单的方法调用，被递交给下一个function。所以，这里在方法之间的records传递中，基本没有序列化以及通信消耗。
任务链可以极大减少本地task之间的通信成本，但是有时候在执行一个管道时，不使用任务链也是合理的。例如，将一个包含多个链式任务的长管道断开，或是将一个链分成两个任务，并将较为消耗资源的function调度到另一个slot上。这些都是合理的。下图描绘了同样一个执行的管道，但未使用任务链。所有function由一个单独的task，在它自身的线程中运行。
任务链默认是开启的。在“控制任务链”一节，我们会介绍如何为某个任务关闭任务链化，以及如何控制单个operator的链行为。
References:
Vasiliki Kalavri, Fabian Hueske. Stream Processing With Apache Flink. 2019
Vasiliki Kalavri, Fabian Hueske. Stream Processing With Apache Flink. 2019