StreamGraph build and submit source code analysis
StreamGraph: The initial graph generated based on the code written by the user through the Stream API. Flink transforms each operator
into a stream conversion (such as SingleOutputStreamOperator, which is a subclass of DataStream), and
registers it in the execution environment to generate StreamGraph.
The main abstract concepts it contains are
1. StreamNode: used to represent The class of operator, and has all relevant attributes, such as concurrency, inbound and outbound, etc.
2. StreamEdge: represents the edge connecting two StreamNodes
1.1 Encapsulation of transformation operator
用户代码 执行map等算子时，以map为例
DataStream.map -> transform -> doTransform -> getExecutionEnvironment().addOperator(resultTransform)
-> transformations.add(transformation)
将代码中的算子全部封装到transformations 这个list里面,后面生成图时从这个transformations里面遍历生成StramGraph
从上方代码可以了解到， map 转换将用户自定义的函数 MapFunction 包装到 StreamMap
这个 Operator 中，再将 StreamMap 包装到 OneInputTransformation，最后该 transformation 存
到 env 中，当调用 env.execute 时，遍历其中的 transformation 集合构造出 StreamGraph
The layered implementation is shown in the figure below.
In addition, not every StreamTransformation will be converted into a physical operation in the runtime layer. A
These are just a logical concept, such as union, split / select, partition and the like. The conversion tree shown in the figure below
will be optimized at runtime to the operation map
union, split/select (removed in 1.12), and the information in partition will be written to the
edge of Source-> Map . Through the source code, you can also find UnionTransformation, SplitTransformation (removed in 1.12),
SelectTransformation (removed in 1.12), PartitionTransformation
has no StreamOperator member variable because it does not contain specific operations , and there are basically other subclasses of StreamTransformation.
1.2 StramGraph build source code analysis
生成StramGraph入口：
StreamExecutionEnvironment.execute(getStreamGraph(jobName));
-> getStreamGraph -> StreamGraph streamGraph = getStreamGraphGenerator().setJobName(jobName).generate();
-> generate -> StreamGraphGenerator.transform(transformations是一个list，依次存放了 用户代码里的 算子)
-> translate -> SimpleTransformationTranslator.translateForStreaming
-> translateForStreamingInternal
区分 map之类的转换算子（OneInputTransformationTranslator）
keyby值类的分区算子（PartitionTransformationTranslator）
-> OneInputTransformationTranslator.translateForStreamingInternal -> translateInternal
-> streamGraph.addOperator(生成StreamNode入口)
-> addOperator-> addNode -> StreamNode vertex = new StreamNode
-> streamGraph.addEdge(生成StreamEdge入口)
-> streamGraph.addEdgeInternal
private void addEdgeInternal(Integer upStreamVertexID,
Integer downStreamVertexID,
int typeNumber,
StreamPartitioner<?> partitioner,
List<String> outputNames,
OutputTag outputTag,
ShuffleMode shuffleMode) {
/*TODO 当上游是侧输出时，递归调用，并传入侧输出信息*/
if (virtualSideOutputNodes.containsKey(upStreamVertexID)) {
int virtualId = upStreamVertexID;
upStreamVertexID = virtualSideOutputNodes.get(virtualId).f0;
if (outputTag == null) {
outputTag = virtualSideOutputNodes.get(virtualId).f1;
}
addEdgeInternal(upStreamVertexID, downStreamVertexID, typeNumber, partitioner, null, outputTag, shuffleMode);
} else if (virtualPartitionNodes.containsKey(upStreamVertexID)) {
/*TODO 当上游是partition时，递归调用，并传入partitioner信息*/
int virtualId = upStreamVertexID;
upStreamVertexID = virtualPartitionNodes.get(virtualId).f0;
if (partitioner == null) {
partitioner = virtualPartitionNodes.get(virtualId).f1;
}
shuffleMode = virtualPartitionNodes.get(virtualId).f2;
addEdgeInternal(upStreamVertexID, downStreamVertexID, typeNumber, partitioner, outputNames, outputTag, shuffleMode);
} else {
/*TODO 真正构建StreamEdge*/
StreamNode upstreamNode = getStreamNode(upStreamVertexID);
StreamNode downstreamNode = getStreamNode(downStreamVertexID);
// If no partitioner was specified and the parallelism of upstream and downstream
// operator matches use forward partitioning, use rebalance otherwise.
/*TODO 未指定partitioner的话，会为其选择 forward 或 rebalance 分区*/
if (partitioner == null && upstreamNode.getParallelism() == downstreamNode.getParallelism()) {
partitioner = new ForwardPartitioner<Object>();
} else if (partitioner == null) {
partitioner = new RebalancePartitioner<Object>();
}
// TODO 健康检查，forward 分区必须要上下游的并发度一致
if (partitioner instanceof ForwardPartitioner) {
if (upstreamNode.getParallelism() != downstreamNode.getParallelism()) {
throw new UnsupportedOperationException("Forward partitioning does not allow " +
"change of parallelism. Upstream operation: " + upstreamNode + " parallelism: " + upstreamNode.getParallelism() +
", downstream operation: " + downstreamNode + " parallelism: " + downstreamNode.getParallelism() +
" You must use another partitioning strategy, such as broadcast, rebalance, shuffle or global.");
}
}
if (shuffleMode == null) {
shuffleMode = ShuffleMode.UNDEFINED;
}
/*TODO 创建 StreamEdge*/
StreamEdge edge = new StreamEdge(upstreamNode, downstreamNode, typeNumber,
partitioner, outputTag, shuffleMode);
/*TODO 将该 StreamEdge 添加到上游的输出，下游的输入*/
getStreamNode(edge.getSourceId()).addOutEdge(edge);
getStreamNode(edge.getTargetId()).addInEdge(edge);
}
}
再来看下对逻辑转换（partition、 union 等）的处理，如下是 transformPartition 函数的源码：
PartitionTransformationTranslator.java
private Collection<Integer> translateInternal(){
for (Integer inputId: context.getStreamNodeIds(input)) {
/*TODO 生成一个新的虚拟id*/
final int virtualId = Transformation.getNewNodeId();
/*TODO 添加一个虚拟分区节点，不会生成StreamNode*/
streamGraph.addVirtualPartitionNode(
inputId,
virtualId,
transformation.getPartitioner(),
transformation.getShuffleMode());
resultIds.add(virtualId);
}
}
对partition 的转换没有生成具体的 StreamNode 和 StreamEdge，而是添加一个虚节点。
当 partition 的下游 transform（如 map）添加 edge 时（调用 StreamGraph.addEdge），会把
partition 信息写入到 edge 中
Case Analysis
DataStream<String> text = env.socketTextStream(hostName, port);
text.flatMap(new LineSplitter()).shuffle().filter(new HelloFilter()).print();
The above program is a simple stream program that divides the source into words by line and filters the output, which contains a logical transformation: random partition shuffle. Analyze how the program generates StreamGraph.
First, a transformation tree will be generated in env and saved with List<Transformation<?>>. Its structure is
as follows.
The symbol * is the input pointer, which points to the upstream transformation, thus forming a transformation
tree. Then, generate
StreamGraph by calling StreamGraphGenerator.generate(env, transformations) . Each transformation is called recursively from bottom to top, that is to say, the processing order is
Source->FlatMap->Shuffle->Filter->Sink
1) The Source processed first, generates the StreamNode of the Source.
2) Then the processed FlatMap generates the StreamNode of the FlatMap, and generates the StreamEdge to connect the upstream
Source and FlatMap. Since the concurrency between upstream and downstream is different (1:4), so here is the Rebalance partition.
3) The Shuffle processed then does not generate actual nodes because it is a logical conversion. Temporarily store partitioner information in virtuaPartitionNodes.
4) When processing the Filter, a StreamNode of the Filter is generated. Found that the upstream is shuffle, find the upstream FlatMap of shuffle,
Create StreamEdge to connect with Filter. And write the information of ShufflePartitioner to StreamEdge.
5) Finally, the sink is processed, the StreamNode of the sink is created, and the StreamEdge is generated to connect to the upstream Filter. Since
the upstream and downstream concurrency is the same (4:4), the Forward partition is selected here.
Finally, you can observe the StreamGraph obtained through UI visualization
