[Source Analysis] Examples of the source and the start of watching a broadcast Broadcast Flink

[Source Analysis] Examples of the source and the start of watching a broadcast Broadcast Flink

0x00 Summary

This paper will analyze source code and examples to explain, led familiar Flink broadcast variable mechanism.

0x01 business needs

1. Scene demand

Blacklist detects the IP filtering. Content IP blacklist will increase or decrease at any time, and therefore can be dynamically configured at any time.

The blacklist is assumed that there are mysql, Flink job will start when the blacklist from mysql load as a variable by the operator to use the sub-Flink.

2. Problem

We do not want to restart the job in order to regain this variable. So we need to be able to dynamically modify a calculation method yard variables.

3. Solution

Broadcast way to solve. Do the configuration dynamically updated.

Different broadcast and common data stream: data broadcasting stream a stream operator can be processed all partitions, and a stream data of the stream can only be considered a partition of the sub-process. Therefore, the characteristics of the broadcast stream also determines suitable for dynamic update configuration.

0x02 Overview

This section has three broadcast difficulties: The procedures; how to customize the function; how to access state. Here's an overview for everyone at first.

Step 1. broadcast using

• Establish MapStateDescriptor
• Back broadcast data stream by DataStream.broadcast method BroadcastStream
• , The traffic data flow and are connected through DataStream.connect BroadcastStream method returns BroadcastConnectedStream
• Processing performed by BroadcastConnectedStream.process processBroadcastElement processElement and Methods

2. The user-defined handlers

• BroadcastConnectedStream.process receive two types of function: KeyedBroadcastProcessFunction and BroadcastProcessFunction
• Two types of function are defined processElement, processBroadcastElement abstract method, but it defines a KeyedBroadcastProcessFunction onTimer method, the default operation is empty, allowing a subclass overrides
• processing the service data flow processElement
• processBroadcastElement processing broadcast data stream

3. Broadcast State

• Broadcast State is always expressed as MapState, namely map format. This is the most common state Flink provides primitives. Hosted a state, the state managed by the state management frame Flink, such ValueState, ListState, MapState the like .
• You must create a user MapStateDescriptorin order to obtain the corresponding state handle. This saves the state name, the type of values held by the state, and may contain the function specified by the user
• checkpoint time will checkpoint broadcast state
• Broadcast State only in memory, there is no RocksDB state backend
• Flink state will be broadcast to each task, note that the state does not spread across the task, modify them in their just role in the task
• downstream tasks sequentially receiving broadcast event may be different, so that it reaches the dependent element are processed in order to be careful when

0x03. Sample Code

1. Sample Code

We start with Flink directly from the source to find the ideal example. The following excerpt Flink source code directly StatefulJobWBroadcastStateMigrationITCase, I would add explanatory notes on the inside.

  @Test
  def testRestoreSavepointWithBroadcast(): Unit = {

    val env = StreamExecutionEnvironment.getExecutionEnvironment
    env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)

    // 以下两个变量是为了确定广播流发出的数据类型，广播流可以同时发出多种类型的数据
    lazy val firstBroadcastStateDesc = new MapStateDescriptor[Long, Long](
      "broadcast-state-1",
      BasicTypeInfo.LONG_TYPE_INFO.asInstanceOf[TypeInformation[Long]],
      BasicTypeInfo.LONG_TYPE_INFO.asInstanceOf[TypeInformation[Long]])

    lazy val secondBroadcastStateDesc = new MapStateDescriptor[String, String](
      "broadcast-state-2",
      BasicTypeInfo.STRING_TYPE_INFO,
      BasicTypeInfo.STRING_TYPE_INFO)

    env.setStateBackend(new MemoryStateBackend)
    env.enableCheckpointing(500)
    env.setParallelism(4)
    env.setMaxParallelism(4)

    // 数据流，这里数据流和广播流的Source都是同一种CheckpointedSource。数据流这里做了一系列算子操作，比如flatMap
    val stream = env
      .addSource(
        new CheckpointedSource(4)).setMaxParallelism(1).uid("checkpointedSource")
      .keyBy(
        new KeySelector[(Long, Long), Long] {
          override def getKey(value: (Long, Long)): Long = value._1
        }
      )
      .flatMap(new StatefulFlatMapper)
      .keyBy(
        new KeySelector[(Long, Long), Long] {
          override def getKey(value: (Long, Long)): Long = value._1
        }
      )

    // 广播流
    val broadcastStream = env
      .addSource(
        new CheckpointedSource(4)).setMaxParallelism(1).uid("checkpointedBroadcastSource")
      .broadcast(firstBroadcastStateDesc, secondBroadcastStateDesc)

    // 把数据流和广播流结合起来
    stream
      .connect(broadcastStream)
      .process(new VerifyingBroadcastProcessFunction(expectedFirstState, expectedSecondState))
      .addSink(new AccumulatorCountingSink)
  }
}

// 用户自定义的处理函数
class TestBroadcastProcessFunction
  extends KeyedBroadcastProcessFunction
    [Long, (Long, Long), (Long, Long), (Long, Long)] {

  // 重点说明，这里的 firstBroadcastStateDesc，secondBroadcastStateDesc 其实和之前广播流的那两个MapStateDescriptor无关。
      
  // 这里两个MapStateDescriptor是为了存取BroadcastState，这样在 processBroadcastElement和processElement之间就可以传递变量了。我们完全可以定义新的MapStateDescriptor，只要processBroadcastElement和processElement之间认可就行。
      
  // 这里参数 "broadcast-state-1" 是name, flink就是用这个 name 来从Flink运行时系统中存取MapStateDescriptor 
  lazy val firstBroadcastStateDesc = new MapStateDescriptor[Long, Long](
    "broadcast-state-1",
    BasicTypeInfo.LONG_TYPE_INFO.asInstanceOf[TypeInformation[Long]],
    BasicTypeInfo.LONG_TYPE_INFO.asInstanceOf[TypeInformation[Long]])

  val secondBroadcastStateDesc = new MapStateDescriptor[String, String](
    "broadcast-state-2",
    BasicTypeInfo.STRING_TYPE_INFO,
    BasicTypeInfo.STRING_TYPE_INFO)

  override def processElement(
                value: (Long, Long),
                ctx: KeyedBroadcastProcessFunction
                [Long, (Long, Long), (Long, Long), (Long, Long)]#ReadOnlyContext,
                out: Collector[(Long, Long)]): Unit = {

    // 这里Flink源码中是直接把接受到的业务变量直接再次转发出去
    out.collect(value) 
  }

  override def processBroadcastElement(
                value: (Long, Long),
                ctx: KeyedBroadcastProcessFunction
                [Long, (Long, Long), (Long, Long), (Long, Long)]#Context,
                out: Collector[(Long, Long)]): Unit = {
    // 这里是把最新传来的广播变量存储起来，processElement中可以取出再次使用. 具体是通过firstBroadcastStateDesc 的 name 来获取 BroadcastState
    ctx.getBroadcastState(firstBroadcastStateDesc).put(value._1, value._2)
    ctx.getBroadcastState(secondBroadcastStateDesc).put(value._1.toString, value._2.toString)
  }
}

// 广播流和数据流的Source
private class CheckpointedSource(val numElements: Int)
  extends SourceFunction[(Long, Long)] with CheckpointedFunction {

  private var isRunning = true
  private var state: ListState[CustomCaseClass] = _

  // 就是简单的定期发送
  override def run(ctx: SourceFunction.SourceContext[(Long, Long)]) {
    ctx.emitWatermark(new Watermark(0))
    ctx.getCheckpointLock synchronized {
      var i = 0
      while (i < numElements) {
        ctx.collect(i, i)
        i += 1
      }
    }
    // don't emit a final watermark so that we don't trigger the registered event-time
    // timers
    while (isRunning) Thread.sleep(20)
  }
}

2. Technical Difficulties

MapStateDescriptor

First, to explain some concepts:

• Flink contains two basic state of: Keyed State and Operator State.
• Keyed State and Operator State and can exist in two forms: the original state and managed.
• Flink managed by the state is the state management frame, such as ValueState, ListState, MapState like.
• i.e. the original state raw state, self-managed by the state of the user specific data structure, do checkpoint frame when using byte [] content to read-write, ignorant of its internal data structures.
• MapState is a managed state: the state value is a map. User putor putAlla method of adding elements.

Returning to our example, the broadcast is OperatorState variable part is to save the state of a managed MapState. GetBroadcastState specific function is achieved in DefaultOperatorStateBackend

So we need to use MapStateDescriptor description broadcast state, more flexible use MapStateDescriptor here, because it is the key, value similar use, so personally feel that value directly using the class, which is more convenient, especially for scala from other languages ​​to students.

processBroadcastElement

// 因为主要起到控制作用，所以这个函数的处理相对简单
override def processBroadcastElement(): Unit = {
    // 这里可以把最新传来的广播变量存储起来，processElement中可以取出再次使用，比如
    ctx.getBroadcastState(firstBroadcastStateDesc).put(value._1, value._2)
}

processElement

// 这个函数需要和processBroadcastElement配合起来使用
override def processElement(): Unit = {
    // 可以取出processBroadcastElement之前存储的广播变量，然后用此来处理业务变量，比如
   val secondBroadcastStateDesc = new MapStateDescriptor[String, String](
    "broadcast-state-2",
    BasicTypeInfo.STRING_TYPE_INFO,
    BasicTypeInfo.STRING_TYPE_INFO)  

    var actualSecondState = Map[String, String]()
    for (entry <- ctx.getBroadcastState(secondBroadcastStateDesc).immutableEntries()) {
      val v = secondExpectedBroadcastState.get(entry.getKey).get
      actualSecondState += (entry.getKey -> entry.getValue)
    }  

   // 甚至这里只要和processBroadcastElement一起关联好，可以存储任意类型的变量。不必须要和广播变量的类型一致。重点是声明新的对应的MapStateDescriptor
   // MapStateDescriptor继承了StateDescriptor，其中state为MapState类型，value为Map类型
}

In combination

Because some restrictions, so the following only to find an example from the Internet for everyone to talk about.

// 模式始终存储在MapState中，并将null作为键。broadcast state始终表示为MapState，这是Flink提供的最通用的状态原语。
MapStateDescriptor<Void, Pattern> bcStateDescriptor = 
  new MapStateDescriptor<>("patterns", Types.VOID, Types.POJO(Pattern.class));

// 能看到的是，在处理广播变量时候，存储广播变量到BroadcastState
 public void processBroadcastElement(Pattern pattern, Context ctx, 
     Collector<Tuple2<Long, Pattern>> out) throws Exception {
   // store the new pattern by updating the broadcast state
   BroadcastState<Void, Pattern> bcState = ctx.getBroadcastState(patternDesc);
   // storing in MapState with null as VOID default value
   bcState.put(null, pattern);
 }

// 能看到的是，在处理业务变量时候，从BroadcastState取出广播变量，存取时候实际都是用"patterns"这个name字符串来作为key。
  public void processElement(Action action, ReadOnlyContext ctx, 
     Collector<Tuple2<Long, Pattern>> out) throws Exception {
   // get current pattern from broadcast state
   Pattern pattern = ctx.getBroadcastState(this.patternDesc)
     // access MapState with null as VOID default value
     .get(null);
   // get previous action of current user from keyed state
   String prevAction = prevActionState.value();
   if (pattern != null && prevAction != null) {
     // user had an action before, check if pattern matches
     if (pattern.firstAction.equals(prevAction) && 
         pattern.secondAction.equals(action.action)) {
       // MATCH
       out.collect(new Tuple2<>(ctx.getCurrentKey(), pattern));
     }
   }
   // update keyed state and remember action for next pattern evaluation
   prevActionState.update(action.action);
 }

0x04. Flink source parsing

1. The logic flow broadcast

 * The life cycle of the Broadcast:
 * {@code
 *  -- 初始化逻辑 -> 用一个BroadcastConnectedStream把数据流和广播流结合起来进行拓扑转换
 *        |   
 *        +---->  businessStream = DataStream.filter.map....
 *        |       // 处理业务逻辑的数据流，businessStream 是普通DataStream  
 *        +---->  broadcastStream = DataStream.broadcast(broadcastStateDesc)
 *        |       // 处理配置逻辑的广播数据流，broadcastStream是BroadcastStream类型
 *        +---->  businessStream.connect(broadcastStream)
 *        |                     .process(new processFunction(broadcastStateDesc))
 *        |       // 把业务流,广播流 结合起来，生成一个BroadcastConnectedStream，然后进行 process
 *        +----------> process @ BroadcastConnectedStream   
 *        |                TwoInputStreamOperator<IN1, IN2, OUT> operator =
 *        |                new CoBroadcastWithNonKeyedOperator<>(clean(function),
 *        |                broadcastStateDescriptors);
 *        |                return transform(outTypeInfo, operator);  
 *        |       // 生成一个类型是TwoInputStreamOperator 的 operator，进行 transform
 *        +----------------> transform @ BroadcastConnectedStream  
 *        |                      transform = new TwoInputTransformation<>(
 *        |       			  	       inputStream1.getTransformation(), // 业务流
 *        |       			  	       inputStream2.getTransformation(), // 广播流
 *        |       			  	       ifunctionName, // 用户的UDF
 *        |       			  	       operator, // 算子 CoBroadcastWithNonKeyedOperator
 *        |       			  	       outTypeInfo);  // 输出类型
 *        |       	      		   returnStream = new SingleOutputStreamOperator(transform);
 *        |       			         getExecutionEnvironment().addOperator(transform)
 *        |       // 将业务流,广播流与拓扑联合起来形成一个转换，加到 Env 中，这就完成了拓扑转换 
 *        |       // 最后返回结果是一个SingleOutputStreamOperator。
 * }


 *  数据结构:
 *  -- BroadcastStream. 
 *  就是简单封装一个DataStream，然后记录这个广播流对应的StateDescriptors  
 public class BroadcastStream<T> {  
	private final StreamExecutionEnvironment environment;
	private final DataStream<T> inputStream;
	private final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors;   
 }
   
 *  数据结构:
 *  -- BroadcastConnectedStream. 
 *  把业务流,广播流 结合起来，然后会生成算子和拓扑
public class BroadcastConnectedStream<IN1, IN2> {
	private final StreamExecutionEnvironment environment;
	private final DataStream<IN1> inputStream1;
	private final BroadcastStream<IN2> inputStream2;
	private final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors;
}  

*  真实计算:
*  -- CoBroadcastWithNonKeyedOperator -> 真正对BroadcastProcessFunction的执行，是在这里完成的
public class CoBroadcastWithNonKeyedOperator<IN1, IN2, OUT>
		extends AbstractUdfStreamOperator<OUT, BroadcastProcessFunction<IN1, IN2, OUT>>
		implements TwoInputStreamOperator<IN1, IN2, OUT> {
  
  private final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors;
	private transient TimestampedCollector<OUT> collector;
	private transient Map<MapStateDescriptor<?, ?>, BroadcastState<?, ?>> broadcastStates;
	private transient ReadWriteContextImpl rwContext;
	private transient ReadOnlyContextImpl rContext;
  
	@Override
	public void processElement1(StreamRecord<IN1> element) throws Exception {
		collector.setTimestamp(element);
		rContext.setElement(element);
    // 当上游有最新业务数据来的时候，调用用户自定义的processElement
    // 在这可以把之前存储的广播配置信息取出，然后对业务数据流进行处理    
		userFunction.processElement(element.getValue(), rContext, collector);
		rContext.setElement(null);
	}

	@Override
	public void processElement2(StreamRecord<IN2> element) throws Exception {
		collector.setTimestamp(element);
		rwContext.setElement(element);
    // 当上游有数据来的时候，调用用户自定义的processBroadcastElement
    // 在这可以把最新传送的广播配置信息存起来  
		userFunction.processBroadcastElement(element.getValue(), rwContext, collector);
		rwContext.setElement(null);
	}  
}

2. DataStream key function

// 就是connect，broadcast，分别生成对应的数据流
public class DataStream<T> {
  protected final StreamExecutionEnvironment environment;
  protected final Transformation<T> transformation;

	@PublicEvolving
	public <R> BroadcastConnectedStream<T, R> connect(BroadcastStream<R> broadcastStream) {
		return new BroadcastConnectedStream<>(
				environment,
				this,
				Preconditions.checkNotNull(broadcastStream),
				broadcastStream.getBroadcastStateDescriptor());
	}
		
	@PublicEvolving
	public BroadcastStream<T> broadcast(final MapStateDescriptor<?, ?>... broadcastStateDescriptors) {
		final DataStream<T> broadcastStream = setConnectionType(new BroadcastPartitioner<>());
		return new BroadcastStream<>(environment, broadcastStream, broadcastStateDescriptors);
	}
}

3. The key data structure MapStateDescriptor

It is mainly used to declare a variety of metadata information. Follow-up can be seen, the system is through MapStateDescriptor name, i.e., the first parameter to store / MapStateDescriptor acquired corresponding State.

public class MapStateDescriptor<UK, UV> extends StateDescriptor<MapState<UK, UV>, Map<UK, UV>> {
	/**
	 * Create a new {@code MapStateDescriptor} with the given name and the given type serializers.
	 *
	 * @param name The name of the {@code MapStateDescriptor}.
	 * @param keySerializer The type serializer for the keys in the state.
	 * @param valueSerializer The type serializer for the values in the state.
	 */  
	public MapStateDescriptor(String name, TypeSerializer<UK> keySerializer, TypeSerializer<UV> valueSerializer) {
		super(name, new MapSerializer<>(keySerializer, valueSerializer), null);
	}

	/**
	 * Create a new {@code MapStateDescriptor} with the given name and the given type information.
	 *
	 * @param name The name of the {@code MapStateDescriptor}.
	 * @param keyTypeInfo The type information for the keys in the state.
	 * @param valueTypeInfo The type information for the values in the state.
	 */
	public MapStateDescriptor(String name, TypeInformation<UK> keyTypeInfo, TypeInformation<UV> valueTypeInfo) {
		super(name, new MapTypeInfo<>(keyTypeInfo, valueTypeInfo), null);
	}

	/**
	 * Create a new {@code MapStateDescriptor} with the given name and the given type information.
	 *
	 * <p>If this constructor fails (because it is not possible to describe the type via a class),
	 * consider using the {@link #MapStateDescriptor(String, TypeInformation, TypeInformation)} constructor.
	 *
	 * @param name The name of the {@code MapStateDescriptor}.
	 * @param keyClass The class of the type of keys in the state.
	 * @param valueClass The class of the type of values in the state.
	 */
	public MapStateDescriptor(String name, Class<UK> keyClass, Class<UV> valueClass) {
		super(name, new MapTypeInfo<>(keyClass, valueClass), null);
	}
}

4. Access Status

In the state transfer between the processElement processBroadcastElement and, by the name of MapStateDescriptor key, to store the Flink . That follows a similar call ctx.getBroadcastState(firstBroadcastStateDesc).put(value._1, value._2). Therefore, we next need to introduce the concept of State under Flink's.

State vs checkpoint

First, distinguish between the two concepts, state generally refers to the state of a particular task / operator of. The checkpoint indicates an Flink Job, in a global snapshot of the state of a particular moment in time, that includes the status of all task / operator of. Flink to achieve fault tolerance and recovery by regularly doing checkpoint.

Flink contains two basic state of: Keyed State and Operator State.

Keyed State

As the name suggests, it is based on the state of the KeyedStream. The state is to bind with a particular key, and each key on the KeyedStream flow, probably corresponds to a state.

Operator State

Keyed State with different, Operator State concurrent with a particular operator of a bound instance, the entire operator has only one state. In comparison, in an operator, there may be a number of key, so that a corresponding plurality of keyed state.

For example, Kafka Connector Flink is, use of the operator state. It will be for each connector instance, to save all the instances of consumer topic (partition, offset) mapping.

Flink hosting its original state and states (Raw and Managed State)

This is another dimension.

Keyed State and Operator State each exist in two forms: Managed and RAW , i.e., the original state and host state.

Managed by the state management Flink frame running state, such as the inside of the hash table or RocksDB. For example, "ValueState", "ListState" and so on. Flink runtime these states will encode and write checkpoint.

For example managed keyed state status of the interface to provide different types of access interfaces, which act on the state of the current key input data. In other words, these states can only KeyedStreamuse, can be stream.keyBy(...)obtained KeyedStream. And we can achieve CheckpointedFunctionor ListCheckpointedto use managed operator state interface.

Raw state i.e. the original state, self-managed by the state of the user specific data structure, stored in the operator's own data structure. checkpoint time, does not know the specific contents Flink, only the sequence of bytes written to the checkpoint.

Usually recommended hosted in the status on DataStream, when implementing a user-defined operator, it will be used to its original state.

Returning to our example, the broadcast is OperatorState variable part is to save the state of a managed MapState. Specific getBroadcastState function is DefaultOperatorStateBackend in implementation .

StateDescriptor

You must create a StateDescriptor, to get a handle to the corresponding state. This saves the state name (you can create multiple states, and they must have a unique name so that you can refer to them), the type of values held by the state, and may contain the function specified by the user, for example ReduceFunction. Depending on the status types, you can create ValueStateDescriptor, ListStateDescriptor, ReducingStateDescriptor, FoldingStateDescriptoror MapStateDescriptor.

By state RuntimeContextvisit, so only rich functions use.

OperatorStateBackEnd

. OperatorStateBackEnd key management OperatorState currently only one implementation: DefaultOperatorStateBackend.

DefaultOperatorStateBackend

DefaultOperatorStateBackend status is Map ways to store. Constructed a PartitionableListState (belonging ListState). OperatorState are stored in memory.

public class DefaultOperatorStateBackend implements OperatorStateBackend {
  
	/**
	 * Map for all registered operator states. Maps state name -> state
	 */
	private final Map<String, PartitionableListState<?>> registeredOperatorStates;

	/**
	 * Map for all registered operator broadcast states. Maps state name -> state
	 */
	private final Map<String, BackendWritableBroadcastState<?, ?>> registeredBroadcastStates;  
  
  /**
	 * Cache of already accessed states.
	 *
	 * <p>In contrast to {@link #registeredOperatorStates} which may be repopulated
	 * with restored state, this map is always empty at the beginning.
	 *
	 * <p>TODO this map should be moved to a base class once we have proper hierarchy for the operator state backends.
	 */
	private final Map<String, PartitionableListState<?>> accessedStatesByName;

	private final Map<String, BackendWritableBroadcastState<?, ?>> accessedBroadcastStatesByName;  // 这里用来缓存广播变量
  
  // 这里就是前文中所说的，存取广播变量的API
	public <K, V> BroadcastState<K, V> getBroadcastState(final MapStateDescriptor<K, V> stateDescriptor) throws StateMigrationException {

		String name = Preconditions.checkNotNull(stateDescriptor.getName());

    // 如果之前有，就取出来
		BackendWritableBroadcastState<K, V> previous =
			(BackendWritableBroadcastState<K, V>) accessedBroadcastStatesByName.get(
      name);

		if (previous != null) {
			return previous;
		}

		stateDescriptor.initializeSerializerUnlessSet(getExecutionConfig());
		TypeSerializer<K> broadcastStateKeySerializer = Preconditions.checkNotNull(stateDescriptor.getKeySerializer());
		TypeSerializer<V> broadcastStateValueSerializer = Preconditions.checkNotNull(stateDescriptor.getValueSerializer());

		BackendWritableBroadcastState<K, V> broadcastState =
			(BackendWritableBroadcastState<K, V>) registeredBroadcastStates.get(name);

		if (broadcastState == null) {
			broadcastState = new HeapBroadcastState<>(
					new RegisteredBroadcastStateBackendMetaInfo<>(
							name,
							OperatorStateHandle.Mode.BROADCAST,
							broadcastStateKeySerializer,
							broadcastStateValueSerializer));
			registeredBroadcastStates.put(name, broadcastState);
		} else {
			// has restored state; check compatibility of new state access

			RegisteredBroadcastStateBackendMetaInfo<K, V> restoredBroadcastStateMetaInfo = broadcastState.getStateMetaInfo();

			// check whether new serializers are incompatible
			TypeSerializerSchemaCompatibility<K> keyCompatibility =
				restoredBroadcastStateMetaInfo.updateKeySerializer(broadcastStateKeySerializer);

			TypeSerializerSchemaCompatibility<V> valueCompatibility =
				restoredBroadcastStateMetaInfo.updateValueSerializer(broadcastStateValueSerializer);

			broadcastState.setStateMetaInfo(restoredBroadcastStateMetaInfo);
		}

		accessedBroadcastStatesByName.put(name, broadcastState); // 如果之前没有，就存入
		return broadcastState;
	}  
}

0x05. Reference

Flink principle and implementation: Detailed Flink in state management https://yq.aliyun.com/articles/225623

Flink broadcast configuration to achieve dynamic update https://www.jianshu.com/p/c8c99f613f10

Flink Broadcast State Practical Guide https://blog.csdn.net/u010942041/article/details/93901918

Talk flink State of Broadcast https://www.jianshu.com/p/d6576ae67eae

Working with State https://ci.apache.org/projects/flink/flink-docs-stable/zh/dev/stream/state/state.html