Broadcast stream is a concept commonly used in stream processing to broadcast static data or configuration information to all stream processing tasks so that these tasks can share or use the data. Let us explain broadcast streaming in detail with a vivid example and write a piece of Java code to demonstrate its usefulness.
Example scene：
Suppose you are developing a real-time e-commerce platform and need to compare the prices of goods in real time in stream processing, and the prices of goods may change at any time. At the same time, you want to be able to dynamically update item prices without stopping the stream processing job and without having to redeploy the job. This is a typical scenario where broadcast streaming can come into play.
Personally, I think that when a data stream requires another constantly changing real-time data to participate in calculation and processing, the constantly changing real-time data can be used as a broadcast stream, and the two can be connected to satisfy the requirement without stopping the original data stream processing task. Meet changing computing needs in real time.
First we define a simple class, the price of Huawei meta60:
package source;
public class price {
//设置商品名称
String a = "华为meta60";
//初始化商品价格
int b = 0;
public price(int b) {
this.b = b;
}
public void setA(String a) {
this.a = a;
}
public String getA() {
return a;
}
public int getB() {
return b;
}
}
Define another class for generating Huawei guidance meta60 random prices in real time:
package source;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import java.util.Random;
//用于广播流
public class YaoBroadcastSource implements SourceFunction<price> {
Random random= new Random();
boolean a=true;
@Override
public void run(SourceContext ctx) throws Exception {
while (a){
//生成随机价格
int l = random.nextInt();
price price1 = new price(l);
ctx.collect(price1);
//定时休眠一段时间，模拟价格的变化过程
Thread.sleep(10000);
}
}
@Override
public void cancel() {
a=false;
}
}
Finally, define a price that generates the current Huawei meta60 in real time:
package source;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import java.util.Random;
public class YaoSource implements SourceFunction<Integer> {
Random random= new Random();
boolean a=true;
@Override
public void run(SourceContext ctx) throws Exception {
while (a){
int l = random.nextInt();
ctx.collect(l);
Thread.sleep(3000);
}
}
@Override
public void cancel() {
a=false;
}
}
Now using the broadcast stream, when the Huawei meta60 guide price is less than the current real-time price of Huawei meta60, it will be output to the downstream alarm:
package Sideoutput;
import org.apache.flink.api.common.restartstrategy.RestartStrategies;
import org.apache.flink.api.common.state.MapStateDescriptor;
import org.apache.flink.api.common.state.ReadOnlyBroadcastState;
import org.apache.flink.api.common.time.Time;
import org.apache.flink.streaming.api.datastream.BroadcastStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.co.BroadcastProcessFunction;
import org.apache.flink.util.Collector;
import source.YaoBroadcastSource;
import source.YaoSource;
import source.price;
import java.util.concurrent.TimeUnit;
public class YaoBroadcastProcessFunction {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment executionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment();
//设置重试次数
// executionEnvironment.setRestartStrategy(RestartStrategies.fixedDelayRestart(3, Time.of(10, TimeUnit.SECONDS)));
//定义广播流
DataStreamSource<price> integerDataStreamSource1 = executionEnvironment.addSource(new YaoBroadcastSource());
// integerDataStreamSource1.map(i -> {System.out.println(i.getA());return i;});
// 创建MapStateDescriptor来描述广播流的状态，“productPrices”是状态名称，String.class是键的类型，Integer.class是值的类型
MapStateDescriptor<String, Integer> productPriceDescriptor = new MapStateDescriptor<>("productPrices", String.class, Integer.class);
//创建广播流
BroadcastStream<price> broadcast = integerDataStreamSource1.broadcast(productPriceDescriptor);
//普通数据流
DataStreamSource<Integer> integerDataStreamSource = executionEnvironment.addSource(new YaoSource());
//将普通数据流和广播流进行关联，从而使广播流能够影响普通数据流
integerDataStreamSource.connect(broadcast)
//第一个Integer代表主流（Main Input）的数据类型，第二个price代表广播流（Broadcast Input）的数据类型，第三个Integer代表函数的输出数据类型
.process(new BroadcastProcessFunction<Integer, price, Integer>() {
@Override
//processElement 方法用于处理主流（Main Input）中的每个元素（事件）。主流通常包含实时数据，你可以在这个方法中对主流中的事件进行处理、转换或过滤，并生成输出。
public void processElement(Integer value, BroadcastProcessFunction<Integer, price, Integer>.ReadOnlyContext ctx, Collector<Integer> out) throws Exception {
//获取广播流中的属性状态
ReadOnlyBroadcastState<String, Integer> broadcastState = ctx.getBroadcastState(productPriceDescriptor);
//获取广播流中的属性状态的值
Integer integer = broadcastState.get("华为meta60");
if (integer == null) {
System.out.println("integer为空指针！！！！！！！！！！！！！！！！！！！！！！！！！！！！！");
} else {
System.out.println("华为meta60指导价格：" + integer.toString() + " 实时华为meta60价格：" + value.toString());
//业务判断逻辑
if (value > integer) {
out.collect(value);
System.out.println(" 实时华为meta60：" + value.toString() + "超出官方指导价格告警！！！！");
}
}
}
@Override
//processElement 方法用于处理主流（Main Input）中的每个元素（事件）。主流通常包含实时数据，你可以在这个方法中对主流中的事件进行处理、转换或过滤，并生成输出。
public void processBroadcastElement(price value, BroadcastProcessFunction<Integer, price, Integer>.Context ctx, Collector<Integer> out) throws Exception {
// 将广播流中的属性值传入到广播流中productPrices属性状态,put方法第一个参数传入key，第二个参数传入value
ctx.getBroadcastState(productPriceDescriptor).put(value.getA(),value.getB());
}
});
executionEnvironment.execute();
}
}
Step1: Create common main data stream and broadcast stream
Step2: Define the configuration properties of the broadcast stream
MapStateDescriptor<String, Integer> productPriceDescriptor = new MapStateDescriptor<>("productPrices", String.class, Integer.class);
Step3: Connect the ordinary main data stream and the broadcast stream, call the process method, and process the events in the ordinary main data stream and the broadcast stream respectively by customizing the processElement method and processBroadcastElement method inherited by BroadcastProcessFunction in the process method.
Step4: Define the broadcast stream configuration attribute value by passing in key and value in the processBroadcastElement method.
Step5: In the processElement method, obtain the corresponding broadcast configuration attribute value through the following method, and perform related business logic processing:
//获取广播流中的属性状态
ReadOnlyBroadcastState<String, Integer> broadcastState = ctx.getBroadcastState(productPriceDescriptor);
//通过key获取广播流中的属性状态的值
Integer integer = broadcastState.get("华为meta60");In summary, the broadcast stream is similar to an auxiliary role, allowing the mainstream to connect dynamically changing values with the mainstream in real time without stopping, to achieve the effect of dynamic condition processing.