Flink tutorial (17) AggregatingState use case of Keyed State state management to find the average value

Series of articles

Flink tutorial (13) Keyed State state management ValueState using temperature difference alarm
Flink tutorial (14) Keyed State state management MapState use case
Flink tutorial (15) Keyed State state management ListState using ValueState to implement
Flink tutorial (16) Keyed State state The ReducingState use case of management to find the maximum value
Flink tutorial (17) The AggregatingState use case of Keyed State state management to find the average value

One, the method of AggregatingState

• AggregatingState needs to be used in conjunction with AggregateFunction
• The add() method adds an element to trigger the AggregateFunction calculation
• get()Get the value of State
  

Two, AggregatingState descriptor

When defining the descriptor, the second parameter requires the AggregateFunction class

//定义描述器
AggregatingStateDescriptor aggregatingStateDescriptor = new AggregatingStateDescriptor(
         "avg-temp",
         new SensorRecordUtils.MyAvgTemp(),
         TypeInformation.of(new TypeHint<Tuple2<Double, Integer>>(){
    
    })
 );

//获取ReducingState
aggregatingState = getRuntimeContext().getAggregatingState(aggregatingStateDescriptor);

Third, the custom AggregateFunction class

Flink calculates the average value. The first parameter of Tuple2 is the sum of the current temperature, and the second parameter is the number of data.

Accumulator.f0 / accumulator.f1 in getResult get the average value.

public static class MyAvgTemp implements AggregateFunction<SensorRecord, Tuple2<Double, Integer>, Double> {
    
    

    @Override
    public Tuple2<Double, Integer> createAccumulator() {
    
    
        return Tuple2.of(0.0, 0);
    }

    @Override
    public Tuple2<Double, Integer> add(SensorRecord value, Tuple2<Double, Integer> accumulator) {
    
    
        Integer currentCount = accumulator.f1;
        currentCount += 1;
        accumulator.f1 = currentCount;
        return new Tuple2<>(accumulator.f0 + value.getRecord(), accumulator.f1);
    }

    @Override
    public Double getResult(Tuple2<Double, Integer> accumulator) {
    
    
        return accumulator.f0 / accumulator.f1;
    }

    @Override
    public Tuple2<Double, Integer> merge(Tuple2<Double, Integer> a, Tuple2<Double, Integer> b) {
    
    
        return new Tuple2<>(a.f0 + b.f0, a.f1 + b.f1);
    }
}

Fourth, the main body of the program

public class Test06_AggregatingState {
    
    

    public static void main(String[] args) throws Exception {
    
    

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        //方便测试，设置为1
        env.setParallelism(1);

        DataStreamSource<String> source = env.socketTextStream(BaseConstant.URL, BaseConstant.PORT);

        /*
        设置watermark和指定时间属性
         */
        SingleOutputStreamOperator<SensorRecord> dataStream = source
                .map(new SensorRecordUtils.BeanMap());

        dataStream
                .keyBy(SensorRecord::getId)
                .process(new MyKeyedProcessFunction())
                .print();

        env.execute();
    }
}

Five, KeyedProcessFunction processing class

public static class MyKeyedProcessFunction extends KeyedProcessFunction<String, SensorRecord, Tuple2<String, Double>> {
    
    

    private transient AggregatingState aggregatingState;

    @Override
    public void open(Configuration parameters) throws Exception {
    
    
        super.open(parameters);

        //定义描述器
        AggregatingStateDescriptor aggregatingStateDescriptor = new AggregatingStateDescriptor(
                "avg-temp",
                new SensorRecordUtils.MyAvgTemp(),
                TypeInformation.of(new TypeHint<Tuple2<Double, Integer>>(){
    
    })
        );

        //获取ReducingState
        aggregatingState = getRuntimeContext().getAggregatingState(aggregatingStateDescriptor);
    }

    @Override
    public void processElement(SensorRecord value, Context ctx, Collector<Tuple2<String, Double>> out) throws Exception {
    
    

        aggregatingState.add(value);
        out.collect(Tuple2.of(value.getId(), (Double) aggregatingState.get()) );
    }
}