Flink 学习六 Flink 窗口计算API

Flink 学习六 Flink 窗口计算API

1.窗口 (window)概念和分类

窗口 window 是处理无限流的核心就是把无界的数据流,按照一定的规则划分成一段一段的有界的数据流(桶),然后再这个有界的数据流里面去做计算;

2.分类体系

2.1 滚动窗口

相邻窗口之间是没有数据重合

• window 大小可以是时间,可以是数据长度
• 按照数据流是否可以是 keyed , 在分类,nonkey window 也叫做global window (并行度为1)

2.2 滑动窗口

滑动步长小于窗口大小,数据会有重合;正常情况下是有数据重合的

• window 大小可以是时间,可以是数据长度
• 按照数据流是否可以是 keyed , 在分类,nonkey window 也叫做global window (并行度为1)

2.3 会话窗口

按照时间分类的一个类别,一段时间没有数据,就重新开一个窗口;

• window 大小可以是时间;
• 按照数据流是否可以是 keyed , 在分类,nonkey window 也叫做global window (并行度为1)

3.窗口计算API

计算的API 主要是根据最后window的类型是否是 Keyed ,

3.1 KeyedWindow

上游数据按照hash key 后计算,并行度可控

stream
       .keyBy(...)               <-  keyed versus non-keyed windows      //按照上面key 分组
       .window(...)              <-  required: "assigner" //window 类别 是 滚动/滑动/会话
      [.trigger(...)]            <-  optional: "trigger" (else default trigger) //数据进入桶触发
      [.evictor(...)]            <-  optional: "evictor" (else no evictor)  // 桶数据擦除策略
      [.allowedLateness(...)]    <-  optional: "lateness" (else zero)  //是否允许数据迟到
      [.sideOutputLateData(...)] <-  optional: "output tag" (else no side output for late data) //测流输出处理迟到数据,补救策略
       .reduce/aggregate/apply()      <-  required: "function" //处理函数,一般是聚合计算
      [.getSideOutput(...)]      <-  optional: "output tag" //

        //滚动窗口  处理时间语义
        dataStream.windowAll(TumblingProcessingTimeWindows.of(Time.seconds(5)));
        //滑动窗口  处理时间语义
        dataStream.windowAll(SlidingProcessingTimeWindows.of(Time.seconds(5),Time.seconds(1)));

        //滚动窗口  事件时间语义
        dataStream.windowAll(TumblingEventTimeWindows.of(Time.seconds(5)));
        //滑动窗口  事件时间语义
        dataStream.windowAll(SlidingEventTimeWindows.of(Time.seconds(5),Time.seconds(1)));
        
        //计数滚动窗口
        dataStream.countWindowAll(100);
        //计数滑动窗口
        dataStream.countWindowAll(100,20);

3.2 NonKeyedWindow

上游数据聚合到一起计算 ,并行度是1

stream 
       .windowAll(...)           <-  required: "assigner"  //窗口类型是全局窗口 ,这个不一样  //window 类别 是 滚动/滑动/会话
      [.trigger(...)]            <-  optional: "trigger" (else default trigger)
      [.evictor(...)]            <-  optional: "evictor" (else no evictor)
      [.allowedLateness(...)]    <-  optional: "lateness" (else zero)
      [.sideOutputLateData(...)] <-  optional: "output tag" (else no side output for late data)
       .reduce/aggregate/apply()      <-  required: "function"
      [.getSideOutput(...)]      <-  optional: "output tag"

        //滚动窗口  处理时间语义
        keyedStream.windowAll(TumblingProcessingTimeWindows.of(Time.seconds(5)));
        //滑动窗口  处理时间语义
        keyedStream.windowAll(SlidingProcessingTimeWindows.of(Time.seconds(5),Time.seconds(1)));

        //滚动窗口  事件时间语义
        keyedStream.windowAll(TumblingEventTimeWindows.of(Time.seconds(5)));
        //滑动窗口  事件时间语义
        keyedStream.windowAll(SlidingEventTimeWindows.of(Time.seconds(5),Time.seconds(1)));

        //计数滚动窗口
        keyedStream.countWindow(100);
        //计数滑动窗口
        keyedStream.countWindow(100,20);

3.3窗口聚合算子

3.3.1 增量窗口聚合算子

数据来一条出库一条,比如累加, min,max,minBy,maxBy,sum,reduce,aggregate

min,max,minBy,maxBy,sum

        KeyedStream<EventLog, Long> keyedStream = streamOperator.keyBy(EventLog::getGuid);
        SingleOutputStreamOperator<EventLog> serviceTime = keyedStream
                .countWindow(10,2) // 创建滑动 count 窗口 
                .min("serviceTime") ; //serviceTime 是符合逻辑的,其他字段是窗口的第一条数据
				.max("serviceTime") ; //serviceTime 是符合逻辑的,其他字段是窗口的第一条数据
				.minBy("serviceTime") ; //serviceTime 是符合逻辑的,其他字段是 最小 serviceTime  所在哪一行
				.maxBy("serviceTime") ; //serviceTime 是符合逻辑的,其他字段是 最大 serviceTime  所在哪一行
				.sum("serviceTime") ; // serviceTime 是符合逻辑的(serviceTime 字段之和),其他字段是窗口的第一条数据

reduce

reduce 函数的返回值写死了,和数据流中一样

        keyedStream.countWindow(10,2).reduce(new ReduceFunction<EventLog>() {
    
    
            @Override
            public EventLog reduce(EventLog value1, EventLog value2) throws Exception {
    
    
                //
                return null;
            }
        })

aggregate

aggregate 函数的返回值可以自定义: 详细见下面 3.4.1

    keyedStream.countWindow(10,2).aggregate(new AggregateFunction<EventLog, Object, Object>() {
    
    
            ......
        })

3.3.2 全量聚合算子

process

详细见下面 3.4.2

相比于apply 更加的强大,可以拿到上下文

apply

        keyedStream.countWindow(10,2).apply(new WindowFunction<EventLog, Object, Long, GlobalWindow>() {
    
    
            /**
             * Evaluates the window and outputs none or several elements.
             *
             * @param key  key
             * @param window  窗口,可以拿到开始和结束
             * @param input The elements in the window being evaluated. //窗口所有数据
             * @param out //收集器
             */
            @Override
            public void apply(Long aLong, GlobalWindow window, Iterable<EventLog> input, Collector<Object> out) throws Exception {
    
    
            }
        })

3.3.3 示例

演示 KeyedWindow 使用的较多,和NonKeyedWindow 区别也不大;

原始需求 :

计算时间窗口内 EventLog 中根据guid 分组的个数

@Data
@AllArgsConstructor
@NoArgsConstructor
public class EventLog {
    
    

    private Long guid; //用户id
    private String event;//用户事件
    private String timeStamp; //事件发生时间
    private Integer serviceTime; //接口时间

}

自定义source 生成数据使用

public class CustomEventLogSourceFunction implements SourceFunction<EventLog> {
    
    

	volatile boolean runFlag = true;
    SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd hh:mm:ss");


	@Override
	public void run(SourceContext<EventLog> sourceContext) throws Exception {
    
    
        Random random = new Random();
        while (runFlag) {
    
    
            EventLog eventLog = new EventLog(Long.valueOf(random.nextInt(5)),"xw"+random.nextInt(3),simpleDateFormat.format(new Date()));
            Thread.sleep(random.nextInt(5)*1000);
            sourceContext.collect(eventLog);
		}

	}

	@Override
	public void cancel() {
    
    
		runFlag = false;
	}
}

aggregate

聚合:窗口中**,拿到每一条数据**,去做聚合计算

public class _03_Window {
    
    

	public static void main(String[] args) throws Exception {
    
    
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd hh:mm:ss");
		// 获取环境
		Configuration configuration = new Configuration();
		configuration.setInteger("rest.port", 8822);
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(configuration).setParallelism(1); //并行度设置为1 好观察
		//DataStream<String> dataStreamSource = env.socketTextStream("192.168.141.141", 9000);
        DataStreamSource<EventLog> dataStream = env.addSource(new CustomEventLogSourceFunction());

		// 创建 Watermark 策略 事件时间推进策略
		WatermarkStrategy<EventLog> watermarkStrategy = WatermarkStrategy
				.<EventLog>forBoundedOutOfOrderness(Duration.ofMillis(0))
				.withTimestampAssigner(new SerializableTimestampAssigner<EventLog>() {
    
    
					@Override
					public long extractTimestamp(EventLog element, long recordTimestamp) {
    
    
                        try {
    
    
                            return sdf.parse(element.getTimeStamp()).getTime();
                        } catch (Exception e) {
    
    
                            return 0;
                        }
                    }
				});
		// 分配wm , 使用事件时间
		DataStream<EventLog> streamOperator = dataStream.assignTimestampsAndWatermarks(watermarkStrategy).disableChaining();

		// 需求 10 s 统计30s 数据 用户行为个数
		streamOperator.keyBy(EventLog::getGuid)
                .window(SlidingEventTimeWindows.of(Time.seconds(30), Time.seconds(10))) // 创建滑动窗口
				// .apply() // 全部窗口数据给你,自定义全窗口运算逻辑,返回计算结果,灵活度更大
                //.process() //全部窗口数据给你,自定义全窗口运算逻辑,返回计算结果,灵活度更大 可以获取上下文,数据更多
				.aggregate(new AggregateFunction<EventLog, Tuple2<Long,Integer>, Tuple2<Long,Integer>>() {
    
     // api 聚合
					// 初始化累加器
					@Override
					public Tuple2<Long,Integer> createAccumulator() {
    
    
						return Tuple2.of(null,0);
					}

					// 累加逻辑
					@Override
					public Tuple2<Long,Integer> add(EventLog value, Tuple2<Long,Integer> accumulator) {
    
    
						return Tuple2.of(value.getGuid(),accumulator.f1+1);
					}

					// 获取结果
					@Override
					public Tuple2<Long,Integer> getResult(Tuple2<Long,Integer> accumulator) {
    
    
						return accumulator;
					}

					// 分区合并 方法
					@Override
					public Tuple2<Long,Integer> merge(Tuple2<Long,Integer> a, Tuple2<Long,Integer> b) {
    
    
						return Tuple2.of(a.f0,a.f1+b.f1);
					}
				}).print();


		env.execute();

	}
}

process

聚合:窗口结束后,拿到窗口中每一条数据,也可以拿到上下文,去做自定义的逻辑处理数据,比 aggregate 更加的自由

keyedStream.window(TumblingProcessingTimeWindows.of(Time.seconds(20)))
                .process(new ProcessWindowFunction<EventLog, Tuple2<Long, Integer>, Long, TimeWindow>() {
    
    
                    @Override
                    public void process(Long aLong,
                                        ProcessWindowFunction<EventLog, Tuple2<Long, Integer>, Long, TimeWindow>.Context context,
                                        Iterable<EventLog> elements, Collector<Tuple2<Long, Integer>> out) throws Exception {
    
    
                        TimeWindow window = context.window();
                        String start = sdf.format(new Date(window.getStart()));
                        String end = sdf.format(new Date(window.getEnd()));
                        Iterator<EventLog> iterator = elements.iterator();
                        int count = 0;
                        while (iterator.hasNext()) {
    
    
                            iterator.next();
                            count++;
                        }
                        System.out.println("start:" + start + ";end:" + end + ";count:" + count);
                        out.collect(Tuple2.of(aLong, count));
                    }
                }).print();

3.4 延迟数据处理

3.4.1 allowdLateness

允许数据迟到 3.5 的图可以参考下

public class _05_Window_allowedLateness {
    
    

    public static void main(String[] args) throws Exception {
    
    
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd hh:mm:ss");
        // 获取环境
        Configuration configuration = new Configuration();
        configuration.setInteger("rest.port", 8822);
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(configuration).setParallelism(1);
        DataStream<String> dataStreamSource = env.socketTextStream("192.168.141.141", 9000);
        // 字符串流==> 对象流

        SingleOutputStreamOperator<EventLog> outputStreamOperator = dataStreamSource.map(new MapFunction<String, EventLog>() {
    
    
            @Override
            public EventLog map(String value) throws Exception {
    
    
                String[] split = value.split(",");
                return new EventLog(Long.valueOf(split[0]), split[1], split[2]);
            }
        });

        // 创建 Watermark 策略 事件时间推进策略
        WatermarkStrategy<EventLog> watermarkStrategy = WatermarkStrategy
                .<EventLog>forBoundedOutOfOrderness(Duration.ofMillis(0))
                .withTimestampAssigner(new SerializableTimestampAssigner<EventLog>() {
    
    
                    @Override
                    public long extractTimestamp(EventLog element, long recordTimestamp) {
    
    
                         return  Long.valueOf(element.getTimeStamp());
                    }
                });
        // 分配wm , 使用事件时间
        DataStream<EventLog> streamOperator = outputStreamOperator.assignTimestampsAndWatermarks(watermarkStrategy)
                .disableChaining();
        OutputTag<EventLog> latedata = new OutputTag<EventLog>("latedata", TypeInformation.of(EventLog.class));
        KeyedStream<EventLog, Long> keyedStream = streamOperator.keyBy(EventLog::getGuid);

        SingleOutputStreamOperator<EventLog> guid = keyedStream
                .window(TumblingEventTimeWindows.of(Time.seconds(30)))
                .allowedLateness(Time.seconds(3))  //允许迟到5s
                .sideOutputLateData(latedata)  //超过允许迟到的值后 ,输出到测流
                .apply(new WindowFunction<EventLog, EventLog, Long, TimeWindow>() {
    
    
                    long count = 0;
                    @Override
                    public void apply(Long aLong, TimeWindow window, Iterable<EventLog> input, Collector<EventLog> out) throws Exception {
    
    
                        for (EventLog eventLog : input) {
    
    
                            count+=eventLog.getGuid();
                        }
                        EventLog eventLog = new EventLog();
                        eventLog.setGuid(count);
                        out.collect(eventLog);
                        String start = sdf.format(new Date(window.getStart()));
                        String end = sdf.format(new Date(window.getEnd()));
                        System.out.println("==> start:" + start + ";end:" + end + ";count:" + count);
                    }
                });

        guid.print("主流输出");
        DataStream<EventLog> sideOutput = guid.getSideOutput(latedata);
        sideOutput.print("测流输出");
        env.execute();

    }
}

3.4.2 window时间&watermark

3.5 窗口的触发机制

3.5.1 trigger 窗口计算触发

窗口计算的触发,是trigger 类决定,不同的 WindowAssigner 对应不同的trigger

Trigger (org.apache.flink.streaming.api.windowing.triggers)
    ProcessingTimeoutTrigger (org.apache.flink.streaming.api.windowing.triggers)
    EventTimeTrigger (org.apache.flink.streaming.api.windowing.triggers)
    CountTrigger (org.apache.flink.streaming.api.windowing.triggers)
    DeltaTrigger (org.apache.flink.streaming.api.windowing.triggers)
    NeverTrigger in GlobalWindows (org.apache.flink.streaming.api.windowing.assigners)
    ContinuousEventTimeTrigger (org.apache.flink.streaming.api.windowing.triggers)
    PurgingTrigger (org.apache.flink.streaming.api.windowing.triggers)
    ContinuousProcessingTimeTrigger (org.apache.flink.streaming.api.windowing.triggers)
    ProcessingTimeTrigger (org.apache.flink.streaming.api.windowing.triggers)

EventTimeTrigger

事件时间触发器

@PublicEvolving
public class EventTimeTrigger extends Trigger<Object, TimeWindow> {
    
    
    private static final long serialVersionUID = 1L;

    private EventTimeTrigger() {
    
    }

    /**
    * 数据来的时候触发
    */
    @Override
    public TriggerResult onElement(
            Object element, long timestamp, TimeWindow window, TriggerContext ctx)
            throws Exception {
    
    
        //窗口的最大时间小于当前的Watermark() 开启新窗口
        if (window.maxTimestamp() <= ctx.getCurrentWatermark()) {
    
    
            // if the watermark is already past the window fire immediately
            return TriggerResult.FIRE;
        } else {
    
     //否则继续
            ctx.registerEventTimeTimer(window.maxTimestamp());
            return TriggerResult.CONTINUE;
        }
    }

    //当事件时间推进的时候 也判断是否开启新窗口
    @Override
    public TriggerResult onEventTime(long time, TimeWindow window, TriggerContext ctx) {
    
    
        return time == window.maxTimestamp() ? TriggerResult.FIRE : TriggerResult.CONTINUE;
    }

     //该类是处理事件时间,不处理ProcessingTime
    @Override
    public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx)
            throws Exception {
    
    
        return TriggerResult.CONTINUE;
    }

    @Override
    public void clear(TimeWindow window, TriggerContext ctx) throws Exception {
    
    
        ctx.deleteEventTimeTimer(window.maxTimestamp());
    }

    @Override
    public boolean canMerge() {
    
    
        return true;
    }

    @Override
    public void onMerge(TimeWindow window, OnMergeContext ctx) {
    
    
        // only register a timer if the watermark is not yet past the end of the merged window
        // this is in line with the logic in onElement(). If the watermark is past the end of
        // the window onElement() will fire and setting a timer here would fire the window twice.
        long windowMaxTimestamp = window.maxTimestamp();
        if (windowMaxTimestamp > ctx.getCurrentWatermark()) {
    
    
            ctx.registerEventTimeTimer(windowMaxTimestamp);
        }
    }

    @Override
    public String toString() {
    
    
        return "EventTimeTrigger()";
    }

    /**
     * Creates an event-time trigger that fires once the watermark passes the end of the window.
     *
     * <p>Once the trigger fires all elements are discarded. Elements that arrive late immediately
     * trigger window evaluation with just this one element.
     */
    public static EventTimeTrigger create() {
    
    
        return new EventTimeTrigger();
    }
}

CountTrigger

数据窗口触发器

public class CountTrigger<W extends Window> extends Trigger<Object, W> {
    
    
    private static final long serialVersionUID = 1L;

    private final long maxCount;

    private final ReducingStateDescriptor<Long> stateDesc =
            new ReducingStateDescriptor<>("count", new Sum(), LongSerializer.INSTANCE);

    private CountTrigger(long maxCount) {
    
    
        this.maxCount = maxCount;
    }

    // 数量超过最大 触发
    @Override
    public TriggerResult onElement(Object element, long timestamp, W window, TriggerContext ctx)
            throws Exception {
    
    
        ReducingState<Long> count = ctx.getPartitionedState(stateDesc);
        count.add(1L);
        if (count.get() >= maxCount) {
    
    
            count.clear();
            return TriggerResult.FIRE;
        }
        return TriggerResult.CONTINUE;
    }
// 不做 onEventTime 的逻辑潘墩
    @Override
    public TriggerResult onEventTime(long time, W window, TriggerContext ctx) {
    
    
        return TriggerResult.CONTINUE;
    }
// 不做 onProcessingTime 的逻辑潘墩
    @Override
    public TriggerResult onProcessingTime(long time, W window, TriggerContext ctx)
            throws Exception {
    
    
        return TriggerResult.CONTINUE;
    }

    @Override
    public void clear(W window, TriggerContext ctx) throws Exception {
    
    
        ctx.getPartitionedState(stateDesc).clear();
    }

    @Override
    public boolean canMerge() {
    
    
        return true;
    }

    @Override
    public void onMerge(W window, OnMergeContext ctx) throws Exception {
    
    
        ctx.mergePartitionedState(stateDesc);
    }

    @Override
    public String toString() {
    
    
        return "CountTrigger(" + maxCount + ")";
    }

    /**
     * Creates a trigger that fires once the number of elements in a pane reaches the given count.
     *
     * @param maxCount The count of elements at which to fire.
     * @param <W> The type of {@link Window Windows} on which this trigger can operate.
     */
    public static <W extends Window> CountTrigger<W> of(long maxCount) {
    
    
        return new CountTrigger<>(maxCount);
    }

    private static class Sum implements ReduceFunction<Long> {
    
    
        private static final long serialVersionUID = 1L;

        @Override
        public Long reduce(Long value1, Long value2) throws Exception {
    
    
            return value1 + value2;
        }
    }
}

结合上面的例子 写一个自定义触发器,当数据流中出现特定值,立刻触发窗口的计算,而不需要等到窗口结束

  class IEventTimeTrigger extends Trigger<EventLog, TimeWindow> {
    
    
    private static final long serialVersionUID = 1L;


      /**
       * watermark 是不是大于窗口结束点 触发新的窗口
       */
    @Override
    public TriggerResult onElement(
            EventLog element, long timestamp, TimeWindow window, TriggerContext ctx)
            throws Exception {
    
    
        if (window.maxTimestamp() <= ctx.getCurrentWatermark()) {
    
    
            // if the watermark is already past the window fire immediately
            return TriggerResult.FIRE;
        } else {
    
    
            ctx.registerEventTimeTimer(window.maxTimestamp());
            if(element.getGuid().equals(1111111)){
    
       //如果数据的GUID 是 1111111 提前触发窗口计算0
                return TriggerResult.FIRE;
            }
            return TriggerResult.CONTINUE;
        }
    }

    @Override
    public TriggerResult onEventTime(long time, TimeWindow window, TriggerContext ctx) {
    
    
        return time == window.maxTimestamp() ? TriggerResult.FIRE : TriggerResult.CONTINUE;
    }

    @Override
    public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx)
            throws Exception {
    
    
        return TriggerResult.CONTINUE;
    }

    @Override
    public void clear(TimeWindow window, TriggerContext ctx) throws Exception {
    
    
        ctx.deleteEventTimeTimer(window.maxTimestamp());
    }

    @Override
    public boolean canMerge() {
    
    
        return true;
    }

    @Override
    public void onMerge(TimeWindow window, OnMergeContext ctx) {
    
    
        long windowMaxTimestamp = window.maxTimestamp();
        if (windowMaxTimestamp > ctx.getCurrentWatermark()) {
    
    
            ctx.registerEventTimeTimer(windowMaxTimestamp);
        }
    }

    @Override
    public String toString() {
    
    
        return "EventTimeTrigger()";
    }

}

3.5.2 evictor窗口计算移出数据

窗口触发前/后 数据的移除机制

Evictor (org.apache.flink.streaming.api.windowing.evictors)
	CountEvictor (org.apache.flink.streaming.api.windowing.evictors)  //数量窗口清除
	DeltaEvictor (org.apache.flink.streaming.api.windowing.evictors)  //滑动窗口清除
	TimeEvictor (org.apache.flink.streaming.api.windowing.evictors)   //时间窗口清除

public interface Evictor<T, W extends Window> extends Serializable {
    
    

//计算之前
    void evictBefore(
            Iterable<TimestampedValue<T>> elements,
            int size,
            W window,
            EvictorContext evictorContext);

//计算之后调用
    void evictAfter(
            Iterable<TimestampedValue<T>> elements,
            int size,
            W window,
            EvictorContext evictorContext);
...............
}

重写一个ITimeEvictor 相对于 TimeEvictor 的改动

class ITimeEvictor<W extends Window> implements Evictor<EventLog, W> {
    
    
......................

	private void evict(Iterable<TimestampedValue<EventLog>> elements, int size, EvictorContext ctx) {
    
    
		if (!hasTimestamp(elements)) {
    
    
			return;
		}

		long currentTime = getMaxTimestamp(elements);
		long evictCutoff = currentTime - windowSize;

		for (Iterator<TimestampedValue<EventLog>> iterator = elements.iterator(); iterator.hasNext();) {
    
    
			TimestampedValue<EventLog> record = iterator.next();
			// if (record.getTimestamp() <= evictCutoff)
			// 添加条件 ,窗口计算移除 Guid().equals(222222)
			if (record.getTimestamp() <= evictCutoff || record.getValue().getGuid().equals(222222)) {
    
    
				iterator.remove();
			}
		}
	}
 .................................
}

3.5.3 调用示例

   keyedStream.window(TumblingEventTimeWindows.of(Time.seconds(30)))
				.trigger(new IEventTimeTrigger()) // 如果数据的GUID 是 1111111 提前触发窗口计算
				.evictor(ITimeEvictor.of(Time.seconds(30), false)) // 计算前移除GUID 是 222222的数据
				.apply(.....)

3.5.4 调用时机

• 数据到达窗口后,调用 Trigger 的 onElement() 方法

• 根据Trigger 的 onElement() 方法 返回值 判断是否要触发窗口计算

• 若触发窗口计算,在计算前调用Evictor(after/before) 来移除某些数据