In the CEP running online, there must be frequent rule changes. It is very inelegant to restart and republish the task every time there is a change. Especially in marketing or risk control scenarios that require high real-time performance, if the rule window is too long (one or two weeks) and the state is too large, the restart time will be extended, and some exceptions that need to be handled will be caused during this period Behavior cannot be detected in time.
1. Implementation Analysis
- External loading: usually the rule engine will have a special rule management module to allow users to create their own rules. For Flink tasks, it is necessary to load rules externally
- Dynamic update: It is necessary to provide timing to detect whether the rules have changed
- Historical state cleaning: In pattern matching, it is a series of continuous changes of NFAState. If the rules change, the historical state needs to be cleaned up
- API: An easy-to-use API needs to be provided externally
2. Code implementation
First implement a user API.
package cep.functions;
import java.io.Serializable;
import org.apache.flink.api.common.functions.Function;
import cep.pattern.Pattern;
/**
* @author StephenYou
* Created on 2023-07-23
* Description: 动态Pattern接口（用户调用API）不区分key
*/
public interface DynamicPatternFunction<T> extends Function, Serializable {
/***
* 初始化
* @throws Exception
*/
public void init() throws Exception;
/**
* 注入新的pattern
* @return
*/
public Pattern<T,T> inject() throws Exception;
/**
* 一个扫描周期:ms
* @return
*/
public long getPeriod() throws Exception;
/**
* 规则是否发生变更
* @return
*/
public boolean isChanged() throws Exception;
}
It is hoped that the calling method of the above API is as follows.
//正常调用
CEP.pattern(dataStream,pattern);
//动态Pattern
CEP.injectionPattern(dataStream, new UserDynamicPatternFunction())So you need to modify the CEP-Lib source code
b. Increase the injectionPattern function.
public class CEP {
/***
* Dynamic injection pattern function
* @param input
* @param dynamicPatternFunction
* @return
* @param <T>
*/
public static <T> PatternStream<T> injectionPattern throws Exception (
DataStream<T> input,
DynamicPatternFunction<T> dynamicPatternFunction){
return new PatternStream<>(input, dynamicPatternFunction);
}
}Add the PatternStream constructor, because it needs to be updated dynamically, so it is necessary to pass in the entire function.
public class PatternStream<T> {
PatternStream(final DataStream<T> inputStream, DynamicPatternFunction<T> dynamicPatternFunction) throws Exception {
this(PatternStreamBuilder.forStreamAndPatternFunction(inputStream, dynamicPatternFunction));
}
}Modify PatternStreamBuilder.build to add the process of calling functions.
final CepOperator<IN, K, OUT> operator = null;
if (patternFunction == null ) {
operator = new CepOperator<>(
inputSerializer,
isProcessingTime,
nfaFactory,
comparator,
pattern.getAfterMatchSkipStrategy(),
processFunction,
lateDataOutputTag);
} else {
operator = new CepOperator<>(
inputSerializer,
isProcessingTime,
patternFunction,
comparator,
null,
processFunction,
lateDataOutputTag);
}Add the corresponding CepOperator constructor.
public CepOperator(
final TypeSerializer<IN> inputSerializer,
final boolean isProcessingTime,
final DynamicPatternFunction patternFunction,
@Nullable final EventComparator<IN> comparator,
@Nullable final AfterMatchSkipStrategy afterMatchSkipStrategy,
final PatternProcessFunction<IN, OUT> function,
@Nullable final OutputTag<IN> lateDataOutputTag) {
super(function);
this.inputSerializer = Preconditions.checkNotNull(inputSerializer);
this.patternFunction = patternFunction;
this.isProcessingTime = isProcessingTime;
this.comparator = comparator;
this.lateDataOutputTag = lateDataOutputTag;
if (afterMatchSkipStrategy == null) {
this.afterMatchSkipStrategy = AfterMatchSkipStrategy.noSkip();
} else {
this.afterMatchSkipStrategy = afterMatchSkipStrategy;
}
this.nfaFactory = null;
}Load Pattern and construct NFA
@Override
public void open() throws Exception {
super.open();
timerService =
getInternalTimerService(
"watermark-callbacks", VoidNamespaceSerializer.INSTANCE, this);
//初始化
if (patternFunction != null) {
patternFunction.init();
Pattern pattern = patternFunction.inject();
afterMatchSkipStrategy = pattern.getAfterMatchSkipStrategy();
boolean timeoutHandling = getUserFunction() instanceof TimedOutPartialMatchHandler;
nfaFactory = NFACompiler.compileFactory(pattern, timeoutHandling);
long period = patternFunction.getPeriod();
// 注册定时器检测规则是否变更
if (period > 0) {
getProcessingTimeService().registerTimer(timerService.currentProcessingTime() + period, this::onProcessingTime);
}
}
nfa = nfaFactory.createNFA();
nfa.open(cepRuntimeContext, new Configuration());
context = new ContextFunctionImpl();
collector = new TimestampedCollector<>(output);
cepTimerService = new TimerServiceImpl();
// metrics
this.numLateRecordsDropped = metrics.counter(LATE_ELEMENTS_DROPPED_METRIC_NAME);
}Status cleaning is divided into two parts: matching status data cleaning and timer cleaning;
Perform state cleanup:
@Override
public void processElement(StreamRecord<IN> element) throws Exception {
if (patternFunction != null) {
// 规则版本更新
if (needRefresh.value() < refreshVersion.get()) {
//清除状态
computationStates.clear();
elementQueueState.clear();
partialMatches.releaseCacheStatisticsTimer();
//清除定时器
Iterable<Long> registerTime = registerTimeState.get();
if (registerTime != null) {
Iterator<Long> iterator = registerTime.iterator();
while (iterator.hasNext()) {
Long l = iterator.next();
//删除定时器
timerService.deleteEventTimeTimer(VoidNamespace.INSTANCE, l);
timerService.deleteProcessingTimeTimer(VoidNamespace.INSTANCE, l);
//状态清理
iterator.remove();
}
}
//更新当前的版本
needRefresh.update(refreshVersion.get());
}
}
}The above is to clear the status and version when processing each piece of data. The next step is to initialize the state and version.
@Override
public void initializeState(StateInitializationContext context) throws Exception {
super.initializeState(context);
//初始化状态
if (patternFunction != null) {
/**
* 两个标识位状态
*/
refreshFlagState = context.getOperatorStateStore()
.getUnionListState(new ListStateDescriptor<Integer>("refreshFlagState", Integer.class));
if (context.isRestored()) {
if (refreshFlagState.get().iterator().hasNext()) {
refreshVersion = new AtomicInteger(refreshFlagState.get().iterator().next());
}
} else {
refreshVersion = new AtomicInteger(0);
}
needRefresh = context.getKeyedStateStore()
.getState(new ValueStateDescriptor<Integer>("needRefreshState", Integer.class, 0));
}
}3. Test verification
Set the Pattern to change every 10s.
PatternStream patternStream = CEP.injectionPattern(source, new TestDynamicPatternFunction());
patternStream.select(new PatternSelectFunction<Tuple3<String, Long, String>, Map>() {
@Override
public Map select(Map map) throws Exception {
map.put("processingTime", System.currentTimeMillis());
return map;
}
}).print();
env.execute("SyCep");
}
public static class TestDynamicPatternFunction implements DynamicPatternFunction<Tuple3<String, Long, String>> {
public TestDynamicPatternFunction() {
this.flag = true;
}
boolean flag;
int time = 0;
@Override
public void init() throws Exception {
flag = true;
}
@Override
public Pattern<Tuple3<String, Long, String>, Tuple3<String, Long, String>> inject()
throws Exception {
// 2种pattern
if (flag) {
Pattern pattern = Pattern
.<Tuple3<String, Long, String>>begin("start")
.where(new IterativeCondition<Tuple3<String, Long, String>>() {
@Override
public boolean filter(Tuple3<String, Long, String> value,
Context<Tuple3<String, Long, String>> ctx) throws Exception {
return value.f2.equals("success");
}
})
.times(1)
.followedBy("middle")
.where(new IterativeCondition<Tuple3<String, Long, String>>() {
@Override
public boolean filter(Tuple3<String, Long, String> value,
Context<Tuple3<String, Long, String>> ctx) throws Exception {
return value.f2.equals("fail");
}
})
.times(1)
.next("end");
return pattern;
} else {
Pattern pattern = Pattern
.<Tuple3<String, Long, String>>begin("start2")
.where(new IterativeCondition<Tuple3<String, Long, String>>() {
@Override
public boolean filter(Tuple3<String, Long, String> value,
Context<Tuple3<String, Long, String>> ctx) throws Exception {
return value.f2.equals("success2");
}
})
.times(2)
.next("middle2")
.where(new IterativeCondition<Tuple3<String, Long, String>>() {
@Override
public boolean filter(Tuple3<String, Long, String> value,
Context<Tuple3<String, Long, String>> ctx) throws Exception {
return value.f2.equals("fail2");
}
})
.times(2)
.next("end2");
return pattern;
}
}
@Override
public long getPeriod() throws Exception {
return 10000;
}
@Override
public boolean isChanged() throws Exception {
flag = !flag ;
time += getPeriod();
System.out.println("change pattern : " + time);
return true;
}
}Print result: as expected
4. Source address
If you find it useful, please give me a little star. ^_^