Interactive Conditioning For Kafka Stream

uğur taş :

I want to create interactive conditions for my kafka stream. I just want to know is it possible.

Example use case is this:

I have user click events that stream over my kafka topic. Users define minimum click count for themselves in click options table and i want to inform them when they reach minimum click count. Kstream filters click counts accordingto limit. Eventlistener consumes topic data which produced by kstream outputs and send notification to user.

How can i define Kstream filter conditions user by user according to their persistent data? And can i change it when persistent data changed?

Bartosz Wardziński :

You need to create two topic:

• user-prefs - with user preferences, where key is user id, and value is minimum number of clicks. (Good practice is to make it compacted)
• clicks - topic where original clicks are send, key is user id and value not important (assume some string)

Using KafkaProducer you send user preference (minimal number of click) to user-prefs, if they will change you need to send new message User clicks go to clicks topic.

Suppose you want to aggregate them for some period (60 seconds).

First you have to group and aggregate clicks, and send final result. After that you join final result with user-prefs, where number of minimal click are kept. Filter is made based on number of aggregated clicks and minimal number of clicks

KStream<String, Long> clicks = builder.<String, String>stream("clicks")
    .groupByKey().windowedBy(TimeWindows.of(Duration.ofSeconds(60)).grace(Duration.ofSeconds(1)))
    .count(Materialized.with(Serdes.String(), Serdes.Long()))
    .suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded()))
    .toStream().map((key, value) -> new KeyValue<>(key.key(), value));

KTable<String, Long> userPrefs = builder.<String, Long>table(
    "user-prefs",
    Consumed.with(Serdes.String(), Serdes.Long())
);

clicks.join(
    userPrefs,
    (userClicks, minUserClicksNumber) -> userClicks >= minUserClicksNumber,
    Joined.with(Serdes.String(), Serdes.Long(), Serdes.Long())
)
    .filter((userName, isSufficientNumberOfClick) -> isSufficientNumberOfClick)
    .map(((key, value) -> new KeyValue<>(key, key)))
    .to("output", Produced.with(Serdes.String(), Serdes.String()));