flink learning state

state function

Keep the historical status of the current key.

state usage

ListState<Integer> vipList = getRuntimeContext().getListState(new ListStateDescriptor<Integer>("vipList", TypeInformation.of(Integer.class)));

There is valueState listState mapstate. rash without setstate

state case

For example, Qidian novels cannot be downloaded. Others can only pirate novels by taking screenshots and extracting text.

The starting point has taken anti-climbing measures. Anti-climbing measures are as follows.

1. If a user turns 1 page in 1 second, or faster, 10 times in a row, then the user is considered a robot.

In the above two cases, the user continuously initiates click events

userId=1 click_time=2023-09-07 00:00:00

userId=1 click_time=2023-09-07 00:00:01

How do we judge 1?

lastClickState retains the time of the user's last click.

clickcntState retains the number of consecutive clicks on the user's 1s1 page.

Compare the next piece of data with the lastClickState.

If the interval is <1s clickcntState +1

If >1s clickcntState is set to 0

At the same time, determine whether the number of clickcntState is >=10. If it is greater, output the userid to the sink.

Come to a demo and talk directly.

package com.chenchi.pojo;

public  class User {
    public Integer userId;
    public Integer vip;

    public long clickTime=System.currentTimeMillis();
    public User() {
    }

    public User(Integer userId, Integer vip) {
        this.userId = userId;
        this.vip = vip;
    }

    public Integer getUserId() {
        return userId;
    }

    public void setUserId(Integer userId) {
        this.userId = userId;
    }

    public Integer getVip() {
        return vip;
    }

    public void setVip(Integer vip) {
        this.vip = vip;
    }

    @Override
    public String toString() {
        return "User{" +
                "userId=" + userId +
                ", vip=" + vip +
                ", clickTime=" + clickTime +
                '}';
    }

    public long getClickTime() {
        return clickTime;
    }

    public void setClickTime(long clickTime) {
        this.clickTime = clickTime;
    }
}

package com.chenchi.pojo;

import org.apache.flink.streaming.api.functions.source.SourceFunction;

import java.util.Random;

public class UserSource implements SourceFunction<User> {
    boolean run = true;
    public UserSource(){}
    int randomBound=1000;
    int interval=1000;
    public UserSource(Integer randomBound){
        this.randomBound=randomBound;
    }
    public UserSource(Integer randomBound,int interval){
        this.randomBound=randomBound;
        this.interval=interval;
    }
    @Override
    public void run(SourceContext<User> sourceContext) throws Exception {
        while (true) {
            Integer userId = new Random().nextInt(randomBound);
            Integer vip = new Random().nextInt(10);
            sourceContext.collect(new User(userId, vip));
            Thread.sleep(interval);
        }
    }

    @Override
    public void cancel() {
        run = false;
    }
}

package com.chenchi.state;

import com.chenchi.pojo.User;
import com.chenchi.pojo.UserSource;
import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
import org.apache.flink.util.Collector;

public class ValueStateDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        CustomProcess customProcess = new CustomProcess();
        DataStreamSource<User> userDataStreamSource = env.addSource(new UserSource(4,100));
        userDataStreamSource.print();
        userDataStreamSource
                .keyBy(user->user.userId)
                .process(customProcess)
                        .print();
        env.execute();
    }

     static class CustomProcess extends KeyedProcessFunction<Integer,User,String> {
        ValueState<Long> lastClickTime;
        ValueState<Integer> cnt;
        @Override
        public void open(Configuration parameters) throws Exception {
            lastClickTime= getRuntimeContext().getState(new ValueStateDescriptor<Long>("click", TypeInformation.of(Long.class)));
            cnt= getRuntimeContext().getState(new ValueStateDescriptor<Integer>("cnt",Integer.class));
            super.open(parameters);
        }

        @Override
        public void processElement(User user, KeyedProcessFunction<Integer, User, String>.Context context, Collector<String> collector) throws Exception {
            Integer userId = user.getUserId();
            long clickTime = user.getClickTime();
            Long last = lastClickTime.value();
            Integer value = cnt.value();
            if (value==null){
                value=0;
            }
            if (last!=null&&clickTime-last<1000){
                //点击太快
                cnt.update(value+1);
            }else {
                //之前可能是不喜欢的页数突然点快了 点击慢就置0
                cnt.update(0);
            }
            //保存当前的listState
            lastClickTime.update(clickTime);
            if (cnt.value()>10){
                collector.collect("userId="+userId+",clickCnt="+cnt.value()+",click太快 注意注意");
            }
        }
    }
}

Print log

10> User{userId=0, vip=0, clickTime=1694083167883}
11> User{userId=0, vip=4, clickTime=1694083167994}
12> User{userId=2, vip=4, clickTime=1694083168102}
1> User{userId=2, vip=7, clickTime=1694083168212}
2> User{userId=2, vip=3, clickTime=1694083168320}
3> User{userId=1, vip=0, clickTime=1694083168428}
4> User{userId=0, vip=7, clickTime=1694083168537}
5> User{userId=2, vip=3, clickTime=1694083168646}
6> User{userId=2, vip=0, clickTime=1694083168757}
7> User{userId=2, vip=3, clickTime=1694083168866}
8> User{userId=0, vip=9, clickTime=1694083168975}
9> User{userId=0, vip=3, clickTime=1694083169084}
10> User{userId=2, vip=7, clickTime=1694083169191}
11> User{userId=0, vip=7, clickTime=1694083169298}
12> User{userId=0, vip=6, clickTime=1694083169406}
1> User{userId=3, vip=9, clickTime=1694083169513}
2> User{userId=0, vip=4, clickTime=1694083169618}
3> User{userId=3, vip=3, clickTime=1694083169726}
4> User{userId=1, vip=8, clickTime=1694083169833}
5> User{userId=2, vip=1, clickTime=1694083169942}
6> User{userId=3, vip=2, clickTime=1694083170050}
7> User{userId=2, vip=8, clickTime=1694083170158}
8> User{userId=1, vip=4, clickTime=1694083170267}
9> User{userId=1, vip=2, clickTime=1694083170374}
10> User{userId=0, vip=1, clickTime=1694083170481}
11> User{userId=3, vip=6, clickTime=1694083170589}
12> User{userId=0, vip=9, clickTime=1694083170696}
1> User{userId=3, vip=1, clickTime=1694083170804}
2> User{userId=1, vip=8, clickTime=1694083170911}
3> User{userId=1, vip=3, clickTime=1694083171018}
4> User{userId=0, vip=7, clickTime=1694083171126}
5> User{userId=1, vip=8, clickTime=1694083171233}
6> User{userId=3, vip=5, clickTime=1694083171341}
7> User{userId=0, vip=8, clickTime=1694083171448}
9> userId=0,clickCnt=11,click太快注意注意

The effect is as expected.

flink learning state

Guess you like