Flink + Redis real-time early warning

As the population dividend cuts slowly, fighting Internet products increasingly fierce, we began to sink optimistic about the potential of the market, a lot of fight, interesting headlines and other manufacturers to pull through new incentives, shopping incentives and other policies first to seize the user, grow up. Other manufacturers have followed each, have launched their own speed version of the product, such as today's headlines speed version, Tencent news Extreme Edition, also by drawing new incentives, reading incentives and other policies to attract users.

For this type of APP, real-time risk control is essential, a more common scenario is real-time risk control anti-cheating brush interfaces. Brush interface is a cheating black production, various operations on the APP, generally corresponding to an interface of the background, the user operates the APP data will be reported by the interface to the background, if black yield obtained by cracking a new APP increase user interface, they will be able to skip directly to the landing APP step back tone interface constructs a false data for profit. For this type of business, we can achieve real-time anti brush function interface by Flink + Redis. Data flow diagram is shown below:

The interface is generally cheating brush across the landing APP operations, direct data transfer interface to send Server side, these users do not exist in the APP reported inside the log, then we can Flink reported the APP real-time Redis writes up new users, and then Server-side user interface reports up Redis in the user for comparison, it is judged if there is not a brush Redis user interfaces.

For this demand, it may require real-time calculation engine can achieve millisecond delay, otherwise it will cause the user's misjudgment and affect the user experience. For this reason we have chosen Flink as the real-time calculation engine.

The main logic of the code as follows:

//配置flink运行环境
val env = StreamExecutionEnvironment.getExecutionEnvironment
//val env = StreamExecutionEnvironment.createLocalEnvironment()
env.enableCheckpointing(1000 * 60 * 5)
env.getCheckpointConfig.setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE)
env.getCheckpointConfig.setMinPauseBetweenCheckpoints(1000 * 60 * 3)
env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
env.setStateBackend(new FsStateBackend(checkPointPath))
env.getConfig.setLatencyTrackingInterval(1000)
env.getConfig.registerTypeWithKryoSerializer(classOf[Log], classOf[ProtobufSerializer])
env.setStreamTimeCharacteristic(EventTime)
env.setParallelism(parallel)
env.getConfig.setLatencyTrackingInterval(1000)

//kafka source,实时消费kafka中日志解析出用户id
val stream = env.addSource(new FlinkKafkaConsumer010[Array[Log]](topic, new LogDeserializationSchema(), properties))
val data = stream.flatMap(x => x)
  .map(log =>{
    val userid = log.getUid.getUuid
    val current_time = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date())
    (userid,current_time)
  }).filter(record=>{
  val userid = record._1
  var flag = false
  if(userid != null && !"".equals(userid)){
    flag = true
  }
  flag
})

//redis sink,将APP上报日志的用户id写入redis供server端匹配
data.addSink(new RedisSink[(String, String)](getJedisClusterConfig, new RedisSinkMapper))
env.execute("newsinfo_active_userid_to_redis")

One of the more important points:

1 Structure kafka source

val stream = env.addSource(new FlinkKafkaConsumer010[Array[Log]](topic, new LogDeserializationSchema(), properties))

All serialized data reported by APP general, we need to define the anti-sequence method, LogDeserializationSchema is a protobuf type of anti-sequence method.

//将kafka中的数据解析为google protobuf 的Log,一个message可能包含多条Log
class LogDeserializationSchema extends AbstractDeserializationSchema[Array[Log]] {
  override def deserialize(message: Array[Byte]): Array[Log] = {
    val data = ArrayBuffer[Log]()
    val input = new ByteArrayInputStream(message)
    while (input.available() > 0) {
      try {
        data += Log.parseDelimitedFrom(input)
      } catch {
        case _: Throwable =>
      }
    }
    input.close()
    data.toArray
  }
}

2 redis sink

Here it is the online open source flink-connector-redis dependent libraries.
More details, see: http://bahir.apache.org/docs/flink/current/flink-streaming-redis

Maven dependency follows

<dependency>
    <groupId>org.apache.bahir</groupId>
    <artifactId>flink-connector-redis_2.11</artifactId>
    <version>1.1-SNAPSHOT</version>
</dependency>

Sink Redis class provides an interface for transmitting data to the Redis. The receiver may use three different methods Redis environment with different types of communication:
a single server Redis
Redis cluster
Redis the Sentinel

Redis Sink core class is RedisMappe is an interface that we want to use when writing your own operating redis class implements three methods in this interface, as shown below:

class RedisExampleMapper extends RedisMapper[(String, String)]{
  override def getCommandDescription: RedisCommandDescription = {
    new RedisCommandDescription(RedisCommand.HSET, "HASH_NAME")
  }

  override def getKeyFromData(data: (String, String)): String = data._1

  override def getValueFromData(data: (String, String)): String = data._2
}
val conf = new FlinkJedisPoolConfig.Builder().setHost("127.0.0.1").build()
stream.addSink(new RedisSink[(String, String)](conf, new RedisExampleMapper))

And redis structure using RedisCommand correspondence relationship setting data structure types.

Flink + Redis real-time early warning

Guess you like