How does Flink obtain Kafka's topic, partition, and offset information

background

After Flink consumes Kafka data, we often convert the data into POJO objects to facilitate subsequent data processing, but each POJO type is different, and a new deserialization class needs to be developed each time. There will be a lot of repeated code, how to implement an inflationary deserialization class solution?

In another scenario, when flink subscribes to multiple kafka topics at the same time, we need to do different logical processing according to different topics. For the data sourced from topic_A, set a property of pojo to 1, and for the data sourced from topic_B, set the property of POJO to 2. How to deal with this?

method

In fact, the above two problems can be solved by implementing the KafkaDeserializationSchema interface.

1, general type anti-ordering

/**
 * @desc 自定义实现kafka的消息反序列化
 * @throws
 */
public class CustomKafkaDeserializationSchema<T>  implements KafkaDeserializationSchema<T> {
    @Override
    public boolean isEndOfStream(T nextElement) {
        return false;
    }

    @Override
    public T deserialize(ConsumerRecord<byte[], byte[]> record) throws Exception {
        TypeReference<T> MAP_TYPE_REFERENCE =new TypeReference<T>(){};
        T bTrack = JSON.parseObject(record.value(),MAP_TYPE_REFERENCE.getType());

        return bTrack;
    }

    @Override
    public TypeInformation<T> getProducedType() {
        return TypeInformation.of(new TypeHint<T>() {
            @Override
            public TypeInformation<T> getTypeInfo() {
                return super.getTypeInfo();
            }
        });
    }
}

2. Custom serialization

// kafka数据源
DataStreamSource<POJO> stream = env.addSource(
        new FlinkKafkaConsumer<POJO>(topics, new CustomKafkaDeserializationSchema<POJO>(), PropertyFileUtils.readProFile("kafka.properties"))
                .setCommitOffsetsOnCheckpoints(true)
                .setStartFromLatest());

3. If you need to obtain kafka-related information, you can read it as shown in the figure below