Article Directory
This blog post is to use Kafka API, consumers - with a time stamp query message
more advanced usage see my blog Kafka Series
Reference:
https://www.orchome.com/451
https://www.w3cschool.cn /apache_kafka/apache_kafka_workflow.html
part of the reference case "Kafka entry and practice"
I. Introduction
kafka concepts related to the introduction see the official documents and other Bowen
official Chinese document
kafka introductory presentation
Kafka consumer API provides offsetsForTimes (Map <TopicPartition, Long> timestampsToSearch) method, into the reference to Map, Key partition is to be queried, Value is the time stamp to be queried, the return value is greater than or equal to the timestamp of the first message offset and a timestamp. If the partition does not exist, this method will always be blocked.
If we want to start spending from a certain time, you can use it to find offsets, after a call to seek (TopicPartition partition, long offset) offset consumption will be reset to the specified offset from the query.
II. To achieve
2.1 introduces dependence
Mainly rely on spring-kafka
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit-dep</artifactId>
<version>4.9</version>
<scope>test</scope>
</dependency>
<!-- kafka start -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.10.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
<version>0.10.1.1</version>
</dependency>
<!-- kafka end -->
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2-beta-5</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.3</version>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.1</version>
</dependency>
</dependencies>
2.2 Consumer - specified time
ConsumerForTime.java
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
/**
* @author 司马缸砸缸了
* @date 2020/1/9 17:09
* @description 根据时间消费
*/
public class ConsumerForTime {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
// 消费者组
props.put(ConsumerConfig.GROUP_ID_CONFIG, "test");
// key序列化方式
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
// value序列化方式
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
// 自动提交偏移量
// 如果设置成true,偏移量由auto.commit.interval.ms控制自动提交的频率
// 如果设置成false,不需要定时的提交offset，可以自己控制offset，当消息认为已消费过了，这个时候再去提交它们的偏移量。
// 这个很有用的，当消费的消息结合了一些处理逻辑，这个消息就不应该认为是已经消费的，直到它完成了整个处理。
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
// 自动提交的频率
props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
// 订阅主题
consumer.assign(Arrays.asList(new TopicPartition("my-topic", 0)));
try {
Map<TopicPartition, Long> timestampsToSearch = new HashMap<TopicPartition, Long>();
// 构造查询的分区
TopicPartition partition = new TopicPartition("my-topic", 0);
// 设置查询12个小时之前消息的偏移量
timestampsToSearch.put(partition, (System.currentTimeMillis() - 12 * 3600 * 1000));
// 会返回时间大于等于查找时间的第一个偏移量
Map<TopicPartition, OffsetAndTimestamp> offsetMap = consumer.offsetsForTimes(timestampsToSearch);
OffsetAndTimestamp offsetTimestamp = null;
// 遍历查询的分区，我们只查询了一个分区
for (Map.Entry<TopicPartition, OffsetAndTimestamp> entry : offsetMap.entrySet()) {
// 若查询时间大于时间戳索引文件中最大记录的索引时间,此时value为空,即待查询时间点之后没有新消息生成
offsetTimestamp = entry.getValue();
if (null != offsetTimestamp) {
System.out.printf("partition = %d, offset = %d,timestamp= %d%n",
entry.getKey().partition(), entry.getValue()
.offset(), entry.getValue().timestamp());
// 重置消费起始偏移量
consumer.seek(partition, entry.getValue().offset());
}
}
while (true) {
// 拉取消息
ConsumerRecords<String, String> records = consumer.poll(1000);
for (ConsumerRecord<String, String> record : records)
System.out.printf(
"partition = %d, offset = %d,key= %s value = %s%n",
record.partition(), record.offset(), record.key(),
record.value());
}
} catch (Exception e) {
e.printStackTrace();
} finally {
consumer.close();
}
}
}
Detailed parameters, please refer to:
http://www.shixinke.com/java/kafka-configuration
https://blog.csdn.net/xiaozhu_you/article/details/91493258
Source Address
-CLOUD-KAFKA-IT CLIENT : the Spring integration kafka tutorial source code. Bowen in this CSDN kafka series.
Recommended items
CLOUD-IT : IT service management platform, integrated basic services, middleware services, alarm monitoring services.
CLOUD-ACTIVITI6-IT : the Activiti tutorial source code. CSDN Activiti Bowen in this series.
CLOUD-elasticsearch-IT : elasticsearch tutorial source code. CSDN elasticsearch Bowen in this series.
CLOUD-KAFKA-IT : the Spring integration kafka tutorial source code. Bowen in this CSDN kafka series.
-CLOUD-KAFKA-IT CLIENT : Client tutorial source Kafka used to live. Bowen in this CSDN kafka series.Open source project, continuously updated, the Star ~ like please
