分享一下我老师大神的人工智能教程!零基础,通俗易懂!http://blog.csdn.net/jiangjunshow
也欢迎大家转载本篇文章。分享知识,造福人民,实现我们中华民族伟大复兴!
为了支持断点续传功能,需要将offset保存在一个地方,下次从这个offset开始。librdkafka提供了本地文件保存的方式。
下面的代码演示了
1. 要用topic config对象设置 offset.store.path和offset.store.method
2. start函数接受参数OFFSET_STORED
std::unique_ptr<RdKafka::Consumer> consumer(RdKafka::Consumer::create(global_conf_.get(), err_)); if (!consumer) { throw KafkaError("Failed to create consumer"); } stringstream stream; stream << topic_name_ << "-" << partition_idx << ".txt"; string file_path = stream.str(); string lasterr; topic_conf_->set("offset.store.path", file_path, lasterr); topic_conf_->set("offset.store.method", "file", lasterr); std::unique_ptr<RdKafka::Topic> topic(RdKafka::Topic::create(consumer.get(), topic_name_, topic_conf_.get(), err_)); // RdKafka::ErrorCode resp = consumer->start(topic.get(), partition_idx, RdKafka::Topic::OFFSET_BEGINNING); RdKafka::ErrorCode resp = consumer->start(topic.get(), partition_idx, RdKafka::Topic::OFFSET_STORED); if (resp != RdKafka::ERR_NO_ERROR) { throw KafkaError(RdKafka::err2str(resp)); }
注意:
1. 当前进程目录下会出现$topic-$partiton-idx.txt文件
2. 运行后正常读取数据后需要过一会儿才会写入数值
这个写入时间可以设置:offset.store.sync.interval.ms
3. 程序启动时,可以先手动将offset写入文件,然后再启动
详细配置参考github
https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md