How zookeeper maintains offset

How zookeeper maintains offset

introduction:

​ SparkStreaming consumes Kafka in two ways, namely Receiver mode and Direct mode. Receiver can maintain the offset by itself. In Direct mode, Executer directly connects to Kafka to consume data. When it is used, it is read and lost. SparkStream generates and topic RDD with the same number of partitions improves efficiency and saves resources. You need to manually maintain the offset. You can use zookeeper, mysql, checkpoint, Hbase, etc. The most used in the company is zookeeper, so I have a rough exploration How zookeeper maintains the offset, there is nothing too low-level, just to understand the process.

Zookeeper node structure:

Let's first understand the node structure of zookeeper.

After starting zookeeper, start the zookeeper client and check the nodes inside

Execute zkCli.sh -server localhost:2181open client

Execute to ls /view all nodes

You can see that we are familiar with brokers (kafka cluster nodes), yarn-leader-election, hadoop-ha, consumers, and hbase.

Since you want to maintain the offset of kafka, you must check consumers (consumers of kafka)

You can see a bunch of consumers, some of these consumers are named by themselves, and some are customized (do not define groupid, automatically assign),

Then turn on the consumers we need and give g001

There is only one directory offsets, which is created by zookeeper itself, and there may be ids, owners in other consumer directories

ids records the list of running consumers under the consumer group

owners record the list of topics consumed by the consumer group

offsets records the offset of each partition of the topic consumed by each consumer under the consumer group

Continue to open the offset to see the row offset of each partition

There are two topic directories, yes, it is the topic in kafka, and under the topic is the partition

What is stored in the partition?

The first is the row offset stored in partition 0, which is 94

So far, you can know which consumer is the consumer, the topic consumed, how many partitions are there, and which one is consumed by each partition

So zookeeper maintains the row offset, which is updating this value 96.

Node information:

cZxid: zxid when the node was created

ctime: node creation time

mZxid: zxid when the node was last updated

mtime: the time when the node was last updated

cversion: update times of child node data

dataVersion: the number of times the data of this node is updated

aclVersion: update times of node ACL (authorization information)

ephemeralOwner: If the node is a temporary node, the ephemeralOwner value represents the session id bound to the node. If the node is not a temporary node, the ephemeralOwner value is 0

dataLength: node data length, in this case the length of hello world

numChildren: the number of child nodes

How to maintain

The preparation of kafka is mainly the name of the consumer, which topic to consume, the address of the kafka node, and the configuration of kafka parameters

//指定组名
val group = "g001"
//创建SparkConf
val conf = new SparkConf().setAppName(this.getClass.getName).setMaster("local")
//创建SparkStreaming，并设置间隔时间
val ssc = new StreamingContext(conf, Duration(5000))
//使用updateStateByKey会用到
ssc.checkpoint("f:/bigdata/cmcc/out")
//指定消费的 topic 名字
val topic = "offsettest"
//指定kafka的broker地址(sparkStream的Task直连到kafka的分区上，用更加底层的API消费，效率更高)
val brokerList = "hadoop01:9092,hadoop02:9092,hadoop03:9092"
//创建 stream 时使用的 topic 名字集合，SparkStreaming可同时消费多个topic
val topics: Set[String] = Set(topic)
//准备kafka的参数
val kafkaParams = Map(
    "metadata.broker.list" -> brokerList,
    "group.id" -> group,
    //从头开始读取数据
    "auto.offset.reset" -> kafka.api.OffsetRequest.SmallestTimeString
)

Zookeeper preparation and directory creation

Zookeeper node address, zookeeper address object, zookeeper client object, storage DStream object

//指定zk的地址，后期更新消费的偏移量时使用(以后可以使用Redis、MySQL来记录偏移量)
val zkQuorum = "hadoop01:2181,hadoop02:2181,hadoop03:2181"

//创建一个 ZKGroupTopicDirs 对象,其实是指定往zk中写入数据的目录，用于保存偏移量
//对应着zookeeper的节点结构，创建目录，其本身就是一个地址
val topicDirs = new ZKGroupTopicDirs(group, topic)

//是zookeeper的客户端，可以从zk中读取偏移量数据，并更新偏移量
val zkClient = new ZkClient(zkQuorum)

//储存从kafka上获取的DStream
var kafkaStream: InputDStream[(String, String)] = null

//如果 zookeeper 中有保存 offset，我们会利用这个 offset 作为 kafkaStream 的起始位置
var fromOffsets: Map[TopicAndPartition, Long] = Map()

Find the number of partitions in the directory

//获取 zookeeper 中的路径 "/g001/offsets/offsettest/"
val zkTopicPath = s"${topicDirs.consumerOffsetDir}"
//查询该路径下是否字节点
//其实就是topic下的分区个数，上文提到的有0和1两个分区，int类型
val children = zkClient.countChildren(zkTopicPath)

According to the number of partitions to determine whether to read from the beginning, or to obtain data based on the offset of each partition

if (children > 0) {//有分区
    for (i <- 0 until children) {
        //获取目录下的partition，就是之前看的partition0的值--94
        val partitionOffset = zkClient.readData[String](s"$zkTopicPath/${i}")
        //将topic和partition对应到TopicAndPartition中，即(offsettest/0)....
        val tp = TopicAndPartition(topic, i)
        //以key-value的形式放入map，即(offsettest/0)：94
        fromOffsets += (tp -> partitionOffset.toLong)
    }
    //这个会将 kafka 的消息进行 transform，最终 kafak 的数据都会变成 (kafka的key, message) 这样的 tuple，其实就是规定一下格式
    val messageHandler = (mmd: MessageAndMetadata[String, String]) => (mmd.key(), mmd.message())

    //通过KafkaUtils创建直连的DStream（fromOffsets参数的作用是:按照前面计算好了的偏移量继续消费数据）
    //[String, String, StringDecoder, StringDecoder,     (String, String)]
    //  key    value    key的解码方式   value的解码方式
    kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder, (String, String)](ssc, kafkaParams, fromOffsets, messageHandler)
} else {
    //如果未保存，根据 kafkaParam 的配置使用最新(largest)或者最旧的（smallest） offset
    kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics)
}

Maintenance update offset

//偏移量的范围
var offsetRanges = Array[OffsetRange]()
//直连方式只有在KafkaDStream的RDD中才能获取偏移量，那么就不能到调用DStream的Transformation
//所以只能子在kafkaStream调用foreachRDD，获取RDD的偏移量，然后就是对RDD进行操作了
//依次迭代KafkaDStream中的KafkaRDD


kafkaStream.foreachRDD { kafkaRDD =>
    //定义zk客户端对象和zk目录对象，因为在算子中，在work端执行，所以要重新定义，不让会出现序列化的问题
    val zkClient = new ZkClient(zkQuorum)
    val topicDirs = new ZKGroupTopicDirs(group, topic)
    
    //只有KafkaRDD可以强转成HasOffsetRanges，并获取到偏移量，多个分区-->list
    offsetRanges = kafkaRDD.asInstanceOf[HasOffsetRanges].offsetRanges
    
    
    for (o <- offsetRanges) {
        //offset的存在目录
        val zkPath = s"${topicDirs.consumerOffsetDir}/${o.partition}"
        //将该 partition 的 offset 更新到 zookeeper
        ZkUtils.updatePersistentPath(zkClient, zkPath, o.untilOffset.toString)
    }
}

There are several attributes of o in the above code

o.untilOffset//消费到了第几行，行偏移量
o.fromOffset//之前的行偏移量
o.partition//所在的分区
o.topic//所在的topic