SparkStreaming踩坑之Kafka重复消费

1.问题描述
使用SparkStreaming连接Kafka的demo程序每次重启，都会从Kafka队列里第一条数据开始消费。

修改enable.auto.commit相关参数都无效。

2.原因分析
demo程序使用"KafkaUtils.createDirectStream"创建Kafka输入流，此API内部使用了Kafka客户端低阶API，不支持offset自动提交（提交到zookeeper）。

"KafkaUtils.createDirectStream"官方文档：

http://spark.apache.org/docs/2.2.0/streaming-kafka-0-8-integration.html

3.对策
方案一）通过zookeeper提供的API，自己编写代码，将offset提交到zookeeper；服务启动时，从zookeeper读取offset，并作为"KafkaUtils.createDirectStream"的输入参数
优点：可与基于zookeeper的监控系统融合，对消费情况进行监控

缺点：频繁的读写offset可能影响zookeeper集群性能，从而影响到Kafka集群的稳定性

方案二）自己编写代码维护offset，并将offset保存到MongoDB或者redis
优点：不影响zookeeper集群性能；可基于MongoDB或者redis自主实现消费情况的监控

缺点：无法与基于zookeeper的监控系统融合

4.代码示例
基于上述方案二，将offset保存到redis，并在服务重启时从redis获取offset，确保不会重复消费。

1）Scala操作redis的工具类

package xxx.demo.scala_test

import redis.clients.jedis.{Jedis, JedisPool, JedisPoolConfig, Protocol}

import org.apache.commons.pool2.impl.GenericObjectPoolConfig

import org.slf4j.LoggerFactory

import com.typesafe.scalalogging.slf4j.Logger

class RedisUtil extends Serializable {

  @transient private var pool: JedisPool = null

  @transient val logger = Logger(LoggerFactory.getLogger("cn.com.flaginfo.demo.scala_test.RedisUtil"))

  def makePool(redisHost: String, redisPort: Int,

               password: String, database: Int): Unit = {

    if (pool == null) {

      val poolConfig = new GenericObjectPoolConfig()

      pool = new JedisPool(poolConfig, redisHost, redisPort, Protocol.DEFAULT_TIMEOUT, password, database)

       

      val hook = new Thread {

        override def run = {

          pool.destroy()

          logger.debug("JedisPool destroyed by ShutdownHook")

        }

      }

      sys.addShutdownHook(hook.run)

    }

  }

  def jedisPool: JedisPool = {

    assert(pool != null)

    pool

  }

   

  def generateKafkaOffsetGroupIdTopicKey(groupId : String, topic : String) : String = {

    groupId + "/" + topic

  }

}

2）初始化redis工具

import kafka.serializer.{StringDecoder, DefaultDecoder}

import kafka.common.TopicAndPartition

import kafka.message.MessageAndMetadata

import org.apache.spark._

import org.apache.spark.streaming._

import org.apache.spark.streaming.StreamingContext._

import org.apache.spark.streaming.kafka._

import org.apache.spark.sql.SparkSession

import org.bson.Document

import com.mongodb.spark.config._

import com.mongodb.spark._

import com.mongodb._

import xxx.demo.model._

import redis.clients.jedis.{Jedis, JedisPool, JedisPoolConfig, Protocol}

import scala.collection.JavaConversions.{mapAsScalaMap}

import org.slf4j.LoggerFactory

import com.typesafe.scalalogging.slf4j.Logger

...

  var redisUtil = new RedisUtil()

  redisUtil.makePool(redisHost, redisPort, redisPassword, redisDatabase)

  var jedisPool = redisUtil.jedisPool

注意，不需要密码验证时，redisPassword必须设置为null，空字符串会报错。

3）从redis获取上次的offset

var kafkaOffsetKey = redisUtil.generateKafkaOffsetGroupIdTopicKey(groupId, kafkaTopicName)

var allOffset: java.util.Map[String, String] = jedisPool.getResource().hgetAll(kafkaOffsetKey)

val fromOffsets = scala.collection.mutable.Map[TopicAndPartition,Long]()

if( allOffset != null && !allOffset.isEmpty() ){

  // Jedis获取的Java Map转换为Scala Map

  var allOffsetScala : scala.collection.mutable.Map[String, String] = mapAsScalaMap[String, String](allOffset)

  for(offset <- allOffsetScala){

    // 将offset传入kafka参数。offset._1 : partition, offset._2 : offset

    fromOffsets += (TopicAndPartition(newsAnalysisTopic, offset._1.toInt) -> offset._2.toLong)

  }

  logger.debug( "fromOffsets : " + fromOffsets.toString() )

}

else{

  // 初次消费

  for( i <- 0 to (newsAnalysisTopicPartitionCount - 1) ){

    fromOffsets += (TopicAndPartition(newsAnalysisTopic, i) -> 0)

  }

  logger.debug( "fromOffsets : " + fromOffsets.toString() )

}

// mutable转换为imutable

var imutableFromOffsets = Map[TopicAndPartition,Long](

  fromOffsets.map(kv => (kv._1, kv._2)).toList: _*

)

4）定义消息过滤器：根据metadata取出需要的字段

val messageHandler: (MessageAndMetadata[String, String]) => (String,String, Long, Int) = (mmd: MessageAndMetadata[String, String]) =>

    (mmd.topic, mmd.message, mmd.offset, mmd.partition)

5）创建kafka输入流

val kafkaParam = Map[String, String](

  "bootstrap.servers" -> kafkaServer,

  "group.id" -> groupId,

  "client.id" -> clientId,

  "auto.offset.reset" -> "smallest",

  "enable.auto.commit" -> "false"

  )

var kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder, (String,String, Long, Int)](ssc, kafkaParam, imutableFromOffsets, messageHandler)

其中ssc为StreamingContext对象

6）业务逻辑代码中，将offset更新到redis

var kafkaOffsetKey = redisUtil.generateKafkaOffsetGroupIdTopicKey(groupId, kafkaTopicName)

// _._1 : topic name, _._2 : message body, _._3 : offset, _._4 : partition

kafkaStream.foreachRDD { rdd =>

  if( !rdd.isEmpty() ){  // 此处判断可防止offsetRanges.foreach循环意外执行

    var offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges

    // 处理数据

    rdd.foreach{ row =>

      logger.info("message : " + row + offsetRanges)

    }

    // 开启Redis事务

    var jedis = jedisPool.getResource()

    var jedisPipeline = jedis.pipelined()

    jedisPipeline.multi()

    // 更新offset

    offsetRanges.foreach { offsetRange =>

      logger.debug("partition : " + offsetRange.partition + " fromOffset:  " + offsetRange.fromOffset + " untilOffset: " + offsetRange.untilOffset)

      jedisPipeline.hset(kafkaOffsetKey, offsetRange.partition.toString(), offsetRange.untilOffset.toString())

    }

    jedisPipeline.exec() //提交事务

    jedisPipeline.sync //关闭pipeline

    jedis.close()

  }

}