代码问题:用idea写的SparkStreaming和Kafka整合,实时从kafka中消费数据,有错误,望指正。

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/D:/jars/webproject/kafka-libs/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/D:/jars/webproject/spark-jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.ExceptionInInitializerError
    at org.apache.spark.streaming.dstream.InputDStream.<init>(InputDStream.scala:78)
    at org.apache.spark.streaming.dstream.ReceiverInputDStream.<init>(ReceiverInputDStream.scala:42)
    at org.apache.spark.streaming.kafka.KafkaInputDStream.<init>(KafkaInputDStream.scala:56)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:91)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:66)
    at cn.tedu.service.Driver$.main(Driver.scala:30)//报的这儿有错
    at cn.tedu.service.Driver.main(Driver.scala)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.1
    at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
    at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
    at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751)
    at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
    at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
    ... 7 more

Process finished with exit code 1

代码如下:
package cn.tedu.service

import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.{SparkConf, SparkContext}

/**
 * 将SparkStreaming和Kafka整合,从kafka中消费数据
 */
object Driver {

  def main(args: Array[String]): Unit = {
    // 如果从Kafka消费数据,Local模式的线程数至少是2个
    //其中一个线程负责SparkStreaming ,另一个负责消费Kafka
    val conf=new SparkConf().setMaster("local[2]").setAppName("kafka")

    val sc=new SparkContext(conf)

    val ssc=new StreamingContext(sc,Seconds(5))

    //zookeeper集群地址
    val zkHosts="hadoop01:2181,hadoop02:2181,hadoop03:2181"
    //定义消费者组名
    val groupId="gp1"
    //指定消费的主题信息,key是主题名,value是消费的线程数
    //可以消费多个主题,比如:Map("weblog"->1,"enbook"->1)
    val topics=Map("weblog"->1)

    //通过整合包提供的工具类,从kafka指定主题中消费数据
    val kafkaSource=KafkaUtils.createStream(ssc,zkHosts,groupId,topics)//这儿
      .map{x=>x._2}

    //TODO 实时数据的处理
    kafkaSource.print

    ssc.start()

    ssc.awaitTermination()
  }
}

发布了5 篇原创文章 · 获赞 3 · 访问量 47

猜你喜欢

转载自blog.csdn.net/qq_41676088/article/details/104113462
今日推荐