Spark Streaming整合Kafka有两种方式:Receiver和Direct方式
两种方式的区别?
Receiver方式:接收固定时间间隔的数据(放在内存中的),使用Kafka高级的API,自动维护偏移量,达到固定的时间才进行处理,效率低并且容易丢失数据。
Direct直连方式:相当于直接连接到Kafka的分区上,使用Kafka底层的API,效率高,需要自己维护偏移量。(常用)
(1)Receiver方式
编写程序实现Receiver方式连接(KafkaReceiverStreaming.scala)
package com.fyy.spark.streaming
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.{SparkConf, streaming}
/**
* @Title: KafkaReceiverStreaming
* @ProjectName SparkStreamingProject
* @Description: Spark Streaming对接kafka的Receiver方式
* @author fanyanyan
*/
object KafkaReceiverStreaming {
def main(args: Array[String]): Unit = {
if(args.length != 4){
System.err.println("请输入参数: <zkQuorum> <group> <topics> <numThreads>")
System.exit(1)
}
val Array(zkQuorum,group,topics,numThreads)= args
val sparkConf = new SparkConf()
val ssc = new StreamingContext(sparkConf, streaming.Seconds(5))
val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap
// Spark Streaming对接Kafka
val mess = KafkaUtils.createStream(ssc,zkQuorum,group,topicMap)
// 进行词频统计(主要的代码逻辑)
val result = mess.map(_._2).flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)
result.print()
ssc.start()
ssc.awaitTermination()
}
}
(2)Direct直连方式
编写代码实现Direct方式连接(KafkaDirectStreaming.scala)
package com.fyy.spark.streaming
import org.apache.commons.codec.StringDecoder
import org.apache.spark.{SparkConf, streaming}
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.kafka.KafkaUtils
/**
* @Title: KafkaDirectStreaming
* @ProjectName SparkStreamingProject
* @Description: Spark Streaming对接Kafka的Direct方式(常用)
* @author fanyanyan
*/
object KafkaDirectStreaming {
def main(args: Array[String]): Unit = {
if(args.length != 2){
System.err.println("请输入参数: <brokers> <topics> <group>")
System.exit(1)
}
val Array(brokers,topics,group)= args
val sparkConf = new SparkConf()
val ssc = new StreamingContext(sparkConf, streaming.Seconds(5))
val topicSet = Set(topics)
val kafkaParams = Map(
"metadata.broker.list" -> brokers,
"group.id" -> group,
// 指定偏移量
"auto.offset.reset" -> kafka.api.OffsetRequest.SmallestTimeString
)
// Spark Streaming对接Kafka(Direct方式)
val mess = KafkaUtils.createDirectStream[String,String,StringDecoder,StringDecoder](
ssc,kafkaParams,topicSet
)
// 进行词频统计(主要的代码逻辑)
val result = mess.map(_._2).flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)
result.print()
ssc.start()
ssc.awaitTermination()
}
}
Spark Streaming集成Kafka官方网站:
http://spark.apache.org/docs/2.2.0/streaming-kafka-0-8-integration.html