spark streaming kafka

SparkStreaming + Kafka 
What • kafka yes, what are the characteristics 
• SparkStreaming + Kafka What are the benefits 
- decoupling 
- buffer

　

 

 

　

 

 

Message queue characteristics of 
producers and consumers modes 
• Reliability Assurance 
- they do not lose the data 
- the consumer does not lose the data: "At least once, a strict"

 

 

Broker 
the n-. brokers, brokers 
vt. middleman to arrange other factors ... 
vi. to negotiate a power-broker

　　

 

 

 

 

 

 

 

 

 

 

 

 

 

kafka deployment 
node2,3,4 based zookeeper 

start three zookeeper 

/opt/sxt/zookeeper-3.4.6/bin/zkServer.sh Start 

configuration kafka 

the tar -zxvf kafka_2.10-0.8.2.2.tgz -C / opt / SXT / 
kafka_2.10-0.8.2.2 / config / 
VI the server.properties 
broker.id = 0 to ## 0 node3 node2 is 1 node4 is 2 
log.dirs = / logs-Kafka 
zookeeper.connect = node2: 2181, node3: 2181 , Node4: 2181 

## scp to node3, node4, modify broker.id = 0 and 

try the next node2 kafka bin start a 
./kafka-server-start.sh ../config/server.properties   
## scripting using background start 

[@ node2 the root shells] # pwd 
/ the root / shells 
[@ node2 the root shells] CAT # start-kafka.sh 
cd /opt/sxt/kafka_2.10-0.8.2.2
bin the nohup / kafka-server-start.sh config / the server.properties> kafka.log 2> &. 1 & 

## to SCP node3,4 

## batch execution node2,3,4 
/root/shells/start-kafka.sh # # successful launch of three kafka cluster.