Basic Concept and Installation

KAFKA

1. Concept

Kafka is a distributed, partitionable, and replicable messaging system.
Kafka summarizes messages by topic.
The programs that publish messages to Kafka topics are called producers. The programs
that subscribe to topics and consume messages are called consumers.
Kafka operates in a cluster and can be composed of one or more services, each of which is called a broker.
Producers pass through the network Send messages to the Kafka cluster, and the cluster serves the messages to consumers.
The client and server communicate through the TCP protocol.

2. Pseudo-distributed installation

1. Unzip the installation package
tar -zxvf kafka.tar.gz

2.cd /conf
kafka comes with zookeeper

to modify the server.properties file
log.dirs=/tmp/kafka-logs
to modify zookeeper.properties
dataDir=/ The two paths of tmp/zookeeper

are in a parallel relationship. There cannot be a parent-child relationship. Otherwise, an error will be reported

. 3. Start

Start zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties &
start kafka:
bin/kafka-server-start.sh config/server.properties

4. Test

Create topic
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic
testView topic
bin/kafka-topics.sh --list --zookeeper localhost:2181
production data
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
consumption data
bin/kafka-console-consumer. sh --zookeeper localhost:2181 --topic test --from-beginning

3. Completely distributed installation

1. Delete the log

cd tmp after the above pseudo-distributed test, rm -fr *.*

2. Copy it to the other two machines On
scp -r kafka root@linux02
scp -r kafka root@linux03

3. Modify the configuration file

vim server.properties

broker.id=0 #Current server number 
Modify the content in the configuration file on each machine, each of which is different

4. Test

4. Java api

package com.study.kafka;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;

import org.junit.Test;

import kafka.consumer.Consumer;
import kafka.consumer.ConsumerConfig;
import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;
import kafka.javaapi.producer.Producer;
import kafka.message.MessageAndMetadata;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;

public class KafkaDemo {

	@Test
	public void put(){
		Properties props = new Properties();
		// fixed, serialized class
		props.put("serializer.class", "kafka.serializer.StringEncoder");
        // machine address
		props.put("metadata.broker.list", "linux01:9092");
        // data input
        Producer<Integer, String> producer = new Producer<>(new ProducerConfig(props));
        producer.send(new KeyedMessage<Integer, String>("park", "from java~~~"));
        
        producer.close();
	}
	
	@Test
	public void get(){
		// declare connection properties
		Properties properties = new Properties();  
		// zookeeper address
		properties.put("zookeeper.connect", "linux01:2181,linux02:2181,linux03:2181");//声明zk  
		// Group name, send data to the same group name, competition between topics--queue mode
		// Different group names, parallel between messages -- publish-subscribe mode
		properties.put("group.id", "g_1");// You must use another group name. If the producer and consumer are in the same group, you cannot access topic data in the same group  
		properties.put("auto.offset.reset", "smallest");
		
		//connect to kafka
		ConsumerConnector consumer = Consumer.createJavaConsumerConnector(new ConsumerConfig(properties));

		// consume data
		Map<String, Integer> confMap = new HashMap<>();
		confMap.put("park", 1); // topic name and amount of data read
		Map<String, List<KafkaStream<byte[], byte[]>>> ms = consumer.createMessageStreams(confMap);
		KafkaStream<byte[], byte[]> ks = ms.get("park").get(0);
		ConsumerIterator<byte[], byte[]> it = ks.iterator();
		while(it.hasNext()){
			MessageAndMetadata<byte[], byte[]> next = it.next();
			byte[] message = next.message();
			String str = new String(message);
			System.out.println(str);
		}
		
		//Disconnect
		consumer.shutdown();
	}
	
	public static void main(String[] args) {

	}

}