CentOS 5.5 Server 32-bit
The virtual machine memory is 4096M (I didn't pay attention at the beginning, the default is 1024M, and when multiple kafkas are turned on, the GC will continue, making the application unusable)
CPU 1*1
Hard disk 20G
1. Download the software
zookeeper: http://www.apache.org/dyn/closer.cgi/zookeeper/
wget http://apache.fayea.com/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz
kafka:http://kafka.apache.org/downloads
wget http://mirrors.hust.edu.cn/apache/kafka/0.10.1.1/kafka_2.11-0.10.1.1.tgz
JDK:jdk-8u121-linux-i586.rpm
wget:http://download.oracle.com/otn-pub/java/jdk/8u121-b13/e9e7ea248e2c4826b92b3f075a80e441/jdk-8u121-linux-i586.rpm?AuthParam=1484725145_218b02b9ed050daba89d99daced369e0
2. Create a user
groupadd dev
useradd -G dev zookeper
passwd zookeeper
useradd -G dev mouth
passwd kafka
3. Install the software
JDK
rpm -install jdk-8u121-linux-i586.rpm
zookeeper
tar -xvf zookeeper-3.4.9.tar.gz
kafka
tar -xvf kafka_2.11-0.10.1.1.tgz
4. Update configuration
zookeeper
cd zookeeper/conf/
mv zoo_sample.cfg zoo.cfg
vi zoo.cfg
Modify the following line
dataDir=/home/zookeeper/data/zookeeper/z01
clientPort=2181 #Because it is a pseudo-cluster deployed on the same virtual machine, the ports cannot conflict, and multiple nodes are modified to different ports
#localhost can be used, it is best to modify the IP assigned to the virtual machine. The former port is used for communication between multiple nodes, and the latter port is used for leader election communication. Because it is a pseudo-cluster, it is necessary to avoid port conflicts. server.x, x represents the current service node
server.1=localhost:2280:2281
server.2=localhost:2282:2283
server.3=localhost:2284:2285
Copy zookeeper to three different directories to form a pseudo-cluster of three nodes
Enter the directory configured by dataDir, in the heart of the myid file, the content of the file is 1, 2 and 3 respectively, which identifies which node the current service is.
kafka
cd kafka/config
#If you need a few brokers, just copy a few copies
cp service.properties service-1.properties
cp service.properties service-2.properties
cp service.properties service-3.properties
cp service.properties service-4.properties
Modify service-x.properties
broker.id=1 # are 1/2/3/4 in order
listeners=PLAINTEXT://192.168.88.129:9091 #Modify the IP assigned to the virtual machine. Since it is a pseudo-cluster, the port needs to be adjusted to avoid conflicts
log.dirs=/home/kafka/data/k01/logs #Because it is a pseudo-cluster, avoid directory conflicts
zookeeper.connect=localhost:2181 #zookeeper's address and port, adjust according to the previous zookeeper configuration (any one of the three nodes can be) because of the pseudo-cluster, zookeeper and kafla are on the same virtual machine, so you can use localhost, it is recommended Modify the IP assigned to the virtual machine
5. Server console verification
#Start zookeeper and enter the bin directory of the three nodes of zookeeper
cd /home/zookeeper/app/zookeeper/z01/bin
./zkServer.sh start
cd /home/zookeeper/app/zookeeper/z02/bin
./zkServer.sh start
cd /home/zookeeper/app/zookeeper/z03/bin
./zkServer.sh start
#Create topics 5 partitions, each partition is copied to 3 brokers (can not be greater than the total number of brokers) Enter the address of kafka/bin zookeeper to choose one
./kafka-topics.sh --create --zookeeper 192.168.88.129:2181 --replication-factor 3 --partitions 5 --topic test
#View topic information and enter the address of kafka/bin zookeeper to choose one
./kafka-topics.sh --describe --zookeeper 192.168.88.129:2181 --topic test
#List the address of topic zookeeper choose one
./kafka-topics.sh --list --zookeeper 192.168.88.129:2181
#Choose one of the address of the producer kafka
./kafka-console-producer.sh --broker-list 192.168.88.129:9091 --topic test
#The address of the consumer kafka chooses a bootstrap-server means that this address is used to discover all broker addresses, it does not mean that it is only for this broker
./kafka-console-consumer.sh --bootstrap-server 192.168.88.129:9091 --topic test --from-beginning
6.java client access
pom.xml file content (remember to add logback/log4j, the log is very helpful for problem discovery)
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.study</groupId>
<artifactId>kafka-client</artifactId>
<version>0.1.0</version>
<properties>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.1.0</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
<version>1.1.8</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.1.8</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.22</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
Producer code example
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "192.168.88.129:9092");//The initial address used to discover other broker addresses
props.put("acks", "all");//Indicates that you need to wait for the confirmation of all followers before the message is committed
props.put("retries", 0);//Indicates no retry
props.put("batch.size", 16384);
props.put("linger.ms", 100);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
String topic = "test";
Producer<String, String> producer = new KafkaProducer<String,String>(props);
for(int i = 0; i < 10; i++){
ProducerRecord<String, String> rec = new ProducerRecord<String, String>(topic, "Key88&:"+i, "Value88&:"+i);//key is used to calculate partition, the default is to use key.hash to calculate
System.out.println(rec);
producer.send(rec, new Callback(){
@Override
public void onCompletion(RecordMetadata metadata, Exception exception) {
if(exception != null){
exception.printStackTrace();
}
System.out.println("Send to server -----Offset: " + metadata.offset() + "-----Topic:" + metadata.topic() + "-----partition:" + metadata.partition());
}
});
}
producer.close();
}
Consumer code example
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "192.168.88.129:9091");//The initial address used to discover other broker addresses
props.put("group.id", "consumer-for-test");//consumer的分组
props.put("enable.auto.commit", "true");//Auto commit
props.put("auto.commit.interval.ms", "1000");
props.put("session.timeout.ms", "10000");//The default heartbeat is 2000. If you want to set session.timeout.ms, it must be > 2000, otherwise it will not be set
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<String,String>(props);
consumer.subscribe(Arrays.asList("test"));//What topics are registered
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records)
System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}
}
During the experiment, some kafka nodes were killed. When there was no leader in a partition, neither the producer nor the consumer could respond normally. Then some kafka nodes were manually started, so that after all partitions had leaders, the producer and consumer could respond normally again.
Remember firewall Open the corresponding port
#Open firewall port /sbin/iptables -I INPUT -p tcp --dport 2181 -j ACCEPT /sbin/iptables -I INPUT -p tcp --dport 2182 -j ACCEPT /sbin/iptables -I INPUT -p tcp --dport 2183 -j ACCEPT /sbin/iptables -I INPUT -p tcp --dport 9091 -j ACCEPT /sbin/iptables -I INPUT -p tcp --dport 9092 -j ACCEPT /sbin/iptables -I INPUT -p tcp --dport 9093 -j ACCEPT /sbin/iptables -I INPUT -p tcp --dport 9094 -j ACCEPT /etc/rc.d/init.d/iptables save /etc/init.d/iptables restart #View firewall /etc/init.d/iptables status
For other examples of Producer and Consumer, refer to
http://kafka.apache.org/0101/javadoc/index.html?org/apache/kafka/streams/KafkaStreams.html
http://kafka.apache.org/0101/javadoc /index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html