Installation and basic operations of Kafka

1. Prerequisite work

Zookeeper must be installed

For a single machine, please refer to: ZooKeeper Single Machine Installation and Configuration
For cluster information, please refer to: Zookeeper cluster installation

2. Kafka installation

Upload kafka_2.11-2.3.1.tgz to the /tools directory

Unzip and install to the /training directory

tar -zvxf kafka_2.11-2.3.1.tgz -C /training/

Configure environment variables and make them take effect.
Execute: vi ~/.bash_profilecommand, add the following information and save and exit:

#kafka
export KAFKA_HOME=/training/kafka_2.11-2.3.1
export PATH=$PATH:$KAFKA_HOME/bin

Environment variables, effective execution:source ~/.bash_profile

Create a log directory to store kafka (created in the kafka installation path.

mkdir /training/kafka_2.11-2.3.1/kafka-logs

Configure server.properties (the file is in the config directory under the kafka installation path) and
execute: vi server.properties
and modify the following information (hadoop001 is the host name):

listeners=PLAINTEXT://hadoop001:9092
log.dirs=/training/kafka_2.11-2.3.1/kafka-logs
zookeeper.connect=hadoop001:2181

Start kafka (you need to start zookeeper first)

kafka-server-start.sh /training/kafka_2.11-2.3.1/config/server.properties

Verify whether Kafka starts successfully.
Check whether port 9092 has been started: netstat -anop|grep 9092
or execute: jps to check whether there is a kafka process
or:

3. Basic operations of Kafka

Create a topic topic, such as test
execution:

kafka-topics.sh --create --zookeeper hadoop001:2181 --replication-factor 1 --partitions 1 --topic test

Start the console producer (you need to open an additional terminal.
Execute:

kafka-console-producer.sh --broker-list hadoop001:9092 --topic test

Start the console consumer (you need to open another terminal.
Execute:

kafka-console-consumer.sh --bootstrap-server hadoop001:9092 --topic test --from-beginning

Test:
Enter some text in the terminal where the producer is located, such as: hello world, and press Enter.

In the terminal where the consumer is located, you will see: hello world sent from the generator, as shown in the figure below:
At this time, execute jps to view, and you will see the following process:
You can check what content is generated in zookeeper at this time , execute: zkCli.sh, you will see the following content, which also shows that zookeeper saves some meta-information of Kafka. Please verify it yourself.