Kafka basic installation and use

Kafka basic installation and use

Official website address: http://kafka.apache.org/

1. Features

• High throughput and low latency
• Scalability, support cluster hot expansion
• Persistence, reliability, message persistence to local disk, support data backup
• Fault tolerance, allowing nodes in the cluster to fail
• High concurrency, supporting thousands of clients to read and write simultaneously

2. Usage scenarios

• Log collection
• Message system
• Operational indicators
• User activity tracking
• Streaming

3. Related concepts

1. Producter
2. Broker server node
3. Topic classifies data
4. partition
5. Consumer
6. replica
7. Zookeeper is used to maintain and coordinate brokers. After the broker changes in the kafka system, zookeeper broadcasts to notify producers and consumers
8. offset message offset id is used to identify the position of the data in a partition

4. Download and use

4.1 zookeeper

Download
link : link: https://pan.baidu.com/s/1ZGzO9NzS00Si2TctHbaCIQ Password: hukj
Unzip
tar -zxvf apache-zookeeper-3.5.9.tar.gz

and configure on the virtual machine

1. Create a data directory
   sudo mkdir data
   and modify the group and user to
   
   modify permissions

sudo chmod -R 777 data

Go to data and use pwd to get the current current directory
2. Go to the conf directory

Modify zoo_sample.cfg to zoo.cfg

Edit and modify the data directory

. Description of the remaining parameters

3. Enter the bin directory

4. Start zookeeper

Execute zkserver.sh

according to the prompt and enter the instructions to start
./bin/zkServer.sh --config ./conf start

4.2 kafka download and install

Download address, version 2.7.0:
Link: https://pan.baidu.com/s/1f6rtRqtLGiqQ80KBgAif3g Password: 8gc5
unzip to the directory you want

1. Create log folder

2. Modify configuration parameters server.properties

Let go of the port

3. Start Kafka
   bin/kafka-server-start.sh config/server.properties
   

The following message appears to indicate successful startup

4. Create a new window for verification
   

5. Production and consumption

5.1 Create a theme

bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic hello --partitions 2 --replication-factor 1

Verify that the theme was created successfully
bin/kafka-topics.sh --zookeeper localhost:2181 —list

View the theme details

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic hello

5.2 Start consumption to receive messages

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic hello
9092 is the service port of Kafka

5.3 Start the production side

Create a new window
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic hello

5.4 Message verification

Perform message input on the production side and view the information output result on the consumer side

6. Problem

Zookeeper startup failure problem

1. Check the log file and find that the class is missing:

2. File permissions and user issues

3. Error in the data directory in the configuration file