【Kafka】Kafka Basic Concept Notes
Article Directory
1. Two modes
As a message queue, Kafka has two modes:
- peer-to-peer mode
- publish/subscribe model
1.1 Point-to-point mode
Features:
- The consumer actively pulls the data, and clears the message after the message is received
1.2 Publish/Subscribe Mode
- There can be multiple topic topics (browsing, likes, favorites, comments, etc.)
- After the consumer consumes the data, the data is not deleted
- Each consumer is independent of each other and can consume data
2. Infrastructure
Kafka's infrastructure:
- To facilitate expansion and improve throughput, a topic is divided into multiple partitions (partitions)
- In conjunction with the partition design, the concept of consumer group is proposed. Each consumer in the group consumes in parallel. A partition can only be consumed by one consumer in the group to avoid repeated consumption.
- To improve availability, add several copies for each partition
- Who is the leader is recorded in Zookeeper, Kafka2.8.0 and later can also be configured not to use ZK
- Consumer Group (CG) : A consumer group consisting of multiple consumers. Each consumer in a consumer group is responsible for consuming data from different partitions. To avoid repeated consumption of detailed messages, a partition can only be consumed by one consumer in the group; consumer groups do not affect each other . All consumers belong to a consumer group, that is, a consumer group is a logical subscriber.
- Broker : A Kafka server is a broker. A cluster consists of multiple brokers. A broker can hold multiple topics.
- Topic : It can be understood as a queue, and both producers and consumers are facing a topic .
- Partition : In order to achieve scalability, a very large topic can be distributed to multiple brokers (ie servers), a topic can be divided into multiple partitions , and each partition is an ordered queue .
- Replica : copy. Each partition of a topic has several copies, a Leader and several Followers.
- Leader : The "primary" of multiple copies of each partition, the object that the producer sends data, and the object that the consumer consumes data are all leaders.
- Follower : The "slave" in multiple copies of each partition, synchronizes data from the Leader in real time, and maintains synchronization with the Leader data. When the Leader fails, a Follower will become the new Leader.
3. Topic command line operation
3.1 View Topic operations
① View the operation Topic command parameters
#在kafka的目录下
bin/kafka-topics.sh
After entering the command line, the console lists all parameters and their meanings:
Summarized as follows:
3.2 Create Topic
②Create a topic named first, the number of partitions is required to be 1, and the cluster has 3 nodes, so the partition copy is set to 3
bin/kafka-topics.sh --bootstrap-server node1:9092 --create --partitions 1 --replication-factor 3 --topic first
3.3 View all Topics
③ View all topics in the current server
bin/kafka-topics.sh --bootstrap-server node1:9092 --list
3.4 View Topic details
④ View the details of the first theme
bin/kafka-topics.sh --bootstrap-server node1:9092 --describe --topic first
Replicas：1,2,0Indicates that replicas exist in three nodesLeader:1Indicates that the Leader copy is stored in the node code-named 1, and the other two stores are Follower copiesIsr:1,2,0Represents a synchronous copy, the follower copy synchronizes the data of the leader copy, and the ISR is a set of copies used in Kafka to ensure data consistency and reliability
3.5 Modify the number of partitions
⑤ Note: the number of partitions can only be increased but not decreased
bin/kafka-topics.sh --bootstrap-server node1:9092 --alter --partitions 3 --topic first
Check the details of the first theme again after modification:
3.6 Delete Topic
⑥Delete first theme
bin/kafka-topics.sh --bootstrap-server node1:9092 --delete --topic first
4. Producer command line operation
4.1 Producer command line operation
View operation producer command parameters
bin/kafka-console-producer.sh
4.2 Send message to topic
bin/kafka-console-producer.sh --bootstrap-server node1:9092 --topic first
A message was sent to the first topic hello world.
5. Consumer command line operation
5.1 View operation consumer command parameters
bin/kafka-console-consumer.sh
5.2 Consume messages in topic
Consume messages from the first topic
bin/kafka-console.consumer.sh --bootstrap-server node1:9092 --topic first
We found that the cursor has been flashing, but did not receive any message, that is because kafka reads the message after the consumer script is started by default, and if you want to read all the data (including historical data), you need to add --from-beginningparameters
bin/kafka-console-consumer.sh --bootstrap-server node1:9092 --from-beginning --topic first
In this way, the historical data is also read out.