As operation and maintenance engineers, we are familiar with Kafka, but that's all. There are still a large number of people who do not know what Kafka does, nor do they know the principles and configuration of Kafka, let alone the advanced applications of Kafka.
We learn Kafka not only because it is a necessary knowledge for operation and maintenance, but also because Kafka is also a "frequent visitor" in interviews. In order to pass the interview smoothly and make the subsequent career smooth , learning Kafka is a top priority.
Today I don’t talk about obscure knowledge, just let everyone use it 几分钟to quickly understand Kafka, 超通俗易懂the manual, and newcomers can completely follow it.
Kafka concepts and basic terms
Kafka is developed by Linkedin. It is a distributed, multi-partition, multi-copy, Zookeeper-based distributed message flow platform. It is also an open source message engine system based on publish-subscribe mode.
Message : The data unit in Kafka is called a message, also known as a record, which can be regarded as a record of a row in a database table
Batch : To improve efficiency, messages are written to Kafka in batches, and a batch refers to a group of messages.
Topic : The type of message is called topic. It can be said that a topic represents a type of message. It is equivalent to classifying messages. Topics are like tables in a database.
Partition : A topic can be divided into several partitions. The partitions in the same topic may not be on one machine, and may be deployed on multiple machines. In this way, the scalability of Kafka can be realized. The partitions in a single topic are ordered. However, there is no guarantee that all partitions in a topic are in order.
Producer : The client application that publishes messages to a topic is called a producer, and the producer is used to continuously send messages to a topic.
Consumer : The client program that subscribes to the topic message is called the consumer (Consumer), and the consumer is used to process the message generated by the producer.
Consumer group : The relationship between producers and consumers is like the relationship between chefs and customers in a restaurant. One chef corresponds to multiple customers, that is, one producer corresponds to multiple consumers. Consumer groups (Consumer Group) refers to a group consisting of one or more consumers.
Kafka's message queue
Kafka's message queue is generally divided into two modes: point-to-point mode and publish-subscribe mode
Kafka supports consumer groups, that is to say, there will be one or more consumers in Kafka. If a message produced by a producer is consumed by a consumer, then this mode is a point-to-point mode. If a
producer Or when messages generated by multiple producers can be consumed by multiple consumers at the same time, such a message queue becomes a message queue in the publish-subscribe mode.
Kafka system architecture
Kafka message sending
After instantiating the producer object, you can start sending messages. There are mainly the following ways to send messages
- simple message sending
- Send messages synchronously
- Send messages asynchronously
Kafka Consumer
Kafka consumers belong to consumer groups. Consumers in a group subscribe to the same topic, and each consumer receives messages from a part of the topic. The following is a Kafka partition consumption diagram.
The topic T1 in the above figure has four partitions, namely partition 0, partition 1, partition 2, and partition 3. We create a consumer group 1, and there is only one consumer in the consumer group , it subscribes to topic T1 and receives all messages in T1. Since one consumer handles the messages sent by four producers to the partition, the pressure is a bit high, and a helper is needed to help share the task, so it evolves into the following figure
Full version PDF display
This note 短小精悍is very suitable 入门使用, the illustrations are clear and easy to understand, and you can take it away for study.
