Introduction to Kafka Architecture

1. The architecture of Kafka

1、Broker

    A kafka cluster contains one or more servers, which are called Brokers

2、Topic

    Each message published to the kafka cluster has a category, and this category is called a topic (a logical concept, the messages of different topics are stored separately. Logically, the messages of a topic are stored on one or more brokers, but Users only need to specify the Topic of the message to produce or consume data without caring where the data is stored)

3、Partition

    Partition is a physical concept, each Topic corresponds to one or more Partitions

4、Producer

    Responsible for sending messages to the broker server of kafka

5、Consumer

    Message consumer, client that reads messages from kafka broker

6、Consumer Group

    Each consumer belongs to a specific consumer group (consumer name can be specified for each consumer, if not specified, it belongs to the default group)

Second, the topology of kafka

A typical Kafka cluster consists of several producers (which can be page views generated by the web, or server logs, system cpu, memory, etc.) and several brokers (Kafka supports horizontal expansion. Generally, the more the number of brokers, the higher the cluster throughput rate. Higher), several consumer groups, and a zookeeper cluster, Kafka manages the cluster configuration through zookeeper, elects the leader, and rebalances when the consumer group changes. Producers use push mode to publish messages to brokers, and consumers use pull mode to subscribe and consume messages from brokers .