Cluster membership
Kafka zookeeper use to maintain information about cluster members. Each broker has a unique identifier broker.id. When the broker starts up, it registers itself to the ID zookeeper. Kafka components Subscribe to zookeeper of brokers / ids / path, when there are added or removed from the cluster broker, these components would be notified.
Controller
The controller has a general election broker function, but also responsible for the partition of the leader, the first cluster of broker started by zookeeper create / controller allow themselves to be controller.
copy
Kafka use the theme to organize the data, each topic is divided into several partitions, each have multiple copies. Those copies are stored on the broker, each broker can save hundreds or thousands of different topics and belong to the partition copy.
There are two types of copy: a copy of the princes and follow copy. All producers and consumers will request a copy by the leader. Followed by a copy of the message to copy copy chief, ready to be the new leader.
Processing the request
kafka provides a binary protocol (based on TCP), developed a format request message broker and how to respond to the request.
All message request contains a standard message header:
- Request type
- Request version
- Correlation ID, for identifying the request message
- Client ID, identification of the client sends a request
Production Request: Manufacturer transmission request containing the client to write the message broker.
Acquisition request: request sent when a consumer needs to read and follow the copy of the message from the broker.
Physical storage
Partition allocation
We will try to copy not on the same broker, strategies and similar DataNode block copy chief.
Document Management
The partition is divided into several fragments, each fragment comprising or week 1GB data segment limit is reached, closes the current file and opens a new file
Segment data is being written called active fragment, active fragments will never be deleted.
file format
In addition to the key value and the offset, the message also contains a message size, a checksum, a message format version number, and the version stamp compression algorithm.
index
To help baroker quickly locate the specified offset, kafka for each partition maintains an index, the index maps to offset clip file and an offset location in the file
Interceptor
Interceptor is mainly used to customize the client control logic. For producers, the interceptor allows users to have the opportunity again before the message is sent before the message producer callback logic and do some customization requirements, such as modifying messages. Producers interceptors allows users to specify a plurality of sequentially applied to the same piece of information so as to form a chain of interception.