As the third in a series of blog Quick Start Kafka herein, this is for everyone to bring the macro and micro analysis of Kafka architecture -
Codeword is not easy, praise after the first look!
Article Directory
Kafka technical architecture
Macro
Macro, Kafka's architecture consists of four parts
1, producers API
It allows applications to publish a stream to one or more kafka topics (topics) record.
2, consumer API
It allows the application to subscribe to one or more topics, topics and processes the received stream record.
3、StreamsAPI
It allows an application processor acts as a stream (stream processor), takes an input stream from one or more topics, and to produce an output flow of one or more topics, can effectively change the input stream to the output stream.
4、ConnectAPI
Construction and operation allows reusable producer or consumer can be connected to the kafka relating to existing application program or data system. For example: a connector connected to a relational database may get change for each table.
Microcosmic
Description : kafka supports message persistence, for the end consumer to pull data pull model, consumer subscription status and is responsible for maintaining client relationships we have, after news consumption is not deleted immediately, preserves the history of the message. Therefore, when support for multi-subscription, the message will store a copy of it.
Internal details
. 1) Producer : message producers, it is the message to the client end kafka broker;
2) Consumer : a message consumer takes the message to the client terminal kafka broker;
. 3) Consumer Group (the CG) : consumer group, composed of a plurality of consumer. Consumers within each group is responsible for consumer consumption data for different partition, a partition can only be consumed by a group of consumers ; simultaneously and consumer groups. All consumers belong to a consumer group, namely consumer group is a subscriber logical .
4) Broker : a kafka server is a broker. A cluster composed of a plurality of broker. A broker can receive a plurality of topic.
5) Topic : can be understood as a queue, producers and consumers are facing a Topic ; every message posted to kafka cluster must have a category (topic)
. 6) the Partition : In order to achieve scalability, a very large topic may be distributed to a plurality Broker (i.e., server), a topic can be divided into a plurality of partition , each partition is an ordered queue;
. 7) segment : the presence of a partition segment among the plurality of file segments, each segment divided into two parts, .log file and .index file, wherein .index file is an index file, primarily for quick partial data among file .log shift of position
8) Replica : copy, when a cluster node fails guarantee, Partition data on the node is not lost, and kafka continue to operate, and provide a copy of kafka mechanism, each partition has a topic number a copy of a leader and a number of follower .
9) Leader : "Lord", multiple copies of each partition object sends data producers , as well as objects of consumer spending data are leader.
10) follower : multiple copies of each partition in the "from" real-time synchronization of data from the leader , and leader of maintaining synchronization data. When the leader fails, a follower will become a new follower.
To Learn More:
1. a broker can be only one leader, a broker can be only one Topic a Partition
2. copy is not responsible for reading the response data, consumer data can only be read from Lead, Follower just to improve fault tolerance
3. consumer It depends on the number of concurrent partition
4. partition data copy number <= Broker's number (a copy of the data is not allowed in the same partition Broker)
Well, finally finished the code word of the ~ this blog can be described as the essence of full , for just learning Kafka's friend can be described as very conscience - if you think you have help, it may take a thumbs up, focus on a point, make more more people benefit 9 (1❛ᴗ❛1) 6.
Summary of knowledge on this blog here, and we will bring the next blog Kafka的命令行操作, so stay tuned ~
