1, kafka architecture model
Based on the basic structure of a producer consumer topic broker, etc.
. 5 , Kafka components introduced
Topic: messages are classified according Topic
Producer: sending a message by
Consumer: Message recipients
broker: kafka each instance (server)
Zookeeper: dependent on the cluster to save meta information.
Topics Components Introduction
Topic: a type of message, the topic is divided into a plurality of each partition (zone), arranged in the cluster configuration file.
partition: the storage level is the logical append log file that contains multiple segment files.
Segment: real file message store, will continue to generate new.
offset: Each message in the file location (offset). a digital offset is long, it is the only mark a message.
partition
1, it is a logical storage level append log file, each segment consisting of a plurality of partition.
2, any news release this partition will be added directly to the end of the log file.
3, each partition corresponds to a list in memory index, each record in the first message segment offset. Such lookup message when the first message is positioned at the position index list, and then reads the file, the block speed.
4, the publisher to send a message topic will be evenly distributed to multiple part, Broker adds the message received on the last segment of the release message to the corresponding part.
partition distribution
1、 partitions分区到不同的server上，一个partition保存在一个server上，避免一个server上的文件过大，同时可以容纳更多的consumer消费,有效提升并发消费的能力。
2、 这个server(如果保存的是partition的leader)负责partition的读写。可以配置备份。
3、 每个partition都有一个server为"leader"，负责读写，其余的相对备份机为follower，follower同步leader数据，负责leader死了之后的接管。n个leader均衡的分散在每个server上。
4、 partition的leader和follower之间监控通过zookeeper完成。
segment
1、 每个segment中存储多条消息，消息id由其逻辑位置决定，即从消息id可直接定位到消息的存储位置，避免id到位置的额外映射。
2、 当某个segment上的消息条数达到配置值或消息发布时间超过阈值时，segment上的消息会被flush到磁盘，只有flush到磁盘上的消息订阅者才能订阅到
3、 segment达到一定的大小（可以通过配置文件设定,默认1G）后将不会再往该segment写数据，broker会创建新的segment。
offset
offset是每条消息的偏移量。
segment日志文件中保存了一系列"log entries"(日志条目),每个log entry格式为"4个字节的数字N表示消息的长度" + "N个字节的消息内容";
每个日志文件都有一个offset来唯一的标记一条消息,offset的值为8个字节的数字,表示此消息在此partition中所处的起始位置.
每个partition在物理存储层面,有多个log file组成(称为segment).
segment file的命名为"最小offset".log.例如"00000000000.log";其中"最小offset"表示此segment中起始消息的offset.
