Kafka information stored by Zookeeper
Kafka-related information stored on the server side of zookeeper
1) /kafka/brokers/ids [0,1,2] records which servers are there
2) /kafka/brokers/topics/first/partitions/0/state records who is the leader, Which servers are available
3) /kafka/controller auxiliary election leader
Kafka Broker overall workflow
1) The broker registers in ZK after startup
2) The controller who registers first has the final say
3) The elected Controller monitors the change of broker nodes
4) The Controller decides the Leader election
5) The controller uploads the node information to ZK
6) Other controllers Synchronize relevant information from zk
7) Assume that the leader in the broker is down
8) The controller monitors node changes
9) Obtains the ISR
10) Elects a new leader (election rules: survival in the ISR is the premise, and the priority in the AR is given priority)
11) Update the leader and ISR
4.1.3 Important parameters of Broker
| parameter name | describe |
|---|---|
| replica.lag.time.max.ms | In the ISR, if the Follower does not send communication or synchronization information to the Leader for a long time, the Follower will remove the ISR. The event threshold, the default is 30s. |
| auto.leader.rebalance.enable | The default is true, automatic Leader Partitioner balance |
| leader.imbalance.per.broker.percentage | The default is 10%, the ratio of unbalanced leader allowed by each broker, if each broker exceeds this value, the controller will trigger leader balance |
| leader.imbalance.check.interval.seconds | The default is 300 seconds, the interval for checking whether the leader load is balanced |
| log.segment.bytes | The log in Kafka is stored in blocks. This configuration refers to the size of the log divided into blocks. The default value is 1G |
| log.index.interval.bytes | The default is 4KB. Whenever a 4KB log (.log) is written in Kafka, an index is recorded in the index file |
| log.retention.hours | Data storage time in kafka, default seven days |
| log.retention.minutes | Data storage time in kafka, minute level, default off |
| log.retention.ms | Data storage time in kafka, millisecond level, off by default |
| log.retention.check.interval.ms | Check whether the data is saved at the timeout interval, the default is 5 minutes |
| log.retention.bytes | Exceeds the total size of the set index log, delete the earliest segment |
| log.cleanup.policy | The default is delete, which means that the deletion strategy is enabled for all data. If the set value is compact, it means that the compression strategy is enabled for all data |
| num.io.threads | The default is 8, the number of threads that load to write to the disk, the real parameter value should account for 50% of the total number of cores |
| num.replica.fetchers | The number of copies pulls the number of threads, this parameter accounts for 50% of the total number of cores |
| num.network.threads | The default is 3, the number of data transmission threads, 2/3 of 50% of the total number of cores for this parameter |
| log.flush.interval.messages | Force the page cache to write the number of disk entries, the default is the maximum value of long, 9223372036854775807, generally not recommended to modify, leave it to the system to manage |
| log.flush.interval.ms | How often, brush data to disk, the default is null, generally not recommended to modify, leave it to the system to manage |