In a system with big data and high concurrency, in order to break through the bottleneck, the system will be horizontally expanded and vertically split to form an independent service. Behind each independent service, there may be a cluster providing external services. This will encounter a problem. The entire system is composed of multiple services (subsystems), and data needs to flow continuously in each service. If the data is transmitted too slowly in the various subsystems, it will form a bottleneck and reduce the performance of the entire system. Thus formed a Kafka-centric solution!
Because the importance of reading Kafka source code is self-evident, today I will share a Pinduoduo Kafka source code note, which is now fully open source for the public! ( In order not to affect everyone's reading experience, free access is placed at the end of the article! )
This note starts from the application scenarios and source environment of Kafka. It not only introduces the core concepts of Kafka, but also conducts an in-depth analysis of the source code of Kafka producers, consumers, and servers. Finally, it introduces the implementation of Kafka commonly used management scripts. , So that readers not only understand Kafka from the macro design, but also deeply understand the detailed design of Kafka. In the process of source code analysis, the experience accumulated by the author's work and the understanding of Kafka design are also interspersed. I hope that readers can draw inferences from one another, not only knowing the truth, but also knowing why.
Chapter 1 Quick Start
1.1 Introduction to Kafka
1.2 Kafka-centric solution
1.3 Kafka core concepts
1.4 Kafka source environment
Chapter 2 Producers
2.1 KafkaProducer usage example
2.2 KafkaProducer analysis
- ProducerInterceptors&cProducerInterceptor
- Kafka cluster metadata
- Serializer&Deserializer
- Partitioner
2.3 RecordAccumulator analysis
- MemoryR ecords
- RecordBatch
- BufferPool
- RecordAccumulator
2.4 Sender analysis
- Create request
- KSelector
- InFlightRequests
- MetadataUpdater
- NetworkClient
Chapter 3 Consumers
3.1 KafkaConsumer usage example
3.2 Delivery guarantee semantics
3.3 Consumer Group Rebalance Design
3.4 KafkaConsumer analysis
- ConsumerNetworkC link
- SubscriptionState
- ConsumerCoordinator.
- PartitionAssignor analysis
- Heartbeat analysis
- Rebalance implementation
- offset operation
- Fetcher
- KafkaConsumer analysis summary
Chapter 4 Kafka Server
4.1 Network layer
- Reactor mode
- SocketServer
- AbstractServer Thread
- Acceptor
- Processor
- RequestChannel
4.2 API layers
KafkaR equestHandler
KafkaApis
4.3 Log storage
- basic concepts
- FileMessageSet
- ByteBufferMessageSet
- OffsetIndex
- LogSegment
- Log
- LogManager
4.4 DelayedOperationPurgatory component
- TimingWheel
- SystemTimer
- Delayed Dperation
- DelayedOperationPurgatory
- DelayedProduce
- DelayedFetch
4.5 Copy mechanism
- Copy
- Partition
- ReplicaManager
4.6 KafkaController
- ControllerChannelManager
- ControllerContext
- ControllerBrokerRequestBatch
- PartitionStateMachine
- PartitionLeaderSelector
- ReplicaStateMachine
- ZooKeeper Listener
- KafkaController initialization and failover
- Handling ContolledShutdownRequest
4.7 GroupCoordinator
- GroupMetadataManager
- GroupCoordinator analysis
4.8 Identity authentication and permission control
- Configure SASL/PLAIN authentication
- Authentication
- Access control
4.9 Kafka monitoring
- Introduction to JMX
- Introduction to Metrics
- Metrics in Kafka
- Kafka's monitoring function
- Monitor KSelector's metrics
Chapter 5 Kafka Tool
5.1 kafka-server-start Screenplay
5.2 kafka-topics script
- Create Topic
- Modify Topic
5.3kafka-preferred-replica-election脚本
5.4 kafka-reassign-partitions Screenplay
5.5 kafka-console -producer Screenplay
5.6 kafka-console-consumer Screenplay
5.7 kafka-consumer-groups Screenplay
5.8 DumpLogSegments
5.9 kafka-producer-perf-test script
5.10 kafka-consumer-perf-test Screenplay
5.11kafka-mirror-maker脚本
to sum up
A large distributed system is like a life, each service in the system is like a skeleton, the data in it is like blood, and Kafka is like a meridian, connecting the entire system. This Kafka source code note shows the realization of Kafka to readers through a large number of design drawings, code analysis, and sample sharing, and helps readers to better study Kafka code.
Those who need to receive this Kafka source notes for free, please help forward this article + follow me, and then add the assistant vx to get it for free!
