Preface
Distributed is one of the necessary skills for programmers. It belongs to the necessary category in the interview process, and it is often used in work. Kafka is a distributed publish-subscribe-based message queue. At present, its charm is endless. For the mystery of Kafka, we still need to explore in detail.
To talk about how familiar with Kafka, I believe that Ali’s bigwigs have the most say, so the content shared today is the "limited notes" provided by Alibaba internally. All the essence of Kafka is written here. I have to Exclamation: As expected of Alibaba's technical officer, I really took it!
Regarding this limited Kafka note, I can only show part of the chapter content and core screenshots in the article. If you need the complete pdf version, you can click here to see the full content and download it for free .
1. Understanding of Kafka
1. Basic concepts of Kafka
2. Installation and configuration
3. Production and consumption
4. Server parameter configuration
2. Producer
1. Client development
- Necessary parameter configuration
- Message sending
- Serialization
- Partitioner
- Producer interceptor
2. Principle analysis
- Overall structure
- Metadata update
3. Important producer parameters
Three, consumers
1. Consumers and consumer groups
2. Client development
- Necessary parameter configuration
- Subscribe to topics and partitions
- Deserialization
- Message consumption
- Displacement submission
- Control or close consumption
- Specify displacement consumption
- Rebalance
- Consumer interceptor
- Multi-threaded implementation
- Important consumer parameters
Fourth, themes and divisions
1. Subject management
- Create theme
- Distribution of partition copies
- View topic
- Edit theme
- Configuration management
- Subject parameters
- Delete topic
2. Getting to know KafkaAdminCilent
- Basic use
- Subject legality verification
3. Partition management
- Election of the first copy
- Partition redistribution
- Replication limit
- Modify the copy factor
4. How to choose the right number of partitions
- Performance testing tools
- The more partitions, the higher the throughput
- Maximum number of partitions
- Considerations
Five, log storage
1. File directory layout
2. Evolution of log format
- v0 version
- v1 version
- Message compression
- Variable length field
- v2 version
3. Log Index
- Offset index
- Timestamp index
4. Log cleanup
- Log deletion
- Log compression
5. Disk storage
- Page cache
- Disk I/O process
- Zero copy
Six, in-depth server
1. Protocol design
2. Time wheel
3. Delay operation
4. Controller
5. Parameter decryption
Seven, in-depth client
1. Partition allocation strategy
2. Consumer Coordinator and Group Coordinator
3. Analysis of _consumer_offsets
4. Affairs
8. Reliability research
1. Copy analysis
2. Log synchronization mechanism
3. Reliability analysis
Nine, Kafka application
1. Command line tools
2.Kafka Connect
3.Kafka Mirror Maker
4.Kafka Streams
10. Kafka monitoring
1. Source of monitoring data
2. Consumption lags behind
3. Synchronize failed partitions
4. Description of monitoring indicators
5. Monitoring module
11. Advanced applications
1. Expiration time (TTL)
2. Delay queue
3. Dead letter queue and retry queue
4. Message routing
5. Message track
6. Message audit
7. Message Broker
8. Selection of message middleware
12. Integration of Kafka and Spark
1.Spark installation and simple application
2.Spark programming model
3.Spark operating structure
4. Introduction to Spark Streaming
5. Integration of Kafka and Spark Streaming
6.Spark SQL
7.Structured Streaming
8. Integration of Kafka and Structured Streaming
to sum up
We always like to pay homage to the great gods of big factories, but in fact, great gods are no more than mortals. Compared with rookie programmers, they spend a little more thought. If you don't work hard, the gap will only get bigger and bigger. In fact, as a programmer, it is necessary to enrich one's knowledge reserve and enhance the depth and breadth of one's knowledge. In my opinion, the rewards will be as much as you pay.
Learning Kafka is not as difficult as imagined. The content in this limited Kafka note will inspire and help you to learn Kafka. If you need this full version of Kafka notes , you only need to ** support ** this article.