Author: Zen and the Art of Computer Programming
1 Introduction
Apache Kafka is an open source distributed messaging system developed and open sourced by LinkedIn. It was originally designed to build a real-time stream processing platform, capable of diversified publish/subscribe and consume data through multiple transport protocols. With the passage of time, Kafka has become a basic component of more and more application fields and is widely used by various companies and organizations. In September 2018, Didi Chuxing announced a pilot message queue service based on Apache Kafka, which will be used to transmit important information and data externally.
This article will elaborate on the exploration and practice of Apache Kafka in the commercialization process of Didi Chuxing from the aspects of the overall framework, Kafka key features, and Kafka usage scenarios, hoping to bring readers more knowledge and experience.
Contributions and suggestions are welcome to discuss some practical experience of Apache Kafka in the commercialization of Didi Chuxing, so as to promote the prosperity and healthy development of the community ecology.
2. Background introduction
2.1 What is Kafka?
Apache Kafka is an open source distributed computing platform whose purpose is to process data streams in real time. Kafka is a high-throughput distributed publish-subscribe message system with the following main characteristics:
- Distributed: Supports deployment on multiple servers in a cluster, making full use of multi-core advantages to improve performance;
- Reliability: support persistence to ensure that messages are not lost;
- Fault tolerance: ensure that messages are not lost or lost in a small amount through the backup mechanism;
- Efficient: messages are sent in batches to reduce network IO consumption;
- Timing: Kafka guarantees the order of messages and can be sorted according to the timestamps of associated events;
- Message engine: Provides a unified message interface, developers can use various languages