Author: Zen and the Art of Computer Programming
1 Introduction
Introduction to message queue
"Message Queue" is a core component that plays a vital role in distributed systems. It can help us solve complex problems, such as asynchronous calls, traffic peak cutting, decoupling, etc. Message queues are often used to buffer data and transfer it to another process or service. There are two types of message queues - point-to-point (PTP) and publish/subscribe (Pub/Sub). A point-to-point type of message queue gets information from receivers, while a publish/subscribe type of message queue allows multiple consumers to receive information at the same time. The most commonly used scenario of message queue is for peak clipping. Assume that a system has tens of thousands of requests per second, but it takes tens or even hundreds of milliseconds to process each request. When there are too many instantaneous accesses, the system load will surge, or even crash. In this case, queuing requests through the message queue can smooth the system pressure. Some of the main features of message queues include:
Asynchronous communication: The message queue does not need to wait for the recipient to respond when delivering messages, so asynchronous communication can be achieved.
Traffic peak shaving: The message queue can limit the system's traffic to a certain extent, so it can avoid occupying too many resources by processing a large number of requests.
Decoupling: The message queue can decouple the dependency between the sender and the receiver, so that the sender no longer depends on the response speed of the receiver.
Final consistency: Since there may be delays in the data of each node in a distributed environment, message queues generally adopt the final consistency mode.
Recoverability: The message queue generally records whether the message is successfully delivered, and if it fails, it can be re-delivered.
Introduction to Kafka
Apache Kafka is an open source distributed publish-subscribe messaging system. Kafka is written in Scala language and is developed and maintained by LinkedIn. Kafka is a distributed, scalable,