Why use a messaging system
- decoupling
It is extremely difficult to predict what needs the project will encounter in the future at the beginning of the project. The message system inserts an implicit, data-based interface layer in the middle of the process, and both processes must implement this interface. This allows you to extend or modify both processes independently, as long as you make sure they obey the same interface constraints.
- redundancy
In some cases, the process of processing the data fails. It will be lost unless the data is persisted. Message queues avoid the risk of data loss by persisting data until they have been fully processed. In the "insert-get-delete" paradigm used by many message queues, before removing a message from the queue, your processing system needs to explicitly indicate that the message has been processed, thus ensuring that your data is kept safe until you are done using it.
- Extensibility
Because message queues decouple your processing, it is easy to increase the frequency of message enqueuing and processing by adding additional processing. No need to change the code, no need to adjust the parameters. Expansion is as easy as turning up the power button.
- Flexibility & Peak Handling
In the case of a surge in traffic, the application still needs to continue to function, but such bursts of traffic are not common; it is undoubtedly a huge waste to invest resources to be on standby at any time based on the standard of being able to handle such peak traffic. Using message queues enables critical components to withstand sudden access pressures without completely crashing due to sudden overloaded requests.
- recoverability
When a part of the system fails, it does not affect the entire system. Message queues reduce the coupling between processes, so even if a process processing a message hangs, messages added to the queue can still be processed after the system is restored.
- order guarantee
In most use cases, the order of data processing is important. Most message queues are inherently ordered and guarantee that data will be processed in a specific order. Kafka guarantees the ordering of messages within a Partition.
- buffer
In any critical system, there will be elements that require different processing times. For example, loading an image takes less time than applying filters. Message queues use a buffer layer to help tasks perform most efficiently - writes to the queue are processed as quickly as possible. This buffering helps control and optimize the speed at which data flows through the system.
- Asynchronous communication
Many times the user does not want or need to process the message immediately. Message queues provide an asynchronous processing mechanism that allows users to put a message on the queue, but not process it immediately. Put as many messages as you want into the queue, and then process them when needed.