The learning process of RocketMQ (1) - Introduction to MQ

1. Simple understanding of MQ

1.1.MQ Introduction:

MQ, message Queue, is a middleware that provides message queuing services, also known as message middleware, and is a set of API software systems that provide the entire process of information production, storage, and consumption.

1.2. The main purpose of MQ:

1. flow clipping

MQ traffic peak clipping refers to using MQ to buffer requests when the system is facing high concurrent requests, so as to avoid excessive pressure on the database or other services.

For example, during Double Eleven and other events, users' operations such as placing orders, rushing to buy, and paying will cause the peak traffic of the system to be concentrated in a short period of time. If the database is directly accessed, it may cause the database to go down or respond slowly. At this time, MQ can be used to receive user requests, and then consume messages at a certain rate, thereby smoothing the system load. This can not only ensure that the user's request will not be lost, but also improve the availability and stability of the system.

Traffic peak shaving is actually the most important function of message queues. To put it bluntly, it can avoid server downtime that may occur during peak hours.

2. Asynchronous decoupling

MQ decoupling refers to the use of MQ to reduce the dependencies between systems, so that data and messages can be exchanged asynchronously between systems. For example, if system A needs to send data to systems B, C, and D, if MQ is not used, then system A needs to directly call the interfaces of systems B, C, and D, which will increase the complexity and risk of system A. If If a system fails or changes, it will affect the normal operation of system A. If MQ is used, system A only needs to send data to MQ, and then systems B, C, and D pull data from MQ, so that system A can be decoupled from systems B, C, and D. This can not only improve the scalability and fault tolerance of the system, but also reduce the development and maintenance costs of the system

1.3. Mainstream products of MQ:

1. ActiveMQ
   is a very old thing, it is said that it has been g
2. RabbitMQ
   RabbitMQ is an open source message middleware developed in Erlang language that implements AMQP (Advanced Message Queuing Protocol). It allows applications to interact asynchronously through messages, improving system reliability, scalability, and flexibility.
3. Kafka
   is widely used in the field of big data and is famous for its high throughput. It is used in real-time computing and log collection scenarios. It does not use the common MQ protocol, but a self-developed protocol.
4. RocketMQ
   is developed in java language. After several years of testing in Alibaba Double Eleven, it borrows from Kafka and also uses a self-developed protocol.
   Performance comparison: Image source https://www.bilibili.com/video/BV1cf4y157sz

1.4. MQ common protocol:

• AMQP (Advanced Message Queuing Protocol), an advanced message queuing protocol, is an application layer standard protocol that provides unified messaging services, designed for message-oriented middleware. It defines the message format, transmission method and exchange rules, and supports multiple languages ​​and platforms. RabbitMQ is an open source message middleware based on the AMQP protocol.
• MQTT (Message Queuing Telemetry Transport), message queue telemetry transmission, is an instant messaging protocol developed by IBM. It is a binary protocol and is mainly used for communication between servers and low-power IoT (Internet of Things) devices. It uses a publish/subscribe model and supports three different quality of service (QoS) levels.
• STOMP (Simple Text Oriented Messaging Protocol), Simple Text Oriented Messaging Protocol, is a text-based message transmission protocol that can communicate between different languages ​​and platforms. It uses frame structure to transmit data, each frame contains a command, an optional header and an optional body.
• XMPP (Extensible Messaging and Presence Protocol) is a protocol based on Extensible Markup Language ( XML ), which is mostly used for instant messaging ( IM ) and online presence detection. Suitable for quasi-instant operations between servers. At its core, it's based on XML streaming, a protocol that may eventually allow Internet users to send instant messages to anyone else on the Internet, even if their operating system and browser are different.
  Advantages: general openness, strong compatibility, scalability, and high security, but the XML encoding format takes up a lot of bandwidth

1.5. The development history of RocketMQ:

• In 2007, Ali started the colorful stone project, and Notify, as the core transaction message circulation system in the project, came into being. The Notify system is the prototype of RocketMQ.

• In 2010, B2B used ActiveMQ as Ali's message core on a large scale, and Ali needed a message system with massive information accumulation.

• At the beginning of 2011, Kafka was open-sourced, Taobao learned from it, and developed Meta MQ.

• In 2012, MetaMQ developed DA to the level of 3.0 and abstracted it to form Rocket MQ, which was open sourced.

• In 2015, based on RocketMQ, Ali launched Aliware MQ, a messaging system specifically for Alibaba Cloud.

• On Double Eleven in 2016, RocketMQ carried the circulation of trillions of messages, crossing a new milestone. On November 28, Alibaba donated RocketMQ to the Apache Software Foundation and became an Apache incubation project.

• On September 25, 2017, Apache announced that RocketMQ was hatched to become the Apache Top-Level Project (TLP), becoming the first top-level project of Internet middleware on Apache in China.