RocketMQ novice learning articles (1) Principles and actual combat that novices can understand!

Learning any technology is a two-step process:

1. Set up the environment

2. helloworld

I'm no exception, just do it directly.

One, the installation of RocketMQ

1. Document

Official website

http://rocketmq.apache.org

GitHub

https://github.com/apache/rocketmq

2. Download

wget https://mirror.bit.edu.cn/apache/rocketmq/4.7.0/rocketmq-all-4.7.0-bin-release.zip

We are based on Centos8, learning for official documents, so the download address is naturally also official.

Go to the official website to find a suitable version to download. Currently, the latest version I have here is version 4.7.0.

http://rocketmq.apache.org/dowloading/releases/

 

https://www.apache.org/dyn/closer.cgi?path=rocketmq/4.7.0/rocketmq-all-4.7.0-bin-release.zip

 

3. Preparation

3.1, unzip

unzip rocketmq-all-4.7.0-bin-release.zip

3.2, install jdk

sudo yum install java-1.8.0-openjdk-devel

4. Start

4.1, start namesrv

cd rocketmq-all-4.7.0-bin-release/bin
./mqnamesrv

4.2, start the broker

cd rocketmq-all-4.7.0-bin-release/bin
./mqbroker -n localhost:9876

Common errors and solutions:

Common error: failed to start broker Cannot allocate memory

[root@node-113b bin]# ./mqbroker -n localhost:9876
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000005c0000000, 8589934592, 0) failed
; error='Cannot allocate memory' (errno=12)#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/local/rocketmq/bin/hs_err_pid1997.log

solution:

It is because the default memory allocation is too large, which exceeds the local memory and directly OOM.

Modify the following two scripts in the bin/ directory

runbroker.sh
runserver.sh

Search in these two scripts and -server -Xmsallocate their memory smaller. If you play by yourself, 512MB is enough, enough!

4.3. Logo for successful startup

The namesrv start successful identification:

The broker startup success flag:

2. Installation of RocketMQ console

There are currently two ways to obtain the console:

1. Go to a third-party website to download the ready-made ones, such as csdn and so on.

2. The official source code package is compiled by itself, and there is no official ready-made one.

We certainly adopt the official method here.

1. Official documents

github repository

https://github.com/apache/rocketmq-externals

Chinese guide

https://github.com/apache/rocketmq-externals/blob/master/rocketmq-console/doc/1_0_0/UserGuide_CN.md

2. Download the source code

https://codeload.github.com/apache/rocketmq-externals/zip/master

3. Modify the configuration (optional)

The file directory after downloading and decompressing is as follows:

The rocketmq-console\src\main\resources\application.propertiesones server.portthat modify the file are just okay. The default is 8080.

4. Compile and package

Enter rocketmq-console, and then use maven to compile and package

mvn clean package -DskipTests

After packaging, we will generate our spring boot jar program under the target, and java -jarstart it directly .

5. Start the console

Throw the compiled and packaged springboot program to the server and execute the following command to start

java -jar rocketmq-console-ng-1.0.1.jar --rocketmq.config.namesrvAddr=127.0.0.1:9876

If you want to start in the background, nohup &

Visit to see the effect:

Three, test

Rocketmq provides us with test tools and test classes, which can be easily tested after installation.

0, preparation

The default test tool provided by rocketmq is in the bin directory, called tools.sh. We need to configure this script before testing, and specify the namesrv address for him, otherwise the following error will occur when testing sending/consuming messages  : connect to null failed :

22:49:02.470 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
java.lang.IllegalStateException: org.apache.rocketmq.remoting.exception.RemotingConnectException: connect to null failed

The configuration is as follows:

vim tools.sh
# 在export JAVA_HOME上面添加如下这段代码
export NAMESRV_ADDR=localhost:9876

1. Send a message

./tools.sh org.apache.rocketmq.example.quickstart.Producer

If it succeeds, you will see the rushing log, because this class will send 1000 messages to TopicTest.

2. Consumer news

./tools.sh org.apache.rocketmq.example.quickstart.Consumer

If it succeeds, you will see the rushing log, because this class will consume all the messages under TopicTest. The 1000 just sent will be consumed.

3. Console

After the transmission is successful, we can naturally come to the control console to see news and consumption information, etc.

 

Four, architecture diagram and roles

1. Architecture diagram

2. Role

2.1、Broker

• Understand as RocketMQ itself

• Broker is mainly used for producer and consumer to receive and send messages

• The broker regularly submits its information to the nameserver

• It is the message storage and forwarding server of message middleware

• When each Broker node is started, it will traverse the NameServer list, establish a long connection with each NameServer, register its own information, and then report it regularly

2.2、Nameserver

• Understand the effect of zookeeper, but he didn’t use zk, instead he wrote a nameserver instead of zk.

• The bottom layer is implemented by netty, which provides routing management, service registration, and service discovery functions. It is a stateless node

• The nameserver is the service discoverer. Each role in the cluster (producer, broker, consumer, etc.) needs to report their status to the nameserver regularly in order to discover each other. If the timeout does not report, the nameserver will remove it from the list.

• Multiple nameservers can be deployed. When multiple nameservers exist, other roles report information to them at the same time to ensure high availability.

• There is no communication between NameServer clusters, and there is no concept of active and standby

• Nameserver memory storage, the broker, topic and other information in the nameserver will not be persisted by default, so it is a stateless node

2.3、Producer

• Message producer

• Randomly select one of the NameServer nodes to establish a long connection to obtain Topic routing information (including queues under topic, which brokers these queues are distributed on, etc.)

• Next, establish a long connection to the master providing topic services (because only the master can write messages in rocketmq), and send heartbeats to the master regularly

2.4、Consumer

• Message consumer

• Obtain Topic routing information through the NameServer cluster, and connect to the corresponding Broker to consume messages

• Since both Master and Slave can read messages, Consumer will establish a connection with both Master and Slave to consume messages

3. The core process

• Brokers are registered on the nameserver

• When the Producer sends a message, it will obtain the topic information of the message from the name server

• Producer establishes long connections to all masters that provide services, and sends heartbeats to masters regularly

• Consumer obtains Topic routing information through the NameServer cluster

• The Consumer will establish connections with all Masters and all Slaves to monitor new messages

Five, core concepts

1、Message

Message carrier. Topic must be specified when Message is sent or consumed. Message has an optional Tag item for filtering messages, and additional key-value pairs can be added.

2、topic

Logical classification of the message. Before sending a message, you must specify a topic to send, that is, the message is sent to this topic. Specify this topic for consumption when consuming messages. It is logical classification.

3、queue

A Topic will be divided into N Queues, and the number is configurable. The message itself is actually stored on the queue, and what consumers consume is also the message on the queue. Let me talk more. For example, if there are 1 topic and 4 queues, and 5 consumers are consuming this topic, then one consumer will be wasted. Because of the load balancing strategy, each consumer consumes 1 queue, 5>4, overflow 1 Yes, this will not work.

4、Tag

Tag is a further subdivision of Topic, as the name implies, tag. Each message can be tagged when it is sent, and it can be filtered according to the tag when consuming, and it can be selectively consumed.

5、Message Model

Message model: Clustering and Broadcasting

6、Message Order

Message order: order (Orderly) and concurrent (Concurrently)

7、Producer Group

Message Producer Group

8、Consumer Group

Message consumer group

Six, ACK

The first thing to be clear is that the ACK mechanism occurs on the Consumer side, not on the Producer side . That is to say, after the Consumer consumes the message, it must confirm the ACK. If it is not confirmed, it means that the consumption has failed. At this time, the Broker will carry out a retry strategy (only the cluster mode will retry). ACK means: Consumer said: ok, I succeeded in consumption. Mark this message as consumed.

7. Consumption model

1. Cluster mode (Clustering)

1.1 Diagram

 

1.2. Features

• Each message only needs to be processed once, and the broker will only send the message to one consumer in the consumer cluster

• When the message is reposted, it is not guaranteed to be routed to the same machine

• The consumption status is maintained by the broker

2. Broadcasting mode (Broadcasting)

2.1 Illustration

 

2.2. Features

• The consumption schedule is maintained by the consumer

• Ensure that every consumer consumes the message once

• Messages of failed consumption will not be reposted

8. RocketMQ-API: https://blog.csdn.net/My_SweetXue/article/details/107381276