RocketMQ setup

MQ: MessageQueue, message queue. Queue is a FIFO first-in, first-out data structure. Messages are sent by the producer to MQ for queuing, and then handed over to the message consumer for processing in the original order. QQ and WeChat are typical MQ.

The functions of MQ mainly include the following three aspects:

asynchronous

Example: If a courier delivers a package directly to the customer's home, it will be very inefficient. After the introduction of Cainiao Post, couriers only need to place the express delivery at Cainiao Post, and then they can continue to send other express delivery. Customers then go to Cainiao Station to pick up the express delivery according to their own schedule.

Function: Asynchronous can improve the response speed and throughput of the system.
decoupling

Example: "Thinking in JAVA" is a classic, but it's all in English and we can't understand it, so we need an editing agency to translate the articles into other languages, so that communication between English and other languages can be completed.

effect:

1. Decoupling between services can reduce the impact between services. Improve the overall stability and scalability of the system.

2. In addition, data distribution can be achieved after decoupling. After the producer sends a message, it can be consumed by one or more consumers, and the increase or decrease of consumers has no impact on the producer.
Peak clipping

Example: The Yangtze River rises every year, but the speed of the downstream outlet is basically stable, so the water will rise. After the Three Gorges Dam is introduced, water can be stored and slowly drained downstream.

Function: Use stable system resources to cope with sudden traffic impacts.

2. Advantages and disadvantages of MQ

The use of MQ above is also the advantage of using MQ. But introducing MQ also has its shortcomings:

Reduced system availability

As more external dependencies are introduced into the system, the stability of the system will become worse. Once MQ goes down, it will have an impact on the business. This requires consideration of how to ensure the high availability of MQ.

Increased system complexity

The complexity of the system will be greatly increased after the introduction of MQ. In the past, synchronous service calls could be made between services. After the introduction of MQ, it will become asynchronous calls, and the data link will become more complex. And it will also bring about other problems. For example: How to ensure that consumption will not be lost? Will it not be called repeatedly? How to ensure the order of messages and other issues.

Message consistency issue

After system A processes the business, it sends messages to systems B and C through MQ for subsequent business processing. What should I do if system B processes successfully but system C fails? This requires consideration of how to ensure the consistency of message data processing.

3. Comparison of features of several major MQ products

Commonly used MQ products include Kafka, RabbitMQ and RocketMQ. Let’s make a simple comparison of these three products, focusing on understanding their applicable scenarios.

4.Start RocketMQ in Windows

1. Download RocketMQ 4.7.1 version

RocketMQ download address: Download | RocketMQ

Pick your own version:

2. Unzip to local disk and configure JAVA_HOME and ROCKETMQ_HOME

Create a new JAVA_HONE environment variable (no longer discussed here)

Create a new ROCKETMQ_HOME environment variable:

Computer->Properties->Advanced System Settings->Environment Variables->System Variables->New:

The location of rocketmq that you decompressed yourself

3. Modify runserver.cmd

Enter the bin directory and modify the JAVA_OPT of runserver.cmd. The size can be adjusted by yourself.

rem set "JAVA_OPT=%JAVA_OPT% -server -Xms2g -Xmx2g -Xmn1g"
set "JAVA_OPT=%JAVA_OPT% -server -Xms256m -Xmx512m"

4. Start the server

After modification, enter the bin directory and enter the start mqnamesrv.cmd command in cmd to start rocketmq's nameserver.
The following interface will appear after success:

If there is a crash problem, it may be that the permissions are insufficient. Start with administrator permissions.

5. Modify runbroker.cmd

Modify the JAVA_OPT of runbroker.cmd to, the size can be adjusted by yourself

rem set "JAVA_OPT=%JAVA_OPT% -server -Xms2g -Xmx2g -Xmn1g"
set "JAVA_OPT=%JAVA_OPT% -server -Xms256m -Xmx512m"

6. Start broker

Enter commandstart mqbroker.cmd -n localhost:9876 -c ../conf/broker.conf autoCreateTopicEnable=trueStart broker

You can also double-click .cmd to start it directly.

(NoautoCreateTopicEnable=true cannot test tools.cmd)
The following window will appear after success

Take one from someone else’s house:

---------------------------------------------------------------

I was very depressed. I was almost finished writing, and then suddenly it jumped for some reason. The blog I had written could not be returned. I can only rewrite from here

------------------------------------------

Okay, continue....

7.Testing

In the RocketMQ installation package, a tools.sh tool is provided that can be used to quickly verify the RocketMQ service on the command line. We enter the RocketMQ installation directory on worker2:

First, you need to configure an environment variable NAMESRV_ADDR to point to the NameServer service we started.

Then start the message producer to send messages: 1000 messages will be sent by default

D:\rocketmq-all-4.2.0-bin-release\bin start cmd

.\tools.cmd  org.apache.rocketmq.example.quickstart.Producer

Sent successfully.

Then start the message consumer to receive messages:

 .\tools.cmd org.apache.rocketmq.example.quickstart.Consumer

At this point, completed.

5. Configure visualization tools

1.Download

github下载地址：GitHub - apache/rocketmq-dashboard: The state-of-the-art Dashboard of Apache RoccketMQ provides excellent monitoring capability. Various graphs and statistics of events, performance and system information of clients and application is evidently made available to the user.

The source code of the old version is rocketmq-console in rocketmq-external, and the new version has been split into dashboard separately.

2. Unzip the source code to your local computer

3. Packing

. Open the command line window in the decompressed directory and execute: mvn clean package -Dmaven.test.skip=true. The execution is successful as shown in the figure.

mvn clean package -Dmaven.test.skip=true

If you get an error when running, add quotes.

mvn clean package "-Dmaven.test.skip=true"

This has been running for a long time, and I have no choice but to run it again and take a screenshot:

I don’t know what happened.

But this time it was much faster, the first time was 9 minutes.

java -jar starts:

Open the browser localhost:8080

Finish! ! !

I have uploaded both the old and new packages. The resource download address after packaging is:

https://download.csdn.net/download/csl12919/87355373https://download.csdn.net/download/csl12919/87355373

6.Centos Install RocketMQ

The installation of RocketMQ is very simple, just upload and decompress it.

Then we prepare a CentOS7 Linux machine to quickly run RocketMQ. The Linux version I am using is as follows:

[oper@worker1 jdk1.8]$ uname -a
Linux worker1 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

We need to create an operating user to run our own programs, separate from the root user. Create an oper user using the root user and create a working directory for him.

[root@worker1 ~]# useradd oper
[root@worker1 ~]# passwd oper 
设置用户密码
[root@worker1 ~]# mkdir /app
[root@worker1 ~]# chown oper:oper /app

Running RocketMQ requires installing JDK first. We use the most stable JDK1.8 version currently. CentOS can use jdk-8u171-linux-x64.tar.gz in the courseware materials, or you can download it from the Oracle official website. Then use FTP to upload to the working directory of the oper user. Extract it to the /app/jdk1.8 directory by the oper user.

[oper@worker1 tools]$ tar -zxvf jdk-8u171-linux-x64.tar.gz
[oper@worker1 tools]$ mv jdk1.8.0_171/ /app/jdk1.8

Configure environment variables. Use vi ~/.bash_profile to edit the file and add the following content below:

export JAVA_HOME=/app/jdk1.8/
PATH=$JAVA_HOME/bin:$PATH:$HOME/.local/bin:$HOME/bin
export PATH

After editing is completed, execute source ~/.bash_profile to make the environment variables take effect. Enter java -version and you can see the following content, indicating that the JDK installation is successful.

[oper@worker1 ~]$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

Then we decompress the downloaded rocketmq-all-4.7.1-bin-release.zip locally and upload it to the /app/rocketmq directory. After completion, configure the bin directory of rocketmq into the environment variables. vi ~/.bash_profile, add the following content, and execute source ~/.bash_profile to make the environment variables take effect:

export JAVA_HOME=/app/jdk1.8/
export ROCKETMQ_HOME=/app/rocketmq/rocketmq-all-4.7.1-bin-release
PATH=$ROCKETMQ_HOME/bin:$JAVA_HOME/bin:$PATH:$HOME/.local/bin:$HOME/bin
export PATH

In this way, the RocketMQ installation is complete. Let's run him.

This ROCKETMQ_HOME environment variable must be configured separately. If not configured, an error will be reported when starting NameServer and Broker.

The purpose of this environment variable is to load several configuration files except broker.conf under $ROCKETMQ_HOME/conf. So in actual situations, you don't need to follow this configuration, but you must be able to find the configuration file.

1 Start NameServer

Starting NameServer is very simple. There is mqnamesrv in the $ROCKETMQ_HOME/bin directory. Directly executing this script can start RocketMQ's NameServer service.

But please note that RocketMQ’s default JVM memory is 4G, which is the best configuration RocketMQ gives us. But usually when we use a virtual machine, there is not enough 4G memory, so we need to adjust the JVM memory size. The way to modify is to directly modify runserver.sh. Use vi runserver.sh to edit this script and find this line in the script to adjust the memory size to 512M.

JAVA_OPT="${JAVA_OPT} -server -Xms512m -Xmx512m -Xmn256m -
XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

Then we start the NameServer service using silent startup:

nohup bin/mqnamesrv &

After the startup is completed, if you see this key log in nohup.out, the startup is successful. And using the jps command, you can see that there is a NamesrvStartup process.

Java HotSpot(TM) 64-Bit Server VM warning: Using the DefNew young collector with the CMS
collector is deprecated and will likely be removed in a future release
Java HotSpot(TM) 64-Bit Server VM warning: UseCMSCompactAtFullCollection is deprecated and
will likely be removed in a future release.
The Name Server boot success. serializeType=JSON

2 Start Broker

The script to start Broker is runbroker.sh. . The default default memory of Broker is 8G. Before starting, if the memory is not enough, you also need to adjust the JVM memory. vi runbroker.sh, find this line and adjust the memory

JAVA_OPT="${JAVA_OPT} -server -Xms512m -Xmx512m -Xmn256m"

Then we need to find $ROCKETMQ_HOME/conf/broker.conf, edit it with vi command, and add a configuration at the bottom:

autoCreateTopicEnable=true

Then also start runbroker.sh in silent startup mode

nohup ./mqbroker &

After the startup is completed, also check the nohup.out log. If there is this key log, it indicates that the startup is successful. And the jps command can see a BrokerStartup process.

The broker[worker1, 192.168.232.128:10911] boot success. serializeType=JSON

3 Command line quick verification

In the RocketMQ installation package, a tools.sh tool is provided that can be used to quickly verify the RocketMQ service on the command line. We enter the RocketMQ installation directory on worker2:

First, you need to configure an environment variable NAMESRV_ADDR to point to the NameServer service we started.

export NAMESRV_ADDR='localhost:9876'

Then start the message producer to send messages: 1000 messages will be sent by default

bin/tools.sh org.apache.rocketmq.example.quickstart.Producer

Then start the message consumer to receive the message:

bin/tools.sh  org.apache.rocketmq.example.quickstart.Consumer

After starting, you can see the consumed messages.

This Consume instruction will not end, it will continue to hang, waiting to consume other messages. We can stop the process using CTRL+C.

4 Close RocketMQ service

# 1.关闭NameServer
sh bin/mqshutdown namesrv
# 2.关闭Broker
sh bin/mqshutdown broker

So far. Finish! ! ! !

I don’t know what’s going on today, but every time I post code, it jumps to the top.

I don’t know if anyone cares.

Let’s get here first.

I was so scared that I clicked save again!

1. What is MQ? Why use MQ?