Message queue, introduction to rabbitmq, common operations, cluster

message queue

Description on Wikipedia: In computer science, message queue is a method of inter-process communication or communication between different threads of the same process. Software storage is used to process a series of inputs, usually from users . This description is very blunt, and it may be a bit difficult to understand for you who have not touched the message queue.

In fact, message queues existed in the 1980s and 1990s, but it was not originally used in the Internet cluster architecture that we are familiar with. In the past decade or so, the Internet has developed too fast, and the user base has grown larger and larger, and the early simple architecture has long been unable to meet the needs. Therefore, smart architects try their best to solve all bottlenecks, and the message queue is a typical representative. It is currently widely used in the field of asynchronous communication.

What is asynchronous? For example, if a user registers an account on the website, it needs to be confirmed by email. The correct steps are: 1) the user fills in the information (including the email address); 2) the server receives the user information and writes it to the database; 3) the server sends a confirmation email to the user; 4) the website prompts the user to complete the registration. If these 4 steps are completed one by one, the third step is the most time-consuming process, which may take less than 1 second or 1 minute. Therefore, such a design will eventually cause individual users to abandon their registration due to impatientness.

If it is an asynchronous idea, then the third step will be taken out separately, which means that no matter whether the user receives the email or not, the website will directly tell the user that the registration is successful. There is nothing wrong with this design, but how to achieve asynchronous? This uses the "message queue" middleware. Let’s also talk about the small example above. In the second step, the server (actually a certain process) in addition to writing user information to the database, it also writes the user’s email address to the message queue, and The mail sending service will fetch this email address from the message queue, and then send the email to that email address. In this process, the service of writing the message queue (a certain process) and the service of reading the message queue (the service sending mail) are not directly related, they only need to concentrate on writing and reading the message queue.

The above small example is a very typical application scenario for message queues. In addition to this scenario, it can also be used in scenarios where traffic peaks are clipped. I think you should have experienced double 11 Taobao second grab products, this is actually traffic peaking. If under normal circumstances, a website structure carries up to 1w people to visit at the same time, but the number of people who are online at the same time when doing activities reaches 100w. At this time, either increase the server or restrict the people who are online. If you increase the server, you must consider the cost. For such a large event once a year, it is impossible to increase the equipment by 99 times for this time. It is not realistic at all, so I have to limit the number of online activities. How to achieve? The message queue can do this. When doing activities, let everyone queue first, and only let 1w people enter the message queue, and these talents in the message queue can enter the next step of commodity payment. The payment of the people who entered the message queue is completed, the number of people in the message queue is reduced, and new people are allowed to enter the message queue.

Another typical application scenario of message queuing is decoupling. In simple terms, decoupling is to separate the two steps associated with each other and complete the final communication through a message queue service. This is actually an asynchronous mechanism.

Disadvantages of message queues

Due to the asynchronous nature of the message queue, the processing efficiency of the entire architecture is directly improved, and the user experience is improved. But everything has two sides, and message queues are accompanied by defects while bringing performance improvements.

• Reduced system availability

After all, in the entire architecture, we added a separate message queuing middleware, so the risk is increased. If the message queuing service hangs, it will inevitably affect the entire architecture.

• Increased system complexity

It was originally a very simple logical design, but it happened to insert a message queue in the middle, so this increases the workload of the programmer, and it is necessary to consider how to ensure that messages are not repeatedly consumed, whether messages are lost, and the order of messages.

• Data consistency cannot be guaranteed

If the message is not correctly written to the message queue, or the service that reads the message does not read the message correctly, this will affect the consistency of the data.

Message queue roll call

The current mainstream message queues are: RabitMQ, ActiveMQ, RocketMQ, Kafka, ZeroMQ, etc. There are also niche ones such as Beanstalk. Of course, Redis, which we have learned before, can also implement the function of message queues.

The following comparison of several major message queues, I extracted from the Internet, after all, I have not fully studied these message queues in depth, and can only learn from the opinions of others.

ActiveMQ is developed based on the JAVA language, and its community is relatively mature, but at present, the performance of ActiveMQ is relatively poor, and the version iteration is very slow, so it is not recommended.

RocketMQ is produced by Ali, Java is an open source project, we can read the source code directly, and then we can customize our company's MQ, and RocketMQ has the actual test of Alibaba's actual business scenarios. The RocketMQ community activity is relatively average, but it's okay. The documentation is relatively simple. Then the interface is not in accordance with the standard JMS specification. Some systems need to modify a lot of code to migrate. There is also the technology introduced by Alibaba. You have to do well in case this technology is abandoned, and the community will be exposed to the risk. If your company has the technical strength, I think it is good to use RocketMQ.

Kafka is written in Scala and Java, and its characteristics are actually very obvious, that is, it only provides fewer core functions, but provides ultra-high throughput, ms-level latency, extremely high availability and reliability, and distributed can be expanded arbitrarily. At the same time, Kafka is best to support a small number of topics to ensure its ultra-high throughput. The only disadvantage of Kafka is the possibility of repeated consumption of messages, which will have an extremely slight impact on data accuracy. In the field of big data and log collection, this slight impact can be ignored. This feature is naturally suitable for big data real-time calculation and logging. collect.

Although RabbitMQ is slightly inferior to Kafka and RocketMQ in terms of throughput, because it is developed based on erlang, it has strong concurrency, extremely good performance, and low latency, reaching the microsecond level. But also because RabbitMQ is developed based on erlang, few domestic companies have the strength to do research and customization of erlang source code level. If the business scenario does not require too much concurrency (one hundred thousand, one million), RabbitMQ must be your first choice among these four message queues. If it is real-time computing, log collection and other scenarios in the field of big data, Kafka is the industry standard, absolutely no problem, the community is very active and will never be pornographic, not to mention the fact that it is almost a factual norm in this field around the world.

ZeroMQ is just a Pattern library for network programming, which models and componentizes common network request forms (group management, link management, publish and subscribe, etc.), in short, above socket and below MQ. For MQ, network transmission is only a part of it, and more things need to be processed are message storage, routing, Broker service discovery and search, transactions, consumption patterns (ack, reinvest, etc.), cluster services, etc.

RabbitMQ/Kafka/ZeroMQ can provide message queue services, but there are big differences.

In a service-oriented architecture, it is a better idea to use the producer/consumer model for asynchronous communication between services through message brokers (such as RabbitMQ / Kafka, etc.).

ZeroMQ is different from RabbitMQ/Kafka in that it is just an asynchronous message library that provides a mechanism similar to a message broker based on sockets. If you use ZeroMQ, you need to transform your own business code, which is not conducive to service decoupling.

RabbitMQ supports AMQP (binary), STOMP (text), MQTT (binary), HTTP (packaged in other protocols) and other protocols. Kafka uses its own protocol.

Both Kafka's own services and consumers need to rely on Zookeeper.

RabbitMQ's performance will decrease when there is a large amount of messages accumulated, but Kafka will not. After all, the original intention of AMQP was not designed to persist massive messages, and Kafka was used to process massive logs at the beginning.

In general, RabbitMQ and Kafka are both excellent distributed message broker services. As long as they are deployed reasonably and not done, they can basically meet any demand under production conditions.

Provide you with a reference document https://www.zhihu.com/question/43557507

Role/noun in message queue

• Broker

Message server, as a server to provide core message services

• Producer

The message producer, the initiator of the business, is responsible for producing the message and transmitting it to the broker

• Consumer

The message consumer, the business processor, is responsible for obtaining messages from the broker and processing business logic

• Topic

Topic, the unified collection of messages in the publish-subscribe mode, different producers send messages to the topic, which are distributed by the MQ server to different subscribers to realize the broadcast of the message

• Queue

Queue, in PTP mode, a specific producer sends a message to a specific queue, and the consumer subscribes to a specific queue to complete the reception of the specified message

• Message

The message body is a data packet encoded according to a fixed format defined by different communication protocols to encapsulate the business data and realize the transmission of the message

Two working modes in message queue

• Point-to-Point

In fact, it is point-to-point, and the process is relatively simple to understand. It is like making a phone call between two people, and these two people have exclusive use of this communication link. One party sends a message and the other party receives it. It's that simple. In point-to-point mode, messages are kept in the queue. One or more consumers can consume messages in the queue, but a particular message can only be consumed by at most one consumer. Once the consumer reads the message in the queue, it disappears from the queue. A typical example of this mode is an order processing system, where each order will be processed by one order processor, but multiple order processors can also work at the same time.

• Pub/Sub

That is, the publish/subscribe model, which is somewhat similar to subscribing to newspapers in our daily lives. For each subscriber, one or more newspapers can be selected. Then these newspapers we subscribe to are equivalent to topics in the publish-subscribe model. There are many individuals who subscribe to newspapers, and some people may subscribe to the same newspaper as me. Multiple people subscribing to the same newspaper is equivalent to multiple people registering in the same topic. For a newspaper publisher, it forms a one-to-many relationship with all subscribers. In this mode, the message is kept in the subject. Unlike the peer-to-peer model, consumers can subscribe to one or more topics and use all messages in that topic. In this mode, the message producer is called the publisher, and the message consumer is called the subscriber.

Introduction to RabbitMQ

Official website:https://www.rabbitmq.com

RabbitMQ is an open source message queue middleware that is widely used worldwide. It is lightweight, easy to deploy, and supports multiple protocols. It is developed based on Erlang and is naturally capable of high concurrency.

RabbitMQ related terms

· Producer

The process or service that generated the message

· Consumer

The process or service that receives the message

· Queue

RabbitMQ is a message queue middleware, and the real storage of message data is the queue, there can be many queues.

· Switch

Similar to a network device switch, it can send messages to different queues based on different keywords.

· Virtual Host

Virtual hosts are similar to Apache virtual hosts. If there is no virtual host, when the data in RabbitMQ becomes larger and larger, the queues become more and more, and there are headaches in management problems, such as queues and switch naming conflicts. , They influence each other and so on. Virtual hosting can solve these problems, without the need for us to deploy multiple RabbitMQ to be responsible for different businesses.

Virtual hosts provide logical grouping and separation of resources. Each virtual host is essentially a mini version of RabbitMQ server. They have their own connections, queues, bindings, switches, and more importantly, their own permissions mechanism. This is a bit The similar server is the same as the virtual machine running on the server.

Install RabbitMQ under CentOS7 (stand-alone)

Official document: https://www.rabbitmq.com/install-rpm.html

Install erlang

curl -s 
https://packagecloud.io/install/repositories/rabbitmq/erlang/script.rpm.sh | sudo bash

`yum install -y erlang```

Install RabbitMQ

rpm --import https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey
rpm --import https://packagecloud.io/gpg.key
rpm --import https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc

vi /etc/yum.repos.d/rabbitmq.repo  ##内容如下
[bintray-rabbitmq-server]
name=bintray-rabbitmq-rpm
baseurl=https://dl.bintray.com/rabbitmq/rpm/rabbitmq-server/v3.7.x/el/7/
gpgcheck=0
repo_gpgcheck=0
enabled=1

yum install -y rabbitmq-server

Start rabbitmq service

systemctl start rabbitmq-server

查看监听端口
netstat -lntp
监听端口为4369，25672

Open the web management console

rabbitmq-plugins enable rabbitmq_management
查看监听端口多了一个15672
此时可以通过 http://ip:15672 来访问rabbitmq的web管理控制台，用户名密码都是guest，但是有个限制，只允许127.0.0.1访问，所以还需在本机配置一个nginx代理

Configure Nginx proxy

vim /usr/local/nginx/conf/vhost/rabbitmq.conf #内容如下
server {
        listen 80;
        server_name ra.jinkai.cc; 

        location /
        {
            proxy_pass http://127.0.0.1:15672;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
}
##到此结束
检测并重置NGINX服务
nginx -t
nginx -s reload
windows hosts添加192.168.111.136 ra.jinkai.cc

Rabbitmq commonly used commands

Virtual machine management

#列出所有的虚拟主机
rabbitmqctl list_vhosts  
[root@jinkai01 ~]# rabbitmqctl list_vhosts
Listing vhosts ...
name
/

#创建虚拟主机
rabbitmqctl add_vhost <虚拟主机名字> 
[root@jinkai01 ~]# rabbitmqctl add_vhost jinkai
Adding vhost "jinkai" ...
[root@jinkai01 ~]# rabbitmqctl list_vhosts
Listing vhosts ...
name
jinkai
/

#删除虚拟主机
rabbitmqctl delete_vhost <虚拟主机名字> 
[root@jinkai01 ~]# rabbitmqctl delete_vhost jinkai
Deleting vhost "jinkai" ...
[root@jinkai01 ~]# rabbitmqctl list_vhosts
Listing vhosts ...
name
/

User Management

#``创建用户
rabbitmqctl add_user <username> <password> 
如：rabbitmqctl add_user user1 user1_passwd #创建user1用户，密码为admin
[root@jinkai01 ~]# rabbitmqctl add_user user1 admin
Adding user "user1" ...

#列出所有用户
rabbitmqctl list_users 
[root@jinkai01 ~]# rabbitmqctl list_users
Listing users ...
user    tags
user1   []
guest   [administrator]

#更改用户密码
rabbitmqctl change_password <username> <password> 
[root@jinkai01 ~]# rabbitmqctl change_password user1 admin123
Changing password for user "user1" ...

#清除用户密码
rabbitmqctl clear_password <username> 
[root@jinkai01 ~]# rabbitmqctl clear_password user1
Clearing password for user "user1" ...

#删除用户
rabbitmqctl delete_user <username> 
[root@jinkai01 ~]# rabbitmqctl delete_user user1
Deleting user "user1" ...
[root@jinkai01 ~]# rabbitmqctl list_users
Listing users ...
user    tags
guest   [administrator]

· (1) Super administrator (administrator)

You can log in to the management console (when the management plugin is enabled), you can view all the information, and you can operate on users and policies.

· (2) Monitoring

You can log in to the management console (when the management plugin is enabled), and you can view the relevant information of the rabbitmq node (number of processes, memory usage, disk usage, etc.)

· (3) Policymaker

You can log in to the management console (when the management plugin is enabled), and you can manage the policy at the same time. But it is not possible to view information about the node.

· (4) Ordinary managers (management)

You can only log in to the management console (when the management plugin is enabled), you cannot see node information, and cannot manage policies.

· (5) Other

Unable to log in to the management console, usually ordinary producers and consumers.

#赋予用户某个角色
rabbitmqctl set_user_tags <username> <rolename> 
如赋予user1用户management角色
rabbitmqctl set_user_tags user1 managemnet
如同时赋予多个角色
rabbitmqctl set_user_tags user2 monitoring management 

rabbitmqctl set_permissions -p <vhostname> <username> <conf> <write> <read> #给用户设置权限
rabbitmqctl set_permissions -p jinkai user1 '.*' '.*' '.*' 
[root@jinkai01 ~]# rabbitmqctl set_permissions -p jinkai user1 '.*' '.*' '.*'
Setting permissions for user "user1" in vhost "jinkai" ...

#针对jinkai虚拟主机给user1用户设置所有的配置、读写queue和exchange的权限。

说明：用户权限指的是用户对exchange，queue的操作权限，包括配置权限，读写权限。配置权限会影响到exchange，queue的声明和删除。读写权限影响到从queue里取消息，向exchange发送消息以及queue和exchange的绑定(bind)操作。例如： 将queue绑定到某exchange上，需要具有queue的可写权限，以及exchange的可读权限；向exchange发送消息需要具有exchange的可写权限；从queue里取数据需要具有queue的可读权限。

#列出某用户的权限，即该用户对哪个虚拟主机有权限
rabbitmqctl list_user_permissions <username> 
[root@jinkai01 ~]# rabbitmqctl list_user_permissions user1
Listing permissions for user "user1" ...
vhost   configure      write   read
jinkai  .*      .*      .*

#列出指定虚拟主机下所有用户的权限，即哪些用户对该虚拟主机有权限
rabbitmqctl list_permissions -p <vhostname> 
[root@jinkai01 ~]# rabbitmqctl list_permissions -p jinkai
Listing permissions for vhost "jinkai" ...
user    configure      write   read
user1   .*      .*      .*

#清除某用户在指定虚拟机上的授权
rabbitmqctl clear_permissions -p <vhostname> <user> 
[root@jinkai01 ~]# rabbitmqctl clear_permissions -p jinkai user1
Clearing permissions for user "user1" in vhost "jinkai" ... 

Plug-in management

#获取RabbitMQ插件列表
rabbitmq-plugins list

#安装RabbitMQ插件
rabbitmq-plugins enable <插件名字>

#卸载某个插件

rabbitmq-plugins disable <插件名字> #卸载某个插件

limit

#``设置虚拟主机的最大连接数
rabbitmqctl set_vhost_limits -p vhost_name '{"max-connections": 256}' 

#不允许客户端连接虚拟主机
rabbitmqctl set_vhost_limits -p vhost_name '{"max-connections": 0}' 

#不限制连接数
rabbitmqctl set_vhost_limits -p vhost_name '{"max-connections": -1}' 

#限制虚拟主机里最大的队列数
rabbitmqctl set_vhost_limits -p vhost_name '{"max-queues": 1024}' 

#不限制队列数
rabbitmqctl set_vhost_limits -p vhost_name '{"max-queues": -1}' 

other

#``列出所有的交换器
rabbitmqctl list_exchanges

#列出所有的绑定，即把exchange和queue按照路由规则绑定起来
rabbitmqctl list_bindings

#分别查看当前系统种存在的Exchange和Exchange上绑定的Queue信息。
rabbitmqctl list_queues 

#查看运行信息
rabbitmqctl status  

RabbitMQ cluster

RabbitMQ itself is written based on Erlang, and Erlang inherently supports distributed (implemented by synchronizing the cookies of each node of the Erlang cluster), so there is no need to implement distributed clusters through ZooKeeper like Kafka.

· Metadata

There are various basic components inside RabbitMQ, including queues, switches, bindings, virtual hosts, etc. They form the basis of AMQP protocol message communication, and these components exist in the form of metadata

· Memory node and disk node

Each node in the cluster is either a memory node or a disk node. If it is a memory node, all metadata information will only be stored in the memory, while the disk node will not only store all metadata in the memory , It will also be persisted to disk. Therefore, when building a cluster, in order to ensure data security and performance, it is best to have both types of nodes

planning

CPU name	ip	Node type
jinkai01	192.168.111.136	Disk node
jinkai05	192.168.111.140	Memory node
jinkai06	192.168.111.141	Memory node

Deploy the cluster

1. Configure hosts and hostname

Set hostname for three machines

hostnamectl set-hostnamejinkai01,

hostnamectl set-hostnamejinkai05,

hostnamectl set-hostnamejinkai06``

The following hosts need to be edited on all three machines

192.168.111.136 jinkai01
192.168.111.140 jinkai05
192.168.111.141 jinkai06

2. Close selinux and firewalld

All three machines must be executed

setenforce 0
systemctl stop firewalld

3. Install rabbitmq

All three machines must be installed, please refer to the steps above

4. Start the service

All three machines start up

systemctl start rabbitmq-server

5. Install the management plugin

All three machines must be turned on

rabbitmq-plugins enable rabbitmq_management

6. Edit the cookie file

#将三台机器的该文件内容编辑为一致
rsync -av /var/lib/rabbitmq/.erlang.cookie jinkai05:/var/lib/rabbitmq/.erlang.cookie 
rsync -av /var/lib/rabbitmq/.erlang.cookie jinkai06:/var/lib/rabbitmq/.erlang.cookie 
如果失败，可以尝试吧对端的文件先删除，再重新执行
vim  /var/lib/rabbitmq/.erlang.cookie
TQYKRRDIBNBEGSIBSBEM

7. Assign Node

jinkai01 is a disk node, jinkai05 and jinkai06 are memory nodes

Execute on both jinkai05 and jinkai06:

停止rabbitmq
rabbitmqctl stop_app  
如果失败可以尝试重启服务，还不行再重启机器试试

将jinkai05和jinkai06作为内存节点连接到aming01
rabbitmqctl join_cluster --ram rabbit@jinkai01

开启rabbitmq
rabbitmqctl start_app

查看集群状态
rabbitmqctl cluster_status

Follow-up operation

Although the cluster has been set up, we can only use two memory nodes to provide services, so we need to make a load balancer. We can use Nginx's tcp proxy function or use haproxy.

Reference document: http://objcoding.com/2018/10/19/rabbitmq-cluster/#top