Do not miss passing through
Click the blue word to follow us
How to ensure reliability?
RabbitMQ provides several features, at the expense of a little performance cost, and provides a guarantee of reliability.
Endurance
When RabbitMQ exits, the message and queue will be cleared by default, so you need to specify its persistence attribute as true when you declare the queue and send the message for the first time, so that RabbitMQ will store the queue, message and status in the local RabbitMQ database. It will be restored after restarting.
durable=true
channel.queueDeclare ("task_queue", durable, false, false, null); // 队列
channel.basicPublish("", "task_queue",
MessageProperties.PERSISTENT_TEXT_PLAIN,
message.getBytes()); // 消息
Note: When the declared queue already exists, trying to redefine its durable is invalid.
Receive reply
The default mode for the client to receive messages is automatic response, but the client can actively respond to messages by setting autoAck to false. When the client rejects this message or disconnects without answering, it will make this message re-enter the queue (before version 2.7.0 it was re-joined to the end of the queue, 2.7.0 and later kept the message in the queue Original location).
autoAck = false;
requeue = true;
channel.basicConsume(queue, autoAck, callback);
channel.basicAck();//应答
channel.basicReject(deliveryTag, requeue); // 拒绝
channel.basicRecover(requeue); // 恢复
Send confirmation
By default, the sender does not care whether the sent message is consumed. The channel can be set to confirm mode, all sent messages will be confirmed once, and the user can check the status according to the confirmation message sent back by the server.
channel.confirmSelect(); // 进入confirm模式
// do publish messages... 每条消息都会被编号，从1开始
channel.getNextPublishSeqNo() // 查看下一条要发送的消息的序号
channel.waitForConfirms(); // 等待所有消息发送并确认
Transaction: and confirm mode can not be used at the same time, and will bring a lot of redundant overhead, resulting in a lot of decline in throughput, so it is not recommended.
channel.txSelect();
try {
// do something...
channel.txCommit();
} catch (e){
channel.txRollback();
}
<a name="ha" /> High availability of message queue (active/standby mode)
Compared with routing and binding, it can be regarded as shared by all nodes. By default, the message queue only exists on the node that declares it for the first time, so that once the node hangs up, there are no unprocessed messages in the queue. Up.
Fortunately, RabbitMQ provides a mechanism to back it up to other nodes. At any time, a master is responsible for processing requests, and other slaves are responsible for backups. When the master dies, the slave created first will be promoted to the master.
command:
rabbitmqctl set_policy ha-all "^ha\."'{"ha-mode":"all"}': Set all queues starting with'ha' to have backups on all nodes.
It can also be configured on the interface.
Note: Since the exclusive type of queue will be deleted when the client and server are disconnected, it is meaningless to set persistent attributes and backups for it.
Order guarantee
Just go to the picture above:
Some things to pay attention to
Cluster configuration:
Multiple nodes in a cluster share an .erlang.cookie file; if RABBITMQ_USE_LONGNAME is not enabled, you need to specify the addresses of other nodes in the hosts file of each node, otherwise you will not find nodes in other clusters.
<a name="cluster_partion" /> Split brain (network partition):
The RabbitMQ cluster is not very good at processing and tolerating network partitions. It is recommended to use federation or shovel plugin to solve it.
However, the situation has already happened, how to solve it? Don't worry, there is still a way to recover.
When the network is intermittent, the communication between nodes will be interrupted, which will cause the cluster to be separated.
In this way, each small cluster will only process its own local connections and messages, resulting in data out of sync. When the network connection is restored, they both think that each other is down -_-||, and it can be judged that there is a network partition. However, RabbitMQ ignores and does not process it by default, causing the two nodes to continue to work independently (routes, binding relationships, queues, etc. can be created and deleted independently, and even the main and standby queues will each have their own master).
The configuration can be changed so that when the connection is restored, it will be automatically restored according to the configuration:
ignore: default, do not do any processing
pause-minority: When disconnecting, determine whether the current node is in the minority (the number of nodes is less than or equal to half), if so, pause until the connection is restored.
{pause_if_all_down, [nodes], ignore | autoheal}: When disconnecting, determine whether there are nodes in the current cluster in nodes, if so, continue to run, otherwise pause until the connection is restored. Under this strategy, when the connection is restored, multiple partitions may survive, so the last parameter determines how to merge them.
autoheal: When the connection is restored, the node with the most client connections is selected as the main state, and other nodes are restarted.
Configuration: cluster configuration
Multiple ack
If the client responds to the same message multiple times, the client cannot receive subsequent messages.
Use with Docker
Implementation of the cluster version: see an example I wrote myself rabbitmq-server-cluster
Comparison of message queue middleware
RabbitMQ：
Advantages: Support many protocols such as: AMQP, XMPP, STMP, STOMP; flexible routing; mature and stable cluster solution; load balancing; data persistence, etc.
Disadvantages: slower speed; relatively heavyweight, the installation needs to rely on the Erlang environment.
Redis ：
Advantages: relatively lightweight and easy to use
Disadvantages: single point problem, single function
Kafka：
Advantages: high throughput; distributed; fast persistence; load balancing; lightweight
Disadvantages: messages will be lost in extreme cases
Finally, attach a test result intercepted on the Internet:
If you are interested in a brief introduction to RabbitMQ, you can continue to read it~
Introduction
Several important concepts
Virtual Host: Contains several Exchanges and Queues, representing a node;
Exchange: Accept messages sent by the client and route the messages to the queue in the server according to the Binding. Exchange is divided into three types: direct, fanout, and topic.
Binding: Connect Exchange and Queue, including routing rules.
Queue: Message queue, which stores messages that have not yet been consumed.
Message: Header+Body
Channel: Channel, execute AMQP commands; one connection can create multiple channels to save resources.
Client
RabbitMQ officially implements many popular language clients, so I won’t list them one by one. Take java as an example, let’s start with the topic directly:
establish connection:
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
A disconnect retry mechanism can be added:
factory.setAutomaticRecoveryEnabled(true);
factory.setNetworkRecoveryInterval(10000);
Create connections and channels:
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
One to one: one producer, one consumer
Producer:
channel.queueDeclare (QUEUE_NAME, false, false, false, null);
channel.basicPublish ("", QUEUE_NAME, null, message.getBytes());
consumer:
Consumer consumer = new DefaultConsumer(channel) {
@Override
public void handleDelivery (String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body)
throws IOException {
String message = new String(body, "UTF-8");
System.out.println(" [x] Received '" + message + "'");
}
};
channel.basicConsume (QUEUE_NAME, autoAck, consumer);
One-to-many: one producer, multiple consumers
The code is the same as above, but there will be multiple consumers, and messages will be sent to each consumer in turn.
If autoAck=false is set, fair distribution can be achieved (that is, for a specific consumer, only a specified number of messages can be sent each time, until one of the messages is answered, then the next one can be sent). Need to add to consumers:
int prefetchCount = 1;
channel.basicQos(prefetchCount);
Others are the same as above.
broadcast
Producer:
channel.exchangeDeclare (EXCHANGE_NAME, "fanout");
String queueName = channel.queueDeclare().getQueue();
channel.queueBind (queueName, EXCHANGE_NAME, "");
channel.basicPublish (EXCHANGE_NAME, "", null, message.getBytes());;
Consumers are the same as above.
Routing: Specify routing rules
Producer:
String queueName = channel.queueDeclare().getQueue();
channel.queueBind(queueName, EXCHANGE_NAME, routingKey);
channel.basicPublish(EXCHANGE_NAME, routingKey, null, message.getBytes());
Consumers are the same as above.
Topics: Routing that supports wildcards
* Can represent a word
#Can represent one or more words
Producer:
channel.exchangeDeclare (EXCHANGE_NAME, "topic");
String queueName = channel.queueDeclare().getQueue();
channel.queueBind (queueName, EXCHANGE_NAME, bindingKey);
Consumers are the same as above.
RPC
In fact, it is a usage of one-to-one mode:
First, the client sends a message to the queue declared by the server, and the message attributes include reply_to and correlation_id
-reply_to is a queue of messages created by the client to receive remote call results
-correlation_id is the identifier of the message, which will be included in the message attribute of the response from the server to know which message is the result.
Then, the server receives the message, processes it, and returns a result to the reply_to queue,
Finally, the client receives the return message and continues to process it downward.
Server
Supports major mainstream operating systems. Here, we take Unix as an example to introduce the following common configurations and commands:
installation
Since RabbitMQ is dependent on Erlang, the latest version of Erlang must be installed first.
Single point installation is relatively simple, just download and unzip. download link
Configuration: (Generally, just use the default.)
$RABBITMQ_HOME/etc/rabbitmq/rabbitmq-env.conf: The default configuration of environment variables (can also be set in the startup script, and the configuration in the startup command shall prevail). Commonly used ones are:
RABBITMQ_NODENAME: Node name, the default is rabbit@$HOSTNAME.
RABBITMQ_NODE_PORT: Protocol port number, the default is 5672.
RABBITMQ_SERVER_START_ARGS: Overwrite some configurations in rabbitmq.config.
$RABBITMQ_HOME/etc/rabbitmq/rabbitmq.config: Core components, plug-ins, erlang services and other configurations, commonly used are:
disk_free_limit: Information such as queue persistence is stored in RabbitMQ's local database, and the default limit is 50000000 (that is, it can only use 50M space at most, and it can be adjusted up if it is not enough, and it also supports the configuration of free space percentage). If it exceeds the limit, it will strike...
vm_memory_high_watermark: memory usage, the default is 0.4 (allow it to use 40% of the memory at most, strike if it exceeds the mark)
Note: If the startup fails, you can view the specific error information in the startup log.
command:
$RABBITMQ_HOME/sbin/rabbitmq-server: startup script, which will print out configuration files, plug-ins, clusters and other information; plus -detached for background startup;
/sbin/rabbitmqctl status: View the startup status
/sbin/rabbitmqctl add_user admin admin: Add a new user admin, password admin; there is only one guest user by default, but only local access.
/sbin/rabbitmqctl set_user_tags admin administrator: set admin as an administrator
/sbin/rabbitmqctl set_permissions -p / admin ".*"".*" ".*" Grant all permissions to admin
/sbin/rabbitmqctl stop: close
Cluster
Cluster nodes share all state and data, such as: users, routing, binding and other information (the queue is a bit special, although it is reachable from all nodes, but only exists on the node where it is first declared. Solution: message High availability of the queue); each node can receive connections and process data.
There are two cluster nodes, disc: default, the information is stored in the local database; ram: when joining the cluster, add the --ram parameter, the information is stored in memory, which can improve performance.
Configuration: (Generally, just use the default.)
$RABBITMQ_HOME/etc/rabbitmq/rabbitmq-env.conf：
RABBITMQ_USE_LONGNAME: default false, (default, the $HOSTNAME behind @ in RABBITMQ_NODENAME is the host name, so the hosts file of each node in the cluster needs to contain the mapping of other node host names to addresses. But if set to true, RABBITMQ_NODENAME can be defined $HOSTNAME in is a domain name)
RABBITMQ_DIST_PORT: cluster port number, default RABBITMQ_NODE_PORT + 20000
$RABBITMQ_HOME/etc/rabbitmq/rabbitmq.config：
cluster_nodes: After setting, it will try to automatically connect to the joined nodes and form a cluster at startup.
cluster_partition_handling: Handling of network partitions.
For more detailed configuration, see: Configuration
command
rabbitmqctl stop_app
rabbitmqctl join_cluster [--ram] nodename@hostname: join the current node to the cluster; the default is to join the cluster with the disc node, plus --ram as the ram node.
rabbitmqctl start_app
rabbitmqctl cluster_status: view cluster status
Note: If you fail to join the cluster, you can check first
The content of $HOME/.erlang.cookie of each node is consistent;
If hostname is a host name, then the mapping between hostname and address needs to be added to the hosts file;
If you are using a domain name, you need to set RABBITMQ_USE_LONGNAME to true.
Note: For the docker version of the cluster, see: rabbitmq-server-cluster
advanced
Introduction to AMQP protocol
RabbitMQ natively supports AMQP 0-9-1 and extends some commonly used functions: AMQP 0-9-1
Contains three layers:
Model layer: The highest layer, which provides commands invoked by the client, such as queue.declare, basic.ack, consume, etc.
Session layer: transfer commands from the client to the server, and then transfer the server's response to the client. The session layer provides reliability, synchronization mechanism and error handling for this transfer process.
Transport layer: It mainly transmits binary data streams, providing frame processing, channel multiplexing, error detection and data representation.
Note: For the support of other protocols, see: Protocols supported by RabbitMQ
Commonly used plugins
Management interface (artifact)
After startup, execute rabbitmq-plugins enable rabbitmq_management->
Visit http://localhost:15672-> to view node status, queue information, etc., and you can even dynamically configure the active/standby strategy of the message queue, as shown below:
Federation
Enabling the Federation plug-in allows the nodes of different clusters to pass messages, thereby simulating a cluster-like effect. This can have several advantages:
Loose coupling: Different clusters that are united together can have their own users, permissions and other information, and do not need to be consistent; in addition, the RabbitMQ and Erlang versions of these clusters can be inconsistent.
The remote network connection is friendly: because the communication follows the AMQP protocol, it has a high tolerance for intermittent network connections.
Customization: You can choose which components to enable federation.
Several concepts:
Upstreams: Define upstream node information, such as:
rabbitmqctlset_parameterfederation-upstreammy-upstream'{"uri":"amqp://server-name","expires":3600000}'定义一个my-upstream
Uri is the address of its upstream node, and multiple upstream nodes do not need to be in the same cluster.
Expires means that the upstream node will cache the message after 3600000ms is disconnected.
Upstream sets: A collection of multiple Upstreams; there is an all by default, which will add all Upstreams.
Policies: Define which exchanges and queues are associated with which Upstream or Upstreamset, such as:
rabbitmqctl set_policy --apply-to exchanges federate-me "^amq\."'{"federation-upstream-set":"all"}' All the exchanges of this node beginning with amq. are combined to the same-named exchange of the upstream node.
Note:
Since the exchange of the downstream node can continue to be the upstream of other nodes, it can be set to loop, broadcast and other forms.
The number of transmission layers is controlled by the max_hops parameter.
To simulate a cluster, you can interconnect multiple nodes in pairs and set max_hops=1.
federated_cluster
federated_broadcast
rabbitmq-plugins enable rabbitmq_federation
If the management interface is enabled, you can add:
rabbitmq-plugins enable rabbitmq_federation_management
In this way, Upstream and Policy can be configured on the interface.
Note: If you use federation in a cluster, you need to enable the Federation plugin on each node of the cluster
Note: For more plug-ins, please see: plug-ins
Previous wonderful recommendations
Summary of Tencent, Ali, Didi Backstage Interview Questions-(including answers)
Interview: The most comprehensive multi-threaded interview questions in history!
The latest Alibaba pushes Java back-end interview questions
JVM is difficult to learn? That's because you didn't read this article seriously
—END—
Follow the author's WeChat public account—"JAVA Rotten Pigskin"
Learn more about java back-end architecture knowledge and the latest interview book
Every thing you order is pretty, I take it seriously as a favorite
After reading this article, remember to give the author a thumbs up + watching it~~~ Everyone's support is the motivation for the author to continue publishing articles.