An overview of MQ (message middleware) and a basic introduction to RabbitMQ

overview

Message middleware (MQ)

• Message queue middleware is an important component in a distributed system. It mainly solves problems such as application decoupling, asynchronous messages, and traffic shaving , and realizes high performance, high availability, scalability, and eventual consistency architecture. Flow shaving: reduce peak pressure (second kill, rush purchase)

• MQ (Message Queue, message queue) is a typical producer and consumer model. Producers continuously produce messages to the message queue, and consumers continuously obtain messages from the queue. Because the production and consumption of messages are asynchronous, and only care about the sending and receiving of messages, there is no intrusion of business logic, so the decoupling of producers and consumers is realized.

• Producer: The party that sends messages to the message middleware

  Consumer: The party that gets the message from the middle

  broker: message middleware

• Common message middleware:

  • RabbitMQ: based on the AMQP protocol, developed in erlang language, with good stability

    An open source message management system that supports mainstream operating systems, such as Linux, Windows, MacOX, etc., and supports multiple development languages, such as Java, Python, Ruby, .NET, PHP, C/C++, node.js, etc.

  • RocketMQ: Based on JMS, Alibaba products, currently handed over to the Apache Foundation

  • ActiveMQ: based on JMS

  • kafka: Distributed message system, high throughput

AMQP and JMS standards

There are two mainstream ways to implement MQ: AMQP and JMS.

• AMQP, or Advanced Message Queuing Protocol, is an application layer standard advanced message queuing protocol that provides unified messaging services. It is an open standard for application layer protocols and is designed for message-oriented middleware.

  The client and the message middleware based on this protocol can transmit messages, and are not limited by different products of the client middleware, different development languages ​​and other conditions.

• JMS, usually mentioned JMS (Java MessageService) actually refers to the JMS API. JMS is an early message standard proposed by Sun, aiming to provide unified message operations for Java applications, including create, send, receive, etc.

  JMS has become part of java Enterprise Edition. From the perspective of use, JMS and JDBC play similar roles, and users can communicate with services that implement JMS and perform related operations according to the corresponding interfaces.

Differences and connections between the two:

• JMS defines a unified interface to unify message operations

  AMQP is a format for unifying data interaction by specifying a protocol

• JMS restricts the use of the Java language

  AMQP is only a protocol and does not specify the implementation method, so it is cross-language

• JMS specifies two message models point-to-point, publish-subscribe

  The message model of AMQP is more abundant (7 types), and there are 5 types commonly used in work

Brief introduction to RabbitMQ

basic concept

• Broker : Represents the message queue server entity

• Message : message, message is anonymous, it consists of message header and message body.

  The message body is opaque, while the message header consists of a series of optional attributes, including routing-key (routing key), priority (priority relative to other messages), delivery-mode (indicating that the message may require persistent storage), etc.

• Publisher : The producer of the message, which is also a client application that publishes the message to the exchange.

• Virtual Host : virtual host, which represents a batch of switches, message queues and related objects.

  Virtual hosts are separate domains of servers that share the same authentication and encryption environment.

  Each vhost is essentially a mini version of RabbitMQ server, with its own queue, switch, binding and permission mechanism.

  The vhost is the basis of the AMQP concept and must be specified when connecting. The default vhost of RabbitMQ is /

• Exchange : An exchange that receives messages from producers and routes them to queues in the server.

• Binding : Binding, used for the association between message queues and exchanges.

  A binding is a routing rule that connects an exchange and a message queue based on a routing key, so an exchange can be understood as a routing table composed of bindings.

• Queue : Message queue, used to save messages until sent to consumers. It is the container of the message and the destination of the message.

  A message can be put on one or more queues. The message is always in the queue, waiting for the consumer to connect to the queue to take it away.

• Connection : A network connection, such as a TCP connection.

• Channel : channel, an independent bidirectional data flow channel in a multiplexed connection.

  The channel is established on the virtual connection in the real TCP connection. AMQP commands are sent through the channel. Whether it is publishing a message, subscribing to a queue or receiving a message, these actions are all completed through the channel.

  Because it is very expensive for the operating system to establish and destroy TCP, the concept of channel is introduced to reuse a TCP connection.

• Consumer : The consumer of the message, which represents a client application that gets the message from the message queue.

message model

RabbitMQ provides 6 message models, but the sixth is actually RPC, not MQ. Among them, 3, 4, and 5 all belong to the subscription model, but the routing methods are different.

Basic message model (direct connection)

Graphic:

• P: Producer, that is, the program to send the message
• C: The consumer, that is, the receiver of the message, will always wait for the message to arrive.
• The red part in the picture: queue (message queue), similar to a mailbox, can cache messages; producers send messages to the queue, consumers get messages from the queue, and the queue is a buffer for storing messages. The queue is limited only by the host's memory and disk, and is essentially a large message buffer.

sample code

• Producer:

  // 连接工具类
  public class ConnectionUtil {
        
        
      // 建立与RabbitMQ的连接
      public static Connection getConnection() throws Exception {
        
        
          //定义连接工厂
          ConnectionFactory factory = new ConnectionFactory();
          //设置服务地址
          factory.setHost("127.0.0.1");
          //端口
          factory.setPort(5672);
          //设置账号信息，用户名、密码、vhost
          factory.setVirtualHost("/saas");
          factory.setUsername("saas");
          factory.setPassword("saas");
          // 通过工程获取连接
          Connection connection = factory.newConnection();
          return connection;
      }
  }
  

  Producer sends message:

  public class Send {
        
        
  
      private final static String QUEUE_NAME = "simple_queue";
  
      public static void main(String[] argv) throws Exception {
        
        
          // 获取到连接
          Connection connection = ConnectionUtil.getConnection();
          // 从连接中创建通道，使用通道才能完成消息相关的操作
          Channel channel = connection.createChannel();
          // 声明（创建）队列
          channel.queueDeclare(QUEUE_NAME, false, false, false, null);
          // 消息内容
          String message = "Hello World!";
          // 向指定的队列中发送消息
          channel.basicPublish("", QUEUE_NAME, null, message.getBytes());
          
          System.out.println(" [x] Sent '" + message + "'");
  
          //关闭通道和连接
          channel.close();
          connection.close();
      }
  }
  

Job (task) model

Work model, Work queues, also known as (Task queues, task model). When message processing is time-consuming, the speed of message production may be far greater than the speed of message consumption. If things go on like this, more and more messages will pile up and cannot be processed in time. At this point, you can use the work model: let multiple consumers bind to a queue and consume the messages in the queue together. Once the messages in the queue are consumed, they will disappear, so the task will not be executed repeatedly.

By default, RabbitMQ will send each message to the next consumer in order. On average, each consumer receives the same number of messages. This way of distributing messages is called round robin.

Disadvantage: When one of the consumption is relatively slow, for example, a consumer has already consumed. But another consumer is still receiving one by one. This will cause the accumulation of information.

Optimization:

• Consumer: To turn off automatic confirmation messages

• The current channel can only consume one message at a time

  // 每次只能消费一个消息|一次只接收一条未确认的消息
  channel.basicQos(1);
  

• Message Confirmation: manually confirm the message

Sample code:

• producer

  public class Provider {
        
        
      private static final String TASK_QUEUE_NAME = "task_queue";
   
      public static void main(String[] argv) throws Exception {
        
        
          // 获取连接对象
          Connection connection = RabbitMQUtils.getConnection();
          // 获取连接中的通道对象
          Channel channel = connection.createChannel();
          // 通过通道声明队列
          channel.queueDeclare(TASK_QUEUE_NAME, true, false, false, null);
          // 发送10条消息
          for (int i = 0; i < 20; i++) {
        
        
              channel.basicPublish("", TASK_QUEUE_NAME, null, (i+"你好啊").getBytes());
          }
      }
  }
  

• Consumers (create 2, the difference is whether to bring the following thread sleep code)

  public class Customer2 {
        
        
      private static final String TASK_QUEUE_NAME = "task_queue";
   
      public static void main(String[] args) throws IOException {
        
        
          // 获取连接对象
          Connection connection = RabbitMQUtils.getConnection();
          // 获取连接中的通道对象
          Channel channel = connection.createChannel();
          // 每次只能消费一个消息
          channel.basicQos(1);
          // 通道绑定消息队列
          channel.queueDeclare(TASK_QUEUE_NAME, true, false, false, null);
          // 获取消息  参数2：消息自动确认，消费者自动向rabbitmq确认消息消费（只要消息队列有消息，就分配给消费者）
          channel.basicConsume(TASK_QUEUE_NAME, false, new DefaultConsumer(channel) {
        
        
              @Override
              public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
        
        
                  System.out.println("消费者-1：" + new String(body));
                  try {
        
        
                      Thread.sleep(1000);
                  } catch (InterruptedException e) {
        
        
                      e.printStackTrace();
                  }
                  // 参数1：确认队列中那个具体消息 参数2：是否开启多个消息同时确认
                  channel.basicAck(envelope.getDeliveryTag(),false);
              }
          });
      }
  }
  

Publish Subscribe Model

• 1 producer, many consumers
• Each consumer has its own queue
• Each queue must be bound to an exchange
• The producer does not send the message directly to the queue, but to the exchange
• The message sent by the producer reaches the queue through the switch, so that a message can be obtained by multiple consumers

X (Exchanges): Exchanges. On the one hand: Receive the message sent by the producer. On the other hand: knowing how to process messages, such as submitting to a particular queue, submitting to all queues, or discarding the message. Exactly how to do this depends on the type of Exchange. Exchange (exchange) is only responsible for forwarding messages, and does not have the ability to store messages, so if there is no queue bound to Exchange, or there is no queue that meets the routing rules, then the message will be lost!

Exchange types are as follows:

• Fanout : Broadcast, hand over the message to all queues bound to the exchange
• Direct : directional, deliver the message to the queue that matches the specified routing key
• Topic : wildcard, send the message to the queue that matches the routing pattern (routing pattern)

Subscription model - Fanout (broadcast) type

Message sending process in broadcast mode:

• can have multiple consumers
• Each consumer has its own queue (queue)
• Each queue must be bound to an Exchange (exchange)
• The message sent by the producer can only be sent to the switch, and the switch decides which queue to send to, and the producer cannot decide
• The exchange sends the message to all bound queues
• Consumers of the queue can get the message. Realize that a message is consumed by multiple consumers

Sample code:

• producer

  public class Provider {
        
        
      public static void main(String[] args) throws IOException {
        
        
          // 获取连接对象
          Connection connection = RabbitMQUtils.getConnection();
          // 获取连接中的通道对象
          Channel channel = connection.createChannel();
   
          // 将通道声明指定交换机 参数1：交换机名称 参数2：交换机类型，fanout是广播类型
          channel.exchangeDeclare("logs","fanout");
          // 发送消息
          channel.basicPublish("logs","",null,"fanout type message".getBytes());
      }
  }
  

• consumer

  public class Customer1 {
        
        
      public static void main(String[] args) throws IOException {
        
        
          Connection connection = RabbitMQUtils.getConnection();
   
          Channel channel = connection.createChannel();
          //通道绑定交换机
          channel.exchangeDeclare("logs","fanout");
          // 创建临时队列
          String queue = channel.queueDeclare().getQueue();
          // 绑定交换机和队列
          channel.queueBind(queue,"logs","");
          // 消费消息
          channel.basicConsume(queue,true,new DefaultConsumer(channel){
        
        
              @Override
              public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
        
        
                  System.out.println("消费者1"+new String(body));
              }
          });
      }
  }
  

  public class Customer2 {
        
        
      public static void main(String[] args) throws IOException {
        
        
          Connection connection = RabbitMQUtils.getConnection();
   
          Channel channel = connection.createChannel();
          //通道绑定交换机
          channel.exchangeDeclare("logs","fanout");
          // 创建临时队列
          String queue = channel.queueDeclare().getQueue();
          // 绑定交换机和队列
          channel.queueBind(queue,"logs","");
          // 消费消息
          channel.basicConsume(queue,true,new DefaultConsumer(channel){
        
        
              @Override
              public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
        
        
                  System.out.println("消费者2"+new String(body));
              }
          });
      }
  }
  

Subscription model - Direct type (routing)

In Fanout mode, a message will be consumed by all subscribed queues. However, in some scenarios, you may want different messages to be consumed by different queues. At this time, the Direct type of Exchange will be used.

Under the Direct model:

• The binding between the queue and the switch cannot be arbitrary binding, but a RoutingKey (routing key) must be specified
• The sender of the message must also specify the RoutingKey of the message when sending the message to Exchange
• Exchange no longer delivers messages to each bound queue, but judges based on the Routing Key of the message. Only when the Routing Key of the queue is exactly the same as the Routing Key of the message, can the message be received

Scenario example: When an error occurs, it needs to be printed on the console and persisted in the log

• P: Producer, send a message to Exchange, when sending a message, a routing key will be specified
• x: Exchange (exchange), receive the message from the producer, and then deliver the message to the queue that exactly matches the routing key
• C1: The consumer whose queue specifies a message whose routing key is error
• C2: The consumer whose queue specifies messages whose routing keys are info, error, and warning

Sample code:

• Producer:

  public class Provider {
        
        
      public static void main(String[] args) throws IOException {
        
        
          // 创建connection
          Connection connection = RabbitMQUtils.getConnection();
          // 创建channel
          Channel channel = connection.createChannel();
          String exchangeName = "logs_direct";
          // 通道声明指定交换机 参数1：交换机名称 参数2：交换机类型，direct是路由模式
          channel.exchangeDeclare(exchangeName,"direct");
          // 发送消息
          channel.basicPublish(exchangeName,"error",null,("这是direct模型发布的基于route" +
                  " kye: ["+"info"+"]发送的消息").getBytes());
      }
  }
  

• consumer:

  public class Customer1 {
        
        
      public static void main(String[] args) throws IOException {
        
        
          // 1.建立连接
          Connection connection = RabbitMQUtils.getConnection();
          // 建立channel
          Channel channel = connection.createChannel();
          String exchangeName = "logs_direct";
          // 绑定交换机和通道
          channel.exchangeDeclare(exchangeName,"direct");
          // 创建队列
          String queue = channel.queueDeclare().getQueue();
          // 基于路由key ,绑定队列和交换机
          channel.queueBind(queue,exchangeName,"error");
          // 消费
          channel.basicConsume(queue,true,new DefaultConsumer(channel) {
        
        
              @Override
              public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
        
        
                  System.out.println("消费者1："+ new String(body));
              }
          });
      }
  }
  

  public class Customer2 {
        
        
      public static void main(String[] args) throws IOException {
        
        
          // 1.建立连接
          Connection connection = RabbitMQUtils.getConnection();
          // 建立channel
          Channel channel = connection.createChannel();
          String exchangeName = "logs_direct";
          // 绑定交换机和通道
          channel.exchangeDeclare(exchangeName,"direct");
          // 创建队列
          String queue = channel.queueDeclare().getQueue();
          // 基于路由key ,绑定队列和交换机
          channel.queueBind(queue,exchangeName,"info");
          channel.queueBind(queue,exchangeName,"error");
          channel.queueBind(queue,exchangeName,"warning");
   
          // 消费
          channel.basicConsume(queue,true,new DefaultConsumer(channel) {
        
        
              @Override
              public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
        
        
                  System.out.println("消费者2："+ new String(body));
              }
          });
      }
  }
  

Subscription Model - Topic Type

Compared with Direct, Topic type Exchange can route messages to different queues according to RoutingKey. It's just that the Topic type Exchange allows the queue to use wildcards when binding the Routing key.

This model Routingkey is generally composed of one or more words, separated by ".", for example: item.insert

通配符：
	* 	匹配不多不少恰好1个词
	# 	匹配0个，1个或多个词
如:
	audit.#    匹配 audit.irs.corporate 或者 audit.irs 等
	audit.*    只能匹配 audit.irs

Sample code:

• producer

  public class Provider {
        
        
      public static void main(String[] args) throws IOException {
        
        
          // 建立连接工厂
          Connection connection = RabbitMQUtils.getConnection();
          // 建立channel
          Channel channel = connection.createChannel();
          String exchangeName = "topics";
          // 交换机
          channel.exchangeDeclare(exchangeName,"topic");
          // 发消息
          String routekey = "user.save";
          channel.basicPublish(exchangeName,routekey,null,("这是topic动态路由模型发布的基于route" +
                  " key :["+routekey+"]").getBytes());
      }
  }
  

• consumer

  public class Customer1 {
        
        
      public static void main(String[] args) throws IOException {
        
        
          Connection connection = RabbitMQUtils.getConnection();
   
          Channel channel = connection.createChannel();
          //通道绑定交换机
          channel.exchangeDeclare("topics","topic");
          // 创建临时队列
          String queue = channel.queueDeclare().getQueue();
          // 绑定交换机和队列
          channel.queueBind(queue,"topics","user.*");
          // 消费消息
          channel.basicConsume(queue,true,new DefaultConsumer(channel){
        
        
              @Override
              public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
        
        
                  System.out.println("消费者1"+new String(body));
              }
          });
      }
  }
  

  public class Customer2 {
        
        
      public static void main(String[] args) throws IOException {
        
        
          Connection connection = RabbitMQUtils.getConnection();
   
          Channel channel = connection.createChannel();
          //通道绑定交换机
          channel.exchangeDeclare("topics","topic");
          // 创建临时队列
          String queue = channel.queueDeclare().getQueue();
          // 绑定交换机和队列
          channel.queueBind(queue,"topics","user.#");
          // 消费消息
          channel.basicConsume(queue,true,new DefaultConsumer(channel){
        
        
              @Override
              public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
        
        
                  System.out.println("消费者2"+new String(body));
              }
          });
      }
  }
  

RabbitMQ console

management interface

Add user

Create Virtual Hosts and assign permissions