RabbitMQ learning summary

About what RabbitMQ is and its concept, if you don't know it, you can check the following blogs.

https://blog.csdn.net/whoamiyang/article/details/54954780

https://www.cnblogs.com/frankyou/p/5283539.html

https://blog.csdn.net/mx472756841/article/details/50815895

Official website introduction: http://www.rabbitmq.com/getstarted.html

The github source code of this article: http://www.cnblogs.com/bluesummer/p/8992225.html

Because I didn't know the switch and AMQP protocol before, I came up to study RabbitMQ. Many concepts are a bit confusing, so it is recommended that you have a basic understanding of some concepts before learning RabbitMQ

Installation and configuration:

下载Erlang：http://www.erlang.org/downloads
下载rabbitmq：http://www.rabbitmq.com/install-windows.html

Environment variable configuration:

新增：ERLANG_HOME= C:\Program Files\erlx.x
新增：RABBITMQ_SERVER=C:\Program Files\RabbitMQ Server\rabbitmq_server-x.x.x
Path中新增：%ERLANG_HOME%\bin;%RABBITMQ_SERVER%\sbin;

Service related commands

rabbitmq-plugins enable rabbitmq_management //Open the management plugin
rabbitmq-service.bat start //Start the service
rabbitmq-service.bat stop //Close the service
rabbitmqctl list_queues //View tasks

rabbitmqctl list_queuesNote that if an error of " unable to perform an operation on node" is reported when executing the command . . . . , you can copy the C:\Users\username\.erlang.cookie.erlang.cookie file to C:\Windows\System32\config\systemprofile\.erlang.cookie to replace, and then restart the service

So far the RabbitMQ service has been installed

Backstage management

After opening the management plugin, we restart the rabbitmq service, open the http://localhost:15672/ background management interface, and the
username and password are both guest

The guest account can only log in through localhost in the latest version. If you want to log in through ip, you need to set the configuration file:

Find the rabbit.app file under /rabbitmq_server-xxx/ebin: Find: loopback_users Delete the <<”guest”>> in it.

The deleted content is: {loopback_users, []} , then restart the service

Regarding the operation of user password management, we can set it in the management page

Default port:

client port communication port 5672
Management port 15672
Internal communication port 25672 between servers
erlang found port: 4369

If you want to modify the default port, you can modify the etc/rabbitmq.config file in the installation directory. There is a default example, and you can change it.

send messages

Let's build an application first, it is recommended to create a winform or wpf program, the console is not very useful here.
The nuget package is referenced in the project:RabbitMQ.Client

Next we write a code to send and receive messages:

  public void SendMsg(string message)
    {
        //这里的端口及用户名都是默认的，可以直接设置一个hostname=“localhost”其他的不用配置
        var factory = new ConnectionFactory() { HostName = "192.168.1.15",Port=5672,UserName= "guest",Password= "guest" };
        //创建一个连接，连接到服务器：
        using (var connection = factory.CreateConnection())
        {
            using (var channel = connection.CreateModel())
            {
                //创建一个名称为hello的消息队列
                //durable:队列持久化，为了防止RabbitMQ在退出或者crash等异常情况下数据不会丢失，可以设置durable为true
                //exclusive：排他队列,只对首次声明它的连接（Connection）可见，不允许其他连接访问，在连接断开的时候自动删除，无论是否设置了持久化
                //autoDelete：自动删除，如果该队列已经没有消费者时，该队列会被自动删除。这种队列适用于临时队列。
                channel.QueueDeclare(queue: "hello", durable: false, exclusive: false, autoDelete: true, arguments: null);
                //channel.BasicConsume("hello", autoAck: true);
               
                var props = channel.CreateBasicProperties();
                //消息持久化，若启用durable则该属性启用
                props.Persistent = true;
                //封装消息主体
                var body = Encoding.UTF8.GetBytes(message);

                channel.BasicPublish(exchange: "", routingKey: "hello", basicProperties: props, body: body);
                Console.WriteLine(" 发送消息{0}", message);
            }
        }
    }
    
     public class Consumer : IDisposable
    {
        public static int _number;
        private static ConnectionFactory factory;
        private static IConnection connection;
        static Receive()
        {
            factory = new ConnectionFactory() { HostName = "localhost" };
        }
        public Receive()
        {
            _number++;
        }
        public void ReceiveMsg(Action<string> callback)
        {
            if(connection==null||!connection.IsOpen)
                connection = factory.CreateConnection();
            IModel _channel = connection.CreateModel();
            _channel.QueueDeclare(queue: "hello", durable: false, exclusive: false, autoDelete: true, arguments: null);
            _channel.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false);
            // 创建事件驱动的消费者
            var consumer = new EventingBasicConsumer(_channel); 
            consumer.Received += (model, ea) =>
            {
                var body = ea.Body;
                var message = Encoding.UTF8.GetString(body);
                callback($"number:{_number}.message:{message}");
                //模拟消息处理需要两秒
                Thread.Sleep(2000);
                //显示发送ack确认接收并处理完成消息，只有在前面进行启用显示发送ack机制后才奏效。
                _channel.BasicAck(ea.DeliveryTag, false);
            };
            //指定消费队列,autoAct是否自动确认
            string result = _channel.BasicConsume(queue: "hello", autoAck: false, consumer: consumer);

            //设置后当所有的channel都关闭了连接会自动关闭
            //connection.AutoClose = true;
        }
        public void Dispose()
        {
            if (connection != null && connection.IsOpen)
                connection.Dispose();
        }
    }

The sender and consumer of a very simple message queue above, explain the basic process:

The publisher calls the send function to first create a connection to the server, then use the connection to create a channel, then use the channel to declare a hello queue, and finally send a message to the default switch. (exchange: "") The empty string is the default exchange, the message route is hello , and the default exchange is the direct type, which exactly matches the name of the queue according to the route name. All queues are bound to the default exchange, and the route name is the name of the queue. So the default exchange sends the message to the queue with the name hello . Then, the ReceiveMsg function is called in the Consumer to get the message from the hello queue. After getting the message, the act function is called to notify the broker that the message has been successfully consumed, and the broker deletes the message, as shown in the following figure

There are some examples on the Internet that use QueueingBasicConsumer to create consumers. I found that it is outdated in the new version, because it is easy to cause a series of problems such as memory overflow and performance degradation. Briefly talk about the processing flow of QueueingBasicConsumer, it receives messages After that, the message will be stuffed into a Queue queue, and then the user will cycle the queue to process the message, but if you process a message very slowly, and the message is sent very quickly, it will cause the message stored in the queue to become more and more More and more, eventually causing memory overflow. So now it is recommended to use EventingBasicConsumer or inherit DefaultBasicConsumer to create consumers, event-driven will not have this problem

The above code needs to pay attention to the following points:

If you want to specify an ip connection through the guest account, you need to modify the loopback_users configuration

We call QueueDeclarethe function to declare a queue. If the queue persistence is set, the queue will still exist even if the service is restarted. If it is not persistent, even if the messages are all consumed, as long as the service is not restarted, the queue will still exist. RabbitMQ does not allow you to redefine an existing queue with different parameters , so either delete the queue or rename a queue, delete the queue can be deleted through the management interface or call the QueueDeletefunction.

If the queue exists, it is enough to declare it once. If the same queue is declared multiple times, there will be no exception, but if the consumer binds a queue that does not exist, an exception will occur: **_channel.BasicConsume**, So the habit is to declare the queue that needs to be monitored in the Worker first

Exclusive queue: It probably means that after declaring an exclusive queue by connecting connectionA, you can only access the queue by connecting to connectionA in the future . Once other connections are accessed, the error of the queue being locked will be reported. This really unexpected application scenario

Queue persistence means that the queue is still there after restarting the service. If you want the messages in the queue to still exist, you need to set message persistence at the same time, but it makes no sense if you only set message persistence without queue persistence. But this does not necessarily guarantee that the message will not be lost. First of all, there must be a message confirmation mechanism to ensure that the message must be correctly consumed. The main problem is that it takes a certain amount of time for the message to be written to the disk. If the service receives the message and hangs up before writing it to the disk, then the message is lost. For this, you can check the articles related to the RabbitMQ cluster.

The messages sent by default require confirmation by the consumer. You can automatically confirm the message by setting autoAct to true, or you can call the BasicAck function to confirm. In short, if the message needs to be confirmed, it must be confirmed after the message processing is completed, otherwise when the consumer connection is closed Unacknowledged messages after that are quickly bounced back.

The consumer I defined above originally wanted to instantiate Receive multiple times to simulate multiple consumers. However, it turned out to be not easy to use. If you want to simulate multiple consumers, you still need to open multiple programs.

The listener of EventingBasicConsumer will create a foreground thread that is running all the time, so in winform, if you close the program, you need to dispose the thread occupied by the connection

round robin scheduling

Round-robin scheduling is to run multiple consumers at the same time. When the number of tasks is large, RabbitMQ will distribute messages to different consumers (Workers) to reduce the pressure. To make RabbitMQ distribute tasks fairly, you need to use the following code in the worker to set the maximum number of unacknowledged messages for a worker

channel.BasicQos(0, 1, false);

Parameter 1 means that this worker will only process one message at the same time. If the current message is not processed (no act), rabbitmq will send the remaining tasks to other workers. If all workers are busy, they need to queue up. span

bind

In the above example, we use the default exchange to send messages. We can exchangeuse the specified exchange by assigning a value, by QueueBindbinding the exchange to the queue

_channel.QueueBind("log1", "logs", "info");

The code to declare a switch is as follows

_channel.ExchangeDeclare("logs", ExchangeType.Direct, false, false);

We bind the queue log1 to the switch: on logs , the route is info , the type of the switch is Direct, and Direct represents the exact match of the route. Now we send a message to the logs switch, the route is info , and the queue log1 will receive the message span

channel.BasicPublish(exchange: "logs", routingKey: "info", basicProperties: props, body: body);

The relationship between queues and switches is many-to-many. There are three commonly used types of switches: Direct, Fanout, Topic, Headers

Direct: requires exact routing key match

Fanout: ignore routing keys and send messages to all queues bound to the switch

Topic: Fuzzy matching, setting routing keys by combining letters with symbols "*" and "#"

Headers: The header type is used less. It also ignores the routing key, but matches the headers of the switch. The headers are the hashtable of key-value pairs. To match the headers set on both sides of the publisher and the consumer, you need to specify whether the matching method is all or any. , the specific code can be found on github

The following shows a relevant code that uses a direct type switch

public class LogDirectPub
{
    public void SendMsg(string message)
    {
        var factory = new ConnectionFactory() { HostName = "192.168.1.15", Port = 5672, UserName = "guest", Password = "guest" };
        //创建一个连接，连接到服务器：
        using (var connection = factory.CreateConnection())
        {
            using (var channel = connection.CreateModel())
            {
                var props = channel.CreateBasicProperties();
                var body = Encoding.UTF8.GetBytes(message);
                channel.BasicPublish(exchange: "logs", routingKey: "info", basicProperties: props, body: body);
                channel.BasicPublish(exchange: "logs", routingKey: "error", basicProperties: props, body: body);
                Console.WriteLine("发送消息{0}", message);
            }
        }
    }
}


public class LogDirectConsumer : IDisposable
{
    private static ConnectionFactory factory;
    private static IConnection connection;
    static LogDirectConsumer()
    {
        factory = new ConnectionFactory() { HostName = "localhost" };
    }
    public void ReceiveMsg(Action<string> callback)
    {
        if (connection == null || !connection.IsOpen)
            connection = factory.CreateConnection();
        IModel _channel = connection.CreateModel();
        _channel.ExchangeDeclare("logs", ExchangeType.Direct, false, false);
        _channel.QueueDeclare(queue: "log1", durable: false, exclusive: false, autoDelete: false, arguments: null);
        _channel.QueueBind("log1", "logs", "info");
        _channel.QueueBind("log1", "logs", "error");
        _channel.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false);
        var consumer = new EventingBasicConsumer(_channel);
        consumer.Received += (model, ea) =>
        {
            var body = ea.Body;
            var message = Encoding.UTF8.GetString(body);
            callback($"log1Write.message:{ea.RoutingKey}:{message}");
            //模拟消息处理需要两秒
            Thread.Sleep(2000);
            _channel.BasicAck(ea.DeliveryTag, false);
        };
        string result = _channel.BasicConsume(queue: "log1", autoAck: false, consumer: consumer);
    }
    public void Dispose()
    {
        if (connection != null && connection.IsOpen)
            connection.Dispose();
    }
}

RabbitMQ Management HTTP API

RabbitMQ has its own set of http/api, the address is http://192.168.1.15:15672/api , you can query all the information configuration you want to check, through these apis, we can realize the monitoring and management of RabbitMQ by ourselves, which is a headache to read in English , here is a Chinese translation document: http://www.blogjava.net/qbna350816/archive/2016/08/13/431575.html

Here is a simple example to get all queues:

    string username = "guest";
    string password = "guest";
    string queuesUrl = "http://localhost:15672/api/queues";
    /// <summary>
    /// 查询所有队列
    /// </summary>
    /// <returns></returns>
    public string GetAllQuenes()
    {
        string jsonContent = GetApiResult(queuesUrl).Result;
        List<QueueModel> queues = JsonConvert.DeserializeObject<List<QueueModel>>(jsonContent);
        return JsonConvert.SerializeObject(queues);
    }

    private async Task<string> GetApiResult(string Url)
    {
        var client = new HttpClient();
        var passByte = Encoding.UTF8.GetBytes(string.Format("{0}:{1}", username, password));
        client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", Convert.ToBase64String(passByte));
        using (HttpResponseMessage response = await client.GetAsync(Url).ConfigureAwait(false))
        {
            string result = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
            return result;
        }
    }

Custom Consumer

I said before that using QueueingBasicConsumer will cause performance problems, but eventconsumer cannot block threads. It is not convenient to use some functions that need to block threads. At this time, we can customize a Consumer to inherit DefaultBasicConsumer, and only need to implement the HandleBasicDeliverfunctions in it. Now, here is a consumer I defined to implement the following Rpc client

public class QueueingConsumer : DefaultBasicConsumer
{
    private IModel _channel;
    private BasicDeliverEventArgs args = new BasicDeliverEventArgs();

    private AutoResetEvent argResetEvent = new AutoResetEvent(false);
    public QueueingConsumer(IModel channel)
    {
        _channel = channel;
    }
    public override void HandleBasicDeliver(string consumerTag,
       ulong deliveryTag,
       bool redelivered,
       string exchange,
       string routingKey,
       IBasicProperties properties,
       byte[] body)
    {
        args = new BasicDeliverEventArgs
        {
            ConsumerTag = consumerTag,
            DeliveryTag = deliveryTag,
            Redelivered = redelivered,
            Exchange = exchange,
            RoutingKey = routingKey,
            BasicProperties = properties,
            Body = body
        };
        argResetEvent.Set();
    }

    public void GetResult(Action<BasicDeliverEventArgs> callback)
    {
        argResetEvent.WaitOne();
        callback(args);
    }

}

RPC implementation

Needless to say what Rpc is, I know it is a remote procedure call anyway. Using RabbitMQ to implement Rpc, the official website has a simple example, but I personally feel that RabbitMQ is not very suitable for Rpc. However, it is quite good to use this example as a learning outcome practice for RabbitMQ. Please see the code below:

public class RpcPub
{
    public async Task<string> SendMsg(string message)
    {
        ConnectionFactory factory = RabbitMQHelper.ConFactory;
        //创建一个连接，连接到服务器：
        using (var connection = factory.CreateConnection())
        {
            using (var channel = connection.CreateModel())
            {
                //定义一个临时的队列，用来接收返回的消息
                string replyQueueName = channel.QueueDeclare().QueueName;
                var consumer = new QueueingConsumer(channel);
                //监听该临时队列，自动act消息
                channel.BasicConsume(queue: replyQueueName, autoAck: true, consumer: consumer);


                string corrId = Guid.NewGuid().ToString();
                var props = channel.CreateBasicProperties();
                //定义ReplyTo让服务端知道返回消息给哪个路由
                props.ReplyTo = replyQueueName;
                //定义CorrelationId作为消息的唯一关联ID
                props.CorrelationId = corrId;

                var messageBytes = Encoding.UTF8.GetBytes(message);
                channel.BasicPublish(exchange: "", routingKey: "rpc_queue", basicProperties: props, body: messageBytes);
                Task<string> result = new Task<string>(() =>
                {
                    while (true)
                    {
                        string replystr = string.Empty;
                        consumer.GetResult((args) =>
                        {
                            if (args.BasicProperties.CorrelationId == corrId)
                            {
                                replystr = Encoding.UTF8.GetString(args.Body);
                            }
                        });
                        if (replystr != string.Empty)
                            return replystr;
                    }
                });
                result.Start();
                return await result;
            }
        }
    }
}


public class RpcConsumer : IDisposable
{

    private ConnectionFactory factory = RabbitMQHelper.ConFactory;
    private IConnection connection;


    public void ReceiveMsg(Action<string> callback)
    {
        if (connection == null || !connection.IsOpen)
            connection = factory.CreateConnection();
        IModel channel = connection.CreateModel();

        channel.QueueDeclare(queue: "rpc_queue", durable: false, exclusive: false, autoDelete: false, arguments: null);
        //channel.BasicQos(0, 1, false);
        var consumer = new EventingBasicConsumer(channel);

        consumer.Received += (model, arg) =>
        {
            var props = arg.BasicProperties;
            var replyProps = channel.CreateBasicProperties();
            replyProps.CorrelationId = props.CorrelationId;
            callback($"接收到消息：{Encoding.UTF8.GetString(arg.Body)}");
            var responseBytes = Encoding.UTF8.GetBytes($"成功接收你的消息：{ Encoding.UTF8.GetString(arg.Body)}");
            channel.BasicPublish(exchange: "", routingKey: props.ReplyTo, basicProperties: replyProps, body: responseBytes);
            channel.BasicAck(deliveryTag: arg.DeliveryTag, multiple: false);
        };
        channel.BasicConsume(queue: "rpc_queue", autoAck: false, consumer: consumer);

    }
    public void Dispose()
    {
        if (connection != null && connection.IsOpen)
            connection.Dispose();
    }

}

Basic process:

When the client sends a message, create an anonymous callback queue channel.QueueDeclare()and listen to the queue.

The client obtains the name of the anonymous queue and sets 2 properties in the request: replyTo=callback queue name; CorrelationId=unique id associated with the request

The client sends a request to the rpc_queue queue.

The RPC server listens to the request in the rpc_queue queue. When the request arrives, the server will process the message, send the returned result to the queue specified by replyTo, and set one attribute in the request: CorrelationId=requested CorrelationId

The queue monitored by the client receives the message, checks whether the correlationId matches the previously generated one, and returns the result if the match is successful.

There are two reasons why the correlationId should be verified. 1. The message may not be sent by the rpc server. 2. If the rpc service suddenly hangs up at a certain stage, a message that does not contain the correlationId may be sent.