Remember an erlang language bug that caused the rabbitmq queue to have no consumers

　　The erlang version used in the company's development and test environment is 19.0.3, and the rabbitmq version is 3.6.10. It has been used stably for nearly a year under cluster conditions without any problems.

　　In order to maintain consistency with offline, the online production environment uses the same version, but after running for a few months, problems occur. The phenomenon is as follows:

　　Within a few days, three queues had problems with no consumers. Check out rabbitmq's log.

operation queue.declare caused a channel exception not_found: failed to perform operation on queue '出现问题的队列' in vhost '/' due to timeout

　　According to https://bugzilla.redhat.com/show_bug.cgi?id=1418668, this bug has been fixed in 3.6.3, but we still have problems.

　　At this time, through the management page, the queue cannot be deleted, and the error is similar, the queue.delete timeout. Based on feedback from https://github.com/rabbitmq/rabbitmq-server/issues/1333

by command

rabbitmqctl eval 'rabbit_amqqueue:internal_delete({resource,<< "where vhost " >>,queue,<< " queue in question " >>}).'

After closing the application and deleting the queue, it is possible to reply.

Finally, keep digging into the error log

     {gen_server2,call,[<0.26274.8>,{init,new},infinity]}},
    [{gen_server2,call,3,[{file,"src/gen_server2.erl"},{line,327}]},
     {rabbit_channel,handle_method,3,
         [{file,"src/rabbit_channel.erl"},{line,1335}]},
     {rabbit_channel,handle_cast,2,
         [{file,"src/rabbit_channel.erl"},{line,459}]},
     {gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1048}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}

=ERROR REPORT==== 2-May-2018::20:26:08 === 
Restarting crashed queue 'problem queue' in vhost '/'.

So far, it is suspected that there is a bug in the erlang language.

Check the rabbitmq official website.

http://www.rabbitmq.com/which-erlang.html, this page introduces the version of erlang required by rabbitmq, and prompts the bugs to fix

These two solved problems are obviously our problems. So erlang and rabbitmq have been upgraded.

In order to ensure smooth upgrade without downtime, we only upgrade erlang to 19.3.6.8 and rabbitmq to 3.6.14. Otherwise, you need to stop the cluster

The specific upgrade is very simple, stop the rabbitmq-server service, and then uninstall erlang. At this time, rabbitmq will also be uninstalled. Then install the new erlang and rabbitmq, after starting the service, it is automatically in the cluster