Message queue selection: rocketmq or rabbitmq

1. At present we use activemq, facing some problems

  • activemq availability based replication leveldb, but activemq from 5. * start, there is no longer any updates leveldb

The LevelDB store has been deprecated and is no longer supported or recommended for use. The recommended store is KahaDB
but the official recommended KahaDB storage solutions, there is no native support for high availability.

  • Our use leveldb process, encounter some weird problem: the queue is consumed, the log does not automatically clean up; only clean up the log after deleting queue; "space usage percentage" does not reduce the log after manual cleaning, activemq record, you need to reboot to correct value; "space usage percentage" Once 100%, the cluster is not available. Official online a lot of people mention a similar issue, but they no longer support the process.
  • leveldb not support deferred messages.
  • After leveldb when a node failover, will restart the "space usage percentage" after the situation has not reduced only to stop the entire cluster, clear all leveldb directory, re-initialized to resolve (the entire cluster is expected to stop taking one minute)

For these reasons, I propose to consider new cohort program: rabbitmq or rocketmq.

2.rocketmq Introduction

  • Nameserver responsible broke automatic discovery (and dubbo similar role in zk); earlier versions nameserver in rocketmq is directly zk do, then develop their own; nameserver no state can be any number of nodes.
  • Producer cluster, n nodes will be appreciated that for an application, the primary role rocketmq Cluster is used after a producer produder hang, other producer may notify continue its outstanding transactions;
  • Consumer cluster, also be interpreted as an application of n nodes, it is to achieve load balancing of the n consumer.
  • A cluster may be n-topic, a topic may have n broken, a broken there may be an n-th master and slave.
  • Producer send a message to the topic, n a broken load balancing, scale-out; after a broken down in the master, master is removed, the message load to the other broken, but the original broken the slave can still provide consumer services.

  • The different Rocketmq queue MQ and the other, not to a specific logical object or activemq producers and consumers in rabbitmq paired, but in order to enhance the concurrency of the message stored in the sharding.
  • Producers send messages to poll each queue, load balancing consumers to consume these queue; consumer queue number for each broken in default node 16, can be customized, the number must exceed the number of consumer, otherwise there will not be extra consumption Case.
  • Rocketmq in each message will add a specific tag, similar to the concept of the queue tag and other mq in for producers and consumers pairing specific message.

Sample Code

public class Producer {
    public static void main(String[] args) throws Exception {
        DefaultMQProducer producer = new DefaultMQProducer("sina-www");
        producer.setNamesrvAddr("10.40.20.200:9876,10.40.20.201:9876");
        producer.start();
        for (int i = 0; i < 100000; i++) {
            Message msg = new Message("topic1", "sina.www.abc", ("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET));
            SendResult sendResult = producer.send(msg);
            System.out.printf("%s%n", sendResult);
        }
        producer.shutdown();
    }
}

public class Consumer {
    public static void main(String[] args) throws InterruptedException, MQClientException {
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("sina-www");
        consumer.setNamesrvAddr("10.40.20.200:9876,10.40.20.201:9876");
        consumer.subscribe("topic1", "sina.www.abc");
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,ConsumeConcurrentlyContext context) {
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();
    }
}

3.rabbitmq Introduction

  • Exchange duties similar switch that provides data exchange, will distribute data to a specific queue.
  • Binding: binding relationship between the exchange and the queue; different types of exchange, plus routekey binging configuration, you can achieve any combination of exchange and distribution rules of the queue.
  • Each cluster node has a same binding and exchange information.
  • Each exchange must be sent to the master Queue message, regardless of whether the master exchange in a node; Master queue and the slave queue automatic synchronization, queue may specify the number of copies, may be less than the number of node; the Node down, the master node copies of other nodes corresponding to the queue will be upgraded to master.
     

The sample program

public class Producer {
    private static String queueName = "sina.goods.updatestock";
    public static void main(String[] args) throws Exception{
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("10.40.20.203");
        factory.setUsername("guest");
        factory.setPassword("guest");
        factory.setVirtualHost("/");
        factory.setPort(5672);
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();
        channel.queueDeclare(queueName, true, false, false, null);
        channel.queueBind(queueName,"lbexchange", queueName);
        for (int i = 0; i < 100000; i++) {
            String message = "hello world + " + i;
            channel.basicPublish("lbexchange", queueName, MessageProperties.PERSISTENT_TEXT_PLAIN, message.getBytes());
            System.out.println(" [x] Sent '" + i + "' ");
        }
        channel.close();
        connection.close();
    }
}

public class Consumer {
    private static String queueName = "sina.goods.updatestock";
    public static void main(String[] args) throws Exception {
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("10.40.20.203");
        factory.setUsername("guest");
        factory.setPassword("guest");
        factory.setVirtualHost("/");
        factory.setPort(5672);
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();
        channel.queueDeclare(queueName, true, false, false, null);
        QueueingConsumer consumer = new QueueingConsumer(channel);
        int prefetchCount = 1;
        channel.basicQos(prefetchCount);
        channel.basicConsume(queueName, true, consumer); 
        while (true) {
            QueueingConsumer.Delivery delivery = consumer.nextDelivery();
            String message = new String(delivery.getBody());
        }
    }
}

4.rabbitmq performance test

因为rocketmq的性能肯定比rabbitmq强,因此只测试了rabbitmq的性能。
测试场景:10个生产者,10个消费者,消费者自动ACK,2节点集群,服务器配置8C8G
压测工具:github.com/rabbitmq/rabbitmq-perf-test
以下分别测试消息大小为10字节和1K、直连主节点和直连从节点,一共4种组合。

消息大小10字节,直连主节点
Cpu 负载:主65%,从25%

消息大小1K,直连主节点
Cpu 负载:主65%,从25%

消息大小10字节,直连从节点
Cpu 负载:主35%,从40%

消息大小1K,直连从节点
Cpu 负载:主30%,从35%

消息积压的表现
关闭消费者,让消息积压,消息大小1K

因为内存限制,消息积压到一定程度要page out到磁盘,因此生产者发送消息会有明显的抖动(page out时会block)。

5.使用总结

Rocketmq

  1. Rocketmq为高并发海量数据设计,架构很先进,支持容量横向扩展。
  2. Rocketmq没有我们传统意义上的queue对象,而是通过tag实现消息的分类。消息的负载数据(enqueue\dequeue\pending等)无法精确到tag维度,只能到broken或consumer group。
  3. 控制台没有包含到原生项目,而是在rocketmq-externals中;控制台更新缓慢,2017年6月第一版后再无新的版本;控制台做得很差
  4. 没有完全遵守AMQP,只支持java客户端;官方文档简陋。

Rabbitmq

  1. 一个2节点的集群能保守提供6-7K/S的请求量,性能满足要求(目前我司还没有单个应用队列吞吐超过该数量的请求);
  2. 准守AMQP 协议,支持多种客户端,如:Python、Ruby、.NET、Java、C、PHP等;
  3. 控制台很友好,很强大,能看每个queue最近24小时的生产、消费、积压走势图
     
发布了24 篇原创文章 · 获赞 25 · 访问量 2万+

Guess you like

Origin blog.csdn.net/sdmei/article/details/90646445