RocketMQ之路由管理

RocketMQ是一个分布式集群系统，每个节点之间都会进行通信，其中的通信逻辑使我们必须理清楚的。涉及到的组件包括：NameServer、Broker、Producer、Consumer，这里以Cluster结构为模型来分析各个节点之间的通信流程。路由管理是RocketMQ的核心功能之一，涵盖了订阅管理，连接管理，负载均衡等一系列功能，代码分布在NameServer、Broker、Producer和Consumer等各组件实现中。RocketMQ有一个特色功能：支持灵活的集群工作方式。Broker、Producer和Consumer都能够在运行时动态扩容或缩容，这都要依赖于强大的路由管理模块。RocketMQ的每个组件都保存了一份各自通信必须的路由信息数据，下面我们结合代码看看：

1、NameServer

public class RouteInfoManager {
    private final HashMap<String/* topic */, List<QueueData>> topicQueueTable;
    private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
    private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
    private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
    private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;
    ......
}

2、Broker

Broker每30S向一个或多个NameServer注册自身信息，

public class ProducerManager {
    ......
    private final HashMap<String /* group name */, HashMap<Channel, ClientChannelInfo>> groupChannelTable =
        new HashMap<String, HashMap<Channel, ClientChannelInfo>>();
}

public class ConsumerManager {
    ......
    private final ConcurrentMap<String/* Group */, ConsumerGroupInfo> consumerTable =
        new ConcurrentHashMap<String, ConsumerGroupInfo>(1024);
}

public class SubscriptionGroupManager extends ConfigManager {
    ......
    private final ConcurrentMap<String, SubscriptionGroupConfig> subscriptionGroupTable =
        new ConcurrentHashMap<String, SubscriptionGroupConfig>(1024);
}

public class TopicConfigManager extends ConfigManager {
    ......
    private final ConcurrentMap<String, TopicConfig> topicConfigTable =
        new ConcurrentHashMap<String, TopicConfig>(1024);
}

3、Consumer

public class MQClientInstance {
    ......
    private final ConcurrentMap<String/* group */, MQConsumerInner> consumerTable = new ConcurrentHashMap<String, MQConsumerInner>();
    private final ConcurrentMap<String/* Topic */, TopicRouteData> topicRouteTable = new ConcurrentHashMap<String, TopicRouteData>();
    private final ConcurrentMap<String/* Broker Name */, HashMap<Long/* brokerId */, String/* address */>> brokerAddrTable = new ConcurrentHashMap<String, HashMap<Long, String>>();
}
public abstract class RebalanceImpl {
    ......
    protected final ConcurrentMap<String/* topic */, Set<MessageQueue>> topicSubscribeInfoTable = new ConcurrentHashMap<String, Set<MessageQueue>>();
    protected final ConcurrentMap<String /* topic */, SubscriptionData> subscriptionInner = new ConcurrentHashMap<String, SubscriptionData>();
    protected final ConcurrentMap<MessageQueue, ProcessQueue> processQueueTable = new ConcurrentHashMap<MessageQueue, ProcessQueue>(64);
}

4、Producer

Producer每隔两分钟会向NameServer请求所有NameServer的Addrs，每隔10秒会向NameServer请求TopicRouteData信息，然后解析生成TopicPublishInfo，重要的结构是TopicPublishInfo结构，里面包含了List<MessageQueue> messageQueueList和TopicRouteData。可以看到messageQueueList是一个List，说明他是针对某一个Topic的，一个MessageQueue包含了topic，brokerName，queueId三个信息。另外TopicPublishInfo还包含了一个TopicRouteData类型的对象，TopicRouteData包含的信息有List<QueueData> queueDatas和List<BrokerData> brokerDatas，这个是个列表，一个QueueData就代表了一个Broker上这个队列的信息，比如读写队列个数，权限，brokerName等，因为部署可能为Cluster，所以需要是列表结构；brokerDatas也是个列表，一个BrokerData就是这个Topic相关的某一个Broker的信息，主要包括brokerName，broker所属的clusterName，broker主从信息HashMap<Long/* brokerId */, String/* broker address */> brokerAddrs，List<BrokerData>就将cluster中这个Topic相关的所有Broker信息汇集了。结合一张图来记忆：

public class MQClientInstance {
    ......
    private final ConcurrentMap<String/* group */, MQProducerInner> producerTable = new ConcurrentHashMap<String, MQProducerInner>();
    // 消息路由表，TopicRouteData里存了Topic对应的Broker类别以及
    private final ConcurrentMap<String/* Topic */, TopicRouteData> topicRouteTable = new ConcurrentHashMap<String, TopicRouteData>();
    private final ConcurrentMap<String/* Broker Name */, HashMap<Long/* brokerId */, String/* address */>> brokerAddrTable = new ConcurrentHashMap<String, HashMap<Long, String>>();
}

public class DefaultMQProducerImpl implements MQProducerInner {
    ......
    // 包含了某个Topic在集群中所有broker数据，以及所有queue和broker的逻辑关系
    private final ConcurrentMap<String/* topic */, TopicPublishInfo> topicPublishInfoTable = new ConcurrentHashMap<String, TopicPublishInfo>();
}

大致了解了每个组件包含的路由信息后，我们需要看看这几个组件之间的拓扑结构，下面这个图中的连线大家务必理解清楚。

主要的通信关系有：
1、每个Broker(包括Master和Slave)每30秒向每个NameServer发送心跳消息(REGISTER_BROKER)，上报Broker信息。
2、每个Slave每60秒向对应Master发送GET_ALL_TOPIC_CONFIG和GET_ALL_SUBSCRIPTIONGROUP_CONFIG消息请求Topic和订阅关系数据。
3、每个Producer和任意一个NameServer建立长连接，并每30秒发送GET_ROUTEINTO_BY_TOPIC消息给NameServer请求相关Topic的路由信息到本地。
4、每个Producer会向每个Master每30秒发送HEART_BEAT上报Producer状况。
5、每个Consumer和任意一个NameServer建立长连接，每30秒发送GET_ROUTEINTO_BY_TOPIC消息给NameServer请求相关Topic的路由信息到本地。
6、每个Consumer会和自身Topic相关的Master和Slave都建立长连接并发送HEART_BEAT上报Consumer状况。

接下来就是Topic的创建

RocketMQ之路由管理

猜你喜欢