【消息队列】RocketMQ 配置文件broker

 

前言

     broker主要涉及broker 配置文件、broker 启动流程、broker延迟消息、broker消息存储。

一、 broker 配置文件

1.1 broker目录结构

├── abort
├── checkpoint
├── commitlog
│   └── 00000000000000000000
├── config
│   ├── consumerFilter.json
│   ├── consumerOffset.json
│   ├── delayOffset.json
│   ├── subscriptionGroup.json
│   ├── topics.json
├── consumequeue
│   ├── SCHEDULE_TOPIC_XXXX
│   │   ├── 0
│   │   │   └── 00000000000000000000
│   │   ├── 1
│   │   │   └── 00000000000000000000
│   │   └── 2
│   │       └── 00000000000000000000
│   └── TopicTest
│       ├── 0
│       │   └── 00000000000000000000
│       ├── 1
│       │   └── 00000000000000000000
│       ├── 2
│       │   └── 00000000000000000000
│       ├── 3
│       │   └── 00000000000000000000
│       ├── 4
│       │   └── 00000000000000000000
│       ├── 5
│       │   └── 00000000000000000000
│       ├── 6
│       │   └── 00000000000000000000
│       └── 7
│           └── 00000000000000000000
├── index
│   └── 20200502170505253
└── lock

• commitlog记录broker真正存储的消息。
• config记录mq的各类配置信息。
• consumequeue记录topic的各个队列的消费情况。
• index记录mq的索引信息，指向commitlog。

 1.2 topic.json

• 记录该broker上各topic信息，包括queue信息、读写权限。

{
    "dataVersion":{
        "counter":8,
        "timestamp":1588575388560
    },
    "topicConfigTable":{
        "TopicTest":{
            "order":false,
            "perm":6,
            "readQueueNums":8,
            "topicFilterType":"SINGLE_TAG",
            "topicName":"TopicTest",
            "topicSysFlag":0,
            "writeQueueNums":8
        },
        "SELF_TEST_TOPIC":{
            "order":false,
            "perm":6,
            "readQueueNums":1,
            "topicFilterType":"SINGLE_TAG",
            "topicName":"SELF_TEST_TOPIC",
            "topicSysFlag":0,
            "writeQueueNums":1
        },
        "%RETRY%consumer_group_test":{
            "order":false,
            "perm":6,
            "readQueueNums":1,
            "topicFilterType":"SINGLE_TAG",
            "topicName":"%RETRY%consumer_group_test",
            "topicSysFlag":0,
            "writeQueueNums":1
        },
        "DefaultCluster":{
            "order":false,
            "perm":7,
            "readQueueNums":16,
            "topicFilterType":"SINGLE_TAG",
            "topicName":"DefaultCluster",
            "topicSysFlag":0,
            "writeQueueNums":16
        },
        "%RETRY%order_consumer":{
            "order":false,
            "perm":6,
            "readQueueNums":1,
            "topicFilterType":"SINGLE_TAG",
            "topicName":"%RETRY%order_consumer",
            "topicSysFlag":0,
            "writeQueueNums":1
        },
        "RMQ_SYS_TRANS_HALF_TOPIC":{
            "order":false,
            "perm":6,
            "readQueueNums":1,
            "topicFilterType":"SINGLE_TAG",
            "topicName":"RMQ_SYS_TRANS_HALF_TOPIC",
            "topicSysFlag":0,
            "writeQueueNums":1
        },
        "broker-a":{
            "order":false,
            "perm":7,
            "readQueueNums":1,
            "topicFilterType":"SINGLE_TAG",
            "topicName":"broker-a",
            "topicSysFlag":0,
            "writeQueueNums":1
        },
        "TBW102":{
            "order":false,
            "perm":7,
            "readQueueNums":8,
            "topicFilterType":"SINGLE_TAG",
            "topicName":"TBW102",
            "topicSysFlag":0,
            "writeQueueNums":8
        },
        "BenchmarkTest":{
            "order":false,
            "perm":6,
            "readQueueNums":1024,
            "topicFilterType":"SINGLE_TAG",
            "topicName":"BenchmarkTest",
            "topicSysFlag":0,
            "writeQueueNums":1024
        },
        "OFFSET_MOVED_EVENT":{
            "order":false,
            "perm":6,
            "readQueueNums":1,
            "topicFilterType":"SINGLE_TAG",
            "topicName":"OFFSET_MOVED_EVENT",
            "topicSysFlag":0,
            "writeQueueNums":1
        }
    }
}

1.3 subscriptionGroup.json

• 记录该broker上各类订阅关系。

{
    "dataVersion":{
        "counter":1,
        "timestamp":1588575546295
    },
    "subscriptionGroupTable":{
        "SELF_TEST_C_GROUP":{
            "brokerId":0,
            "consumeBroadcastEnable":true,
            "consumeEnable":true,
            "consumeFromMinEnable":true,
            "groupName":"SELF_TEST_C_GROUP",
            "notifyConsumerIdsChangedEnable":true,
            "retryMaxTimes":16,
            "retryQueueNums":1,
            "whichBrokerWhenConsumeSlowly":1
        },
        "consumer_group_test":{
            "brokerId":0,
            "consumeBroadcastEnable":true,
            "consumeEnable":true,
            "consumeFromMinEnable":true,
            "groupName":"consumer_group_test",
            "notifyConsumerIdsChangedEnable":true,
            "retryMaxTimes":16,
            "retryQueueNums":1,
            "whichBrokerWhenConsumeSlowly":1
        },
        "CID_ONSAPI_OWNER":{
            "brokerId":0,
            "consumeBroadcastEnable":true,
            "consumeEnable":true,
            "consumeFromMinEnable":true,
            "groupName":"CID_ONSAPI_OWNER",
            "notifyConsumerIdsChangedEnable":true,
            "retryMaxTimes":16,
            "retryQueueNums":1,
            "whichBrokerWhenConsumeSlowly":1
        },
        "CID_ONSAPI_PERMISSION":{
            "brokerId":0,
            "consumeBroadcastEnable":true,
            "consumeEnable":true,
            "consumeFromMinEnable":true,
            "groupName":"CID_ONSAPI_PERMISSION",
            "notifyConsumerIdsChangedEnable":true,
            "retryMaxTimes":16,
            "retryQueueNums":1,
            "whichBrokerWhenConsumeSlowly":1
        },
        "order_consumer":{
            "brokerId":0,
            "consumeBroadcastEnable":true,
            "consumeEnable":true,
            "consumeFromMinEnable":true,
            "groupName":"order_consumer",
            "notifyConsumerIdsChangedEnable":true,
            "retryMaxTimes":16,
            "retryQueueNums":1,
            "whichBrokerWhenConsumeSlowly":1
        },
        "TOOLS_CONSUMER":{
            "brokerId":0,
            "consumeBroadcastEnable":true,
            "consumeEnable":true,
            "consumeFromMinEnable":true,
            "groupName":"TOOLS_CONSUMER",
            "notifyConsumerIdsChangedEnable":true,
            "retryMaxTimes":16,
            "retryQueueNums":1,
            "whichBrokerWhenConsumeSlowly":1
        },
        "CID_ONS-HTTP-PROXY":{
            "brokerId":0,
            "consumeBroadcastEnable":true,
            "consumeEnable":true,
            "consumeFromMinEnable":true,
            "groupName":"CID_ONS-HTTP-PROXY",
            "notifyConsumerIdsChangedEnable":true,
            "retryMaxTimes":16,
            "retryQueueNums":1,
            "whichBrokerWhenConsumeSlowly":1
        },
        "FILTERSRV_CONSUMER":{
            "brokerId":0,
            "consumeBroadcastEnable":true,
            "consumeEnable":true,
            "consumeFromMinEnable":true,
            "groupName":"FILTERSRV_CONSUMER",
            "notifyConsumerIdsChangedEnable":true,
            "retryMaxTimes":16,
            "retryQueueNums":1,
            "whichBrokerWhenConsumeSlowly":1
        },
        "CID_ONSAPI_PULL":{
            "brokerId":0,
            "consumeBroadcastEnable":true,
            "consumeEnable":true,
            "consumeFromMinEnable":true,
            "groupName":"CID_ONSAPI_PULL",
            "notifyConsumerIdsChangedEnable":true,
            "retryMaxTimes":16,
            "retryQueueNums":1,
            "whichBrokerWhenConsumeSlowly":1
        }
    }
}

1.4 delayOffset.json

• 记录该broker延迟队列的消费位移情况，1:1000中1表示延迟粒度，1000表示位移。

{
    "offsetTable":{1:1000,2:1000,3:1000
    }
}

1.5 consumerOffset.json

• 记录该broker上针对每个topic的每个consumer group的针对每个queue的消费位移。

{
    "offsetTable":{
        "TopicTest@please_rename_unique_group_name_4":{0:125,1:125,2:125,3:125,4:125,5:125,6:125,7:125
        },
        "TopicTest@order_consumer":{0:165,1:165,2:111,3:145,4:145,5:145,6:145,7:145
        },
        "TopicTest@consumer_group_test":{0:540,1:540,2:520,3:520,4:520,5:520,6:520,7:520
        },
        "%RETRY%consumer_group_test@consumer_group_test":{0:0
        },
        "%RETRY%please_rename_unique_group_name_4@please_rename_unique_group_name_4":{0:0
        },
        "%RETRY%order_consumer@order_consumer":{0:0
        }
    }
}