背景
架构部署
32C 128G 1.8T 4主4从架构部署
rocketmq延迟服务生产者生产数据，消费者无法正常消费到数据，查看日志有重大发现，所有的主节点均有这报错
| #vim /data2/logs/rocketmqlogs/storeerror.log |
| 020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 |
发现消费的位点全部错乱，导致消费者无法常消费到数据
解决方法
先将broker节点的流量清除，2表示只写权限，4表示只读权限，6表示读写权限
| #bin/mqadmin updateBrokerConfig -b brokerip:10911 -n nameserver地址:9876 -k brokerPermission -v 4 |
查看broker节点流量是否为0,需要等待broker完全没有流量了才可以进行关闭broker节点操作，以防止丢数据
| #bin/mqadmin clusterList -n nameserver:9876 |
关闭broker节点进程
| #bin/mqshutdown broker |
将delayOffset.json 和delayOffset.json.bak文件拷走 此文件用于记录消费位点的，拷走后，重启应用进程会自动生产
#mv /data1/rocketmq/store/config/delayOffset.json* /data2/ |
将SCHEDULE_TOPIC_XXXX 文件拷走，用于调度任务，拷走后，重启应用进程会自动生产
| #mv /data1/rocketmq/store/consumequeue/SCHEDULE_TOPIC_XXXX/ /data2/ |
将rocketmq的日志文件全部删除
#rm -rf /data2/logs/rocketmqlogs/* |
然后将rocketmq进程启动
#cd /usr/local/rocketmq/bin/ #nohup sh mqbroker -c /usr/local/rocketmq/conf/2m-2s-async/broker-a.properties & |
把权限重新赋值
| #bin/mqadmin updateBrokerConfig -b brokerip:10911 -n nameserver地址:9876 -k brokerPermission -v 6 |
等节点流量恢复重，按上面的操作，将集群所有主节点都操作一遍，就能正常消费数据了，且storeerror.log日志也没在报错了