In Milvus's cloud-native architecture, the message queue (Log Broker) has a long way to go. It must not only have streaming data persistence, support TT synchronization, event notification, etc., but also ensure incremental data when working nodes recover from system crashes. of integrity .
In the architecture of Milvus, everything is built around message queues and follows the principle of structured log storage. The role of message queues in Milvus can be compared to the role of WAL (Redo Log) in traditional databases. Before Milvus 2.3, Milvus officially supported RocksMQ (Standalone mode limited, Milvus official MQ system based on RockDB), as well as traditional MQ such as Pulsar and Kafka.
In Milvus Standalone mode, RocksMQ is the simplest MQ deployment solution compared to Pulsar and Kafka. However, because RocksMQ is based on RocksDB, its performance is average in scenarios with large message bodies and massive messages. At the same time, RocksMQ requires rocksdb and frequently calls CGO, which brings additional performance burden. In Milvus 2.3, Milvus Standalone recently introduced a new stand-alone MQ implementation based on NATS, providing users with more MQ choices in different usage scenarios.
This article will introduce how to use the new MQ and compare it with other MQ.
01.
What is NATS
NATS is a distributed system connection technology implemented by GO. It supports cross-system communication modes such as Request-Reply and Publish-Subscribe. It supports data persistence through the underlying JetStream and has built-in RAFT to provide distributed capabilities. If you want to know NATS systematically, you can check the official website: https://nats.io/ .
NATS supports many Features. In Milvus 2.3 standalone mode, Milvus uses the stand-alone version of NATS+JetStream+PubSub mode to provide MQ capabilities. At the same time, Nats-server is Embedding into the Milvus program, and NATSMQ can be implemented without additional NATS deployment.
02.
How to enable NATS
In Milvus 2.3, a new control option is introduced mq.typeas an MQ type. In order to maintain upward compatibility, NATS will not enter the default MQ selection priority and needs to be mq.type=natsmqforced. After the Milvus instance is started, if you see the following log, it means that Milvus uses NATS as MQ.
[INFO] [dependency/factory.go:83] ["try to init mq"] [standalone=true] [mqType=natsmq]
03.
Detailed explanation of NATS configuration items
The current configuration items support the following customized configuration capabilities:
natsmq:
server: # server side configuration for natsmq.
port: 4222 # 4222 by default, Port for nats server listening.
storeDir: /var/lib/milvus/nats # /var/lib/milvus/nats by default, directory to use for JetStream storage of nats.
maxFileStore: 17179869184 # (B) 16GB by default, Maximum size of the 'file' storage.
maxPayload: 8388608 # (B) 8MB by default, Maximum number of bytes in a message payload.
maxPending: 67108864 # (B) 64MB by default, Maximum number of bytes buffered for a connection Applies to client connections.
initializeTimeout: 4000 # (ms) 4s by default, waiting for initialization of natsmq finished.
monitor:
trace: false # false by default, If true enable protocol trace log messages.
debug: false # false by default, If true enable debug log messages.
logTime: true # true by default, If set to false, log without timestamps.
logFile: /tmp/milvus/logs/nats.log # /tmp/milvus/logs/nats.log by default, Log file path relative to .. of milvus binary if use relative path.
logSizeLimit: 536870912 # (B) 512MB by default, Size in bytes after the log file rolls over to a new one.
retention:
maxAge: 4320 # (min) 3 days by default, Maximum age of any message in the P-channel.
maxBytes: # (B) None by default, How many bytes the single P-channel may contain. Removing oldest messages if the P-channel exceeds this size.
maxMsgs: # None by default, How many message the single P-channel may contain. Removing oldest messages if the P-channel exceeds this limit.
server.portserver.port: Since Nats is a CS mode program, it currently does not support the use of socket solutions that do not occupy ports, such as unix sockets. Currently , the port as the NATS Server needs to be specified in Milvus. If a port conflict occurs, Milvus cannot start normally. Fill inserver.port=-1and you can randomly select the port.
storeDir: Used to specify the storage directory of the underlying JetStream persistence mechanism. It is recommended to mount this directory on a high-performance SSD to improve the read and write throughput of Milvus. If Milvus fails to start, please check whether the directory exists or the directory's usage permissions.
maxFileStore: Used to limit the storage upper limit of JetStream. If the upper limit is exceeded, writing will be prohibited.
maxPayload: The hard size limit of a single message, the largest messagechunksupported by Milvus5MB, so this configuration should be kept above5MBand leave a certain margin, otherwise Milvus may refuse to write.
initializeTimeout: Used to control the startup timeout configuration of Nats Server when Milvus starts. If the following log appears, you can increase this configuration appropriately.
[WARN] [nmq/nmq_server.go:77] ["nmq is not ready within timeout"]
monitor：用于配置 NATS 的独立日志，建议在日常运行环境中启用 trace 日志。
retention：用于控制 NATS 消息的保持机制，由于当前 Milvus 的消息保持机制与消息消费机制尚未实现同步。因此请务必保持该配置有充足的余量，让 Milvus 可以在消息被消息队列清退前消费完毕，否则 Milvus 可能会出现数据丢失的情况。
以上的绝大部分配置都与 NATS 官方配置对齐，如果需要了解更多的配置，或者希望 Milvus 引入更多的定制化配置，可以查看 NATS 的官方文档 https://docs.nats.io/running-a-nats-service/config。
04.
RocksMQ 迁移至 NATS
可以采用 Milvus 的通用 MQ 迁移方案。
停止 Milvus 一切的写入操作。
调用
FlushALL操作，等待FlushALL完毕后，关闭Milvus进程。
修改配置项
mq.type=natsmq，以及natsmq下相关需要修改的配置项（如果出现端口冲突或者目录权限等问题）
启动 2.3 版本的 Milvus 进程：
日志应当出现
mqType=natsmq日志。
natsm.server.storeDir配置的目录下应当出现 jetstream 文件夹。
可选：备份并清理原
rocksmq.path存储目录下的文件数据。
05.
NATS 和 RocksMQ 对比
Pub/Sub 性能对比
测试平台与方案
M1 Pro Chip / Memory: 16GB
启动 MQ，同时对一个 Topic 进行订阅和发布随机内容数据包，循环 N 次发布操作后，直到订阅得到最后一次发布结果时，测试结束。
测试结果
NATS 为纯 GO 的实现，而 RocksMQ 使用 CGO 与 rocksdb 的 lib 交互。因此 NATS 的绝大部分内存由 GO 的 GC 接管，而 RocksMQ 依赖 lib 自身的内存管理。NATS 的内存开销会更高。
在数据包较小（小于 64kb）的场景下，RocksMQ 不论在内存、CPU 还是在响应速度都具备较大优势。(1kb message rocksmq 有1x 以上性能优势)
在数据包较大的场景下(大于 64kb)，NATS 在内存充足以及理想的 GC 场景下，在响应速度方面有较大的优势(5MB message NATS 有 1x 以上性能优势)
存储方面，RocksMQ 当前有 Zstd 压缩加持，消耗的磁盘空间更少(NATS 未开压缩)
Milvus 集成测试
在 1 亿级别的向量对比测试中：NATS 可以支持更低的检索延迟。
在数据量较少的场景下：NATS 与 RocksMQ 的差距不大。
以上就是关于 Milvus 新消息队列 NATS 的全部内容，如果大家有任何疑问都可以跟我们沟通，下一篇我们会继续讲解 Milvus 2.3 的新功能——MMap，敬请期待。
本文分享自微信公众号 - ZILLIZ（Zilliztech）。
如有侵权，请联系 [email protected] 删除。
本文参与“OSC源创计划”，欢迎正在阅读的你也加入，一起分享。