Preface
This article briefly introduced the nsq infrastructure, mainly focusing on how to deploy a healthy, highly available distributed message queue in production.
For detailed introduction, please refer to the official document: https://nsq.io
1 Introduction
- nsq is a highly available, single-point and distributed message queue.
- nsq is the classic producer-consumer model, including two broker roles, nsqd and lookupd, but the respective numbers can be expanded.
- The producer delivers topical messages directly to nsqd, and the messages are directly stored in the server where nsqd is located.
- Consumers do not directly ask nsqd for the corresponding topic, but ask for topic information through lookupd. When requesting, they can use the channel to mark the requester channel.
2. Sample code
import "github.com/nsqio/go-nsq"
Producer-production news
var producer *nsq.Producer
var nsqdAddrTCP = "localhost:4150"
var conf *nsq.Config
conf = nsq.NewConfig()
producer, er = nsq.NewProducer(nsqdAddrTCP, conf)
if er != nil {
log.Println(er.Error())
return
}
// 并发发布3000个消息
for i := 0; i < 3000; i++ {
go func(i int) {
er := producer.Publish("go-nsq_testcase", []byte(fmt.Sprintf("hello,everyone_%d", i)))
if er != nil {
log.Println(er.Error())
return
}
}(i)
Consumer-consumer news
type ConsumerT struct{
}
//处理消息
func (*ConsumerT) HandleMessage(msg *nsq.Message) error {
fmt.Println("receive", msg.NSQDAddress, "message:", string(msg.Body))
return nil
}
func main() {
var nsqlookupdAddr = "localhost:4161" // help consumer find topics
var channel = "channel_1" // 消费者渠道标记
var consumers []*nsq.Consumer
// 创建100个消费者
consumers = make([]*nsq.Consumer, 100)
for i, _ := range consumers {
consumers[i], e = nsq.NewConsumer("go-nsq_testcase", channel, conf)
if e != nil {
log.Println(e.Error())
return
}
consumers[i].SetLogger(nil, 0)
consumers[i].AddHandler(&ConsumerT{
}) // 添加消费者接口
if e = consumers[i].ConnectToNSQLookupd(nsqlookupdAddr);e!=nil {
log.Println(e)
return
}
}
}
3.nsq architecture
3-7 lookupd-hundreds of nsqd-tens of thousands of consumers
This ratio is healthy
3.1 How consumers find topics through lookupd
When nsqd is launched, it must be bound to a lookupd:
$ nsqd --lookupd-tcp-address=127.0.0.1:4160
In other words, the consumer must know in advance which nsqd the topic he wants is posted to, and the message can be obtained through the corresponding lookupd.
3.2 If the load of the nsqd server delivered by the producer is too high, how to increase the node balance.
Assuming that the order data is delivered to nsqd1-lookupd1, consumers consume the order data through lookupd1. When the backlog of order data exceeds 5 million and the nsqd1 server is too tired, then at this time, a new server should be opened and a nsqd2 should be run.
$ nsqd --lookupd-tcp-address=127.0.0.1:4160
Then, as long as nsqd2 and nsqd1 are bound to the same lookupd, then consumers can directly consume the data on the newly added nsqd2.
In short, you can dynamically expand nodes by ensuring the following:
- Directly add a nsqd node, and bind the node to the same lookupd
3.3 How does nsq solve the problem of repeated delivery
Unfortunately, nsq cannot solve the problem of repeated delivery, and the client needs to handle it by itself. You can refer to the database unique key, redis SETNX. But the premise is that the message itself has a unique id.
3.4 How does nsq solve the execution order
Unfortunately, nsq also does not guarantee the order of messages. So what if consumers have order requirements?
Solution: Mark the message weight and use the redis ranking mechanism (zadd-add, zrange-list, zcard-length, complexity logN)
- In the message, add the tag header, which contains the following information:
{
"total_number": 10,
"number":1
}
- When consumers consume
// 自带去重，不怕重复消费
conn.Do("ZADD", "话题名", number, "消息")
// 插入后检查长度
length ,_ := redis.Int64(conn.Do("ZCARD", "话题名"))
if length == total_number {
messagss, _ := redis.Bytes(conn.Do("ZRANGE", "话题名",0, -1))
// 按照messages顺序消费完。
for i,v:=range message{
handle(v)
}
}
// 最后，设置定期销毁该定序队列
// 该步不可忽视，因为可能出现单个消息重复投递，如果在消费完10个数据后，重复数据到达消费者这里，
// 那么该话题就会永久保留一个多余的消息，不会失效。
conn.Do("Expire", "话题名", 60 *60 * 24)
3.5 nsqd hangs, will the message be lost?
Under the default configuration, it will be lost, but nsqd can be configured to be persistent.
nsqd --lookupd-tcp-address=127.0.0.1:4160 --mem-queue-size 0
After mem-queue-size is set to 0, it will be persisted, and restart will not cause loss.
3.6 Is there a running order for nsqd and lookupd?
Yes, lookupd first, then nsqd