Go uses RabbitMQ to complete message asynchronous communication practice problems and solve records

foreword

Background: Due to the requirements of the blockchain project, the high-concurrency big data environment at the same time cannot synchronize the data results on the chain in real time and return it to the business backend (the business backend implemented by java), so consider using asynchronous communication middleware. In order to build rabbitMQ by ourselves to complete our requirements and replace it with the method of API request return, we will do technical transformation of the project. It involves a series of issues such as cross-language communication, message middleware, mode selection, message body reception conversion joint debugging, etc., this is the only record.
Requirement:
The business backend (java) provides the information structure on the chain [producer – send message] –> go to the backend of go to call the blockchain chain method [consumer – consume message] –> call the chain smart contract to return the result After that, return the uplink result according to different situations [producer – send message to java backend]

technology choice

• Message middleware: rabbitMQ (no more details)
  mode selection: Routing routing mode (change and adjust the version, one of the reasons for subsequent pitfalls)
  switch type: direct (direct connection)
  to ensure message reliability: select manual confirmation ack method (another pitfall )
• Business backend: java
• On-chain business processing: go
• Blockchain bottom layer: fisco-bcos

RabbitMQ

RabbitMQ is an open source message broker software (message queue middleware) that implements the Advanced Message Queuing Protocol AMQP (Advanced Message Queuing Protocol) using the Erlang programming language. The role of message queue middleware: decoupling, peak cutting, asynchronous processing, cache storage, message Communication, improve system scalability.

RabbitMQ Features

• Reliability: through some mechanisms such as persistence, transmission confirmation, etc. to ensure the reliability of message delivery
• Scalability: Multiple RabbitMQ nodes can form a cluster
• High availability: the queue can be mirrored in the RabbitMQ cluster, so that even if some nodes hang up, the queue can still be used
• Multiple protocols: natively supports AMQP, and also supports STOMP, MQTT and other protocols
• Rich clients: all our commonly used programming languages ​​support RabbitMQ
• Management interface: comes with a WEB management interface
• Plug-in mechanism: RabbitMQ provides many plug-ins, which can be expanded as needed

problem and its solution

Question 1. Historical messages cannot be consumed in Routing mode

Due to the requirements of the project, we chose the routing routing mode, but during the test, we found that the go backend cannot get the historical data in the history queue (that is, the messages initiated by the java backend when the consumption coroutine is not enabled), and can only get After the consumer coroutine is started, add the newly added message body to the queue.

Question 2: Manual ack fails in Routing mode

A derivative of the previous question, the message body that can be obtained after the consumption coroutine is enabled, when the automatic ack confirmation is selected, the consumption still does not confirm the successful consumption, and the manual ack still fails, which is distressing

Question 1, Question 2 Error Causes

The reason for the final investigation turned out to be the follow-up chain reaction caused by a small change to the routing mode.
In the official website or other reference blog posts, the producer of the routing mode only needs to give the switch name exchangeName and routingkey when creating the mq instance. At this time, the messages will be placed in the temporary queue, that is, they will be created immediately if they are needed, and they will not be used. will be deleted automatically. The declaration of the temporary queue is very simple. You only need to write the queue name as an empty string when declaring the queue, so that the temporary queue will be assigned a random name, such as: amq.gen-JzTY20BRgKO-HjmUJj0wLg. The temporary queue will be deleted when the connection is closed

var (
	mqURL = viper.GetString("RabbitMQ.Url") //连接信息读取配置文件 amqp:// 账号 密码@地址:端口号/vhost
)

//RabbitMQ结构体
type RabbitMQ struct {
	//连接
	conn    *amqp.Connection
	channel *amqp.Channel
	//队列
	QueueName string
	//交换机名称
	ExChange string
	//绑定的key名称
	Key string
	//连接的信息，上面已经定义好了
	MqUrl string
}
//rabbitmq的路由模式。
//主要特点不仅一个消息可以被多个消费者消费还可以由生产端指定消费者。
//这里相对比订阅模式就多了一个routingkey的设计，也是通过这个来指定消费者的。
//创建exchange的kind需要是"direct",不然就不是roting模式了。
//创建rabbitmq实例，这里有了routingkey为参数了。
func NewRabbitMqRouting(exchangeName string, routingKey string) *RabbitMQ {
	return NewRabbitMQ("", exchangeName, routingKey)
}

//创建RabbitMQ结构体实例
func NewRabbitMQ(queueName string, exchange string, key string) *RabbitMQ {
	rabbitMQ := &RabbitMQ{QueueName: queueName, ExChange: exchange, Key: key, MqUrl: mqURL}
	var err error
	//创建rabbitMQ连接
	rabbitMQ.conn, err = amqp.Dial(rabbitMQ.MqUrl)
	rabbitMQ.failOnErr(err, "创建rabbit的路由实例的时候连接出现问题！")
	rabbitMQ.channel, err = rabbitMQ.conn.Channel()
	rabbitMQ.failOnErr(err, "创建rabbitmq的路由实例时获取channel出错")
	return rabbitMQ
}
//错误处理函数
func (r *RabbitMQ) failOnErr(err error, message string) {
	if err != nil {
		log.Fatalf("%s:%s", message, err)
		panic(fmt.Sprintf("%s:%s", message, err))
	}
}
//断开channel和connection
func (r *RabbitMQ) Destory() {
	r.channel.Close()
	r.conn.Close()
}

However, in order to facilitate subsequent positioning of the problem and many java blog posts about message middleware, colleagues in java back-end development chose to create a specified queue name and bind it to the switch. There is no problem for the producer to send messages, but when I receive and consume messages, I encounter the situation described in Problem 1 and Problem 2

solution

The consumer method is slightly different from the official one. Comment out the code for creating switches, creating queues, binding queues and switches, and get the consumption information directly from the channel of the mq instance (prerequisite: the queue needs to be created before execution , switch, binding switch and queue relationship can be created in the control panel or code, in short, it must be there first)

//路由模式接收信息-测试
func (r *RabbitMQ) ReceiveRouting() {
	尝试创建交换机，不存在创建
	//err := r.channel.ExchangeDeclare(
	//	//交换机名称
	//	r.ExChange,
	//	//交换机类型 广播类型
	//	"direct",
	//	//是否持久化
	//	true,
	//	//是否字段删除
	//	false,
	//	//true表示这个exchange不可以被client用来推送消息，仅用来进行exchange和exchange之间的绑定
	//	false,
	//	//是否阻塞 true表示要等待服务器的响应
	//	false,
	//	nil,
	//)
	//r.failOnErr(err, "failed to declare an excha"+"nge")
	2试探性创建队列，创建队列
	//q, err := r.channel.QueueDeclare(
	//	"", //随机生产队列名称
	//	false,
	//	false,
	//	true,
	//	false,
	//	nil,
	//)
	//r.failOnErr(err, "Failed to declare a queue")
	绑定队列到exchange中
	//err = r.channel.QueueBind(
	//	q.Name,
	//	//在pub/sub模式下，这里的key要为空
	//	r.Key,
	//	r.ExChange,
	//	false,
	//	nil,
	//)
	//消费消息
	message, err := r.channel.Consume(
		"queue_data_chain_reply",
		"",
		false, // 是否自动确认消费
		false,
		false,
		false,
		nil,
	)
	forever := make(chan bool)
	go func() {
		for d := range message {
			//chainDataRepBody := new(model.ChainDataRep)
			//err = json.Unmarshal(d.Body, &chainDataRepBody)
			d.Ack(true) // 更改为手动确认ack
			result := make(map[string]interface{})
			err = json.Unmarshal(d.Body, &result)
			if err != nil {
				fmt.Println(err)
			}
			for k, v := range result {
				fmt.Println(k)
				fmt.Println(v)
			}
			//fmt.Printf("消费的id是%s", chainDataRepBody.ID)
			fmt.Printf("消费的数据是%s", d.Body)
			//log.Printf("消费消息:%s,", d.Body)
		}
	}()
	fmt.Println("退出请按 Ctrl+C")
	<-forever
}

Just like the above test code, it can solve the historical consumption cannot be consumed, and save the manual confirmation ack

Question 3. How to modify the manual ack confirmation of rabbitMQ

Q: The problem of message confirmation, assuming that a consumer is processing a message, and the message is interrupted suddenly before the message is processed (maybe the consumer process is killed), at this time the message that the consumer is processing is lost, should How to solve this message loss problem?

solution

Answer: At this time, it has nothing to do with the producer, but mainly the settings on the consumer side. The autoAck automatic confirmation field under the Consume method in the code on the consumer side is set to false, indicating that we do not perform automatic confirmation. Instead, manually confirm after the message is processed, so if the message is interrupted before it is processed, and the RabbitMQ server does not receive the confirmation of the message, then the message in the queue will be redistributed again. The consumer side is modified as follows (the code is the same as above)

Question 4. The type of transmission information is converted in java\go

Because the amqp message body of go is []byte type, so our object or structure needs to be transferred between go and its dataBytes, err := json.Marshal(data) to byte type storage, and use json after receiving .Unmarshal() can be used after deserialization and parsing

result := make(map[string]interface{})
err = json.Unmarshal(d.Body, &result)

But the go language I use needs to communicate with the java backend. The range of java byte is -128-127, while golang byte is an alias of uint8, and the range is 0-255, so there will be problems of type mismatch conversion

solution

java producer sends information –> go consumer consumption consumption

When the java backend acts as a producer, the message sent is still put into the message queue with the json type, and the message body json.Unmarshal() taken from go can be converted into the specified object structure

Go producer sends message –> java consumer consumes message

When go is a producer, send a message to serialize the object json.Marshal(), but you need to modify the routing mode to send the content type of the amqp.Publishing message to application/json (official cases or other blog posts are text/plain type )

Routing mode to send messages

//路由模式发送信息
func (r *RabbitMQ) PublishRouting(message []byte) {
	//第一步，尝试创建交换机，与pub/sub模式不同的是这里的kind需要是direct
	err := r.channel.ExchangeDeclare(
		//交换机名称
		r.ExChange,
		//交换机类型 广播类型
		"direct",
		//是否持久化
		true,
		//是否字段删除
		false,
		//true表示这个exchange不可以被client用来推送消息，仅用来进行exchange和exchange之间的绑定
		false,
		//是否阻塞 true表示要等待服务器的响应
		false,
		nil,
	)
	r.failOnErr(err, "路由模式，尝试创建交换机失败")
	//发送信息
	err = r.channel.Publish(
		r.ExChange,
		//要设置
		r.Key,
		false,
		false,
		amqp.Publishing{
			//类型
			//ContentType: "text/plain", // byte类型
			ContentType: "application/json", // 更改为json类型
			//消息
			Body: message,
		})
}

Question 5. In the logical code of the coroutine consumption message, sending the message to MQ fails

Because the business needs to start the coroutine to process the message of the queue, it needs to send the processing result to another queue (that is, to send the message as the producer of another MQ instance in undertaking the MQ consumer processing business logic), Because it is a manual ack mode in consumption, my previous processing sequence is

1. Receive the message body as a consumer in the coroutine
2. Handle business logic
3. Encapsulate return structure
4. As a producer, send a return message to another queue
5. Manually ack confirms the message
   At this time, a problem is encountered. The message has been confirmed to be consumed, but the message has not been sent to another queue.

solution

Replace the code in steps 4 and 5, first manually ack to confirm the consumption, and then send the consumption to solve the problem. Although I don’t understand the logic of this, it is only used as a record of the problem. If you know, welcome to communicate.

Question 6. Some fields in the receiving mq message body are missing

A low-level error, but the record reminds myself that there are two fields of int type in the object structure interaction with the java backend, which are used as the chain status judgment, but after go is sent to the queue as a producer, the java backend When receiving as a consumer, it is found that the two int-type fields are missing, and neither key nor value exists.

solution

I thought it was another strange problem caused by language inconsistency on int type transmission. Later, I found out that I forgot to change the json command in the structure (for example, as shown in the figure below). The json of the two structures is the same, resulting in the transmission It is missing in the mq queue, a low-level mistake, but it is easy to make such a low-level mistake if you are confused, and it still bothers you, record to remind yourself! !

Question 7. The pressure test causes the message channel to be blocked and an infinite loop, the message body is lost, and the manual ack fails

I originally thought that solving the above-mentioned problems can basically meet the business requirements, but I did not pass the pressure test. When we stress-tested key business messages into the queue-consumption-return to this logic, when using the tps2000 intensity stress test, consumption occurred When there are about 1,000 pieces of information, I get a piece of information without a message body, and then the manual ack still fails, causing the channel to be blocked and the infinite loop to be stuck.

Solution: Set the qos parameter prefetch to solve the problem of message congestion

// 启用QoS，每次预取5条消息，避免消息处理不过来，全部堆积在本地缓存里
r.channel.Qos(
		//每次队列只消费一个消息 这个消息处理不完服务器不会发送第二个消息过来
		//当前消费者一次能接受的最大消息数量-自定义设置
		1,
		//服务器传递的最大容量
		0,
		//如果为true 对channel可用 false则只对当前队列可用
		false,
	)

Turn on QoS, when the RabbitMQ queue reaches 5 Unacked messages, no more messages will be pushed to the Consumer; the solution is not timely ack
prefetch refers to the maximum number of unacked messages that a single consumer can consume

q sets a buffer for each consumer, the size is prefetch. Every time a message is received, MQ will push the message to the buffer, and then push it to the client. When receiving an ack message (consumer issues a baseack command), mq will vacate a position from the buffer, and then add a new message. But if the buffer is full at this time, MQ will enter a blocked state.
To describe it more specifically, suppose the prefetch value is set to 10, and there are two consumers. That is to say, each consumer will pre-fetch 10 messages from the queue each time and cache them locally for consumption. At the same time, the unacked number of the channel becomes 20. The delivery order of Rabbit is to deliver 10 messages to consumer1 first, and then deliver 10 messages to consumer2. If there is a new message to be delivered at this time, first judge whether the unacked number of the channel is equal to 20, if so, the message will not be delivered to the consumer, and the message will continue to stay in the queue. Afterwards, the consumer acks a message, and the unacked is equal to 19 at this time, and Rabbit judges which consumer's unacked is less than 10, and delivers it to which consumer.

After setting manual ack and then cooperating with qos, my problem can be solved, and the pressure test will not cause similar problems again.

excellent wheels

After I solved all the above problems, I found that if a large amount of data from the client is uploaded to the chain at the same time, because the consumption rate of mq is not ideal due to the complex processing, it will take half an hour to an hour to complete the consumption of 2000 messages, although there will be no other abnormalities Wrong, but the performance is still too poor, and the application experience is inevitably not good enough, so after my search, I found an excellent wheel. The boss has packaged it very well, and considering the connection pool channel reuse, error retry, dead Processing of other aspects such as permanent queues, cluster polling, etc. I have to say that the above is indeed the available version that I integrated based on the official website case and the limited go data about rabbitmq, but the performance is not good enough. I learned a lot after referring to the big guy's code, so I will record and share a wave Good wheels for everyone.
tym_hmm / rabbitmq-pool-go
I pulled the code to the local, and then adjusted it according to the project requirements. I went through my own pressure test, and the performance directly stepped up to n levels. The boss also wrote that it was applied to the production environment. When 5200W requests qbs 3000 , The connection pool shows that there is no pressure, and the cluster deployment has also been done. The personal test is easy to use. (ps: The premise of using it must also digest the official basic writing method and the meaning of some parameters)

reference article

Some article references, including some mode examples are very detailed
go operation RabbitMQ
rabbitMQ official website routing mode go language
[go-zero] go-zero and amqp go integrate Rabbitmq to achieve message push go message queue (best practice)
golang implements rabbitmq message queue
rabbitmq series Article
amqp
rabbitmq uses
Golang's message queue - RabbitMQ uses
RabbitMQ FAQ
RabbitMq qos prefetch message congestion problem