RabbitMQ official tutorial two Work Queues (GOLANG language)

Chart

In the first tutorial, we wrote a program to send and receive messages from the named queue. In this section, we will create a work queue that will be used to assign time-consuming tasks across multiple worker.

The main idea of the work queue (also known as Task Queue) is to avoid the immediate implementation of resource-intensive tasks (such as lengthy mail, file handling, etc.), and had to wait for it to complete. Instead, we schedule the task after the completion of (asynchronously). Our task is packaged and sent to the message queue. In worker process running in the background and eventually you will get the task to execute the job. When you run a lot of worker, they will share the task.

This concept is particularly useful in Web applications because Web applications, it is impossible to handle complex tasks in a short period of time HTTP request.

ready

In the previous section of this tutorial, we have sent a message with "Hello World!" Is. Now, we will send a string representation of complex tasks. We do not have a real job, for example, you want to resize the image to be rendered or pdf files, so we pretend time-consuming task - using time.Sleep function to fake it. We will point string as its complexity. Each point will account for "work" in a second. For example, the virtual task Hello ... description will spend three seconds.

We send.go slightly modified example of a code to allow transmission of any message from the command line. The program will task to arrange our work queue, so we named it new_task.go:

#new_task.go

package main

import (
    "github.com/streadway/amqp"
    "log"
    "os"
    "strings"
)

func main(){
    // 连接RabbitMQ服务器
    conn, err := amqp.Dial("amqp://guest:guest@localhost:5672/")
    failOnError(err, "Failed to connect to RabbitMQ")
    defer conn.Close()
    // 创建一个channel
    ch, err := conn.Channel()
    failOnError(err, "Failed to open a channel")
    defer ch.Close()

    q, err  := ch.QueueDeclare(
        "hello",            // 队列名称
        false,          // 是否持久化
        false,      // 是否自动删除
        false,          // 是否独立
        false,nil,
        )
    failOnError(err, "Failed to declare a queue")

    body := bodyForm(os.Args)
    err = ch.Publish(
        "",     // exchange
        q.Name, // routing key
        false,  // mandatory
        false,  // immediate
        amqp.Publishing {
            DeliveryMode:amqp.Persistent,
            ContentType: "text/plain",
            Body:        []byte(body),
        })
    failOnError(err, "Failed to publish a message")
    log.Printf(" [x] Sent %s", body)
}

func bodyForm(args []string) string{
    var s string
    if (len(args) < 2) || os.Args[1] == "" {
        s = "hello"
    } else {
        s = strings.Join(args[1:], " ")
    }
    return s
}

// 帮助函数检测每一个amqp调用
func failOnError(err error, msg string)  {
    if err != nil {
        log.Fatalf("%s: %s", msg, err)
    }
}


#worker.go

package main

import (
    "bytes"
    "github.com/streadway/amqp"
    "log"
    "time"
)

func main(){
    // 连接RabbitMQ服务器
    conn, err := amqp.Dial("amqp://guest:guest@localhost:5672/")
    failOnError(err, "Failed to connect to RabbitMQ")
    defer conn.Close()
    // 创建一个channel
    ch, err := conn.Channel()
    failOnError(err, "Failed to open a channel")
    defer ch.Close()
    // 监听队列
    q, err  := ch.QueueDeclare(
        "hello",            // 队列名称
        false,          // 是否持久化
        false,      // 是否自动删除
        false,          // 是否独立
        false,nil,
    )
    failOnError(err, "Failed to declare a queue")
    // 消费队列
    msgs, err := ch.Consume(
        q.Name, // queue
        "",     // consumer
        true,   // auto-ack
        false,  // exclusive
        false,  // no-local
        false,  // no-wait
        nil,    // args
    )
    failOnError(err, "Failed to register a consumer")

    forever := make(chan bool)

    go func() {
        for d := range msgs {
            log.Printf("Received a message: %s", d.Body)
            // 统计string中的`.`来表示执行时间
            dot_count := bytes.Count(d.Body, []byte("."))
            t := time.Duration(dot_count)
            time.Sleep(t * time.Second)
            log.Printf("Done")
        }
    }()

    log.Printf(" [*] Waiting for messages. To exit press CTRL+C")
    <-forever
}

// 帮助函数检测每一个amqp调用
func failOnError(err error, msg string)  {
    if err != nil {
        log.Fatalf("%s: %s", msg, err)
    }
}

Open two worker is ready to accept the data queue

# shell 1
go run worker.go
# => [*] Waiting for messages. To exit press CTRL+C

# shell 2
go run worker.go
# => [*] Waiting for messages. To exit press CTRL+C

Performing transmission data new_task.go


# shell 3
go run new_task.go First message.
go run new_task.go Second message..
go run new_task.go Third message...
go run new_task.go Fourth message....
go run new_task.go Fifth message.....

The results obtained are as follows
Results Figure

By default, RabbitMQ will in turn send each message to the next consumer. On average, each consumer will receive the same number of messages. This way to distribute messages is called a cycle.

Message acknowledgment

Mission may take several seconds. You may be wondering, if one consumer to start a lengthy task but only partially completed and die, what happens. Using our current code, RabbitMQ once sent a message to consumers, it will immediately be marked for deletion. In this case, if you kill a worker, we will lose the message being processed. We will also lose all the messages sent to that particular staff but not yet processed.

但是我们不想丢失任何任务。如果一个worker死亡，我们希望将任务交付给另一个worker。

为了确保消息永不丢失，RabbitMQ支持消息确认。消费者发送回一个消息确认，告知RabbitMQ特定的消息已被接收，处理，并且RabbitMQ可以自由删除它。

如果使用者宕机（其通道已关闭，连接已关闭或TCP连接丢失）而没有发送确认，RabbitMQ将了解消息未完全处理，并将重新排队。如果同时有其他消费者在线，它将很快将其重新分发给另一个消费者。这样，您可以确保即使worker偶尔宕机也不会丢失任何消息。

RabbitMQ没有任何消息超时设置；消费者死亡时，RabbitMQ将重新传递消息。即使处理一条消息花费非常非常长的时间也没关系。

在本教程中，我们将通过为“ auto-ack”参数传递一个false来使用手动消息确认，然后一旦在任务完成后使用d.Ack（false）从worker发送适当的确认。

#worker.go修改部门代码
msgs, err := ch.Consume(
  q.Name, // queue
  "",     // consumer
  false,  // 是否自动消息确认
  false,  // exclusive
  false,  // no-local
  false,  // no-wait
  nil,    // args
)
failOnError(err, "Failed to register a consumer")

forever := make(chan bool)

go func() {
  for d := range msgs {
    log.Printf("Received a message: %s", d.Body)
    dot_count := bytes.Count(d.Body, []byte("."))
    t := time.Duration(dot_count)
    time.Sleep(t * time.Second)
    log.Printf("Done")
    d.Ack(false)
  }
}()

log.Printf(" [*] Waiting for messages. To exit press CTRL+C")
<-forever

使用此代码，我们可以确保，即使您在处理消息时使用CTRL + C杀死worker，也不会丢失任何信息。 worker后不久，所有未确认的消息将重新发送。

消息确认必须与消息发送在同一channel上。尝试使用其他channel进行确认将导致通道级协议异常。

忘记消息确认

忘记设置消息确认是一个常见的错误。这是一个很容易犯的错误，但是后果很严重。当您的客户端退出时，消息将被重新发送（可能看起来像是随机重新发送），但是RabbitMQ将占用越来越多的内存，因为它将无法释放任何未确认的消息。

为了调试这种错误，您可以使用rabbitmqctl打印messages_unacknowledged字段：

sudo rabbitmqctl list_queues name messages_ready messages_unacknowledged

消息持久化

我们已经学会了如何确保即使消费者死亡，任务也不会丢失。但是，如果RabbitMQ服务器停止，我们的任务仍然会丢失。

RabbitMQ退出或宕机时，它将丢失队列和消息，除非您告知不要这样做。要确保消息不会丢失，需要做两件事：我们需要将队列和消息都标记为持久。

首先，我们需要确保RabbitMQ永远不会丢失我们的队列。为此，我们需要将其声明为持久的：

q, err := ch.QueueDeclare(
  "hello",      // name
  true,         // durable
  false,        // delete when unused
  false,        // exclusive
  false,        // no-wait
  nil,          // arguments
)
failOnError(err, "Failed to declare a queue")

尽管此命令本身是正确的，但在我们当前的设置中将无法使用。这是因为我们已经定义了一个名为hello的队列，该队列并不持久。 RabbitMQ不允许您使用不同的参数重新定义现有队列，并且将向尝试执行此操作的任何程序返回错误。但是有一个快速的解决方法-让我们声明一个名称不同的队列，例如task_queue：

q, err := ch.QueueDeclare(
  "task_queue", // name
  true,         // durable
  false,        // delete when unused
  false,        // exclusive
  false,        // no-wait
  nil,          // arguments
)
failOnError(err, "Failed to declare a queue")

这种持久的选项更改需要同时应用于生产者代码和消费者代码。

在这一点上，我们确保即使RabbitMQ重新启动，task_queue队列也不会丢失。现在，我们需要使用amqp.Persistent选项amqp.Publishing来将消息标记为持久消息。

err = ch.Publish(
  "",           // exchange
  q.Name,       // routing key
  false,        // mandatory
  false,
  amqp.Publishing {
    DeliveryMode: amqp.Persistent,
    ContentType:  "text/plain",
    Body:         []byte(body),
})

关于消息持久性的说明

将消息标记为持久性并不能完全保证不会丢失消息。尽管它告诉RabbitMQ将消息保存到磁盘，但是RabbitMQ接受消息并且尚未保存消息时，还有很短的时间。而且，RabbitMQ不会对每条消息都执行fsync（2）－它可能只是保存到缓存中，而没有真正写入磁盘。持久性保证并不强，但是对于我们的简单任务队列而言，这已经绰绰有余了。如果您需要更强有力的保证，则可以使用publisher confirms。

公平分发

您可能已经注意到，调度仍然无法完全按照我们的要求进行。例如，在有两名worker的情况下，当所有奇数的消息都很重，偶数消息很轻时，一个worker将一直忙碌而另一位worker将几乎不做任何工作。但是，RabbitMQ对此一无所知，并且仍将平均分配消息。

发生这种情况是因为RabbitMQ在消息进入队列时才调度消息。它不会查看使用者的未确认消息数。它只是盲目地将每第n条消息发送给第n个使用者。

Results Figure

为了解决这个问题，我们可以将预取计数的值设置为1。这告诉RabbitMQ一次不要给一个worker发送多条消息。换句话说，在处理并确认上一条消息之前，不要将新消息发送给worker。而是将其分派给不忙的下一个worker。

err = ch.Qos(
  1,     // prefetch count
  0,     // prefetch size
  false, // global
)
failOnError(err, "Failed to set QoS")

Notes on the size of the queue

If all staff are busy, your queue is full. You will need to pay attention to this, might add more staff, or have some other strategies.

The final code

new_task.go

package main

import (
        "log"
        "os"
        "strings"

        "github.com/streadway/amqp"
)

func failOnError(err error, msg string) {
        if err != nil {
                log.Fatalf("%s: %s", msg, err)
        }
}

func main() {
        conn, err := amqp.Dial("amqp://guest:guest@localhost:5672/")
        failOnError(err, "Failed to connect to RabbitMQ")
        defer conn.Close()

        ch, err := conn.Channel()
        failOnError(err, "Failed to open a channel")
        defer ch.Close()

        q, err := ch.QueueDeclare(
                "task_queue", // name
                true,         // durable
                false,        // delete when unused
                false,        // exclusive
                false,        // no-wait
                nil,          // arguments
        )
        failOnError(err, "Failed to declare a queue")

        body := bodyFrom(os.Args)
        err = ch.Publish(
                "",           // exchange
                q.Name,       // routing key
                false,        // mandatory
                false,
                amqp.Publishing{
                        DeliveryMode: amqp.Persistent,
                        ContentType:  "text/plain",
                        Body:         []byte(body),
                })
        failOnError(err, "Failed to publish a message")
        log.Printf(" [x] Sent %s", body)
}

func bodyFrom(args []string) string {
        var s string
        if (len(args) < 2) || os.Args[1] == "" {
                s = "hello"
        } else {
                s = strings.Join(args[1:], " ")
        }
        return s
}

worker.go


package main

import (
        "bytes"
        "github.com/streadway/amqp"
        "log"
        "time"
)

func failOnError(err error, msg string) {
        if err != nil {
                log.Fatalf("%s: %s", msg, err)
        }
}

func main() {
        conn, err := amqp.Dial("amqp://guest:guest@localhost:5672/")
        failOnError(err, "Failed to connect to RabbitMQ")
        defer conn.Close()

        ch, err := conn.Channel()
        failOnError(err, "Failed to open a channel")
        defer ch.Close()

        q, err := ch.QueueDeclare(
                "task_queue", // name
                true,         // durable
                false,        // delete when unused
                false,        // exclusive
                false,        // no-wait
                nil,          // arguments
        )
        failOnError(err, "Failed to declare a queue")

        err = ch.Qos(
                1,     // prefetch count
                0,     // prefetch size
                false, // global
        )
        failOnError(err, "Failed to set QoS")

        msgs, err := ch.Consume(
                q.Name, // queue
                "",     // consumer
                false,  // auto-ack
                false,  // exclusive
                false,  // no-local
                false,  // no-wait
                nil,    // args
        )
        failOnError(err, "Failed to register a consumer")

        forever := make(chan bool)

        go func() {
                for d := range msgs {
                        log.Printf("Received a message: %s", d.Body)
                        dot_count := bytes.Count(d.Body, []byte("."))
                        t := time.Duration(dot_count)
                        time.Sleep(t * time.Second)
                        log.Printf("Done")
                        d.Ack(false)
                }
        }()

        log.Printf(" [*] Waiting for messages. To exit press CTRL+C")
        <-forever
}