Application of beanstalkd message queue in production environment

Beanstalkd is a high-performance message queue middleware. This blog post will introduce the use of this stuff.

First, let everyone understand the features and working scenarios of Beanstalkd through concepts.

        Beanstalkd is a lightweight message middleware. Its biggest feature is that it positions itself as a work-queue based on pipes (tubes) and tasks (jobs):

Beanstalkd supports task priority (priority), delay (delay) ), time-to-run and reserved (buried), can well support distributed background tasks and timed task processing.

Its internal implementation uses libevent, and a lightweight communication protocol similar to memcached is used between the server and the client, which has high performance.

Despite being an in-memory queue, beanstalkd provides a binlog mechanism, and when beanstalkd is restarted, the current task status can be restored from the recorded local binlog.

Pipeline (tube):

       Pipeline is similar to message topic (topic), in a Beanstalkd can support multiple pipelines, each pipeline has its own publisher (producer) and consumer (consumer). Pipelines do not affect each other.

Jobs:

       Beanstalkd replaces the concept of messages with jobs. Unlike messages, tasks have a series of states:

Beanstalkd


READY - tasks that need to be processed immediately, and will automatically become the current task when the delayed (DELAYED) task expires;

DELAYED-Delayed execution of the task, when the consumer processes the task, you can delay the execution by putting the message back on the DELAYED queue again;

RESERVED-The task that has been acquired by the consumer and is being executed. Beanstalkd is responsible for checking if the task is completed within TTR (time-to-run);

BURIED - reserved tasks: the task will not be executed and will not disappear unless someone "kicks" it back to the queue;

DELETED - the message is completely deleted . Beanstalkd no longer maintains these messages.

Task priority (priority): A

       task (job) can have 0~2^32 priorities, with 0 representing the highest priority. beanstalkd uses Min-max heap to process task priority sorting. Consumers who call the reserve command at any time can always get the task with the current highest priority, and the time complexity is O(logn).

Delayed tasks ( delay):

       There are two ways to delay the execution of tasks (job): the producer specifies the delay when releasing the task; or when the task is processed, the consumer puts the task into the queue again for delayed execution (RELEASE with <delay>) . This mechanism can implement distributed Java.util.Timer. The advantage of this distributed timing task is that if a consumer node fails, the task time-to-run can ensure that the task is transferred to another Node executes.

Task timeout resend (time-to-run):

       After Beanstalkd returns the task to the consumer: the consumer must send delete / release / bury within the preset TTR (time-to-run) time to change the task status; otherwise, Beanstalkd will consider the message processing failure, and then hand over the task to another The consumer node executes. If the consumer is expected to be unable to complete the task within the TTR (time-to-run) time, the touch command can also be sent, its role is to let Beanstalkd recalculate the TTR (time-to-run) from the system time.

Task reservation (buried ) ):

       If the task cannot be executed for some reason, the consumer can put the task into the buried state and let Beanstalkd keep these tasks. The administrator can query the reserved tasks through the peek buried command and perform manual intervention. Simply, kick <n> kicks n reserved tasks back to the queue at once.

Beanstalkd protocol:

       Beanstalkd adopts a memcached-like protocol, where the client interacts with the server through text commands. These commands can be simply grouped into three groups:   

       Producer - use <tube> / put <priority> <delay> <ttr> [bytes]: 

       The producer selects a pipe (tube) with use, and then publishes tasks to the pipe with the put command (job)   

       .Consumer - watch <tubes> / reserve / delete <id> / release <id> <priority> <delay> / bury <id>

       Consumers use watch to select multiple tubes, and then use the reserve command to get tasks to be executed, which is blocking. The client does not return until there is a task to execute. When the task is processed, the consumer can completely delete the task (DELETE), release the task for others to process (RELEASE), or retain (BURY) the task.

       Maintenance class - peek job / peek delayed / peek ready / peek buried / kick <n>

is used to maintain the task state inside the pipeline, and get the task without changing the task state. The state of these tasks can be changed with consumer commands.

Buried tasks can be "kicked" back to the queue with the kick command.

          Protocol document: https://raw.github.com/kr/beanstalkd/master/doc/protocol.txt


Inadequate Beanstalkd:

Beanstalkd does not provide active-standby synchronization + failover mechanism, and there is a risk of becoming a single point in the application. In practical applications, a database can be used to provide persistent storage for tasks (jobs).