Understanding Redis client-output-buffer-limit parameters and source code analysis

overview

Redis's client-output-buffer-limit can be used to forcibly disconnect clients that cannot read data from the redis server fast enough.
The protection mechanism rules are as follows:

1. [hard limit] Size limit, when a client buffer exceeds the set value, the connection will be closed directly.
2. [soft limit] Duration limit, close the connection when a client buffer takes up too much space for a period of time.

This parameter is generally used in the following types of clients:

• Ordinary client, including monitor
• slave client during master-slave synchronization
• client in Pub/Sub mode
  

Configuration introduction and analysis

The configuration syntax for this parameter:

client-output-buffer-limit <class> <hard limit> <soft limit> <soft seconds>

Configuration example:

# 普通client buffer限制 
client-output-buffer-limit normal 0 0 0
# slave client buffer限制
client-output-buffer-limit slave 256mb 64mb 60
# pubsub client buffer限制
client-output-buffer-limit pubsub 32mb 8mb 60

• client-output-buffer-limit normal 0 0 0

Setting hard limit and soft limit to 0 at the same time means turning off the limit.

• client-output-buffer-limit slave 256mb 64mb 60

This configuration means that for the slave client, if the output-buffer occupies 256M or exceeds 64M for 60s, the client connection will be closed.

• client-output-buffer-limit pubsub 32mb 8mb 60

This configuration means that for the Pub/Sub client, if the output-buffer occupies 32M or exceeds 8M for 60s, the client connection will be closed.

General description:
In general, for ordinary clients, there is no limit to client-output-buffer, because the server will only send data when the client requests data, and there will be no backlog.
In the scenario where the server actively sends data and the client handles it, this is usually processed asynchronously, and a buffer will be allocated to "temporarily store" unprocessed data. If the server sends data faster than the client processes data, it will A buffer backlog occurs. For clients used as Pub/Sub and slave, the server will actively push data to them, so it is necessary to set the limit of client-output-buffer.

Example analysis

Let's take a detailed analysis of the slave client during master-slave synchronization.
When redis is in master-slave synchronization, the master will create an output buffer for the slave. Before the master saves the rdb, transfers the rdb file to the slave, and the slave finishes loading the rdb, the master will write all the received write commands to the output buffer in the memory.
If the saving, transmission, and loading of rdb take too long, or there are too many write commands during this period, the buffer limit may be exceeded and the connection between the master and slave may be disconnected. In this case, you need to adjust the configuration appropriately  client-output-buffer-limit slave.

Source code analysis - use of output buffer during master-slave synchronization

Based on redis5.0 version source code

The redis server sends data to the client through addReply, see  https://github.com/redis/redis/blob/5.0/src/networking.c#L190-L211 for the following source code

/* Add the object 'obj' string representation to the client output buffer. */
void addReply(client *c, robj *obj) {
    if (prepareClientToWrite(c) != C_OK) return;

    if (sdsEncodedObject(obj)) {
        if (_addReplyToBuffer(c,obj->ptr,sdslen(obj->ptr)) != C_OK)
            _addReplyStringToList(c,obj->ptr,sdslen(obj->ptr));
    } else if (obj->encoding == OBJ_ENCODING_INT) {
        char buf[32];
        size_t len = ll2string(buf,sizeof(buf),(long)obj->ptr);
        if (_addReplyToBuffer(c,buf,len) != C_OK)
            _addReplyStringToList(c,buf,len);
    } else {
        serverPanic("Wrong obj->encoding in addReply()");
    }
}

At the beginning of the function, it will be prepareClientToWrite(c)judged whether the data needs to be written into the output buffer of the client. Let's see under what conditions the data will be written into the output buffer of the client, and it will be returned  C_OK.

/* This function is called every time we are going to transmit new data
 * to the client. The behavior is the following:
 *
 * If the client should receive new data (normal clients will) the function
 * returns C_OK, and make sure to install the write handler in our event
 * loop so that when the socket is writable new data gets written.
 *
 * If the client should not receive new data, because it is a fake client
 * (used to load AOF in memory), a master or because the setup of the write
 * handler failed, the function returns C_ERR.
 *
 * The function may return C_OK without actually installing the write
 * event handler in the following cases:
 *
 * 1) The event handler should already be installed since the output buffer
 *    already contains something.
 * 2) The client is a slave but not yet online, so we want to just accumulate
 *    writes in the buffer but not actually sending them yet.
 *
 * Typically gets called every time a reply is built, before adding more
 * data to the clients output buffers. If the function returns C_ERR no
 * data should be appended to the output buffers. */
int prepareClientToWrite(client *c) {
    /* If it's the Lua client we always return ok without installing any
     * handler since there is no socket at all. */
    if (c->flags & (CLIENT_LUA|CLIENT_MODULE)) return C_OK;

    /* CLIENT REPLY OFF / SKIP handling: don't send replies. */
    if (c->flags & (CLIENT_REPLY_OFF|CLIENT_REPLY_SKIP)) return C_ERR;

    /* Masters don't receive replies, unless CLIENT_MASTER_FORCE_REPLY flag
     * is set. */
    if ((c->flags & CLIENT_MASTER) &&
        !(c->flags & CLIENT_MASTER_FORCE_REPLY)) return C_ERR;

    if (c->fd <= 0) return C_ERR; /* Fake client for AOF loading. */

    /* Schedule the client to write the output buffers to the socket, unless
     * it should already be setup to do so (it has already pending data). */
    if (!clientHasPendingReplies(c)) clientInstallWriteHandler(c);

    /* Authorize the caller to queue in the output buffer of this client. */
    return C_OK;
}

/* Return true if the specified client has pending reply buffers to write to
 * the socket. */
int clientHasPendingReplies(client *c) {
    return c->bufpos || listLength(c->reply);
}

void clientInstallWriteHandler(client *c) {
    /* Schedule the client to write the output buffers to the socket only
     * if not already done and, for slaves, if the slave can actually receive
     * writes at this stage. */
    if (!(c->flags & CLIENT_PENDING_WRITE) &&
        (c->replstate == REPL_STATE_NONE ||
         (c->replstate == SLAVE_STATE_ONLINE && !c->repl_put_online_on_ack)))
    {
        c->flags |= CLIENT_PENDING_WRITE;
        listAddNodeHead(server.clients_pending_write,c);
    }
}

Since the function returns by default C_OK, we only need to check which types of cases are not returned C_OK, that is C_ERR, the data will not be written into the output buffer of the client.
The situation returned C_ERR:

• The client is a fake client (for loading AOF files)
• The client is a master
• The state of the slave is SLAVE_STATE_ONLINE and its callback function failed ((c->replstate == SLAVE_STATE_ONLINE && !c->repl_put_online_on_ack)), or the state of the slave is REPL_STATE_NONE

If the client should not receive new data, because it is a fake client (used to load AOF in memory), a master or because the setup of the write handler failed, the function returns C_ERR.

When the master saves and sends the rdb file, the status of the slave is as follows, so the write commands during this period will be saved in the output buffer of the slave. Since the callback function is not set, the data will not be sent to the slave, but only stored in the output buffer created by the master for the slave.

#define SLAVE_STATE_WAIT_BGSAVE_START 6 /* We need to produce a new RDB file. */
#define SLAVE_STATE_WAIT_BGSAVE_END 7 /* Waiting RDB file creation to finish. */
#define SLAVE_STATE_SEND_BULK 8 /* Sending RDB file to slave. */

So when will the slave be "flushed" from the output buffer? Until the master completely sends the rdb file to the slave, the master will  sendBulkToSlaveperform related operations in the function. See the following source code: https://github.com/redis/redis/blob/5.0/src/replication.c#L876-L930

void sendBulkToSlave(aeEventLoop *el, int fd, void *privdata, int mask) {
    // 此处省略部分源码

    // rdb 文件已完全发送给 slave 
    if (slave->repldboff == slave->repldbsize) {
        close(slave->repldbfd);
        slave->repldbfd = -1;
        aeDeleteFileEvent(server.el,slave->fd,AE_WRITABLE);
        putSlaveOnline(slave);
    }
}

void putSlaveOnline(client *slave) {
    slave->replstate = SLAVE_STATE_ONLINE;
    slave->repl_put_online_on_ack = 0;
    slave->repl_ack_time = server.unixtime; /* Prevent false timeout. */
    if (aeCreateFileEvent(server.el, slave->fd, AE_WRITABLE,
        sendReplyToClient, slave) == AE_ERR) {
        serverLog(LL_WARNING,"Unable to register writable event for replica bulk transfer: %s", strerror(errno));
        freeClient(slave);
        return;
    }
    refreshGoodSlavesCount();
    serverLog(LL_NOTICE,"Synchronization with replica %s succeeded",
        replicationGetSlaveName(slave));
}

Here, the slave state  SLAVE_STATE_ONLINEwill be repl_put_online_on_ackchanged to 0, (are you familiar with it, by the way, it is clientInstallWriteHandlerthe content judged above). At the same time, a callback function sendReplyToClientwill be set to send all the write operations in the output buffer created by the master for the slave to the slave. At the same time, the change of the slave state will make subsequent write operations on the master to be pushed to the slave normally (directly, without going through the output buffer).

Summarize

This time, we  client-output-buffer-limitlearned about its usage scenarios through the parameters, and focused on a simple source code analysis of the output buffer writing situation during master-slave synchronization. That's all for today's study, we'll continue it another day.

reference content

1. redis client-output-buffer-limit setting - Piao Piao Xue - Blog Garden

Author: Max Fang

Source: Understanding Redis client-output-buffer-limit parameters and source code analysis - immaxfang - 博客园