redis Reconnecting,last destination was /ip:port

Table of contents

1. Problem background

2. Investigation process

2.1 The reasons for redis reconnection are as follows:

2.2 Troubleshooting ideas

2.3 Cause of the problem

3. Solution

1. Increase the maximum number of connections to the Redis server

2. Optimize the connection usage of the application

3. Consider using a connection pool to manage Redis connections.

4. The fastest and most convenient way is to assign different index libraries to different users.

1. Problem background

In the past few days, the program in our production environment has been reporting alarms. After checking the logs, I found that the strange thing is that redis reconnection always occurs during the program execution of scheduled tasks, and it occurs at irregular intervals. This makes me puzzled. Its solution.

2. Investigation process

The error content is: Reconnecting, last destination was /ip:port

The meaning of this error is: reconnection is in progress, and the last reconnection target is /ip:port

Error content

2.1 The reasons for redis reconnection are as follows:

Initial connection: When the initial connection is made to the Redis server, a handshake occurs and the connection is established.
Disconnection: If the connection to Redis is unexpectedly disconnected (such as a network failure), the Redis client will attempt to automatically reconnect.
Connection timeout: If the connection between the Redis client and the server times out, the client will try to reconnect.
Maximum connection limit: If the Redis server has reached the maximum connection limit and there are no available connection slots, new connection requests will be delayed until there are available connection slots.

2.2 Troubleshooting ideas

First we exclude the initial connection since the program has been running stably for some time.

Secondly, we exclude disconnection because the redis service can provide data normally only before and after the error is reported. ’

Secondly, I connected to redis through logs and redis client tools and the connection timeout was eliminated normally.

Finally, there is a maximum connection limit left. Because our system's business volume is not large, basically no other business will connect to redis, and this maximum limit is also excluded.

2.3 Cause of the problem

Unexpectedly, it was because of the "maximum connection limit" that we overlooked a problem: this redis has many users providing services to other users. Are you wondering what other users' affairs have to do with your system? The problem arises here,

Detailed explanation of the maximum number of redis limits:

当 Redis 服务器已达到最大连接数限制，并且没有可用的连接槽位时，新的连接请求会被延迟，直到有可用的连接槽位。

这是因为 Redis 服务器有一个默认的最大连接数限制。一旦达到该限制，任何新的连接请求都会处于排队状态，直到有旧的连接关闭或释放出连接槽位为止。

在排队等待期间，新的连接请求将被暂时挂起，而不会立即获得连接。这意味着连接请求可能会经历一段延迟，直到有可用的连接槽位为止。延迟的时间取决于系统的负载情况和连接的释放速度。

In the above explanation, we know that redis has a maximum limit. If it is exceeded, it will enter the queue waiting state.

The most important key points are:

在 Redis 中，不同的账号可以访问相同的库，而不是每个账号都指向独立的库。

Based on the above explanation, that is to say, no matter which account is used, when switching to the third database, they will point to the same database, not an independent library.

This means that if both Account A and Account B switch to a third database, they will share the same data. For modification or query operations performed by account A, account B can also see the same data.

Because the redis service we use also provides services to other platforms, and then this "other platform" directly reaches the redis service's maximum connection, which causes other connections to enter the waiting state, causing the program to report an error.

3. Solution

1. Increase the maximum number of connections to the Redis server

This can be achieved by modifying the maxclients parameter in the Redis configuration file. Note that increasing the maximum number of connections may impact server performance and resource consumption, so this needs to be weighed carefully.

2. Optimize the connection usage of the application

Ensure the timely release and reuse of connections to avoid continued long connections or invalid connections occupying connection slots.

3. Consider using a connection pool to manage Redis connections.

Connection pooling can help maintain a set of reusable connections between applications and Redis servers, reduce connection creation and release overhead, and improve connection utilization.