Troubleshooting and solutions for Redis Cluster clusters

1. Introduction

1.1 Introduction

Redis is an open source memory data structure storage system that can be used as a database, cache and message middleware. It supports a variety of data structures (such as strings, hash tables, lists, sets, etc.) and functions (such as transactions, distributed locks, Lua scripts, etc.), and can meet the needs of different scenarios.

1.2 Redis Cluster

Redis Cluster is a distributed solution provided by Redis, which mainly consists of the following components:

• A cluster of multiple nodes (each node operates independently)
• Routing layer (each cluster node will keep a copy of the entire cluster node state table, and route access requests by compiling the state table information of each node in the cluster and using the CRC16 algorithm to hash the Key and calculate the node ID)
• Data replication (data synchronization between nodes to ensure data consistency)

2. Cluster troubleshooting

2.1 Fault classification and cause analysis

• cluster unreachable

  Possible Causes:

  • network failure
  • Cluster node failure
  • Cluster configuration error

• Cluster node goes offline

  Possible Causes:

  • Node down or reboot
  • An error occurred in the network connection

• Cluster data is missing or inconsistent

  Possible Causes:

  • Data loss due to node downtime
  • The network transmission is abnormal, resulting in data inconsistency

2.2 Troubleshooting process

If the Redis Cluster cluster fails, you can follow the steps below to troubleshoot step by step:

1. Check the log files to see which node in the cluster failed what.
2. CLUSTER INFOExecute or command on the faulty node INFOto check whether the status information of the node (node ​​ID, slot allocation, load status, etc.) is normal.
3. Execute CLUSTER NODESthe command to view the status information of all nodes in the entire cluster, and restart or remove the offline nodes immediately.
4. Use CLUSTER RESET(use with caution) or CLUSTER FAILOVERcommand to manually switch between master and slave.
5. Analyze the logs to find problems. If data inconsistencies are found, they can be CLUSTER REPLICATEresolved by executing commands.
6. Modify the configuration file and restart the node.

The above are the general steps for Redis Cluster troubleshooting, and the specific troubleshooting method needs to be determined according to the actual situation. At the same time, in the production environment, it is also very important to ensure the stability of data backup and monitoring systems.

3. Cluster failure solution

Node state recovery

Redis active/standby switching

• Redis master/standby switching is implemented based on Redis sentinel. When Sentinel detects that the running master node is down, it will elect a new master node and point other slave nodes to the new master node. At the same time, Sentry can also monitor the health status of the master node.

  //Java实现Redis主备切换示例代码
  JedisSentinelPool redisSentinelPool = new JedisSentinelPool(masterName, sentinels,
  jedisPoolConfig);
  Jedis jedis = redisSentinelPool.getResource();
  jedis.set("foo", "bar");
  String value = jedis.get("foo");
  

Node data synchronization

• When the master node goes offline, the new master node will synchronize data from other slave nodes to keep the data consistent. At this time, if too many slave nodes need to be synchronized, it may cause synchronization blocking. In order to solve this problem, new nodes can be added to share the synchronization pressure.

Node expansion

Expansion principles and methods

• When expanding Redis Cluster, the following principles need to be followed:

  • Try to choose an even number of nodes to achieve a balance between the number of active and standby nodes.
  • The new node needs to be on the same subnet as the existing node. Otherwise, it will lead to communication across network segments and increase network delay.

  There are two specific methods for capacity expansion:

  • add node

  • Expand capacity by changing node configuration, for example: modify maxmemoryorport

    //Java添加新节点示例代码
    Jedis jedis = new Jedis("127.0.0.1", 6379);
    jedis.clusterMeet("192.168.1.100", 6379);
    jedis.clusterAddSlots(0, 16383);
    

Node state synchronization after expansion

• After the node expansion is completed, you need to execute reshardthe command in the Cluster environment to add the new node to the hash slot, and make the number of hash slots of each node as balanced as possible. At the same time, a command needs to be executed on each node replicatefor the new node to sync all the data.

  //Java执行reshard命令示例代码
  Jedis jedis = new Jedis("127.0.0.1", 6379);
  jedis.clusterSetSlotImporting(nodeId, slotId, targetNodeId);
  jedis.clusterSetSlotMigrating(nodeId, slotId, targetNodeId);
  jedis.clusterSetSlotNode(slotId, targetNodeId);
  jedis.clusterDelSlot(nodeId, slotId);
  

4. Redis Cluster High Availability Solution

4.1 Implementation of multi-active architecture

The multi-active architecture refers to the deployment of Redis Cluster clusters in multiple data centers (or computer rooms). Through the Redis Cluster Replication (master-slave replication) mechanism, data synchronization between Redis Cluster nodes in each data center can be performed in real time. In this way, in some special cases, such as a certain data center fails to work normally, the Redis Cluster nodes in other data centers can still continue to provide external services, thus achieving high availability of the entire Redis Cluster cluster.

4.2 Redis Cluster cluster and distributed lock implementation

Distributed lock (Distributed Lock) is a typical distributed system problem, and Redis Cluster has unique advantages in the implementation of distributed lock. In Redis Cluster, you usually use open source components such as Redlock or Redisson, and use the high availability and high performance characteristics of Redis Cluster to easily realize the function of distributed locks. In Redis Cluster, we can use the SET KEY VALUE NX PX command to realize the locking operation of the distributed lock, and use the EVAL script 1 key value command to execute the Lua script to realize the unlocking of the distributed lock.

5. Operation and maintenance management

5.1 Redis cluster monitoring

Redis cluster is a very complex system, so the entire life cycle of Redis cluster must be fully monitored in practical applications. In order to give you a more intuitive understanding, here is a brief list of the important monitoring indicators of the Redis cluster:

• Redis Cluster node status (including node availability and replication status)
• Redis Cluster overall performance (including request response time, throughput and other indicators)
• Redis Cluster memory usage
• Error logs and information about connecting with clients

5.2 Redis security settings

The security of Redis in operation and maintenance management is very important, especially when storing some sensitive data. Therefore, you need to pay attention to the following points when using Redis:

• Configure Redis access password
• Configure the IP whitelist of Redis Cluster
• Disable dangerous commands (such as FLUSHALL and FLUSHDB)
• Configure the persistence mode of Redis (such as AOF or RDB) to ensure data security

6. Application practice

6.1 Rational use of Redis Cluster cluster capabilities

Redis Cluster has high performance while ensuring high availability, so the capabilities of Redis Cluster can be fully utilized in applications. For example, we can use Redis Cluster to carry the data of various message queues, counters, current limiters and other special scenarios, so that the application logic is simpler and the performance is higher.

6.2 Performance optimization for business scenarios

In actual application scenarios, it is very necessary to optimize the performance of Redis Cluster according to different business requirements. For example, when the amount of writing is large, you can consider splitting the database to reduce pressure, optimizing the Redis Cluster storage structure to improve read and write performance, etc. In addition, advanced features such as Redis Cluster's Pipeline can be used to further optimize performance.