redis partition
The benefits of partitioning
1. Allows us to construct larger databases by utilizing the sum of the memory of multiple computers.
2. Through multi-core and multiple computers, it allows us to expand computing power; through multiple computers and network adapters, it allows us to expand network bandwidth.
Disadvantages of Partitioning
1. Operations involving multiple keys are generally not supported. For example, when two sets are mapped to different redis instances, you cannot perform the intersection operation on the two sets.
2. Redis transactions involving multiple keys cannot be used.
3. When using partitions, data processing is more complicated, for example, you need to process multiple rdb/aof files, and backup persistent files from multiple instances and hosts.
4. Adding or removing capacity is also more complicated. Most redis clusters support the ability to add and delete nodes at runtime with transparent data balancing, but other systems like client partitions, proxies, etc. do not support this feature. However, a technique called pre-sharding can help.
Pre-Sharding
Through the above introduction, we know that Redis partitioning is problematic. Unless we just use Redis as a cache, it is very troublesome to add machines or delete machines.
However, usually our Redis capacity changes are very common in practical applications. For example, if I need 10 Redis machines today, I may need 50 machines tomorrow.
Pre-Sharding Principle
Given that Redis is a very lightweight service (each instance only occupies 1M), a simple solution to the above problem is:
We can start multiple Redis instances, even though it is a physical machine, we can start multiple instances at the beginning. We can choose some instances from it, say 32 or 64 instances to be our working cluster. When one physical machine does not have enough storage, we can move the general instance to our second physical machine, and then we can ensure that the number of Redis instances in the cluster remains unchanged, and we can achieve the purpose of expanding the machine.
How to move Redis instance?
When we need to move the Redis instance to a separate machine, we can do it through the following steps:
Simple
1. Run Redis-Server on the new physical machine;
2. The Redis-Server belongs to a Redis-Server in the (slaveof) shard list (assuming it is called RedisA);
3. After the master-slave replication (Replication) is completed, change the IP and port of RedisA in the client shard list to the IP and port of Redis-Server on the new physical machine;
4. Stop RedisA.
detailed
1. Start a new Redis instance on the new physical machine.
2. Use the new physical machine as the slave machine to be moved.
3. Stop the client.
4. Update the IP address of the Redis instance that will be moved.
5. Send the SLAVEOF ON ONE command to the slave machine.
6. Start the Redis client with the new IP.
7. Shut down the Redis instance that is no longer in use.
method of partition
1. Range partitioning . Partitioning by range is to map a certain range of objects to a specific Redis instance, but there must be a mapping table from the range to the instance. This table needs to be managed, and also needs a mapping table of various objects, which is usually not a good method for Redis
Hash partition. Applies to any key, and does not need to be object_name: this form
2. Convert the key to a number with a hash function, such as using the crc32 hash function . Executing crc32(foobar) on key foobar will output an integer like 93024922
Modulo this integer, convert it to a number between 0-3, and you can map this integer to one of four Redis instances. 93024922 % 4 = 2, which means the key foobar should be stored in the R2 instance