This article was first published on the official account: More AI (power_ai), welcome to pay attention, programming and AI dry goods will be delivered in time!
If you have n cache servers, a common load balancing method is to use the following hashing method:
server index = hash(key) % N , where N is the size of the server pool.
Let's go through an example to illustrate how this works. As shown in Table 5-1, we have 4 servers and 8 string keys and their hashes.
To get the server that stores a key, we perform the modulo operation f(key) % 4 . For example, hash(key 0) % 4 = 1 means that the client must contact server 1 to get the cached data. Figure 5-1 shows the distribution of keys based on Table 5-1.
This approach works well when the pool of servers is a fixed size and the data is evenly distributed. However, problems arise when new servers are added, or existing servers are removed. For example, if server 1 goes offline, the server pool size becomes 3. Using the same hash function, we get the same hash value for the key. But applying the modulo operation will result in a different server index because the number of servers is reduced by 1. The results obtained by applying hash% 3 are shown in Table 5-2:
Figure 5-2 shows the new key distribution based on Table 5-2.
As shown in Figure 5-2, most keys are reassigned, not just those originally stored on the offline server (Server 1). This means that when server 1 is offline, most cache clients will connect to the wrong server for data. This caused a storm of cache misses. Consistent hashing is an effective technique to alleviate this problem.
consistent hashing
Quote from Wikipedia: "Consistent hashing is a special kind of hashing such that when the size of the hash table changes and using consistent hashing, on average only k/n keys need to be remapped, where k is the number of keys where n is the number of slots. In contrast, in most traditional hash tables, a change in the number of array slots results in almost all keys needing to be remapped [1]".
Hash spaces and hash rings
Now that we understand the definition of consistent hashing, let's understand how it works. Assuming that SHA-1 is used as the hash function f, the output range of the hash function is: x0, x1, x2, x3, …, xn . In cryptography, the hash space of SHA-1 is from 0 to 2^160 - 1. That is, x0 corresponds to 0, xn corresponds to 2^160 - 1, and all other hash values fall between 0 and 2^160 - 1. Figure 5-3 shows the hash space.
By connecting the two ends, we get a hash ring as shown in Figure 5-4:
hash server
Using the same hash function f, we map servers to rings based on their IP or name. Figure 5-5 shows four servers being mapped to the hash ring.
hash key
It is worth mentioning that the hash function used here is different from the one in the "rehashing problem", and there is no modulo operation. As shown in Figure 5-6, 4 cache keys (key0, key1, key2, and key3) are hashed onto the hash ring.
server lookup
To determine which server a key is stored on, we search clockwise from the key position on the ring until we find a server. Figure 5-7 illustrates this process. Clockwise, key 0 is stored on server 0 ; key1 is stored on server 1 ; key2 is stored on server 2 ; key3 is stored on server 3 .
add server
Using the above logic, adding a new server only requires reassigning a subset of keys.
In Figure 5-8, after adding server 4 , only key0 needs to be reassigned. k1, k2, and k3 are still on the same server. Let's take a closer look at this logic. key0 was stored on server 0 until server 4 was added . Now, key0 will be stored on server 4 because server 4 is the first server it encounters clockwise from key0's location on the ring . Other keys do not need to be redistributed according to the consistent hashing algorithm.
remove server
When a server is removed, only a small number of keys need to be redistributed through consistent hashing. In Figure 5-9, when server 1 is removed, only key1 must be mapped to server 2 . The rest of the keys are unaffected.
Two problems in the basic method
The consistent hashing algorithm was proposed by Karger et al. of MIT [1]. The basic steps are as follows:
- Map servers and keys onto the ring using a uniformly distributed hash function.
- To find out which server a key maps to, go clockwise from the key location to the first server on the ring.
There are two problems with this approach. First, it is not possible to maintain the same size partitions for all servers on the ring, given that servers may be added or removed. A partition is a hash space between adjacent servers. The partition size on the ring to which each server is assigned can be very small or quite large. In Figure 5-10, if s1 is removed, the partition of s2 (highlighted by the double-headed arrow) is twice as large as the partitions of s0 and s3 .
Second, the bond distribution on the ring may be non-uniform. For example, if the servers map to the locations listed in Figure 5-11, most of the keys are stored on server 2 . However, server 1 and server 3 do not have any data.
A technique known as virtual nodes or replicas is used to solve these problems.
virtual node
A virtual node refers to an actual node, and each server is represented by multiple virtual nodes on the ring. In Figure 5-12, both server 0 and server 1 have 3 virtual nodes. This 3 is chosen arbitrarily; in real systems, the number of virtual nodes is much higher. Instead of using s0 , we use s0_0, s0_1 , and s0_2 to represent server 0 on the ring . Likewise, s1_0, s1_1 and s1_2 represent server 1 on the ring . With virtual nodes, each server is responsible for multiple partitions. The partition (edge) with label s0 is managed by server 0 . On the other hand, the partition labeled s1 is managed by server 1 .
To find out on which server a key is stored, we go clockwise from the key's location to the first virtual node encountered on the ring. In Figure 5-13, to find out which server k0 is stored on, we find the virtual node s1_1 clockwise from the location of k0 , which points to server 1 .
As the number of virtual nodes increases, the distribution of keys becomes more even. This is because as the number of virtual nodes increases, the standard deviation becomes smaller, resulting in a balanced data distribution. Standard deviation measures how spread out the data is. The results of an experiment in online research [2] show that when there are one hundred or two hundred virtual nodes, the standard deviation is between 5% (200 virtual nodes) and 10% (100 virtual nodes) of the mean. When we increase the number of virtual nodes, the standard deviation becomes smaller. However, we need more space to store the data of virtual nodes. This is a tradeoff, and we can adjust the number of virtual nodes to suit our system needs.
find the affected keys
When adding or removing a server, some data needs to be redistributed. How do we find the affected ranges to reassign keys?
In Figure 5-14, server 4 is added to the ring. The affected range starts at s4 (the newly added node) and moves counterclockwise until a server is found ( s3 ). Therefore, keys located between s3 and s4 need to be reassigned to s4 .
When a server ( s1 ) is removed as shown in Figure 5-15, the affected range starts at s1 (the removed node) and moves counterclockwise around the circle until a server ( s0 ) is found. Therefore, keys located between s0 and s1 must be reassigned to s2 .
Summarize
In this chapter, we discuss consistent hashing in depth, including why it is needed and how it works. The benefits of consistent hashing include:
- Minimize redistribution of keys when servers are added or removed.
- Because the data is more evenly distributed, it is easier to scale out.
- Alleviate hotkey issue. Excessive access to specific shards can lead to server overload. Imagine Katy Perry, Justin Bieber, and Lady Gaga all on the same shard. Consistent hashing alleviates this problem by distributing data more evenly.
Consistent hashing is widely used in real-world systems, including some notable ones:
- Partitioning component of Amazon's Dynamo database [3]
- Data partitioning across clusters in Apache Cassandra [4]
- Discord chat application[5]
- Akamai Content Delivery Network[6]
- Maglev Network Load Balancer [7]
Congratulations on getting this far! Give yourself a thumbs up now. here you go!
References
[1] Consistent hashing: https://en.wikipedia.org/wiki/Consistent_hashing
[2] Consistent hashing:
https://tom-e-white.com/2007/11/consistent-hashing.html
[3] Dynamo: Amazon's highly available key-value store:
https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
[4] Cassandra - a decentralized structured storage system:
http://www.cs.cornell.edu/Projects/ladis2009/papers/Lakshman-ladis2009.PDF
[5] How to scale Discord Elixir to 5 million concurrent users:
https://blog.discord.com/scaling-elixir-f9b8e1e7c29b
[6] CS168: Modern Algorithm Toolbox Lesson 1: Introduction and Consistent Hashing: http://theory.stanford.edu/~tim/s16/l/l1.pdf
[7] Maglev: A Fast and Reliable Software Network Load Balancer:
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44824.pdf
Hello, I am Shisan, a veteran driver who has been developing for 7 years, and a foreign company for 5 years in the Internet for 2 years. I can beat Ah San and Lao Mei, and I have also been ruined by PR comments. Over the years, I have worked part-time, started a business, took over private work, and mixed upwork. Made money and lost money. Along the way, my deepest feeling is that no matter what you learn, you must keep learning. As long as you can persevere, it is easy to achieve corner overtaking! So don't ask me if it's too late to do what I do now. If you still have no direction, you can follow me [public account: More AI (power_ai)], where I will often share some cutting-edge information and programming knowledge to help you accumulate capital for cornering and overtaking.