In a distributed storage system, data needs to be distributed and stored on multiple devices. Sharding is a technique used to determine the distribution of data on multiple storage devices. Data sharding has three purposes:
-
Uniform distribution, that is, the amount of data on each device should be as close as possible;
-
Load balancing, that is, the amount of requests on each device should be as close as possible;
-
Minimize data migration during expansion and shrinkage.
Data slicing method
Data sharding generally uses Key or Key hash value to calculate the distribution of Key. Several common methods of data sharding are as follows:
-
Divide the number segment. This is generally applicable to the case where the Key is an integer type. Each device stores the same size number range. For example, put the data with the Key [1, 10000] on the first device, and the Key is [10001, 20000 ] The data on the second device, and so on. This method is very simple to implement, and it is more convenient to expand the capacity. It is sufficient to increase the equipment exponentially. For example, if there are N devices, then add N devices to expand the capacity, and migrate half of the data from each old device to the new device. The device with the original number segment [1, 10000] only retains the data of the number segment [1, 5000] after the expansion, and migrates the data with the number segment [5001, 10000] to a newly added device. The disadvantage of this method is that the data may be unevenly distributed. For example, the data volume of the small size segment may be larger than the data size of the large size segment. The heat of the same number segment may also be different, resulting in unbalanced load of each device; and expansion It is also not flexible enough to only increase the equipment exponentially.
-
Modulo. This method first calculates the hash value of the Key, and then modulo the number of devices (Integer Key can also be modulo directly with Key), assuming there are N devices, numbered 0 ~ N-1, through Hash (Key) % N can determine the device number where the data is located. This method is also very simple to implement, the data distribution and load will be relatively uniform, and any number of devices can be added to expand the capacity. The main problem is that when the capacity is expanded, a large amount of data migration will occur. For example, from N devices to N + 1, most of the data will be migrated between the devices.
-
Search table. Store the mapping relationship between the key and the device in the search table. The data distribution can be determined by looking up the search table. The search table here can also be more flexible. You can store the mapping relationship for each key, or you can combine methods such as number segmentation. Reduce the capacity of the search table. In this way, the data can be evenly distributed, load balanced, and the amount of data migration can be reduced. The disadvantage is that the space required to store the search table may be relatively large, and in order to ensure that the amount of data migration caused by expansion and contraction is relatively small, the algorithm for determining the mapping relationship is also relatively complicated.
-
Consistent hash. Consistent Hashing (Consistent Hashing) is a distributed hashing (DHT) implementation algorithm proposed by the Massachusetts Institute of Technology in 1997. The design goal is to solve the hot spot problem in the Internet. The details of the method For the introduction, please refer to http://blog.csdn.net/sparkliang/article/details/5279393 . The algorithm of consistent hashing is simple and ingenious, it is easy to achieve uniform data distribution, and its monotonicity also ensures that the data migration of expansion and contraction is relatively small.
Through the above comparison, in this system, a consistent hash method is selected for data sharding.
virtual server
In order to make the system more scalable, the concept of storage layer VServer (virtual server) is proposed here. A VServer is a logical storage server and a storage unit of a distributed storage system. Multiple physical devices can be deployed. One VServer, one VServer supports one write process and multiple read processes.
Through the way of VServer, there will be the following benefits:
-
Improve stand-alone performance. In order not to introduce a complicated locking mechanism, a single-write process design is adopted. If a single machine has only one write process, the write concurrency capability will be limited, and the storage resources (memory, hard disk) on the single machine are divided into multiple storage units through VServer , So that multiple write processes can work at the same time, which greatly improves the stand-alone write concurrency capability.
-
Deployment scalability is better. The VServer method is very flexible in deployment. You can determine the number of VServers according to the resource situation of a single machine. Configure different VServer numbers for different models, so that different models can make full use of the resources on the machine, even in a system In using multiple models, the load of the machine can also be balanced.
Application of consistent hashing
Data sharding is implemented at the interface layer, the purpose is to divide the data evenly to different VServers. With the existence of the interface layer, the logical layer addressing is much lighter. The addressing storage layer VServer is all responsible for the interface layer. The logical layer only needs to randomly select an interface layer machine to access.
The interface layer uses a consistent hashing cut-ring algorithm to implement data sharding. In the cut-ring algorithm, in order to distribute data evenly to each VServer, each VServer needs to have multiple VNodes (virtual nodes). A key addressing process is shown in the following figure. First, find the corresponding VNode on the hash ring according to Hash (Key), and then determine the VServer to which it belongs according to the mapping table of VNode and VServer.
It can be known from the above search process that the distribution of VNode on the hash ring and the mapping relationship between VServer and VNode need to be calculated offline in advance. In order to make the calculation result universal, that is, a system with any number of VServers can use the result to obtain a consistent hash mapping table, which requires the result to be machine-independent, such as not using IP to calculate VNode. Hopefully. Before calculating, you need to determine the number of VNodes included in each VServer and the maximum number of VServers supported by a system. A simple method is similar to the method mentioned in the above link, but it cannot be related to IP. You can use the VServer and VNode numbers to calculate the hash value, such as Hash ("1 # 1"), Hash ("1 # 2 ") ... This method requires that the number of VNodes contained in a VServer is relatively large. It takes about 500 to make the data on each VServer more uniform. Of course, there are other ways to make a VServer contain a smaller number of VNodes, and make the data distribution deviation within a certain range.
Google has proposed a new consistent hash algorithm Jump Consistent Hash, this algorithm has zero memory consumption, even distribution, fast, and only 5 lines of code, the advantages are very obvious, detailed introduction can be found here http: //my.oschina. net / u / 658658 / blog / 424161 . Compared with the method described above, one of the biggest differences is that when expanding and redistributing data, in the above method, the data on a VNode of the new machine will only come from the VNode on an old machine, and this method It will come from VNode on all old machines. This problem may cause some design complications, so it should be carefully considered when using it.