If you want to use data in large-scale distributed storage systems require high concurrent data evenly distributed scalable so consistency hash algorithm can solve this problem perfectly consistent hash algorithm again in many areas cache hadoop ES distributed database
Consistency principle Hash Algorithm
Consistency Hash algorithm using modulo method, consistency Hash algorithm is modulo 2 32 side. That is, the consistency of the entire Hash algorithm Hash space into a virtual ring, a Hash function value space of 0 to 2 ^ 32--1 (a 32-bit unsigned integer), the entire hash ring as follows: hash value is non-negative integer, for example, a property of the cluster node name hash value takes place on the ring, taking a hash value of the key data is also placed on the ring, find clockwise into its nearest node on it, the whole ring organization in the clockwise direction, the first point representative of the right ring 0:00 n1 server, and so on. We will carry on each server using a hash Hash, you can select specific IP or hostname of the server as a key hash, so each server to determine the location of a hash of the ring, for example, we have three machines, use after hashing the IP address in the loop spatial position as shown:
Key data using the same hash function Hash value is calculated, and determines the position of the data on the ring, find a clockwise direction from the position of the ring, which is to be targeted to the server server encountered.
FIG following three data O1, O2, O3 after hashing, the space position on the ring is as follows: O1 -> n1 O2 -> n2 O3 -> n3
Fault tolerance and scalability consistency Hash Algorithm
Now, suppose we n3 is down, we can see from the figure, n1, n2 will not be affected, only the object O3 is relocated to n1. So we find that consistency Hash algorithm, if one server is unavailable, only the affected data to its server data between a server before other ring space will not be affected. as the picture shows:
Now our system adds a server n4, as shown in Figure
Hash algorithm to increase or decrease the consistency of the nodes are only a small portion of the data re-positioning loop space, good fault tolerance and scalability. In the case where the serving node identity Hash Algorithm too easily because the uneven distribution of the node cause data inclined plane (most cached objects cached on a server) problem, as shown:
Then we found a large dataset on node A, and node B only a small amount of data. In order to solve the problem of data skew, consistency Hash algorithm introduces virtual node mechanism that calculates a plurality of hash for each server node, each calculated results are placed in a location service node, called a virtual node. No specific operation may be added to the server host name or IP to achieve, as shown in FIG implemented: for example, a node n1 our virtual virtual nodes 100, the virtual node on the ring is the same token n2, n3 100 also has a virtual node,
When data is placed over how to determine to which server it
When the data over the first ring node to find the corresponding virtual, and then find the corresponding virtual server nodes, so by adding the virtual node may achieve uniform distribution of the data, the more uniform the more virtual node data, a server generally we placed 200 virtual nodes can be
Data localization algorithm change, only need to add a step: a virtual node mapping to the actual point. So, after adding virtual nodes, even in the rare cases service node, it can be done uniformly distributed data. The above situations are evenly distributed over the case of data, there are actually a Hash algorithm consistent data skew problem
Algorithm interface class
public interface IHashService {
Long hash(String key);
}
复制代码Algorithm interface implementation class
public class HashService implements IHashService {
/**
* MurMurHash算法,性能高,碰撞率低
*
* @param key String
* @return Long
*/
public Long hash(String key) {
ByteBuffer buf = ByteBuffer.wrap(key.getBytes());
int seed = 0x1234ABCD;
ByteOrder byteOrder = buf.order();
buf.order(ByteOrder.LITTLE_ENDIAN);
long m = 0xc6a4a7935bd1e995L;
int r = 47;
long h = seed ^ (buf.remaining() * m);
long k;
while (buf.remaining() >= 8) {
k = buf.getLong();
k *= m;
k ^= k >>> r;
k *= m;
h ^= k;
h *= m;
}
if (buf.remaining() > 0) {
ByteBuffer finish = ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN);
finish.put(buf).rewind();
h ^= finish.getLong();
h *= m;
}
h ^= h >>> r;
h *= m;
h ^= h >>> r;
buf.order(byteOrder);
return h;
}
}
复制代码Simulation machine node
public class Node<T> {
private String ip;
private String name;
public Node(String ip, String name) {
this.ip = ip;
this.name = name;
}
public String getIp() {
return ip;
}
public void setIp(String ip) {
this.ip = ip;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
/**
* 使用IP当做hash的Key
*
* @return String
*/
@Override
public String toString() {
return ip;
}
}
复制代码Consistency Hash operation
public class ConsistentHash<T> {
// Hash函数接口
private final IHashService iHashService;
// 每个机器节点关联的虚拟节点数量
private final int numberOfReplicas;
// 环形虚拟节点
private final SortedMap<Long, T> circle = new TreeMap<Long, T>();
public ConsistentHash(IHashService iHashService, int numberOfReplicas, Collection<T> nodes) {
this.iHashService = iHashService;
this.numberOfReplicas = numberOfReplicas;
for (T node : nodes) {
add(node);
}
}
/**
* 增加真实机器节点
*
* @param node T
*/
public void add(T node) {
for (int i = 0; i < this.numberOfReplicas; i++) {
circle.put(this.iHashService.hash(node.toString() + i), node);
}
}
/**
* 删除真实机器节点
*
* @param node T
*/
public void remove(T node) {
for (int i = 0; i < this.numberOfReplicas; i++) {
circle.remove(this.iHashService.hash(node.toString() + i));
}
}
public T get(String key) {
if (circle.isEmpty()) return null;
long hash = iHashService.hash(key);
// 沿环的顺时针找到一个虚拟节点
if (!circle.containsKey(hash)) {
SortedMap<Long, T> tailMap = circle.tailMap(hash);
hash = tailMap.isEmpty() ? circle.firstKey() : tailMap.firstKey();
}
return circle.get(hash);
}
}
复制代码Test category
public class TestHashCircle {
// 机器节点IP前缀
private static final String IP_PREFIX = "192.168.0.";
public static void main(String[] args) {
// 每台真实机器节点上保存的记录条数
Map<String, Integer> map = new HashMap<String, Integer>();
// 真实机器节点, 模拟10台
List<Node<String>> nodes = new ArrayList<Node<String>>();
for (int i = 1; i <= 10; i++) {
map.put(IP_PREFIX + i, 0); // 初始化记录
Node<String> node = new Node<String>(IP_PREFIX + i, "node" + i);
nodes.add(node);
}
IHashService iHashService = new HashService();
// 每台真实机器引入100个虚拟节点
ConsistentHash<Node<String>> consistentHash = new ConsistentHash<Node<String>>(iHashService, 500, nodes);
// 将5000条记录尽可能均匀的存储到10台机器节点上
for (int i = 0; i < 5000; i++) {
// 产生随机一个字符串当做一条记录，可以是其它更复杂的业务对象,比如随机字符串相当于对象的业务唯一标识
String data = UUID.randomUUID().toString() + i;
// 通过记录找到真实机器节点
Node<String> node = consistentHash.get(data);
// 再这里可以能过其它工具将记录存储真实机器节点上，比如MemoryCache等
// ...
// 每台真实机器节点上保存的记录条数加1
map.put(node.getIp(), map.get(node.getIp()) + 1);
}
// 打印每台真实机器节点保存的记录条数
for (int i = 1; i <= 10; i++) {
System.out.println(IP_PREFIX + i + "节点记录条数：" + map.get(IP_PREFIX + i));
}
}
}
复制代码Results are as follows: