Redis cluster with distributed introduction and set up Redis-Cluster

1 Redis Cluster

1.1 What is a cluster

Cluster is a network of many servers. It refers to the concentration of multiple servers together to achieve the same business .

1.2 Why cluster

One server can not meet the needs of the development, the need for multiple servers to support. This time we need to do cluster, but the cluster is often accompanied

The distributed.

1.3 features and the ability to cluster

1, a cluster of two key features:

Scalability Performance may be limited to a single cluster ---- service entity, the new service entity may be dynamically added to the cluster, thereby enhancing the performance of the cluster. Dynamically added server
high availability through clustering ---- redundant service for the client entity from easily met out of service warning. In a cluster, the same service can be provided by multiple service entities. If a service entity fails, another service entity will take over the service entity failure. Clusters provide service from a wrong entity to restore function to another service entity enhances application availability
2, a cluster of two capabilities:

---- Load balancing can load balance tasks are evenly distributed computing and network resources in a clustered environment.

Error Recovery ---- For some reason, the resources to perform a task fails, the other service resources to perform the same task entity then complete the task. This can not work because of an entity's resources to another entity resources to complete the task of continuing transparent process is called error recovery.

2 Distributed

The different services are distributed in different places, separate web applications and database services, the different business do not get anything different clusters.

Distributed features:

　　1. prevents single point of failure

　　2. The concurrent treatment of high (too many requests, a server can not handle)

　　3. processing large amounts of data (too much memory data, a server to handle, however)

3 Redis cluster scheme

3.1 Master-slave replication

Master-slave synchronization, separate read and write, standby switching.

principle:

• Connecting the main server from the server, transmits SYNC command;
• After receipt of the SYNC master server name, begin BGSAVE RDB command file and generates all the write command buffer after execution of the recording;
• Master BGSAVE executed, all sent from the server to the snapshot file, and continue to record write command is executed during transmission;
• After receiving the snapshot files from the server to discard all the old data, after loading snapshot received;
• Began to transmit buffer write commands from the server after the primary server snapshot has been sent;
• Loaded from the server to complete the snapshot start command request received, and performs a write buffer command from the host server; (from the server initialization completion)
• Each master server will execute a write command in the same write commands sent from the server, receive and execute the write command received (after the completion of the operation from the server initialization) from the server

advantage:

• Support replication master-slave, the host will automatically synchronize data to the slave, may be separate read and write

Disadvantages:

• Redis is more difficult to support online capacity expansion, online capacity expansion reaches the upper limit will become very complicated in the cluster capacity.
• Redis not have automatic fault tolerance and recovery, from a host machine downtime will result in the distal end portion of the read or write request fails, or the need to wait for the machine to restart the manual switching to restore the front end IP.
• Host down, down the front part of the data has failed to synchronize to a slave, also introduce data inconsistencies after switching the IP, reduces the availability of the system.

3.2 sentry mode

Automated system monitoring and fault recovery.

Advantages: Sentinel model is based on a master-slave mode, the master from all the advantages of having a Sentinel model. Can automatically switch from the main, the system is more robust, more available.

Disadvantages: expansion more difficult, more service space

3.3 Redis-Cluster Cluster

3.3.1 What is the Redis-Cluster

Redis Sentinel model basically been highly available, separate read and write, but in this mode redis each server stores the same data, it is wasteful of memory, so the added cluster mode on redis3.0, to achieve the redis distributed storage, which means that on each node redis different memory contents.

Why build Redis cluster. Redis data is stored in memory, and the memory of our computers are generally not large, which means that Redis is not suitable for large data storage for storing large data is Hbase Hadoop ecosystem or MogoDB. Redis is more suitable for highly concurrent processing, storage capacity of one device is very limited, but multiple devices to collaborate, you can let the memory many times larger, which requires use of the cluster.

There are a variety of ways to build Redis cluster, for example using a client patch, Twemproxy, Codis and so on, but after redis 3.0 version supports redis-cluster cluster, it is the solution Redis Crown, Redis-Cluster-free center structure, save data for each node and the entire state of the cluster, each node and all other nodes. Redis-cluster architecture which is shown below:

 

 

 

Redis client node is connected, without an intermediate layer proxy. The client need not be connected to any available cluster node is connected to all the nodes in the cluster.

All nodes are interconnected and redis (PING-PONG mechanism), the internal transmission speed using the binary protocol and bandwidth optimization.

3.3.2 distributed storage mechanism - groove

redis-cluster mapping all of the physical node to [0-16383] the slot, cluster responsible for maintaining.

16384 clusters built Redis hash slot, when a key-value to be placed in the cluster Redis, Redis crc16 algorithm key using the first calculated result, and the results of the remainder number of 16384, such that each key will correspond to a No. 0-16383 hash between the grooves, redis will be substantially equal to the hash slot mapped to different nodes according to the number of nodes.

For example three nodes: the value of the distribution channel is as follows:

SERVER1: 0-5460

SERVER2: 5461-10922

SERVER3: 10923-16383

3.3.3 Fault Tolerance - Voting

1) the electoral process is the master of all involved in the cluster, if more than half of the master node to communicate with the failure of more than (cluster-node-timeout), believes that a node failure, automatically triggers a failover operation. Failed node corresponding node automatically upgrade from the main node

(2) When the entire cluster is unavailable (cluster_state: fail)?

If the cluster master hang arbitrary, and is not the current master Slave. Enter the cluster fail state, as may be appreciated that slot cluster mapping [0-16383] enters fail state when not completed.

 

 4 integrated environment to build

4.1 Preparations

(1) Redis 3.2

Redis need six servers. Build a pseudo-cluster.

Redis need six instances.

You need to run at different ports 6379-6384

(2) Ruby language runtime environment that we need to use the ruby ​​script to build clusters

Ruby, a quick and easy object-oriented (Object Oriented Programming) scripting language, in the 1990's by the Japanese Yukihiro Matsumoto (Yukihiro Matsumoto) development, compliance with the GPL and Ruby License. It was inspired and features from Perl, Smalltalk, Eiffel, Ada, and Lisp language. By the Ruby language itself has developed a JRuby (Java platform), IronRuby (.NET platform) and other platforms Ruby language alternatives. Ruby author on February 24, 1993 to start writing Ruby, until December 1995 was officially released to the public fj (newsgroups). Because Perl pronunciation and June birthstone pearl (pearl), and thus to Ruby ruby ​​birthstone of July (ruby) name.

Ruby (3) RubyGems referred gems, Ruby is a component for a packaging system for packaging

(4) Redis of Ruby driver redis-xxxx.gem

(5) create Redis cluster tool redis-trib.rb

4.2 began to build

(1) copy of the server 6, and the port number in turn to 6379,6380,6381,6382,6383,6384.

(2) set the following properties redis.windows.conf profile:

cluster-enabled yes

cluster-config-file nodes-6379.conf

cluster-node-timeout 15000

appendonly yes

(3) write the startup script startup.bat, says:

title redis-6379

redis-server.exe redis.windows.conf

(4) install Ruby, redis clusters need to use.

(5) Rudy Redis installed drivers redis-xxxx.gem

Download: Rubygems official Download

下载zip文件，下载后解压。当前目录切换到解压目录中，如 D:\redis-cluster\rubygems-3.0.6\rubygems-3.0.6然后命令行执行 ruby setup.rb

再用 GEM 安装 Redis ：切换到redis安装目录，需要在命令行中，执行 gem install redis

 

 

 （6）启动每个节点并执行集群构建脚本，点击每个节点start.bat进行启动，拷贝redis-trib.rb到6379的redis节点。

执行命令：

redis-trib.rb create --replicas 1 127.0.0.1:6379 127.0.0.1:6380 127.0.0.1:6381 127.0.0.1:6382 127.0.0.1:6383 127.0.0.1:6384

在出现 Can I set the above configuration? (type ‘yes’ to accept): 请确定并输入 yes 。

4.3 集群环境测试

4.3.1 命令测试

• redis-cli –c –h ”地址” –p “端口号” ; c 表示集群
• cluster info 查看集群的信息
• info replication 查看当前服务器的信息
• cluster nodes 查看各个节点分配slot

4.3.2 java代码测试

@Test
    public void testCluster() throws IOException, InterruptedException {
        Set<HostAndPort> nodes = new HashSet<>();
        //添加服务器
        nodes.add(new HostAndPort("127.0.0.1", 6379));
        nodes.add(new HostAndPort("127.0.0.1", 6380));
        nodes.add(new HostAndPort("127.0.0.1", 6381));
        nodes.add(new HostAndPort("127.0.0.1", 6382));
        nodes.add(new HostAndPort("127.0.0.1", 6383));
        nodes.add(new HostAndPort("127.0.0.1", 6384));
        JedisCluster cluster = new JedisCluster(nodes);
        try {
            String res = cluster.get("name");
            System.out.println(res);
//            cluster.quit();
        } catch (Exception e) {
            e.printStackTrace();
//            cluster.quit();
        }
    }
}

1.1.1.   分布存储机制-槽