Redis basics (b)

Redis Affairs

redis in transaction is a set of commands, commands in the transaction either all executed or not executed, Redis by MULTI, DISCARD, EXEC WATCH and
to implement the transaction function four commands, multi indicate on things, exec denote something execution, after the implementation of exec returns the result of transaction execution, discard a waiver of all transaction execution order, cleared the transaction already in the queue and exit queue, watch for monitoring a given key, if the key is to modify other clients, will not It will execute the transaction.

127.0.0.1:6379> multi
OK
127.0.0.1:6379> set key 1
QUEUED
127.0.0.1:6379> get key
QUEUED
127.0.0.1:6379> exec
1) OK
2) "1"

127.0.0.1:6379> multi
OK
127.0.0.1:6379> set key1 1
QUEUED
127.0.0.1:6379> discard
OK
127.0.0.1:6379> get key1
(nil)

Here I modified the monitored key in another client, the client transaction does not lead to execution

127.0.0.1:6379> set key 1
OK
127.0.0.1:6379> watch key 
OK
127.0.0.1:6379> multi
OK
127.0.0.1:6379> incr key  
QUEUED
127.0.0.1:6379> incr key  #客户端2
(integer) 2
127.0.0.1:6379> exec
(nil)
127.0.0.1:6379> get key
"2"

Because the transaction will be executed in the exclusive server, so try to avoid performing too many commands in a transaction, so as not to clog servers

Redis Pipline

cs mode is a redis tcp server, use and the like http request response protocol. Client can initiate a connection request command through a plurality of socket. Each request command issued after the client usually blocks and waits redis service processing, redis dealt with the request command will result through a response packet back to the client. If the network latency is large, it will take too much time, redis provides pipline can solve this problem, redis can send multiple messages in pipline the process without waiting for a reply to each message.
Here I use python library of redis wrote a demo to demonstrate the effect of using pipline

from redis import Redis
import time
conn=Redis(host="60.205.177.100",port="6379")

def usepipline():
    start_time=time.time()
    pipline=conn.pipeline()
    for i in range(300):
        pipline.incr("key")
    pipline.execute()
    print("usepipline:",time.time()-start_time)
def withoutpipline():
    start_time=time.time()
    for i in range(300):
        conn.incr("key1")
    print("withoutpipline:",time.time()-start_time)
usepipline()
withoutpipline()

Response results

usepipline: 1.2412519454956055
withoutpipline: 7.2261717319488525

You can see the use of pipline effect is obvious

Redis publish subscription model

Redis is achieved by PUBLISH, SUBSCRIBE commands such as subscription and publishing model, the publisher can publish messages to multiple channels, subscribers can subscribe to multiple channels, of course, a channel can have multiple subscribers, publishers and subscribers this isolates may allow for greater scalability and more dynamic network topology.

command

Send a message to the channel

publish channel message

E.g

It returns the number of subscribers to the message received

127.0.0.1:6379> publish CCTV1 worldnews
(integer) 0
127.0.0.1:6379> publish CCTV1 chinanews
(integer) 0

Subscriptions

subscribe channel [channel ...]

E.g

127.0.0.1:6379> subscribe CCTV1 
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "CCTV1"
3) (integer) 1
1) "message"
2) "CCTV1"
3) "chinanews"

Unsubscribe Channel

UNSUBSCRIBE [channel [channel ...]]

E.g

127.0.0.1:6379> UNSUBSCRIBE CCTV1 CCTV2
1) "unsubscribe"
2) "CCTV1"
3) (integer) 0
4) "unsubscribe"
5) "CCTV2"
6) (integer) 0

Subscription model

redis supports glob way to subscribe to multiple channels

PSUBSCRIBE pattern [pattern ...]

E.g

127.0.0.1:6379> publish CCTV2 chinanew
(integer) 1
127.0.0.1:6379> publish CCTV1 worldnews
(integer) 1
127.0.0.1:6379> PSUBSCRIBE CCTV*
Reading messages... (press Ctrl-C to quit)
1) "psubscribe"
2) "CCTV*"
3) (integer) 1
1) "pmessage"
2) "CCTV*"
3) "CCTV2"
4) "chinanews"
1) "pmessage"
2) "CCTV*"
3) "CCTV2"
4) "chinanew"

Unsubscribe mode

PUNSUBSCRIBE [pattern [pattern ...]]

E.g

127.0.0.1:6379> PUNSUBSCRIBE CCTV*
1) "punsubscribe"
2) "CCTV*"
3) (integer) 0

python demo

redis_pub.py

from redis import Redis
import time
conn=Redis(host="60.205.177.100",port="6379")
def publish():  
    while True:       
        conn.publish("CCTV3","test")

redis_sub.py

from redis import Redis
import time
conn=Redis(host="60.205.177.100",port="6379")
def subscribe():
    subscribe=conn.pubsub()
    subscribe.subscribe('CCTV3')
    message=subscribe.parse_response()
    print(message)

Redis replication

Although there aof rdb do and persistence, but in order to prevent a single point of failure, which requires multiple copies of data replication to ensure data security

The basic features of replication redis

• Redis using asynchronous replication, and at a frequency of once per second acknowledgment to the master server copy progress.
• A plurality of master server from the server. From the server from the server may be a plurality, it may also be cascade-like structure from the server to the server from another.
• Redis blocking replication in the primary server. This means that when one or more copies or to perform an initial synchronization section resynchronization, the primary server will continue to process the query.
• Basically copied from the server is not blocked. When executed copy of the initial synchronization, if the replica-serve-stale-data parameters to yes, the server can use the data sets from the old version to process the query. Or configure replica-serve-stale-data parameter to no, when copying is disconnected, the client sends a mistake, however, after the initial synchronization, remove the old version of the data set from the server and load the new version of the data set that within a period of time, the connection request will be blocked.
• Replication can be used simply redundant data (data redundancy), may be treated by having a plurality of read-only command requests from the server to improve scalability (Scalability)
• You can use the master database replication to avoid the implementation of persistence, but to copy from the server after the server from persistence, persistence but it is better to open the master server, as restarting the primary server will start an empty data set, If you try to synchronize with the server, resulting in data from the server it will also be cleared.
• Disadvantages are also obvious, Redis master-slave mode still has a single point of risk, hang up once the primary server will not be able to write data operation

redis Replication Works

Fully synchronized

Master server sends back to the RDB file from the server. During rdb receiving data from the server, the server will save the new data to the primary copy client buffer when rdb After receiving files from the server, it is stored on disk, and load it into memory. After loading rdb file, if opened aof, will be rewritten from the server to operate. The new primary server will buffer the data sent from the server

Partial synchronization

When the interruption from the main connection, transmitting the offset of the last copy, and masterID recorded from the server to the master server using psync command buffer If offset still present since the last replication master server, the master server and masterID the masterID consistent, the last position will start incremental replication disconnected from the buffer, otherwise it will happen completely synchronized

psync command

psync masterID offset

When the connection is established from the server to the main server, judgment is not the first copy, and if so, will send psync? -1 full synchronization, if not, will send psync masterID offset try partial synchronization, masterID if sent by consistent and offset the primary server is present in the copy buffer primary server, will be part of sync, otherwise it will be a full synchronization
first we see the master server and offset masterID

Then copy masterID and offset to the slave, during which I wrote in two primary data server

You can see from the server has been the primary server data

redis replication configuration

As the main server set a password, you need to specify a password doomed master server from the server
vim /etc/redis/6379.conf

masterauth 123456

Since my redis version 5.0, the command synchronization commands are not the same than before, but also compatible with the previous synchronous command

the first method:

Modify the configuration file
vim /etc/redis/6379.conf

replicaof masterip port   #5.0版本支持的命令
或者
slaveof masterip port

The second method:

In the redis client execute commands, synchronization commands executed on the client after the restart will fail

replicaof masterip port   #5.0版本支持的命令
或者
slaveof masterip port

Once configured, masterID information from the master is the same, which is the main information server

This is the information from the server

Master-slave synchronization, the data can not be written from the server

When we disconnect the current connection to the database, build and another master server master-slave synchronization, masterid also changed, will be fully synchronized again

Redis Sentinel

In redis master-slave mode, once the primary server goes down, the need for manual intervention from a server-based conversion server, also need to monitor the status of redis, protracted and painstaking, but also cause some time service is unavailable for some when some scenarios, this approach is not desirable, since version 2.8 Redis provides RedisSentinel tools, multiple master server by heartbeat and their manner under all and offline from the server, the server a master it automatically failover embodiment, the Sentinel will select a server from the primary server and lifting it, the other remaining from the server instance is automatically reconfigured to use the new primary server.

Sentinel function

• monitor. Sentinel will continue to check whether the work from the master node as expected.
• Notice. Sentinel node notify customers via API end of problems
• Automatic failover. When the primary server is not working properly, the Sentinel operation can fail, in this operation from the upgrade server is the primary server, the other server from the server to the new master copy.
• Configuring provider. When the client initialization, by connecting to the Sentinels, acquires the current address of the master server node.

Sentinel building

Preparing the Environment

192.168.179.131：6379 master
192.168.179.132：6379 slave
192.168.179.134：6379 slave
192.168.179.131:26379 sentinel
192.168.179.132:26379 sentinel
192.168.179.134:26379 sentinel

Master-slave node configuration

#192.168.179.131
port 6379
daemonize yes
logfile /var/log/redis_6379.log
dbfilename dump.rdb

#192.168.179.132
port 6379
daemonize yes
logfile /var/log/redis_6379.log
dbfilename dump.rdb
replicaof 192.168.179.131 6379

#192.168.179.134
port 6379
daemonize yes
logfile /var/log/redis_6379.log
dbfilename dump.rdb
replicaof 192.168.179.131 6379

Test whether the configuration from the master copy

Sentinel node configuration

Three sentinel nodes of the same configuration, the configuration of the last sentence mean monitored 192.168.179.131 the primary node, node name is mymaster, and requires at least two sentinel nodes agree to in order to determine the master node failures and automatically migrate

port 26379
daemonize yes
sentinel monitor mymaster 192.168.179.131 6379 2

Start sentinel node

The following two commands can start, you can see the Sentinel has been launched successfully

./src/redis-sentinel sentinel.conf
./src/redis-server sentinel.conf --sentinel

Failover testing

Here I will redis 192.168.179.131 to turn off, view the replication information from the node, you can see the master node has switched to 192.168.179.132 redis

The 192.168.179.131 above redis started again, view node status has been changed from the node

Sentinel Management Command

• info sentinel: get basic information about all master node monitoring
• sentinel masters: Get all the information of the master node is monitored
• sentinel master <master-name>: Gets the specified details of the master node
• Obtaining details from the specified node to the master node: sentinel slaves <master-name>
• Get the specified details of the master node sentinel nodes: sentinel sentinels <master-name>
• sentinel get-master-addr-by-name mymaster: get the master node IP address and port
• sentinel is-master-down-by-addr: Sentinel nodes between the primary node can ask whether this command offline, thereby to judge whether the objective offline
• sentinel failover <master-name>: Forced master node failover
• sentinel ckquorum: Check the available amount Sentinel
• sentinel flushconfig: mandatory to write the configuration file
• sentinel remove <master-name>: Cancel monitors specified master node

Redis cluster model

The above also said, Redis primary if the primary server will not be able to shoot down a write operation from mode, even if the sentry mode provides high availability Redis, but the face of a large amount of data scene, Redis single point is less able to meet this He demanded
Redis cluster is a distributed, fault-tolerant Redis implementation, the cluster can use the function of a subset of the ordinary that can be used stand-alone Redis functions.
Redis Redis clusters are distributed implementation, there is no central node or a proxy node, the cluster of the cluster wherein a primary design goal is to achieve linear scalability. Fault tolerance through the use of the cluster is the master node and the slave node two roles implemented: the master node and the slave node using the same server implemented, their function is exactly the same, but normally only for the replaced node failure The master node. If you do not ensure that the "first write after reading" consistent operations, you can use to perform read-only queries from the node.
Unlike stand-alone cluster Redis Redis as support for multi-database functionality, use only the default cluster 0 database, and you can not use SELECT index command.

Redis job content nodes in the cluster

• Key to save the data.
• Recording cluster state, including the key mapping to the correct node.
• Automatic discovery of other nodes, node identification is not working properly, and elect a new master node from the node.

Key distribution model

Redis cluster key space is divided into 16,384 slots (slot), the maximum number of cluster nodes is 16,384. When a cluster is in a "stable" state, the cluster each hash slot will not move when you need to add a node when only need to transfer some hash slots to other nodes on the new node, you need to be deleted when a node, put a hash slot for this node metastasis to other nodes on it. A master node may have any number of nodes from which the node from the master node for a network disconnection or node failure occurs, replacement of the master node.

Cluster Node Properties

Each node in the cluster has a unique ID, which is a hexadecimal representation of a random number 160 generated by the / dev / urandom when the node is first started.
Node ID will save it to the configuration file, as long as the configuration files are not deleted, the node will always adopt this ID.
Node ID for identifying each node in the cluster. A node may change its IP address and port number, without changing the node ID. Clusters can automatically identify a change in IP / port number, and this information is known to other nodes through the broadcasting Gossip protocol.
The following information is associated with each node has, and the node will transmit this information to other nodes:

• IP address and TCP port number used by the node.
• Flags (flags) node.
• Node is responsible for processing the hash tank.
• Last time the node transmits the connection packet PING (Packet) is used cluster.
• Last time the node receives the packet in the PONG reply.
• The cluster node is marked as offline time.
• From the number of that node.
• If the node is a slave node, then it records the node ID of the master node. If this is a master node, then the master node ID in this column is 0000000.

Wherein a portion of the above information can be obtained by (master node or the slave node may be) CLUSTER NODES command sent to any node in the cluster.

Cluster data consistency

Redis cluster is unable to guarantee strong consistency of data

1. One reason is because Redis cluster is asynchronous replication, when the client node to write data to a master server, the master server will first respond to the client, then after a write request is sent to the slave node, in this process If the primary node goes down, but have not received a write request from the node, then this data will be permanently lost, of course, you can refresh the data in reply to the client before by forcing the database to disk, but it will lead to performance reduce.
2. Another is to bring the network partition (network partition), when Redis cluster network partition occurs, the client is still on the master node is small partitions write operation, when a cluster node timeout after reaching the time limit, in large partition from the node that will replace the master node is small partition called the new master node, while the node timeout during which time the client is written to the master node of data will be lost

Failure detection node

1. When a node sends a PING command to another node, the target node fails to return to respond to the PING command in the node timeout, then the node will mark the destination node sends a command to PFAIL (may be dead).
2. Every time a node sends PING commands to other nodes, which will be broadcast at random information it knows about three nodes, the information inside one of which is whether a node has been marked as PFAIL or FAIL.
3. Most of the other node if the primary node has a node labeled PFAIL, and also believe that the cluster nodes into a failed state, then the node will state that the failed node is marked as FAIL.
4. Once a node is marked as FAIL, information about the node has failed will be broadcast to the entire cluster, all received this information failed node node will be marked as FAIL.

Cluster Setup

Edit the configuration file to enable the cluster nodes

Each cluster instance must be turned on after the restart redis examples

cluster-enabled yes
cluster-config-file nodes-6379.conf
cluster-node-timeout 15000

Distribution groove and connected to cluster nodes

./redis-cli --cluster create 192.168.179.131:6379 192.168.179.131:6380 192.168.179.134:6379 192.168.179.134:6380 192.168.179.132:6379 192.168.179.132:6380 --cluster-replicas 1

You can view information about the cluster nodes, any node within the cluster address to enter

./redis-cli --cluster check 192.168.179.132:6379

Cluster Verification

When we set a key, will use CRC16 algorithm modulo obtain slot belongs, then the node key assigned to the hash slot section, when acquiring a node get the data, but also by this method to the corresponding data

./redis-cli -c -h 192.168.179.132 -p 6380
192.168.179.132:6380> set key 1
-> Redirected to slot [12539] located at 192.168.179.132:6379
OK
./redis-cli -c -h 192.168.179.134 -p 6379
192.168.179.134:6379> get key
-> Redirected to slot [12539] located at 192.168.179.132:6379
"1"

---

No personal welcome attention to the public, "the story of the development of operation and maintenance."