redis note 3

redis persistence mechanism

redis provides two persistence strategies

RDB

RDB persistence strategy: According to the rules of data synchronization timing memory to disk

snapshot

redis under specified circumstances will trigger a snapshot

1. Configure their own snapshots rules

save <seconds> <changes>

The default configuration rules

save 900 1  when the key number is changed within 900 seconds is greater than 1, it is a snapshot

save 300 10

save 60 10000

1. Perform a manual save or bgsave

save: synchronous execution memory data to disk operation, this operation will block the client's request

bgsave: asynchronously in the background to perform snapshot operations, this operation will not block the client's request

1. When the execution flushall

Clear all data in memory, as long as the rules of the snapshot is not empty, that there is the first rule. Redis will then perform a snapshot

1. When executed copy

The principle snapshots

1: redis using the fork function copy of a copy of the current process (the child)

2: The parent process continues to receive and process commands sent by the client, and the child began to data memory write temporary files on your hard disk

3: When the child process all the data written to the RDB will replace the old file with the temporary file, so far, a snapshot operation is complete.  

 Note: RDB does not modify the file during the snapshot process redis, only after the end of the snapshot will replace the old file into the new, that is to say at any time RDB files are complete. This allows us to redis database backup of files through a scheduled backup RDB, RDB file is a compressed binary file, space is often less than the data in memory, more conducive to transmission.

RDB advantages and disadvantages

1. Use RDB way to achieve persistence, once Redis abnormal exit, you will lose the last snapshot of all data changes in the future. We need this time according to the specific application scenario, the automatic snapshot provided by the combination of conditions to the way data loss may occur in the control range can be accepted. If the data is relatively important, you want to minimize the loss, you can use the AOF way for persistence
2. RDB Redis can maximize performance: the parent process to do is fork out when you save a file RDB child process, the child process will then handle all subsequent preservation work, the parent process order execution any disk I / O operating. While this is a disadvantage, if the data set when larger, fork can be time-consuming, resulting in a server stops processing client requests a period of time;

practice

The modified redis.conf appendonly yes; change instruction executed after the restart of the data, generated in the corresponding bin directory .aof files, records all AOF file operation command

The following two parameters can be done to optimize the aof file

auto-aof-rewrite-percentage 100 represents more than once on how many percent of the aof file size will be rewritten when the current aof file size. If you do not rewritten before, when to start aof file size prevail

auto-aof-rewrite-min-size 64mb aof minimum limit for allowing the overwrite file size, the file size is less than 64mb time, you do not need to be optimized

AOF

AOF each write command execution Redis can be added to a disk file, this process will obviously reduce the performance of Redis, but in most cases this influence is acceptable, in addition to a faster hard drive can improve the performance of AOF

practice

Redis is not turned on by default persistence AOF (append only file) mode can be enabled by appendonly parameters, found in redis.conf in appendonly yes

After a change in the data Redis commands executed after each turn AOF persistence, Redis command will write the AOF file on your hard disk. AOF file save location and file of the same location RDB are set by the dir argument, the default file name is apendonly.aof. Appendfilename appendonlyh.aof can modify the properties of the redis.conf

aof rewritten principle

When Redis AOF may become too large file size, automatically in the background rewriting of AOF: AOF new file after overwriting command contains a minimal set of data required to restore the current collection. Rewrite the entire operation is absolutely safe, because in the process of creating a new Redis AOF file, the command will continue to append to an existing file AOF inside, even during downtime rewrite existing AOF files will not be lost . Once the new AOF file is created, Redis will switch from the old to the new file AOF AOF file and start the new AOF file append operation. AOF save the file in an orderly manner all write operations performed on the database, the write operation to save the format Redis protocol, so the contents of AOF file is very easy to read people, to file for analysis (parse) is also very easy

Synchronous disk data

redis each time you change the time data, aof mechanisms can speak commands to record aof file, but in fact the caching mechanism operating systems, real-time data is not written to the hard disk, but into the hard disk cache. Then flushed to go through the hard disk saved to a file caching mechanism

# Appendfsync always write every execution will be synchronized, but this is the safest way is relatively low efficiency

appendfsync everysec every second execution

# Appendfsync no do not take the initiative to synchronize the operation, to be performed by the operating system, this is the fastest but least secure way

How to repair damaged files after aof  

The server may shut down when the program is being AOF file for writing, if downtime caused AOF file error (corrupt), then restart Redis in AOF will refuse to load this file, ensuring consistency of the data will not be destroyed.

When this happens, you can use the following methods to repair the mistakes of AOF documents:

1. Create a backup for the existing AOF file.
2. Redis using the supplied redis-check-aof program, AOF file to the original repair.

redis-check-aof --fix

Redis server restart, wait AOF file server load repair and data recovery.

RDB and AOF, how to choose

In general, if the data security requirements are very high, you should use both persistent feature. If you can afford the loss of data within a few minutes, you can only use RDB persistence. AOF many users use only persistent, but does not recommend this approach: since the timing generating RDB Snapshot (Snapshot) is very easy database backup and recovery speed RDB data set than AOF is also faster recovery.

Two kinds of persistence strategies can be used simultaneously, you can use one of them. If you use it, when you restart Redis, it will give priority to the use of AOF file to restore data

 

Clusters

Copy (master, slave)

 

Configuration process

Modify 11.140 and 11.141 of redis.conf file, increase slaveof masterip masterport

This increase can slaveof 192.168.11.138 6379 will redis server from the server to slave

Enter info in the client replication view server status

The principle

1. The first slave or after reconnection to the master, sends a command to the master SYNC
2. SYNC received master time, it will do two things

a) execution bgsave (rdb snapshot file)

b) master will receive the new changes to the command stored in the buffer

The disadvantage
is no way to master the dynamic election

Copy the way

1. Rdb file-based replication (first connection or reconnection time)
2. No hard copy
3. Incremental Replication

PSYNC master run id. offset

Sentinel mechanism

sentinel

1. Monitoring master and salve is operating properly
2. If the master fails, one of which will be upgraded to master data salve

Cluster (after redis3.0 function)

The hash value of the key modulo the number of servers.

hash

Principle of cluster

In Redis Cluster, the concept of using Sharding slot (groove), is divided into a total of 16,384 slots, which somewhat similar idea of ​​speaking in front of the pre sharding. For each key to enter the Redis performed in accordance with a hash key, which is assigned to 16,384 in a one slot. hash algorithm used is also relatively simple, is the modulo CRC16 16384. Redis each node in the cluster (node) is responsible for sharing a part of this 16384 slot, that is, each slot corresponds to a node handles. When the dynamic node node added or subtracted, grooves need to be redistributed to 16,384, the key groove also migrate. Of course, this process, in the current implementation, the state is still in the semi-automatic, manual intervention is required. Redis cluster, to ensure that the 16,384 slots corresponding node are working properly, if a node fails, then it will be responsible for the failure of the slots, the entire cluster will not work. To increase the accessibility of the cluster, it is officially recommended program from the master node arranged structure, i.e., a primary master node, hanging from the n-th slave nodes. At this time, if the primary node fails, Redis Cluster will be selected according to the election algorithm from the slave node in a node-based rise, the entire cluster to continue to provide services. This is very similar to the server node by Sentinel surveillance infrastructure into a master-slave structure, but Redis Cluster itself provides the ability to failover for fault tolerance.

 

The concept slot (slot), and in redis clusters have a total of 16,384 slots,

CRC16 algorithm according to the key, and then the result of modulo 16384. If there are three nodes

node1  0 5460

node2 5461 10922

node3 10923 16383

New node

node4  0-1364,5461-6826,10923-12287

Delete Node

First of moving data node to other nodes before you can delete

 

Cluster solutions available on the market

1. redis shardding and jedis client to support shardding operation SharddingJedis; increase and decrease problems nodes; pre shardding

3 virtual machines redis. But I deployed nine nodes. Each deployment of three redis increase cpu utilization

9 split into separate virtual machine server 9

1. codis based redis2.8.13 branch developed a codis-server
2. twemproxy open source solutions provided by twitter