A. Redis install stand-alone and enterprise-class redis Start Program
installation:
[root@localhost ~]# tar zxf redis-3.2.3.tar.gz
[root@localhost ~]# cd redis-3.2.3/
[root@localhost redis-3.2.3]# make&& make install
[root@localhost redis-3.2.3]# cd utils/
[root@localhost utils]# ./install_server.sh
# Accordance with the default position to (go on like carriage return)
Production environment Start Program:
As a daemon process should redis system to run each time the system is started, the system should start together
If it is not the kind we used above # ./install_server.sh mounted on the need to do these following:
First, we generate a configuration file for redis # cd /usr/local/src/redis-3.2.3/utils/ # cp redis_init_script /etc/init.d/ # vim /etc/init.d/redis_init_script Here are just need to modify some (if not directly modify the default on it) REDISPORT=6379 EXEC=/usr/local/bin/redis-server CLIEXEC=/usr/local/bin/redis-cli PIDFILE=/var/run/redis_${REDISPORT}.pid CONF="/etc/redis/${REDISPORT}.conf" # mv /etc/init.d/redis_init_script /etc/init.d/redis_6379 |
# Said here about the way we use ./install_server.sh default installation this step has helped us do a good job
Once you've configured the service to start the test:
# service redis_6379 start
Let him start automatically with the boot configuration:
1. First, we configure two-line comments (top position):
# vim /etc/init.d/redis_6379
#!/bin/sh
# chkconfig: 2345 90 10
# description: Redis is a persistent key-value database
2. Perform boot from the start command:
# chkconfig redis_6379 on
redis-cli use:
Stop redis:
# redis-cli shutdown
Connection redis:
# redis-cli -h 127.0.0.1 -p 6379 -a PASSWORD
Redis test port is normal
# redis-cli ping
Creating a key:
127.0.0.1:6379> set key1 value1
OK
View a key:
127.0.0.1:6379> get key1
"value1"
To delete a key:
127.0.0.1:6379> of key1
(integer) 1
redis technology, including 4:
Redis use various data structures and commands
Redis using special solutions, automatically pub / sub messaging system, distributed lock, input, etc.
redis daily management of related commands
redis and deploy enterprise-class cluster architecture
Two .redis persistence
1.redis significance for the generation of persistence in the environment for disaster recovery:
redis persistent significance Recovery
For example, you deployed a redis, do cache cache, saved some of the more important things
Without persistence, redis suffers a catastrophic failure and they will lose all data
If the data by persistent disk got up, and then periodically synchronize the backup to the cloud storage server up, then we can guarantee that all the data is not lost all or part of the data can be restored.
redis persistence divided into: RDB, AOF two ways
Persistence is mainly to do disaster recovery, data recovery, which can also be classified into a highly available link to.
For instance, your whole redis hung up, then redis No, you have to do is to make redis become available, become available as soon as possible
Restart redis, can make it available as soon as possible, but if you do not have redis persistence, is available a lot of useless sql to mysql directly killed
RDB and AOF two kinds 2.redis persistence mechanism of introduction:
RDB persistence mechanism, performs data redis periodically persistence
AOF mechanism for each data write command as a log, written to a log file in append-only mode , when redis restart, can reconstruct the entire data set by an instruction written in the log replay AOF
If you use RDB and two types of persistence mechanisms AOF , then when redis restart, AOF will be used to reconstruct the data because the data is more complete AOF
If we want to redis only as a pure memory cache to use, you can disable all the persistence mechanism RDB and AOF
RDB persistent way:
It can be understood as data snapshots
At regular intervals, generating a complete snapshot of the data in memory redis
AOF persistent way:
The equivalent of mysql binlog log mode
reids every 1 second (default) will be forced to call the operating system, written to a file to
AOF unlimited files will write data to a file inside, in order to prevent this from happening redis will wait until the AOF large data files to a certain extent, when creating a new AOF file, and then expanded to delete files before AOF
To prevent AOF continues to expand, redis of AOF rewrite mechanisms:
First, we can only take up redis redis provides 1G of memory cache to store data,
100w probably saved data when that data Nianman 1G,
This time redis will clear some of the less frequently used data according to their own LRU algorithm,
This time redis data in memory is 50w strip, AOF data file is 100w strip,
redis continued for additional data when the data redis memory when they reach 100w bar, AOF file has 150w of data,
This time will be shot redis rewrite the continued expansion of AOF documents
First, create a new file AOF out, writing the current redis memory 100w pieces of data into a new file to AOF,
That expansion is completed after writing before the AOF file will be deleted
AOF rewrite mechanism redis ensure AOF file will not last expansion, when AOF file expands to a certain extent, will be shot rewrite redis
RDB and AOF two kinds 3.redis persistence pros and cons of contrast mechanisms:
RDB advantages:
(1) RDB generates a plurality of data files, each data file represents the data in a time of redis this way multiple data files, very suitable for cold backup.
Long time to generate the snapshot file to control the fixing redis
RDB data coldest prepared, in the worst case, data recovery time, faster than AOF
(2) RDB to provide literacy services redis outside, the impact is very small, you can make redis maintain high performance, because redis main process requires only fork a child process, so that the implementation of disk IO operations to child processes can persist RDB
(3) with respect to the AOF persistence mechanism, the direct-based RDB data files to restart and restore redis process, faster
AOF instruction is stored, you need to replay
RDB, is a data file, the recovery time can be directly loaded into memory
RDB disadvantages:
(1) RDB in the event of failure, loss of data will be more than AOF, because RDB is timed snapshot file, AOF is a real-time backup data
This issue, but also RDB biggest drawback is not suitable to do the program first priority recovery.
(2) RDB doing a snapshot of time, particularly if a large amount of data may result in suspension of service provided by the client milliseconds, or even seconds
Do not let the general RDB snapshot interval too long, otherwise the RDB is generated every time a file is too large, the performance of redis itself may
influential
AOF advantages:
(1) AOF can better protect data is not lost, AOF will usually every second, once fsync operation executed by a background thread, loss of data up to 1 second
(2) AOF log files are written in append-only mode, so there is no disk addressing overhead, write performance is very high, and the file is not easily damaged
(3) AOF log file is too large even when the background appears minimal rewriting operation, it will not affect the client to read and write because of which will guide the rewrite log when compressed, created out of a need to restore data log out. when you create a new log file, the old log file is written as usual. when the new log file is created, and then exchange the old log files.
(4) AOF recorded by way of command, very suitable for emergency recovery of accidentally deleted catastrophic. As long as no rewrite operation can restore data instantly
The biggest advantage is: better data protection
AOF disadvantages:
More (1) for the same data is, AOF log file often than RDB data snapshot file
After (2) AOF turn, support redis write QPS will reduce
(3) before AOF happened bug, and now by the way will rewrite some of robust
(4) The only drawback is relatively large, so recovery time will be relatively slow, do not make the cold standby
RDB and AOF in the end how to choose:
RDB recommendations and AOF open simultaneously:
(1) Do not just use RDB, as it will cause you to lose a lot of data
(2) Do not just use the AOF, because, AOF not suitable for cold standby, nor RDB recovery speed, AOF is a bug occurred
(3) the integrated use of AOF and RDB two kinds of persistence mechanism
AOF as large as possible to ensure data security
RDB used for cold backup, when the damage is not available in the AOF file, you can use fast data recovery process to RDB
Configuring the redis RDB persistence
RDB persistence configuration:
# vim /etc/redis/6379.conf
save 60 1000
# Every 60 seconds to detect that, if there are more than 1000 key changes occurred dump.rdb went generate a new file,
This document is dump.rdb redis complete snapshot of the data memory, this operation is also referred to as snapshotting, the snapshot may be invoked manually save or bgsave command, synchronous or asynchronous execution snapshot generation rdb
#save can set up multiple, simultaneous entry into force
dbfilename dump.rdb
# RDB持久化的文件名称(因为名字是用的一个,所以每次快照之后,都会把之前快照的文件给替换掉)
# 意思是dump.rdb 文件只会有一个,新的会覆盖老的
dir /var/lib/redis/6379
# dump.rdb(rdb快照文件)存储的位置
改完配置记得重启:
# redis-cli SHUTDOWN
# service redis_6379 start
RDB持久化机制的工作流程:
(1)redis根据配置自己的尝试去生成rdb快照文件
(2)fork一个子进程出来
(3)子进程尝试将数据dump到临时的rdb快照文件中
(4)完成rdb快照文件的生成之后,就会替换掉之前的旧的快照文件
知识点补充:
通过 redis-cli SHUTDOWN 这种方式去关闭redis,其实是一种安全的退出方式, redis在退出的时候会将内存中的数据立即的生成一份完整的rdb快照
恢复测试:
(1) 写入数据 --- 立即通过 redis-cli SHUTDOWN关闭 -- 查看dump.rdb 文件是否生成 -- 启动redis查看数据是否还在
(2)定义 save 5 1 -- 插入数据,等待5秒 -- kill -9 杀掉redis -- 启动查看数据是否还在
5.配置redis的AOF持久化
默认是打开RDB持久化方式的,AOF默认是关闭的
AOF持久化配置:
# vim /etc/redis/6379.conf
appendonly yes
#打开AOF持久化,在我们生成环境中,AOF是一定要打开的,除非说,你丢几分钟的数据也无所谓
#AOF持久化策略(三种)
# appendfsync always
#每写入一条数据,立即将这个数据对应的写日志fsync到磁盘上去,性能非常差(大概2000QPS)
appendfsync everysec
#默认的AOF持久化策略(生产环境一般这么配置)
#每秒将 os cache中的数据fsync到磁盘(QPS可以到2w)
# appendfsync no
# 仅仅将数据写入到os cache中就撒手不管了,等待系统自动执行fsync操作,不可控
dir /var/lib/redis/6379
# dir即指定了 RDB快照文件的位置,也指定了 aof 日志文件的位置
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
# AOF rewrite操作,说一下这两个参数是什么意思:
比如说上一次AOF rewrite之后是 128mb,然后就会接着往进去写,如果发现超过了之前的100%,256mb, 就会去判读是否超过了 min-size 配置的参数(256>64),如果超过就会重写
改完之后记得重启:
# redis-cli SHUTDOWN
# service redis_6379 start
这里再说一下,RDB 和 AOF同时开启的情况下,启动的时候redis 会加载 AOF中的数据
QPS : 每秒钟的请求数量
AOF rewrite:
这里再说一下为啥要进行AOF rewrite操作:
redis中的数据其实是有限的,很多数据可能会自动过期,可能会被用户删除,可能会被redis自动清理掉
但是AOF文件中还是保存着这部分数据,时间一长数据就会不断膨胀
所有redis会通过rewrite操作来控制 aof文件的不断暴增
我们这边还是用上面的那张图,来看一下,为啥要做AOF rewrite:
AOF rewrite工作原理:
(1)redis fork一个子进程
(2)子进程基于当前内存中的数据,构建日志,开始往一个新的临时的AOF文件中写入日志
(3)redis主进程,接收到client新的写操作之后,在内存中写入日志,同事新的日志也继续写入旧的AOF文件
(4)子进程写完新的日志文件之后,主进程将内存职工的新日志再次追加到新的AOF文件中
(5)用新的日志文件替换掉旧的日志文件
AOF文件修复:
如果AOF文件有破损的话,可以用 redis-check-aof --fix 做修复
# redis-check-aof --fix appendonly.aof
AOF和RDB同时工作:
1.如果RDB在执行快照操作的时候,不会执行AOF的 rewrite操作
反之,如果AOF 在执行rewrite操作的时候,ROD的快照操作也不会执行
总而言之就是,不会同时执行,RDB快照操作 和 AOF rewrite操作
2.如果RDB在执行快照操作,我们手动的去执行 AOF rewrite操作,呢么等RDB的快照操作完成之后AOF
rewrite操作才会去执行
3.如果同时用RDB和AOF两种持久化机制,重启redis的时候,会优先使用AOF进程数据恢复,因为其中的
日志更完整
知识点补充:
如果RDB中有部分数据,AOF中有部分数据,RDB中的数据不会被加载
三.企业级备份方案
在企业中,RDB的生成策略,用默认的其实也差不多
如果你希望你的RDB最多丢失一分钟的数据,呢么尽量每个一分钟都去生成一个快照
save 60 10000 这个调策略根据自己实际需求去改
AOF一定要打开,用默认设置也差不多:
fsync的策略用 everysec 策略就可以(每隔一秒redis调用系统强刷一次fsync)
auto-aof-rewrote-min-size: 根据自己的数据量来调整一下就好了
几台 redis 承受几十万的 QPS,完全没有问题
数据备份方案:
(1)写crontab定时调度备份脚本去做数据备份
(2)每个小时都copy一份rdb的备份到一个目录中去,仅仅保留最近48小时的备份
(3)每天都保留一份当日的rdb备份,到一个目录中去,仅仅保留最近一个月的备份
(4)每次copy备份的时候,都把太旧的备份给删了
(5)每天晚上将当前服务器上所有的数据备份,发送一份到远程的云服务器上去
每小时copy一次备份，删除48小时前的数据
crontab -e
0 * * * * sh /usr/local/redis/copy/redis_rdb_copy_hourly.sh
#vim /usr/local/redis/copy/redis_rdb_copy_hourly.sh
#!/bin/sh cur_date=`date +%Y%m%d%k` rm -rf /usr/local/redis/snapshotting/$cur_date mkdir -p /usr/local/redis/snapshotting/$cur_date cp /var/redis/6379/dump.rdb /usr/local/redis/snapshotting/$cur_date del_date=`date -d -48hour +%Y%m%d%k` rm -rf /usr/local/redis/snapshotting/$del_date |
每天copy一次备份
crontab -e
0 0 * * * sh /usr/local/redis/copy/redis_rdb_copy_daily.sh
# vim /usr/local/redis/copy/redis_rdb_copy_daily.sh
#!/bin/sh cur_date=`date +%Y%m%d` rm -rf /usr/local/redis/snapshotting/$cur_date mkdir /usr/local/redis/snapshotting/$cur_date cp /var/redis/6379/dump.rdb /usr/local/redis/snapshotting/$cur_date del_date=`date -d -1month +%Y%m%d` rm -rf /usr/local/redis/snapshotting/$del_date |
每天一次将所有数据上传一次到远程的云服务器上去
crontab -e
0 0 * * * sh /usr/local/redis/copy/redis_rdb_scp.sh
# vim /usr/local/redis/copy/redis_rdb_scp.sh
#!/bin/sh ssh root@远程主机 "rm -rf /opt/backup/snapshotting/*" scp /usr/local/redis/snapshotting/* root@远程主机:opt/backup/snapshotting/ |
数据恢复方案:
(1)如果是redis进程挂掉，那么重启redis进程即可，直接基于AOF日志文件恢复数据
(2)如果是redis进程所在机器挂掉，那么重启机器后，尝试重启redis进程，尝试直接基于AOF日志文件进行数据恢复
AOF没有破损，也是可以直接基于AOF恢复的
AOF append-only，顺序写入，如果AOF文件破损，那么用redis-check-aof fix
(3)如果redis当前最新的AOF和RDB文件出现了丢失/损坏，那么可以尝试基于该机器上当前的某个最新的RDB数据副本进行数据恢复
当前最新的AOF和RDB文件都出现了丢失/损坏到无法恢复，一般不是机器的故障，人为
找到RDB最新的一份备份，小时级的备份可以了，小时级的肯定是最新的，copy到redis里面去，就可以恢复到某一个小时的数据(要注意目录下面不要有 aof 的破损文件,要暂时关闭AOF持久化)
操作:
1.停止redis
2.清除/var/redis/6379/下面的破损文件
3.再配置文件中关闭 AOF持久化
4.把备份的 rdb文件 传到 /var/redis/6379/ 目录下面
5.启动redis,确认数据恢复正常
6.直接在命令行热修改redis配置打开aof，
打开之后redis就会将内存中的数据对应的日志，写入aof文件中
127.0.0.1:6379> config set appendonly yes
7.稍等片刻(等待redis将内存中的数据写入到刚刚生成的AOF文件),之后再次停止redis
8.在配置文件中开启 AOF 持久化
9.启动redis
(4)如果当前机器上的所有RDB文件全部损坏，那么从远程的云服务上拉取最新的RDB快照回来恢复数据
(5)如果是发现有重大的数据错误，比如某个小时上线的程序一下子将数据全部污染了，数据全错了，那么可以选择某个更早的时间点，对数据进行恢复
举个例子，12点上线了代码，发现代码有bug，导致代码生成的所有的缓存数据，写入redis，全部错了,找到一份11点的rdb的冷备，然后按照上面的步骤，去恢复到11点的数据，不就可以了吗