Redis
Redis basics
Redis (Remote Dictionary Server), the remote dictionary service, is an open source log-based, Key-Value database written in ANSI C language, supporting the network, memory-based or persistent, and providing APIs in multiple languages. From March 15, 2010, the development of Redis is hosted by VMware. Since May 2013, the development of Redis has been sponsored by Pivotal.
It is a non-relational database . It is a memory/cache database. The data is stored in memory , but the data can be written to the hard disk regularly .
redis advantage of
support memory cache, a high speed data read and write
support persistence save, you can store objects
clustered, distributed
data types more
support data backup
is atomic
support queue
Compile and install Redis (come on! Show it!!)
[root@localhost bao]# yum -y install gcc gcc-c++ make
[root@localhost bao]# rz -E
rz waiting to receive.
[root@localhost bao]# ls
redis-5.0.7.tar.gz
[root@localhost bao]# tar zxvf redis-5.0.7.tar.gz
…省略解压过程…
[root@localhost bao]# cd redis-5.0.7/
[root@localhost redis-5.0.7]# ls
00-RELEASENOTES deps README.md runtest-moduleapi tests
BUGS INSTALL redis.conf runtest-sentinel utils
CONTRIBUTING Makefile runtest sentinel.conf
COPYING MANIFESTO runtest-cluster src
[root@localhost redis-5.0.7]# make ##没有con啥的脚本文件，那就直接make吧
[root@localhost redis-5.0.7]# make install PREFIX=/usr/local/redis ##指定安装目录
[root@localhost redis-5.0.7]# cd /usr/local/redis/
[root@localhost redis]# ls
bin
[root@localhost redis]# cd bin/
[root@localhost bin]# ls ##看看这六个文件齐不齐
redis-benchmark redis-check-rdb redis-sentinel
redis-check-aof redis-cli redis-server
[root@localhost bin]# cd /bao/redis-5.0.7/utils/
[root@localhost utils]# ls
build-static-symbols.tcl hashtable redis_init_script.tpl
cluster_fail_time.tcl hyperloglog redis-sha1.rb
corrupt_rdb.c install_server.sh releasetools
create-cluster lru speed-regression.tcl
generate-command-help.rb redis-copy.rb whatisdoing.sh
graphs redis_init_script
[root@localhost utils]# ./install_server.sh ##若没有要求可以一直回车，直到设置可执行文件的八个路径需要手动设置
Welcome to the redis service installer
This script will help you easily set up a running redis server
Please select the redis port for this instance: [6379]
Selecting default: 6379
Please select the redis config file name [/etc/redis/6379.conf]
Selected default - /etc/redis/6379.conf
Please select the redis log file name [/var/log/redis_6379.log]
Selected default - /var/log/redis_6379.log
Please select the data directory for this instance [/var/lib/redis/6379]
Selected default - /var/lib/redis/6379
Please select the redis executable path [] /usr/local/redis/bin/redis-server ##手敲，bin可执行文件
Selected config:
Port : 6379 ##端口
Config file : /etc/redis/6379.conf ##主配置文件
Log file : /var/log/redis_6379.log ##日志文件
Data dir : /var/lib/redis/6379 ##默认接口的数据文件
Executable : /usr/local/redis/bin/redis-server ##可执行脚本文件
Cli Executable : /usr/local/redis/bin/redis-cli ##连接终端命令脚本文件
Is this ok? Then press ENTER to go on or Ctrl-C to abort. ##ok就回车
Copied /tmp/6379.conf => /etc/init.d/redis_6379
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful! ##安装 成功！
[root@localhost utils]# cd /etc/init.d/ ##在init.d下可以看到redis的脚本，证明可以被service控制
[root@localhost init.d]# ls |grep redis
redis_6379
[root@localhost init.d]# service redis_6379 stop
/var/run/redis_6379.pid does not exist, process is not running
[root@localhost init.d]# service redis_6379 start
Starting Redis server...
[root@localhost init.d]# netstat -ntuap |grep 6379
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 77279/redis-server
Set the listening port to see if the database can be successfully entered
[root@localhost init.d]# vim /etc/redis/6379.conf
# Examples: ##这个下面的两个注释掉的bind是redis给的模板，不用修改
#
# bind 192.168.1.100 10.0.0.1
# bind 127.0.0.1 ::1
70 bind 127.0.0.1 20.0.0.6 ##要改的在第70行
[root@localhost init.d]# service redis_6379 restart
Stopping ...
Waiting for Redis to shutdown ...
Redis stopped
Starting Redis server...
[root@localhost init.d]# ln -s /usr/local/redis/bin/* /usr/local/bin
[root@localhost init.d]# redis-cli -h 20.0.0.6 -p 6379
20.0.0.6:6379> ##可以正常进入数据库了
Basic commands! !
[root@localhost init.d]# redis-cli -h 20.0.0.6 -p 6379
20.0.0.6:6379> help set ##查看命令帮助
SET key value [expiration EX seconds|PX milliseconds] [NX|XX]
summary: Set the string value of a key
since: 1.0.0
group: string
20.0.0.6:6379> set gunpla 00Q ##创建 key键 值
OK
20.0.0.6:6379> set gundam EXIA
OK
20.0.0.6:6379> keys * ##查看所有 键
1) "gunpla"
2) "gundam"
20.0.0.6:6379> RENAME gunpla gaoda ##重命名 gunpla 为 gaoda
OK
20.0.0.6:6379> del gundam ##删除 gundam 键
(integer) 1
20.0.0.6:6379> keys *
1) "gaoda"
Library operations
20.0.0.6:6379> SELECT 10 ##切换到 第 10 库
OK
20.0.0.6:6379[10]> keys *
(empty list or set)
20.0.0.6:6379[10]> set GUNDAM00 00RAISE
OK
20.0.0.6:6379[10]> MOVE GUNDAM00 9 ##移动 键 到 第9库
(integer) 1 ##操作状态码 成功返回 非0，不成功返回 0
20.0.0.6:6379[11]> SELECT 9
OK
20.0.0.6:6379[9]> KEYS *
1) "GUNDAM00"
20.0.0.6:6379[9]> GET GUNDAM00 ##查看 键值
"00RAISE"
20.0.0.6:6379[9]> FLUSHALL ##清空库方法 ①
OK
20.0.0.6:6379[9]> keys *
(empty list or set)
20.0.0.6:6379[9]> SELECT 0
OK
20.0.0.6:6379> set caicai 180
OK
20.0.0.6:6379> keys *
1) "caicai"
20.0.0.6:6379> FLUSHDB ##清空库方法 ②
OK
20.0.0.6:6379> KEYS *
(empty list or set)
Stress test
Use the redis-benchmark test tool
-h to specify the host IP
-p to specify the port
-c to specify the number of concurrent connections
-n to specify the number of requests
-d to specify the value of SET/GET in byte data size
-q to force redis to exit, only display the value of query/sec
[root@localhost init.d]# redis-benchmark -h 20.0.0.6 -p 6379 -c 100 -n 10000 ##并发量 100，请求量 10000
……省略部分
====== SET ====== ##写入速度
10000 requests completed in 0.12 seconds ##耗时
100 parallel clients
3 bytes payload
keep alive: 1
97.86% <= 1 milliseconds
99.24% <= 2 milliseconds
99.79% <= 3 milliseconds
100.00% <= 3 milliseconds
86956.52 requests per second
====== GET ====== ##读取速度
10000 requests completed in 0.11 seconds ##耗时
100 parallel clients
3 bytes payload
keep alive: 1
99.92% <= 1 milliseconds
100.00% <= 1 milliseconds
87719.30 requests per second
省略部分……
[root@localhost init.d]# redis-benchmark -h 20.0.0.6 -p 6379 -q -d 100
##-q 强制退出redis，-d 指定字节数量
…省略部分
SET: 85616.44 requests per second
GET: 86655.11 requests per second
省略部分…
hash structure
20.0.0.6:6379> hset yaoshi Asuo yihao
(integer) 1
20.0.0.6:6379> hset demaxiya gaolun 190
(integer) 1
20.0.0.6:6379> hset demaxiya kuiyin 160
(integer) 1
20.0.0.6:6379> hset demaxiya huangzi 185
(integer) 1
20.0.0.6:6379> hget demaxiya gaolun
"190"
20.0.0.6:6379> hget demaxiya huangzi
"185"
Redis data persistence
Data Persistence Overview
Redis originally runs in memory, and data is lost when the machine is powered off.
In order to protect Redis data and prevent system failures, the data in Redis needs to be written to the hard disk. This operation is called persistence
Persistence classification
- RBD method: the method of creating a snapshot, so that you can get a copy of all the data files of the snapshot
- AOF mode: write the executed write command to the last line of the file, and record the data changes in a log mode
RBD
RBD is the default persistence method of redis. The
default backup file name is dump.rdb, and the location is /var/lib/redis/6379. The
trigger conditions are as follows.
In the specified time interval, execute the specified number of write operations (can be changed in the configuration file)
execute save Or bgsave command to
execute flushall command to clear all data in the database and
execute shutdown command to ensure that the server is shut down normally without losing any data
Pros and cons
- Suitable for large-scale data recovery
- If the business does not require high data integrity and consistency, RDB is the preferred choice
- Data integrity and consistency are not high
- Occupy some memory during backup
Configuration file display
[root@localhost init.d]# vim /etc/redis/6379.conf
217 # save ""
218
219 save 900 1 ##900秒之内至少进行了一次写操作
220 save 300 10 ##300秒内至少进行了10次写操作
221 save 60 10000 ##60秒内至少进行10000次写操作
253 # The filename where to dump the DB
254 dbfilename dump.rdb ##RBD文件名
251 rdbchecksum yes ##压缩功能开启
264 dir /var/lib/redis/6379 ##RDB文件位置
[root@localhost init.d]# cd /var/lib/redis/6379/
[root@localhost 6379]# ls
dump.rdb
AOF
Redis is not enabled by default to
make up for the lack of RDB (data inconsistency)
to record each write operation in the form of a log, and append it to the file.
Redis restart will execute the write instruction from front to back according to the content of the log file to complete the data. Return to work
Restore data according to the AOF file.
Copy the appendonly.aof file to the bin directory of the redis installation directory and restart the redis service.
Overview of AOF rewrite mechanism
The working principle of AOF is to append write operations to the file, and the redundant content of the file will become more and more.
When the size of the AOF file exceeds the set threshold, Redis will change the content of the AOF file compression
Principle
Redis will fork a new process, read the data in the memory (not read the old file), and rewrite it to a temporary file, and finally replace the old aof file
Configuration file display
[root@localhost ~]# vim /etc/redis/6379.conf
700 appendonly yes ##默认为no，改成yus开启持久化
704 appendfilename "appendonly.aof" ##文件名称
729 # appendfsync always ##同步持久化，每次发生数据变化会立刻写入磁盘
730 appendfsync everysec ##默认推荐，每秒异步记录次，默认值
731 # appendfsync no ##不同步，交给操作系统决定如何同步
752 no-appendfsync-on-rewrite no ##在日志进行BGREWRITEAOF时， 如果设置为yes表示新写操作不进行同步fsync,只暂存在缓冲区里，避免造成磁盘I0操作冲突，等重写完成后在写入，redis中默认为no
771 auto-aof-rewrite-percentage 100 ##当前AOF文件大小是上次日志重写时AOF文件大小两倍时，发生BGREWRITEAOF操作
772 auto-aof-rewrite-min-size 64mb ##当前AOF文件执行BGREWRITEAOF命令的最小值，避免刚开始启动Reids时由于文件尺寸较小导致频繁的BGREWRITEAOF.当AOF文件到达64M的时候，发生BGREWRITEAOF操作
796 aof-load-truncated yes ##忽略最后一条可能存在问题的指令
Copy the appendonly.aof file to the bin directory of the redis installation directory, restart the redis service, and you can restore the data
[root@localhost ~]# cd /var/lib/redis/6379/
[root@localhost 6379]# ls
appendonly.aof dump.rdb
Redis performance management
127.0.0.1:6379> INFO MEMORY
# Memory
used_memory:853504 ##Redis使用其分配器分配的内存大小，已经使用了的内存大小
used_memory_human:833.50K ##用户数据所占用的内存，就是你缓存的数据的大小
used_memory_rss:12820480 ##操作系统分配给Redis实例的内存大小，表示该进程所占物理内存的大小
used_memory_rss_human:12.23M
used_memory_peak:853504
used_memory_peak_human:833.50K
used_memory_peak_perc:100.01%
used_memory_overhead:841280
used_memory_startup:791416
used_memory_dataset:12224
used_memory_dataset_perc:19.69%
allocator_allocated:1608480
allocator_active:2019328
allocator_resident:9019392
total_system_memory:1907970048
total_system_memory_human:1.78G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.26
allocator_frag_bytes:410848
allocator_rss_ratio:4.47
allocator_rss_bytes:7000064
rss_overhead_ratio:1.42
rss_overhead_bytes:3801088
mem_fragmentation_ratio:15.78 ##内存碎片率，内存疑惑行为???
mem_fragmentation_bytes:12007992
mem_not_counted_for_evict:98
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:49694
mem_aof_buffer:98
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
Memory fragmentation rate
Operating system allocates memory Memory value used_ RSS redis divided value memory used
Used _memory calculated
memory fragmentation is inefficient dispensing OS / recovery due to the physical memory
is not contiguous physical memory allocation
tracking memory fragmentation rate appreciated redis The resource performance of the instance is very important. It
is reasonable that the memory fragmentation rate is slightly greater than 1. This value indicates that the memory fragmentation rate is relatively low. The
memory fragmentation rate exceeds 1.5, indicating that redis consumes 150% of the actual physical memory required, of which 50% is memory fragmentation Rate If the
memory fragmentation rate is lower than 1, it means that Redis memory allocation exceeds physical memory, and the operating system is currently swapping memory.
I saw an article written here that is quite detailed. The picture is also taken by him. Click here for the link
Solution to high fragmentation
1. Restarting the Redis server can invalidate the additional memory fragments and re-use them as new memory, so that the operating system can restore efficient memory management and modify the memory allocator.
Redis supports several different memory allocators of glibc's malloc, jemalloc11, and tcmalloc. Each allocator has a different implementation in memory allocation and fragmentation. It is not recommended for ordinary administrators to modify the Redis default memory allocator, because this requires a complete understanding of the differences between these memory allocators, and also recompile Redis.
2. New instructions can be used to manually recover memory fragments and configure monitoring for redis 4.0 and above. Better performance.
Warning This feature is experimental. However, stress testing was conducted even in production, and was manually tested by multiple engineers for a period of time.
3. Automatic (real-time) defragmentation allows the Redis server to compress the space between small data allocation and data release in memory, thereby allowing memory to be reclaimed .
[root@localhost 6379]# vim /etc/redis/6379.conf
1353 activedefrag yes
Memory usage
The memory usage of the redis instance exceeds the maximum available memory, and the operating system will start to
exchange memory and swap space to
avoid memory exchange.
Choose the size of the cached data
. Use the Hash data structure as much as possible.
Set the key expiration time
Recycle KEY
Ensure that redis's limited memory resources are allocated reasonably.
When the memory usage reaches the set maximum threshold, you need to choose a key recycling strategy.
By default, the recycling strategy is to prohibit deleting the
redis.conf configuration file to modify the maxmemory-policy attribute value
volatile-lru:使用LRU算法从已设置过期时间的数据集合中淘汰数据
volatile-ttl:从已设置过期时间的数据集合中挑选即将过期的数据淘汰（建议使用）
volatile-random:从已设置过期时间的数据集合中随机挑选数据淘汰
allkeys-lru:使用LRU算法从所有数据集合中淘汰数据
allkeys-random:从数据集合中任意选择数据淘汰
no-enviction:禁止淘汰数据