Introduction to Redis, commonly used commands and optimization

1. Overview of relational database and non-relational database

1. Relational database

(1) A relational database is a structured database, created on the basis of a relational model (two-dimensional table model), and is generally record-oriented.
(2) SQL statement (standard data query language) is a language based on relational database, used to perform retrieval and operation of data in relational database.
Note: Mainstream relational databases include Oracle, MySQL, SQL Server, Microsoft Access, DB2, etc.

2. Non-relational database

(1) NoSQL (NoSQL = Not Only SQL), which means "not just SQL", is the general term for non-relational databases.
(2) All databases other than mainstream relational databases are considered non-relational.
Note: The mainstream NoSQL databases include Redis, MongBD, Hbase, CouhDB, etc.

Second, the difference between relational databases and non-relational databases

1. Different data storage methods

The main difference between relational and non-relational databases is the way data is stored. Relational data is naturally in table format, so it is stored in the rows and columns of the data table. Data tables can be stored in association with each other, and it is easy to extract data.
In contrast, non-relational data is not suitable for storage in the rows and columns of the data table, but is grouped together in large chunks. Non-relational data is usually stored in data sets, like documents, key-value pairs, or graph structures. Your data and its characteristics are the primary factors influencing the choice of data storage and retrieval methods.

2. Different expansion methods

The biggest difference between SQL and NoSQL databases may be in the way of expansion. Of course, it must be expanded to support the increasing demand.
To support more concurrency, the SQL database is scaled vertically, that is to say, to increase the processing power and use a faster computer, so that the same data set can be processed faster. Because the data is stored in relational tables, the performance bottleneck of the operation may involve many tables, all of which need to be serviced by improving computer performance. Although the SQL database has a lot of room for expansion, it will definitely reach the upper limit of vertical expansion in the end.
The NoSQL database is scaled horizontally. Because non-relational data storage is naturally distributed, the expansion of NoSQL databases can share the load by adding more ordinary database servers (nodes) to the resource pool.

3. Different support for transactional

If data operations require high transactionality or complex data queries need to control the execution plan, then traditional SQL databases are your best choice in terms of performance and stability. SQL database supports fine-grained control of transaction atomicity and is easy to roll back transactions.
Although NoSQL databases can also use transaction operations, they cannot be compared with relational databases in terms of stability, so their real shining value is in the scalability of operations and the processing of large amounts of data.

Three, non-relational database

It can be used to deal with the three high problems of Web2.0 pure dynamic website type.
(1) High performance-high concurrent read and write requirements for the database
(2) Huge Storage-high-efficiency storage and access requirements for massive data
(3) High Scalability && High Availability-the
relationship between the high scalability and high availability requirements of the database Databases and non-relational databases have their own characteristics and application scenarios. The close combination of the two will bring new ideas to the development of Web2.0 databases. Let relational databases focus on relationships, and non-relational databases focus on storage. For example, in a MySQL database environment where read and write are separated, frequently accessed data can be stored in a non-relational database to improve access speed.
Summary:
Relational database:
instance -> database -> table (table) -> record row (row), data field (column)
non-relational database:
instance -> database -> collection (collection) -> key-value pair ( key-value)
Non-relational databases do not need to manually build databases and collections (tables).

Four, Redis introduction

Redis is an open source NoSQL database written in C language.
Redis runs on memory and supports persistence. It uses a key-value (key-value pair) storage form, which is an indispensable part of the current distributed architecture.
The Redis server program is a single-process model, that is, multiple Redis processes can be started at the same time on a server, and the actual processing speed of Redis is completely dependent on the execution efficiency of the main process. If only one Redis process is running on the server, when multiple clients access at the same time, the processing capacity of the server will be reduced to a certain extent; if multiple Redis processes are opened on the same server, Redis is improving the concurrent processing capacity At the same time, it will put a lot of pressure on the server's CPU. That is: in the actual production environment, it is necessary to decide how many Redis processes to start according to actual needs. If you have higher requirements for high concurrency, you may consider opening multiple processes on the same server. If CPU resources are tight, a single process can be used.

1. Redis advantages

(1) Very high data read and write speed: the data read speed can reach up to 110,000 times/s, and the data write speed can reach up to 81,000 times/s.
(2) Support rich data types: support key-value, Strings, Lists, Hashes, Sets and Ordered Sets and other data type operations.
(3) Support data persistence: the data in the memory can be saved in the disk, and it can be loaded again for use when restarting.
(4) Atomicity: All Redis operations are atomic.
(5) Support data backup: namely data backup in master-salve mode.
Redis is a memory-based database, and caching is one of its most commonly used scenarios. In addition, the common application scenarios of Redis also include the operation of obtaining the latest N data, ranking applications, counter applications, storage relationships, real-time analysis systems, and log records.

Five, Redis installation and deployment

systemctl stop firewalld
setenforce 0

yum install -y gcc gcc-c++ make

tar zxvf redis-5.0.7.tar.gz -C /opt/

cd /opt/redis-5.0.7/
make
make PREFIX=/usr/local/redis install

cd /opt/redis-5.0.7/utils
./install_server.sh
……
慢慢回车
Please select the redis executable path []
手动输入
/usr/local/redis/bin/redis-server

Selected config:
Port           : 6379								#默认侦听端口为6379
Config file    : /etc/redis/6379.conf				#配置文件路径
Log file       : /var/log/redis_6379.log			#日志文件路径
Data dir       : /var/lib/redis/6379				#数据文件路径
Executable     : /usr/local/redis/bin/redis-server	#可执行文件路径
Cli Executable : /usr/local/bin/redis-cli			#客户端命令工具

ln -s /usr/local/redis/bin/* /usr/local/bin/

/etc/init.d/redis_6379 stop				#停止
/etc/init.d/redis_6379 start			#启动
/etc/init.d/redis_6379 restart			#重启
/etc/init.d/redis_6379 status			#状态

Modify the configuration /etc/redis/6379.conf parameters

vim /etc/redis/6379.conf

70行，添加 监听的主机地址
bind 127.0.0.1 192.168.177.11				

93行，Redis默认的监听端口
port 6379									

137行，启用守护进程
daemonize yes							

159行，指定 PID 文件
pidfile /var/run/redis_6379.pid				

167行，日志级别
loglevel notice								

172行，指定日志文件
logfile /var/log/redis_6379.log				

/etc/init.d/redis_6379 restart

Six, Redis command tool

1. Redis-cli command line tool

redis-server		用于启动 Redis 的工具
redis-benchmark		用于检测 Redis 在本机的运行效率
redis-check-aof		修复 AOF 持久化文件
redis-check-rdb		修复 RDB 持久化文件
redis-cli	 		Redis命令行工具

Syntax: redis-cli -h host -p port -a password
-h specifies the remote host
-p specifies the port number of the Redis service
-a specifies the password, and the database password is not set, the -a option can be omitted.
Note: If you do not add any options, Then use 127.0.0.1:6379 to connect to the Redis database on this machine

2. Redis-benchmark test tool

redis-benchmark is the official Redis performance testing tool that can effectively test the performance of Redis services.
Basic test syntax: redis-benchmark [option] [option value]

-h	 指定服务器主机名。
-p	 指定服务器端口。
-s	 指定服务器 socket
-c	 指定并发连接数。
-n  指定请求数。
-d	 以字节的形式指定 SET/GET 值的数据大小。
-k	 1=keep alive 0=reconnect 。
-r	 SET/GET/INCR 使用随机 key, SADD 使用随机值。
-P  通过管道传输请求。
-q	 强制退出 redis。仅显示 query/sec 值。
–csv	 以 CSV 格式输出。
-l	生成循环，永久执行测试。
-t	仅运行以逗号分隔的测试命令列表。
-I	Idle 模式。仅打开 N 个 idle 连接并等待。

Send 100 concurrent connections and 100000 requests to the Redis server with IP address 192.168.177.8 and port 6379 to test performance

redis-benchmark -h 192.168.177.8 -p 6379 -c 100 -n 100000

Test the performance of accessing data packets with a size of 100 bytes

redis-benchmark -h 192.168.177.8 -p 6379 -q -d 100

Test the performance of the Redis service on this machine during set and lpush operations

redis-benchmark -t set,lpush -n 100000 -q

3. Common commands for Redis database

set		存放数据，命令格式为 set key value
get		获取数据，命令格式为 get key
keys 	命令可以取符合规则的键值列表，通常情况可以结合*、?等选项来使用。
exists 	命令可以判断键值是否存在。
del 	命令可以删除当前数据库的指定 key。
type 	命令可以获取 key 对应的 value 值类型。

rename 命令是对已有 key 进行重命名。（覆盖）
命令格式：rename 源key 目标key

renamenx 命令的作用是对已有 key 进行重命名，并检测新名是否存在，如果目标 key 存在则不进行重命名。（不覆盖）
命令格式：renamenx 源key 目标key

使用config set requirepass password命令设置密码
使用config get requirepass命令查看密码（一旦设置密码，必须先验证通过密码，否则所有操作不可用）

4. Redis multi-database commonly used commands

Redis supports multiple databases. Redis contains 16 databases by default, and the database names are sequentially named with numbers 0-15. Multiple databases are independent of each other and do not interfere with each other.
(1) Switch between multiple databases
Command format: select serial number After
using redis-cli to connect to the Redis database, the database with serial number 0 is used by default.

127.0.0.1:6379> select 10			#切换至序号为 10 的数据库
127.0.0.1:6379[10]> select 15		#切换至序号为 15 的数据库
127.0.0.1:6379[15]> select 0		#切换至序号为 0 的数据库

(2) Move data between multiple databases

格式：move 键值 序号
例：
set edg 17
get edg

select 5
get edg

select 0
move edg 5
get edg

select 5
get edg

Clear the data in the database
FLUSHDB: Clear the current database data
FLUSHALL: Clear all the database data

Seven, Redis high availability

1. In a web server, high availability refers to the time during which the server can be accessed normally. The measurement standard is how long it can provide normal services (99.9%, 99.99%, 99.999%, etc.).
However, in the context of Redis, the meaning of high availability seems to be broader. In addition to ensuring the provision of normal services (such as master-slave separation, rapid disaster recovery technology), it is also necessary to consider the expansion of data capacity, and the safety of data will not be lost.
2. In Redis, the technologies to achieve high availability mainly include persistence, master-slave replication, sentinel and cluster
(1) persistence: persistence is the simplest method of high availability (sometimes not even classified as a means of high availability) , The main function is data backup, that is, the data is stored on the hard disk to ensure that the data will not be lost due to the exit of the process.
(2) Master-slave replication: Master-slave replication is the basis of high-availability Redis. Sentinel and clusters are all based on master-slave replication to achieve high availability. Master-slave replication mainly implements multi-machine backup of data, as well as load balancing and simple failure recovery for read operations. Defects: failure recovery cannot be automated; write operations cannot be load balanced; storage capacity is limited by a single machine.
(3) Sentinel: On the basis of master-slave replication, Sentinel realizes automatic failure recovery. Defects: Write operations cannot be load balanced; storage capacity is limited by a single machine.
(4) Clustering: Through clustering, Redis solves the problem that write operations cannot be load balanced, and storage capacity is limited by a single machine, and realizes a relatively complete high-availability solution.
Note: Persistence function: Redis is an in-memory database, and data is stored in memory. In order to avoid permanent loss of data after the Redis process exits abnormally due to server power failure and other reasons, it is necessary to periodically convert the data in Redis in some form ( Data or commands) are saved from the memory to the hard disk; when Redis is restarted next time, the persistent file is used to achieve data recovery. In addition, for disaster backup, the persistent files can be copied to a remote location.

Eight, Redis provides two ways for persistence

1, RDB endurance

RDB persistence refers to the generation of snapshots of the data in the current process in the memory to the hard disk (hence also called snapshot persistence) within a specified time interval, using binary compression for storage, and the saved file suffix is ​​rdb; when Redis is restarted When, you can read the snapshot file to restore the data.
(1) Trigger mode
a, manual trigger
save command and bgsave command can generate RDB file.
The save command will block the Redis server process until the RDB file is created. During the blocking period of the Redis server, the server cannot process any command requests.
The bgsave command will create a child process, and the child process is responsible for creating the RDB file, and the parent process (ie, the Redis main process) continues to process the request.

During the execution of the bgsave command, only the fork child process will block the server. For the save command, the whole process will block the server. Therefore, save has been basically abandoned, and the use of save must be eliminated in the online environment.
b. Automatic triggering
When RDB persistence is automatically triggered, Redis will also choose bgsave instead of save for persistence.
The most common situation for automatic triggering is to pass save mn in the configuration file, specifying that bgsave will be triggered when n changes occur within m seconds.

vim /etc/redis/6379.conf

219行以下三个save条件满足任意一个时，都会引起bgsave的调用

save 900 1 ：当时间到900秒时，如果redis数据发生了至少1次变化，则执行bgsave
save 300 10 ：当时间到300秒时，如果redis数据发生了至少10次变化，则执行bgsave
save 60 10000 ：当时间到60秒时，如果redis数据发生了至少10000次变化，则执行bgsave

254行指定RDB文件名

dbfilename dump.rdb

264行指定RDB文件和AOF文件所在目录

dir /var/lib/redis/6379

242行是否开启RDB文件压缩

rdbcompression yes

(2) Execution process
a. The Redis parent process first judges whether it is currently executing save or a child process of bgsave/bgrewriteaof. If it is executing, the bgsave command will return directly. The child processes of bgsave/bgrewriteaof cannot be executed at the same time, mainly based on performance considerations: two concurrent child processes execute a large number of disk write operations at the same time, which may cause serious performance problems.
b. The parent process executes the fork operation to create a child process. During this process, the parent process is blocked and Redis cannot execute any commands from the client.
c. After the parent process forks, the bgsave command returns the "Background saving started" message and no longer blocks the parent process , And can respond to other commands
d. The child process creates an RDB file, generates a temporary snapshot file based on the memory snapshot of the parent process, and atomically replaces the original file after completion
. The child process sends a signal to the parent process to indicate completion, and the parent process updates statistics
(3) Loading the
RDB file at startup is automatically executed when the server starts, and there is no special command. However, because AOF has a higher priority, when AOF is turned on, Redis will load AOF files first to restore data; only when AOF is turned off, will the RDB file be detected when the Redis server is started and automatically loaded. The server is blocked while loading the RDB file until the loading is complete.
When Redis loads the RDB file, it will verify the RDB file. If the file is damaged, an error will be printed in the log and Redis will fail to start.

2. AOF persistence

(1) The difference between AOF and RDB: RDB persistence is to write process data to a file, while AOF persistence is to record each write and delete command executed by Redis into a separate log file, and query operations will not be recorded; When Redis restarts, execute the commands in the AOF file again to restore the data.
Compared with RDB, AOF has better real-time performance, so it has become the mainstream persistence solution.
(2) Enable AOF
Redis server defaults to enable RDB and disable AOF; to enable AOF, you need to configure in the configuration file:

vim /etc/redis/6379.conf

700行修改，开启AOF

appendonly yes

704行指定AOF文件名称

appendfilename "appendonly.aof"

796行是否忽略最后一条可能存在问题的指令

aof-load-truncated yes

/etc/init.d/redis_6379 restart

(2) Execution process
a. Command append (append)
Redis first appends the write command to the buffer instead of directly writing to the file, mainly to avoid writing directly to the hard disk every time a write command is issued, causing hard disk IO to become a Redis load Bottleneck.
The format of the command addition is the protocol format of the Redis command request. It is a plain text format that has the advantages of good compatibility, strong readability, easy processing, simple operation and avoiding secondary overhead. In the AOF file, except for the select command used to specify the database (for example, select 0 to select the database No. 0) is added by Redis, the rest are write commands sent by the client.
b. File writing (write) and file synchronization (sync)
Redis provides a variety of AOF cache synchronization file strategies. The strategies involve the write function and fsync function of the operating system. The description is as follows:
In order to improve the efficiency of file writing, In modern operating systems, when a user calls the write function to write data into a file, the operating system usually temporarily stores the data in a memory buffer, and when the buffer is filled or exceeds the specified time limit, the buffer is actually saved The data is written to the hard disk. Although this operation improves efficiency, it also brings security issues: if the computer is shut down, the data in the memory buffer will be lost; therefore, the system also provides synchronization functions such as fsync and fdatasync, which can force the operating system to immediately put the buffer in the buffer. The data is written to the hard disk to ensure data security.
Note: There are three synchronization methods for the synchronization file strategy of the AOF buffer area.
● appendfsync always: After the command is written to aof_buf, the system fsync operation is immediately called to synchronize to the AOF file, and the thread returns after the fsync is completed. In this case, each write command must be synchronized to the AOF file, and hard disk IO becomes a performance bottleneck. Redis can only support about a few hundred TPS writes, which severely reduces the performance of Redis; even with solid state drives (SSD), It can only process tens of thousands of commands per second, and it will greatly reduce the lifespan of the SSD.
●appendfsync no: After the command is written to aof_buf, the system write operation is called, and the AOF file is not synchronized with fsync; the synchronization is taken care of by the operating system, and the synchronization period is usually 30 seconds. In this case, the time of file synchronization is uncontrollable, and there will be a lot of data accumulated in the buffer, and data security cannot be guaranteed.
●appendfsync everysec: After the command is written to aof_buf, the system write operation is called, and the thread returns after the write is completed; the fsync synchronization file operation is called once per second by a dedicated thread. Everysec is a compromise between the aforementioned two strategies. It is a balance between performance and data security. Therefore, it is the default configuration of Redis and our recommended configuration.
c. File rewrite (rewrite)
With the passage of time, the Redis server executes more and more write commands, and the AOF file will become larger and larger; too large AOF file will not only affect the normal operation of the server, but also cause data Recovery takes too long.

File rewriting is to rewrite the AOF file within a specified period to reduce the volume of the AOF file. It should be noted that AOF rewrite is to convert the data in the Redis process into write commands and synchronize to the new AOF file; it will not perform any read or write operations on the old AOF file!

Another point to note about file rewriting is: for AOF persistence, although file rewriting is strongly recommended, it is not necessary; even without file rewriting, data can be persisted and started in Redis Import at any time; therefore, in some implementations, automatic file rewriting is turned off, and then executed at a certain time of day through a timed task.
The reason why file rewriting can compress AOF files is:
● Expired data is no longer written to the file
● Invalid commands are no longer written to the file: For example, some data is repeatedly set (set mykey v1, set mykey v2), some The data is deleted (sadd myset v1, del myset) etc.
●Multiple commands can be combined into one: For example, sadd myset v1, sadd myset v2, sadd myset v3 can be combined into sad myset v1 v2 v3.

From the above content, it can be seen that since the commands executed by AOF are reduced after rewriting, file rewriting can not only reduce the space occupied by the file, but also speed up the recovery.

File rewriting triggers are divided into manual triggers and automatic triggers:
● Manual trigger: directly call the bgrewriteaof command. The execution of this command is similar to that of bgsave: both fork subprocesses perform specific tasks, and they are only blocked when fork.
●Automatic trigger: automatically execute BGREWRITEAOF by setting the auto-aof-rewrite-min-size option and auto-aof-rewrite-percentage option. Only when the two options of auto-aof-rewrite-min-size and auto-aof-rewrite-percentage are met at the same time, will the AOF rewrite be automatically triggered, that is, the bgrewriteaof operation.
●auto-aof-rewrite-percentage 100: When the current AOF file size (ie aof_current_size) is twice the AOF file size (aof_base_size) when the log was rewritten last time, the BGREWRITEAOF operation occurred
●auto-aof-rewrite-min-size 64mb: The minimum value of the current AOF file to execute the BGREWRITEAOF command to avoid frequent BGREWRITEAOF
file rewriting due to the small file size when Reids is first started as follows:
a. The Redis parent process first determines whether there is currently a child process that is executing bgsave/bgrewriteaof. If it exists, the bgrewriteaof command will return directly. If there is a bgsave command, it will be executed after the bgsave execution is complete.
b. The parent process executes a fork operation to create a child process, and the parent process is blocked in this process.
c1. After the parent process forks, the bgrewriteaof command returns the "Background append only file rewrite started" message and no longer blocks the parent process, and can respond to other commands. All Redis write commands are still written into the AOF buffer and synchronized to the hard disk according to the appendfsync strategy to ensure that the original AOF mechanism is correct.
c2. Since the fork operation uses copy-on-write technology, the child process can only share the memory data during the fork operation. Since the parent process is still responding to commands, Redis uses the AOF rewrite buffer (aof_rewrite_buf) to save this part of the data to prevent this part of data from being lost during the generation of a new AOF file. In other words, during the execution of bgrewriteaof, Redis write commands are simultaneously appended to the aof_buf and aof_rewirte_buf buffers.
d. According to the memory snapshot, the child process writes to the new AOF file according to the command merging rules.
e1. After the child process finishes writing the new AOF file, it sends a signal to the parent process, and the parent process updates the statistical information, which can be viewed through info persistence.
e2. The parent process writes the data in the AOF rewrite buffer to the new AOF file, thus ensuring that the database state saved in the new AOF file is consistent with the current state of the server.
e3. Replace the old file with the new AOF file to complete the AOF rewrite.
(3) Load at startup.
When AOF is turned on, Redis will load the AOF file first to restore data when it starts; RDB will be loaded only when AOF is turned off File recovery data.
When AOF is turned on, but the AOF file does not exist, it will not be loaded even if the RDB file exists.
When Redis loads the AOF file, it will verify the AOF file. If the file is damaged, an error will be printed in the log and Redis will fail to start. But if the end of the AOF file is incomplete (sudden shutdown of the machine may cause the end of the file to be incomplete), and the aof-load-truncated parameter is enabled, a warning will be output in the log, and Redis will ignore the end of the AOF file and start successfully. The aof-load-truncated parameter is enabled by default.

3. The advantages and disadvantages of RDB and AOF

● RDB persistence
Advantages: RDB files are compact, small in size, fast in network transmission, suitable for full copy; recovery speed is much faster than AOF. Of course, compared with AOF, one of the most important advantages of RDB is that it has a relatively small impact on performance.

Disadvantages: The fatal disadvantage of RDB files is that the persistence of data snapshots determines that real-time persistence is inevitably impossible. As data is becoming more and more important today, a large amount of data loss is often unacceptable, so AOF persistence Become the mainstream. In addition, RDB files need to meet a specific format, which has poor compatibility (for example, the old version of Redis is not compatible with the new version of RDB files).
For RDB persistence, on the one hand, the Redis main process will be blocked when bgsave is performing a fork operation, on the other hand, writing data to the hard disk by the child process will also bring IO pressure.

● AOF Persistence
Corresponding to RDB persistence, AOF has the advantage of supporting second-level persistence and good compatibility. The disadvantage is that the file is large, the recovery speed is slow, and the performance impact is large.
For AOF persistence, the frequency of writing data to the hard disk is greatly increased (second level under the everysec strategy), and the IO pressure is greater, and it may even cause additional blocking problems in AOF.
The rewriting of the AOF file is similar to the bgsave of RDB, and there will be blocking when fork and the IO pressure of the child process. Relatively speaking, because AOF writes data to the hard disk more frequently, it has a greater impact on the performance of the Redis main process.

Nine, Redis performance management

View Redis memory usage

redis-cli -h 192.168.177.8 -p 6379
192.168.177.8:6379> info memory

1. Memory fragmentation rate
●It is reasonable for the memory fragmentation rate to be slightly greater than 1, this value indicates that the memory fragmentation rate is relatively low. The
memory fragmentation rate exceeds 1.5, indicating that Redis consumes 150% of the actual physical memory required, of which 50% is the memory fragmentation rate . You need to enter the shutdown save command on the redis-cli tool and restart the Redis server.
● If the memory fragmentation rate is lower than 1, it means that the Redis memory allocation exceeds the physical memory, and the operating system is swapping memory. Need to increase available physical memory or reduce Redis memory usage.
2. Memory usage rate
Methods to avoid memory swaps
● Choose to install Redis instances for the size of cached data
● Use Hash data structure storage as much as possible
● Set the expiration time of the key
3. Internally recycle the key to
ensure a reasonable allocation of redis's limited memory resources.
When the set maximum threshold is reached, a key recycling strategy needs to be selected. By default, the recycling strategy is to prohibit deletion.

vim /etc/redis/6379.conf

598取消注释

maxmemory-policy noenviction

● Volatile-lru uses the LRU algorithm to eliminate data
from the data set with the expiration time set ● volatile-ttl selects the data that is about to expire
from the data set with the expiration time set ● Volatile-random from the data set with the expiration time set Randomly select data to eliminate
●allkeys-lru Use the LRU algorithm to eliminate data from all data sets
●allkeys-random select data from the data set arbitrarily to eliminate
● noenviction prohibit elimination of data