Redis configuration and optimization of NoSQL
- One, relational database and non-relational database
- Two, the difference between relational database and non-relational database
- Third, the background of non-relational databases
- Four, Redis introduction
- Five, Redis installation and deployment
- Six, Redis command tool
- Seven, redis-cli command line tool
- Eight, redis-benchmark test tool
- Nine, Redis database commonly used commands
-
- The keys command can take a list of key values that comply with the rules, usually in combination with *,? and other options.
- The exists command can determine whether the key value exists
- The del command can delete the specified key of the current database
- The type command can get the value type corresponding to the key
- The rename command is to rename an existing key. (cover)
- The function of the renamenx command is to rename the existing key and check whether the new name exists. If the target key exists, it will not be renamed. (Not covered)
- The function of the dbsize command is to view the number of keys in the current database
- Use the config set requirepass yourpassword command to set the password
- Use the config get requirepass command to view the password (Once the password is set, the password must be verified first, otherwise all operations are unavailable)
- Ten, Redis multi-database commonly used commands
- Eleven, Redis high availability
One, relational database and non-relational database
1.1 Relational database
A relational database is a structured database, created on the basis of a relational model (two-dimensional table model), and is generally record-oriented.
SQL statement (standard data query language) is a language based on relational database, used to perform retrieval and operation of data in relational database.
Mainstream relational databases include Oracle, MySQL, SQL Server, Microsoft Access, DB2, etc.
1.2 Non-relational database
NoSQL (NoSQL = Not Only SQL), which means "not just SQL", is a general term for non-relational databases.
Except for the mainstream relational databases, all databases are considered non-relational.
The mainstream NoSQL databases include Redis, MongBD, Hbase, Memcached, etc.
Two, the difference between relational database and non-relational database
2.1 Different data storage methods
The main difference between relational and non-relational databases is the way data is stored . Relational data is naturally in table format , so it is stored in the rows and columns of the data table . Data tables can be stored in association with each other, and data can be easily extracted.
In contrast, non-relational data is not suitable for storage in the rows and columns of the data table, but is grouped together in large chunks. Non-relational data is typically stored in the data set , just the document, or of the key structure of FIG . Your data and its characteristics are the primary factors that influence the choice of data storage and retrieval methods.
2.2 Different expansion methods
The biggest difference between SQL and NoSQL databases may be in the way of expansion. Of course, it must be expanded to support the increasing demand.
To support more concurrency, the SQL database is scaled vertically , that is to say, to increase the processing power and use a faster computer, so that the same data set can be processed faster. Because the data is stored in relational tables, the performance bottleneck of the operation may involve many tables, which need to be overcome by improving computer performance. Although the SQL database has a lot of room for expansion, it will eventually reach the upper limit of vertical expansion.
The NoSQL database is scaled horizontally . Because non-relational data storage is naturally distributed , the expansion of NoSQL databases can share the load by adding more ordinary database servers (nodes) to the resource pool .
2.3 Different support for transactional
If data operations require high transactionality or complex data queries need to control the execution plan, then traditional SQL databases are your best choice in terms of performance and stability. SQL database supports fine-grained control of transaction atomicity and is easy to roll back transactions.
Although NoSQL databases can also use transaction operations, they cannot be compared with relational databases in terms of stability, so their real shining value is in the scalability of operations and the processing of large amounts of data.
Third, the background of non-relational databases
It can be used to deal with the three high problems of Web2.0 pure dynamic website type.
(1) High performance-high concurrent read and write requirements for the database
(2) HugeStorage-high-efficiency storage and access requirements for massive data
(3) High Scalability && High Availability-high scalability and high availability requirements for the database
Both relational databases and non-relational databases have their own characteristics and application scenarios. The close combination of the two will bring new ideas to the development of Web2.0 databases. Let relational databases focus on relations, and non-relational databases focus on storage. For example, in a MySQL database environment where read and write are separated, frequently accessed data can be stored in a non-relational database to improve access speed.
to sum up
Relational Database
Example -> database -> table -> record row (row), data field (column)
Non-relational database:
Instance -> database -> collection -> key-value
non-relational databases do not need to manually build databases and collections (tables).
Four, Redis introduction
Redis is an open source NoSQL database written in C language.
Redis runs on memory and supports persistence. It uses a key-value (key-value pair) storage form, which is an indispensable part of the distributed architecture today.
The Redis server program is a single-process model , that is, multiple Redis processes can be started at the same time on a server, and the actual processing speed of Redis is completely dependent on the execution efficiency of the main process.
If only one Redis process is running on the server, when multiple clients access at the same time, the processing capacity of the server will be reduced to a certain extent;
if multiple Redis processes are started on the same server, Redis is improving the concurrent processing capacity At the same time, it will put a lot of pressure on the server's CPU. That is: in the actual production environment, it is necessary to decide how many Redis processes to start according to actual needs.
If the requirements for high concurrency are higher, you may consider opening multiple processes on the same server.
If CPU resources are tight, a single process can be used.
4.1 Redis has the following advantages
(1) Very high data read and write speed: the data read speed can reach up to 110,000 times/s, and the data write speed can reach up to 81,000 times/s.
(2) Support rich data types: support key-value, Strings, Lists, Hashes, Sets and Ordered Sets and other data type operations.
(3) Support data persistence: the data in the memory can be saved in the disk, and can be loaded again for use when restarting.
(4) Atomicity: All Redis operations are atomic.
(5) Support data backup: namely data backup in master-salve mode.
Redis is a memory-based database, and caching is one of its most commonly used scenarios. In addition, common application scenarios of Redis also include the operation of obtaining the latest N data, ranking applications, counter applications, storage relationships, real-time analysis systems, and log records.
Five, Redis installation and deployment
systemctl stop firewalld
setenforce 0
yum install -y gcc gcc-c++ make
tar zxvf redis-5.0.7.tar.gz -C /opt/
cd /opt/redis-5.0.7/
make
make PREFIX=/usr/local/redis install
#由于Redis源码包中直接提供了Makefile 文件，所以在解压完软件包后，不用先执行./configure进行配置，可直接执行make与make install 命令进行安装。
#在/utils里执行软件包提供的install_server.sh脚本文件设置Redis服务所需要的相关配置文件
cd /opt/redis-5.0.7/utils
./install_server.sh
...... #一直回车
Please select the redis executable path [/usr/local/bin/redis-server] /usr/local/redis/bin/redis-server
#需要手动修改为/usr/local/redis/bin/redis-server，注意要一次性正确输入
Selected config:
Port ：6379 #默认侦听端口为6379
Config file ：/etc/redis/6379.conf #配置文件路径
Log file ：/var/log/redis_6379.1og #日志文件路径
Data dir ：/var/lib/redis/6379 #数据文件路径
Executable ：/usr/local/redis/bin/redis-server #可执行文件路径
Cli Executable ：/usr/local/bin/redis-cli #客户端命令工具
#把redis的可执行程序文件放入路径环境变量的目录中便于系统识别
ln -s /usr/local/redis/bin/* /usr/local/bin/
#当install_server.sh 脚本运行完毕，Redis 服务就已经启动，默认侦听端口为6379
netstat -natp | grep redis
5.1 Redis service control
/etc/init.d/redis_6379 stop #停止
/etc/init.d/redis_6379 start #启动
/etc/init.d/redis_6379 restart #重启
/etc/init.d/redis_6379 status #状态
#修改配置/etc/redis/6379.conf参数
vim /etc/redis/6379.conf
bind 127.0.0.1 192.168.238.10 #70行，添加监听的主机地址
port 6379 #93行，Redis默认的监听端口
daemonize yes #137行，启用守护进程
pidfile /var/run/redis_6379.pid #159行，指定PID 文件
loglevel notice #167行，日志级别
logfile /var/log/redis_6379.log #172行，指定日志文件
/etc/init.d/redis_6379 restart
Six, Redis command tool
| Redis command tool | effect |
|---|---|
| redis-server | Tools for starting Redis |
| redis-benchmark | Used to detect the operating efficiency of Redis on the machine |
| redis-check-aof | Repair AOF persistent files |
| redis -check- rdb | Repair RDB persistent files |
| redis-cli | Redis command line tool |
Seven, redis-cli command line tool
语法：redis-cli -h host -p port -a password
-h：指定远程主机
-p：指定Redis服务的端口号
-a：指定密码，未设置数据库密码可以省略-a选项
若不添加任何选项表示，则使用127.0.0.1:6379 连接本机上的Redis 数据库
redis-cli -h 192.168.238.10 -p 6379
Eight, redis-benchmark test tool
redis-benchmark is an official Redis performance testing tool that can effectively test the performance of Redis services.
基本的测试语法: redis-benchmark [选项] [选项值]。
-h：指定服务器主机名。
-p：指定服务器端口。
-s：指定服务器socket
-c：指定并发连接数。
-n：指定请求数。
-d：以字节的形式指定SET/GET值的数据大小。
-k：1=keep alive 0=reconnect。
-r：SET/GET/INCR 使用随机key,SADD 使用随机值。
-P：通过管道传输<numreq>请求。
-q：强制退出redis。仅显示query/sec值。
--csv：以CSV格式输出。
-l：生成循环，永久执行测试。
-t：仅运行以逗号分隔的测试命令列表。
-I：Idle模式。仅打开N个idle连接并等待。
向IP地址为192.168.238.10、 端口为6379 的Redis 服务器发送5个并发连接与100000 个请求测试性能
redis-benchmark -h 192.168.238.10 -p 6379 -c 5 -n 100000
测试存取大小为100字节的数据包的性能
redis-benchmark -h 192.168.238.10 -p 6379 -q -d 100
测试本机上Redis 服务在进行set与lpush 操作时的性能
redis-benchmark -t set, lpush -n 100000 -q
Nine, Redis database commonly used commands
set：存放数据，命令格式为set key value
get：获取数据，命令格式为get key
127.0.0.1:6379> set teacher zhangsan
OK
127.0.0.1:6379> get teacher
" zhangsan"
The keys command can take a list of key values that comply with the rules, usually in combination with *,? and other options.
127.0.0.1:6379> set k1 1
127.0.0.1:6379> set k2 2
127.0.0.1:6379> set k3 3
127.0.0.1:6379> set v1 4
127.0.0.1:6379> set v5 5
127.0.0.1:6379> set v22 5
127.0.0.1:6379> KEYS * #查看当前数据库中所有键
127.0.0.1:6379> KEYS v* #查看当前数据库中以v开头的数据
127.0.0.1:6379> KEYS v? #查看当前数据库中以v开头后面包含任意一位的数据
127.0.0.1:6379> KEYS v?? #查看当前数据库中以v开头v开头后面包含任意两位的数据
The exists command can determine whether the key value exists
127.0.0.1:6379> exists teacher #判断teacher 键是否存在
(integer) 1 # 1表示teacher 键是存在
127.0.0.1:6379> exists tea
(integer) 0 #0表示tea键不存在
The del command can delete the specified key of the current database
127.0.0.1:6379> keys *
127.0.0.1:6379> del v5
127.0.0.1:6379> get v5
The type command can get the value type corresponding to the key
127.0.0.1:6379> type k1
string
The rename command is to rename an existing key. (cover)
命令格式：rename 源key 目标key
When using the rename command to rename, the target key will be renamed regardless of whether the target key exists, and the value of the source key will overwrite the value of the target key. In actual use, it is recommended to use the exists command to check whether the target key exists, and then decide whether to execute the rename command to avoid overwriting important data.
127.0.0.1:6379> keys v*
1) "v1"
2) "v22"
127.0.0.1:6379> rename v22 v2
0K
127.0.0.1:6379> keys v*
1) "v1"
2) "v2"
127.0.0.1:6379> get v1
" 4"
127.0.0.1:6379> get v2
127.0.0.1:6379> rename v1 v2
0K
127.0.0.1:6379> get v1
(nil)
127.0.0.1:6379> get v2
"4"
The function of the renamenx command is to rename the existing key and check whether the new name exists. If the target key exists, it will not be renamed. (Not covered)
命令格式: renamenx 源key 目标key
127.0.0.1:6379> keys *
127.0.0.1:6379> get teacher
"zhangsan"
127.0.0.1:6379> get v2
"4"
127.0.0.1:6379> renamenx v2 teacher
(integer) 0
127.0.0.1:6379> keys *
127.0.0.1:6379> get teacher
"zhangsan"
127.0.0.1:6379> get v2
"4"
The function of the dbsize command is to view the number of keys in the current database
127.0.0.1:6379> dbsize
Use the config set requirepass yourpassword command to set the password
127.0.0.1:6379> config set requirepass 123456
Use the config get requirepass command to view the password (Once the password is set, the password must be verified first, otherwise all operations are unavailable)
127.0.0.1:6379> auth 123456
127.0.0.1:6379> config get requirepass
Ten, Redis multi-database commonly used commands
Redis supports multiple databases. Redis contains 16 databases by default. The database names are sequentially named with numbers 0-15.
Multiple databases are independent of each other and do not interfere with each other.
10.1 Switch between multiple databases
命令格式：select 序号
使用redis-cli 连接Redis 数据库后，默认使用的是序号为0的数据库。
127.0.0.1:6379> select 10 #切换至序号为10的数据库
127.0.0.1:6379[10]> select 15 #切换至序号为15的数据库
127.0.0.1:6379[15]> select 0 #切换至序号为0的数据库
10.2 Moving data between multiple databases
格式：move 键值序号
127.0.0.1:6379> set k1 100
OK
127.0.0.1:6379> get k1
"100"
127.0.0.1:6379> select 1
0K
127.0.0.1:6379[1]> get k1
(nil)
127.0.0.1:6379[1]> select 0 #切换至目标数据库0
OK
127.0.0.1:6379> get k1 #查看目标数据是否存在
"100"
127.0.0.1:6379> move k1 1 #将数据库0中k1移动到数据库1中
(integer) 1
127.0.0.1:6379> select 1 #切换至目标数据库1
0K
127.0.0.1:6379[1]> get k1 #查看被移动数据
"100"
127.0.0.1:6379[1]> select 0
OK
127.0.0.1:6379> get k1 #在数据库0中无法查看到k1的值
(nil)
10.3 Clear data in the database
FLUSHDB：清空当前数据库数据
FLUSHALL：清空所有数据库的数据，慎用!
Eleven, Redis high availability
In a web server, high availability refers to the time during which the server can be accessed normally. The measurement standard is how long it can provide normal services (99.9%, 99.99%, 99.999%, etc.).
However, in the context of Redis, high availability The meaning of is seems to be broader. In addition to guaranteeing the provision of normal services (such as master-slave separation, rapid disaster recovery technology), it is also necessary to consider the expansion of data capacity and data security from loss.
In Redis, the technologies to achieve high availability mainly include persistence, master-slave replication, sentinel, and cluster. The following explains their roles and what problems they solve.
●Persistence: Persistence is the simplest high-availability method (sometimes not even classified as a high-availability method). The main function is data backup, that is, data is stored on the hard disk to ensure that data will not be lost due to process exit.
●Master-slave replication: Master-slave replication is the basis of high-availability Redis. Sentinel and clusters are all based on master-slave replication to achieve high availability. Master-slave replication mainly implements multi-machine backup of data, as well as load balancing for read operations and simple failure recovery. Defects: failure recovery cannot be automated; write operations cannot be load balanced; storage capacity is limited by a single machine.
●Sentinel: On the basis of master-slave replication, the sentinel realizes automatic failure recovery. Defect: Write operations cannot be load balanced: storage capacity is limited by a single machine.
●Cluster: Through clustering, Redis solves the problem that write operations cannot be load balanced, and storage capacity is limited by a single machine, and achieves a relatively complete high-availability solution.
11.1 Redis persistence
Persistence function: Redis is an in-memory database, and data is stored in memory. In order to avoid permanent loss of data after the Redis process exits abnormally due to server power failure, etc., it is necessary to periodically convert the data in Redis in some form (data or Command) Save from memory to hard disk: When Redis restarts next time, use persistent files to restore data. In addition, for disaster backup, you can copy persistent files to a remote location.
11.1.1 Redis provides two ways to persist
● RDB persistence: The principle is to save Reids' database records in memory to disk regularly.
● AOF persistence (append only file): The principle is to write the operation log of Reids to a file in an appended manner, similar to MySQL binlog.
Because AOF persistence has better real-time performance, that is, less data is lost when the process exits unexpectedly, AOF is the current mainstream persistence method, but RDB persistence still has its place.
11.2 RDB endurance
RDB persistence refers to the generation of snapshots of the data in the current process in the memory to the hard disk (hence also called snapshot persistence) within a specified time interval, using binary compression for storage, and the saved file suffix is rdb; when Redis is restarted , You can read the snapshot file to restore the data.
11.2.1 Trigger conditions
RDB persistence triggers are divided into manual triggers and automatic triggers.
(1) Manually trigger the
save command and bgsave command to generate RDB files.
The save command will block the Redis server process until the RDB file is created. During the blocking period of the Redis server, the server cannot process any command requests.
The bgsave command creates a child process. The child process is responsible for creating the RDB file, and the parent process (ie, the Redis main process) continues to process the request.
During the execution of the bgsave command, only the fork child process will block the server. For the save command, the whole process will block the server. Therefore, save has been basically abandoned, and the online environment must prevent the use of save.
(2) Automatic triggering
When RDB persistence is automatically triggered, Redis will also choose bgsave instead of save for persistence.
save m n
The most common situation for automatic triggering is to pass save mn in the configuration file, specifying that bgsave will be triggered when n changes occur within m seconds.
vim /etc/ redis/6379. conf
-- 219行--以下三个save条件满足任意- - 个时，都会引起bgsave的调用
save 900 1 ：当时间到900秒时， 如果redis数据发生了至少1次变化，则执行bgsave
save 300 10 ：当时间到300秒时，如果redis数据发生了至少10次变化，则执行bgsave
save 60 10000 ：当时间到60秒时，如果redis数据发生了至少10000次变化，则执行bgsave
--254行--指定RDB文件名
dbfilename dump.rdb
-- 264行--指定RDB文件和AOF文件所在目录
dir /var/lib/redis/6379
-- 242行--是否开启RDB文件压缩
rdbcompression yes
##Other automatic trigger mechanism##
In addition to savemn, there are some other situations that will trigger bgsave:
●In the master-slave replication scenario, if the slave node performs a full copy operation, the master node will execute the bgsave command and save the rdb file Sent to the slave node.
● When the shutdown command is executed, rdb persistence is automatically executed.
11.2.2 Execution process
(1) The Redis parent process first judges whether it is currently executing save, or the child process of bgsave/bgrewriteaof, if it is executing, the bgsave command returns directly.
The child processes of bgsave/bgrewriteaof cannot be executed at the same time, mainly based on performance considerations: two concurrent child processes simultaneously perform a large number of disk write operations, which may cause serious
performance problems.
(2) The parent process executes the fork operation to create a child process. During this process, the parent process is blocked, and Redis cannot execute any commands from the client.
(3) After the parent process forks, the bgsave command returns the "Background saving started" message and no longer blocks The parent process, and can respond to other commands
(4) The child process creates an RDB file, generates a temporary snapshot file based on the memory snapshot of the parent process, and atomically replaces the original file after completion
(5) The child process sends a signal to the parent process to indicate completion, and the parent Process update statistics
11.2.3 Load at startup
The loading of the RDB file is automatically executed when the server starts, and there is no special command. However, because AOF has a higher priority, when AOF is turned on, Redis will load AOF files first to restore data; only when AOF is turned off, RDB files will be detected when the Redis server is started and automatically loaded. The server is blocked while loading the RDB file until the loading is completed.
When Redis loads the RDB file, it will verify the RDB file. If the file is damaged, an error will be printed in the log and Redis will fail to start.
