Introduction to Redis
1.1 Getting to know Redis for the first time
Redis (Remote dictionary server) is an open source (BSD license), is a NoSQL database based on key-value pairs, is a data structure server for memory storage, and can be used as a high-speed cache and message queue agent.
Unlike many key-value pairs databases, redis supports strings (string), hash tables (hash), lists (list), sets (set), ordered sets (zset), bitmaps (bitmaps), HyperLogLogs, GEO and other data structures and algorithms are composed, so redis can meet many application scenarios, and redis will store all the data in the memory, so the read and write performance is very amazing. Not only that, Redis can also save the data in the memory to the hard disk in the form of snapshots and logs, so that the data in the memory will not be "lost" in the event of a power failure or machine failure. In addition to the above functions, Redis also provides key expiration, pipeline, sentinel, built-in replication, Lua scripts, LRU recovery, and transactions. At the same time, it provides high availability through Redis Sentinel and automatic partitioning through Redis Cluster.
In 2008, when Salvatore Sanfilippo, the author of Redis, was developing a website called LLOOGG, he needed to implement a high-performance queue function. It was first implemented using MySQL, but later found that no matter how optimized the SQL statement is, the performance of the website cannot be improved. Going up, coupled with his own shame, so he decided to build a database exclusively for LLOOGG, this is the predecessor of Redis. Later, Salvatore Sanfilippo opened the source code of Redis 1.0! to GitHub, perhaps even he did not expect that Redis would become so popular afterwards. If someone asks the authors of Redis who are using Redis, from the official company statistics of Redis, there are many heavyweight companies using Redis, such as Twitter, Instagram, Stack Overflow, GitHub, etc. in foreign countries, and even more domestically. Too much, if you only count from the volume, Sina Weibo can be said to be the world's largest Redis user, except for Sina Weibo. Companies like Alibaba, Tencent, Baidu, Sohu, Youku Tudou, Meituan, Xiaomi, and Vipshop are all users of Redis.
1.2 Redis features
The reason why Redis is favored by so many companies is bound to excel. Let's talk about the eight features of Redis.
1. Fast speed.
Under normal circumstances, Redis executes commands very fast. The official figure is that the read and write performance can reach 100,000/sec. Of course, this also depends on the performance of the machine, but I will not discuss the performance of the machine here. The difference, just analyze what makes Redis so fast, which can be roughly summarized into the following three
points:
- Putting data in memory is the main reason Redis is fast.
- Redis is implemented in C language. Generally speaking, the programs implemented in C language are closer to the operating system, and the execution speed is relatively faster.
- Redis uses a single-threaded architecture to prevent competition problems that may arise from multiple threads.
2. Data structure server based on key-value pairs
Almost all programming languages provide dictionary-like functions, such as map in Java, dict in Python, similar to this way of organizing data is called a key-value-based way. Different from many key-value pair databases, the value in Redis can be not only a string, but also a specific data structure, which not only facilitates the development in many application scenarios, but also improves development efficiency. The full name of Redis is REmote Dictionary Server. It mainly provides five data structures: string, hash, list, set, and ordered set. At the same time, on the basis of string, it has evolved bitmaps (Bitmaps) and HyperLogLog. This kind of magical "data structure", and with the continuous development of LBS (Location Based Service), Redis 3.2 version added people related to GEO (geographical information positioning), in short, with the help of these data structures , Developers can develop various "interesting" applications.
3. Rich functions
In addition to 5 data structures, Redis also provides many additional functions:
- Provides the key expiration function, which can be used to implement caching.
- Provides a publish and subscribe function, which can be used to implement a message system.
- Support Lua script function, you can use Lua to create new Redis commands.
- Provides a simple transaction function, which can guarantee transaction characteristics to a certain extent.
- Provides a pipeline (Pipeline) function, so that the client can transmit a batch of commands to Redis at once, reducing network overhead.
5. Multiple client languages
Redis provides a simple TCP communication protocol, many programming languages can be easily connected to Redis, and because Redis is widely recognized by the community and major companies, there are many client languages that support Redis. , Almost covering mainstream programming languages, such as Java, PHP, Python, C, C++, Nodejs, etc.
6. Persistence
Generally speaking, it is not safe to put data in memory. Once power failure or machine failure occurs, important data may be lost. Therefore, Redis provides two persistence methods: RDB and AOF. Two strategies are used to save the data in the memory to the hard disk, so as to ensure the durability of the data.
7. Master-slave replication
Redis provides the replication function, which realizes multiple Redis copies of the same data. The replication function is the basis of distributed Redis.
8. High-availability and distributed
Redis officially provides a high-availability implementation Redis Sentinel from version 2.8, which can ensure the failure detection and automatic failover of Redis nodes. Redis officially introduced the distributed implementation of Redis Cluster from version 3.0. It is a true distributed implementation of Redis, providing high availability, read-write and capacity scalability.
1.3 Redis usage scenarios
We have already understood many features of Redis, so let's take a look at what are the typical application scenarios of Redis?
1.3.1 What can Redis do?
1. Caching The
caching mechanism is used in almost all large-scale websites. Proper use of caching can not only speed up data access, but also effectively reduce the pressure on back-end data sources. Redis provides key value expiration time settings, and also provides flexible control of maximum memory and elimination strategies after memory overflow. It can be said that a reasonable cache design can escort the stability of a website.
2, the ranking system ranking system exists in almost all sites, for example, according to rankings Redu ranking, according to rankings released time, according to
the calculated complex dimensions according to rankings, Redis provides a list and ordered collection Data structure, reasonable use of these data structures can easily
build various ranking systems.
3. The application of the
counter The role of the counter in the website is very important. For example, the video website has the number of plays and the e-commerce website has the number of views. The large amount of concurrency is a challenge to the performance of traditional relational data. Redis naturally supports the counting function and the counting performance is also very good, which can be said to be an important choice for the counter system.
4. Social network
likes/dislikes, fans, mutual friends/likes, push, pull-down refresh, etc. are essential features of social networking sites, because social networking sites are usually visited relatively large, and traditional relational data is not suitable for saving this type The data structure provided by Redis can implement these functions relatively easily.
5.
Message queuing system The message queuing system can be said to be an indispensable basic component of a large-scale website, because it has features such as business decoupling and non-real-time business Xiaofeng. Redis provides the function of publish and subscribe and the function of blocking queue. Although it is not powerful enough compared with professional message queue, it can basically meet the general message queue function.
1.3.2 What Can't Redis Do
In fact, like any technology, each technology has its own application scenarios and boundaries, which means that Redis is not a panacea. There are many problems that are suitable for it to solve, but there are also many problems that are not suitable for it to solve.
We can analyze from the perspective of data scale and data coldness and heat. From the perspective of data scale, data can be divided into large-scale data and small-scale data. We know that Redis data is stored in memory. Although memory is now cheap enough, if the amount of data is very large, for example, how many data are there every day? If you use Redis to store billions of user behavior data, it is basically a bottomless pit, and the economic cost is quite high. From the perspective of hot and cold data, data is divided into hot data and cold data. Hot data usually refers to data that requires frequent operations, and vice versa. For example, for a video website, basic video information is basically data that is frequently manipulated in each business line, and the user's viewing record is not necessarily the data that needs to be accessed frequently. I will not discuss the difference in data scale between the two for the time being. From the perspective of hot and cold data, video information is hot data, and user viewing records are cold data. If you put these cold data in Redis, it is basically a waste of memory, but for some hot data, you can put it in Redis to speed up the reading and writing, and it can also reduce the load on the back-end storage, which can be said to be twice the result with half the effort. Therefore, Redis is not a panacea. I believe that with our gradual learning of Redis, we will be able to understand the real usage scenarios of Redis.
1.4 Install and deploy Redis
wget http://download.redis.io/releases/redis-5.0.9.tar.gz
[root@redis-01 ~]# tar zxf redis-5.0.9.tar.gz -C /usr/local/src/
[root@redis-01 ~]# cd /usr/local/src/redis-5.0.9/
[root@redis-01 redis-5.0.9]# make
//cp configuration files and commands to a custom directory
[root@redis-01 ~]# mkdir /usr/local/redis/{conf,bin} -p
[root@redis-01 ~]# cp /usr/local/src/redis-5.0.9/src/redis* /usr/local/redis/bin/
[root@redis-01 ~]# cp /usr/local/src/redis-5.0.9/redis.conf /usr/local/redis/conf/
//Delete redundant commands
[root@redis-01 ~]# cd /usr/local/redis/bin/
[root@redis-01 bin]# ls
redisassert.h redis-check-aof redis-check-rdb.c redis-cli.o redis-trib.rb
redis-benchmark redis-check-aof.c redis-check-rdb.o redismodule.h
redis-benchmark.c redis-check-aof.o redis-cli redis-sentinel
redis-benchmark.o redis-check-rdb redis-cli.c redis-server
[root@redis-01 bin]# rm -rf *.o
[root@redis-01 bin]# rm -rf *.c
[root@redis-01 bin]# ls
redisassert.h redis-check-aof redis-cli redis-sentinel redis-trib.rb
redis-benchmark redis-check-rdb redismodule.h redis-server
The command is in: /usr/local/redis/bin/ The
configuration file is in: /usr/local/redis/conf/
Change configuration file
[root@redis-01 bin]# mkdir /data/redis -p
[root@redis-01 bin]# vim /usr/local/redis/conf/redis.conf
.....
bind 127.0.0.1 192.168.1.40
daemonize yes
pidfile /data/redis/redis.pid
logfile "/data/redis/redis.log"
dir /data/redis
pid is placed in: /data/redis/redis.pid
log is placed in: "/data/redis/redis.log" dir /data/redis
The important options of the configuration file are explained as follows
Prot: Port
Logfile: Log file
Loglevel verbose: Log level: debug \verbose\notice\warning Log output method logfilestdout
Rdbcomperssion yes: Specify whether to store data in the local database to save the cpu can be closed but the database file is huge Dbfilename
dump.rdb: Specify The file name of the local database defaults to dump.rdb
Slaveof: This machine is the slave of the master and slave, set the master server ip and port, and synchronize data to the master. Masterauth When the master sets password protection, the slave server can connect to the master password Requirepass
foobared: the number of concurrent connections , The maximum number of client connections at the same time, 0 means no limit
Maxmemory: specify that each update operation will be logged, the default is NO
Appendfilename appendonly.aof: specify the file name of the update log Everysec synchronizes once per second (default value, speed Compromise) No: The system synchronizes the data cache to the disk (the no method is fast) Always After each update operation, manually call fsync to write the data to the disk (very slow, but safe)
Vm-enabled no: Whether to enable virtual memory Mechanism (default is no)
Vmswap-file /path/redis.swap: virtual memory file path
Vm-max-memeory 0: greater than the following number into virtual memory
Glueoutputbuf yes: (Turn on the pipeline) Set whether to merge smaller packages into one package and send Dir when responding to the client. Working directory Bind-the bound host address protected-mode yes. The protection module is turned on by default. Timeout 0 How many seconds are disconnected. 0 Default Turn off tcp-keepalive300 The tcp connection timeout time is 300 seconds. daemonize no Whether to start redis as a daemon. Default NO databases 16 The database number is up to 0-16 can be set by yourself always-show-logo yes The default is turned on to display logo save 900 1 900 seconds Yes 1 change save save 300 10 changes saved in 10300 seconds save 60 10000 60 seconds saved 1W changes
redis -h host -p port connects to redis interactively
[root@redis-01 bin]# ln -s /usr/local/redis/bin/* /usr/sbin/
[root@redis-01 bin]# redis-server /usr/local/redis/conf/redis.conf
[root@redis-01 bin]# netstat -anput | grep redis
tcp 0 0 192.168.1.71:6379 0.0.0.0:* LISTEN 6265/redis-server 1
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 6265/redis-server 1
[root@redis-01 ~]# redis-cli -h 192.168.1.71 -p 6379
192.168.1.71:6379> SHUTDOWN
Check the log at this time and found some problems, so it needs to be optimized.
//View/data/redis/redis.log needs to be optimized as follows. A total of four items need to be optimized.
【第一条】6265:M 18 Mar 2021 15:08:41.324 * Increased maximum number of open files to 10032 (it was originally set to 1024).
【第二条】6265:M 18 Mar 2021 15:08:41.326 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
【第三条】6265:M 18 Mar 2021 15:08:41.326 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
【第四条】6265:M 18 Mar 2021 15:08:41.326 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
Optimize the first
[root@redis-01 ~]# ulimit -n
1024
[root@redis-01 ~]# vim /etc/security/limits.conf
....
* - nofile 10032
//Switch to another terminal
[root@redis-01 ~]# ulimit -n
10032
Optimize the second
[root@redis-01 ~]# echo "511" > /proc/sys/net/core/somaxconn
[root@redis-01 ~]# vim /etc/sysctl.conf
....
net.core.somaxconn = 511
[root@redis-01 ~]# sysctl -p
net.core.somaxconn = 511
Optimize the third
[root@redis-01 ~]# vim /etc/sysctl.conf
[root@redis-01 ~]# systemctl -p
.....
vm.overcommit_memory = 1
[root@redis-01 ~]# sysctl -p
vm.overcommit_memory = 1
net.core.somaxconn = 511
Optimize Article 4
[root@redis-01 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@redis-01 ~]# vim /etc/rc.local
......
echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@redis-01 ~]# chmod +x /etc/rc.local
Restart redis
[root@redis-01 ~]# > /data/redis/redis.log #清空日志文件方便查看
[root@redis-01 ~]# kill -9 `cat /data/redis/redis.pid `
[root@redis-01 ~]# redis-server /usr/local/redis/conf/redis.conf
Check the log and you will find that there is no warning message
[root@redis-01 ~]# cat /data/redis/redis.log
.....此处省略日志信息