One, redis application
1. Record the number of post likes, comments and clicks (hash)
2. Record the user's post ID list, so as to quickly display the user's post list (zset)
3. Record the title, abstract, author and cover information of the post, and display on the user list page (hash)
4. Record the like user ID list and comment ID list of the post for display and de-duplication (hash)
5. Cache the content of recent hot posts (post content takes up a lot of space) to reduce database pressure (hash)
6. Record the related article ID of the post, and recommend related posts according to the content (list)
7. If the post ID is incremented by an integer, redis can be used to assign the post ID (counter)
8. The relationship between collections and posts (zset)
9. Record hot list post ID list, total hot list and category hot list (zset)
10. Cache user behavior history and filter malicious behavior (zset, hash)
11. Ensure that the same user will not win twice (set)
Two, redis data structure
string (string), list (list), hash (dictionary), set (set), zset (ordered set)
- string
The internal string is a dynamic string, similar to the dynamic expansion of ArrayList. This reduces the overhead of frequent memory allocation. When the string length is less than 1M, the capacity will be doubled; when it is greater than 1M, only 1M will be added; the maximum length is 512M.
scenes to be used:
Cache user login information. The token is used as the key, and the user information is serialized into a string using JSON, and then deserialized when obtained.
Common commands
set name value #存值
get name #取值
exists name #判断
del name
mset name1 name2 value1 value2 #批量取
mget name1 name2 ..... #批量取
expire name m秒 #m秒后过期
setex name m秒 value #存值，并且设置过期时间
setnx name value #存值，如果name已经存在，就返回0
set name 1 #设置为整数
incr name #还可以自增
incrby name 整数 #加
- list
Similar to LinkedList. Insert and delete blocks, the query is slow.
scenes to be used
Often used for asynchronous queue implementation (first in, first out)
Common commands
rpush name value1 value2... #入队
llen name #队列长度
lpop name #出队 value1
- hash (dictionary)
Similar to HashMap
scenes to be used
Each field in the storage structure can be stored separately. The expiration time is for the real hash object, not for individual subkeys.
Common commands
hset key filed1 value1
hset key filed2 value2 #存
hget key filed1 #取
- set
sadd, smembers, scard
- zset (ordered set)
zadd, zrange,zrank, zrem,zcard
Three, HyperLogLog
Scenarios: scenarios with low requirements for estimation and accuracy (statistics of the PV and UV of the website)
Commands pfadd, pfcount, pfmerge
The memory footprint is smaller than set, and there is a certain error
Four, bloom filter
Principle: Bloom filter is a BIT array
Scenario: De-duplication of information recommendation (filtering the information that has been read when the Weibo recommendation is refreshed), spam filtering, crawler system filtering crawled content, solving cache penetration problems
Bloom filter can determine that a certain data must not exist, but cannot determine that it must exist (inaccurate SET)
It occupies very little memory, and inserts and queries are fast enough.
Disadvantages, data cannot be deleted; as the data increases, the false positive rate will increase
Redisson implementation
Five, Reids 6 elimination strategies
Volatile-lru: From the data set with expiration time set, select the least recently used data to release;
allkeys-lru: From the data set (including the set expiration time and the data set without expiration time), select the most recently unused data to release;
Volatile-random: randomly select a data to release from the data set with expiration time set;
allkeys-random: randomly select a piece of data from the data set (including set expiration time and unset expiration time) to enter and release;
volatile-ttl: From the data set with expiration time set, select the data that will expire soon for release operation;
noeviction: Do not delete any data (but redis will release it according to the reference counter). If the memory is not enough, an error will be returned directly.
The default strategy is noeviction The
recommended strategy is volatile-lru
Configure the number of samples through maxmemory-samples, the default is 5
Cache elimination algorithm (LFU, LRU, ARC, FIFO, MRU)
6. Redis persistence scheme:
RDB default mode (RDB persistence means persisting by creating a snapshot, saving the full amount of data at a certain point in time.)
AOF (Append-Only-File persistence is to record all the instructions that change the database state, and save it to the AOF file in the form of append)
If Redis is only used as a cache server, such as caching after the database queries data, then you don't need to consider persistence, because the cache service fails and can be restored from the database.
Seven, cache and database data consistency (concurrent competition problem)
Delay double delete strategy (redis.del is performed before and after writing the library, and a reasonable delay time is set.)
Read binlog analysis, use message queue (rabbitmq, kafka), push to update the redis cache data of each station
8. Cache penetration
Phenomenon: The data corresponding to the data (key) requested by a large number of concurrent users does not exist in redis and the database, resulting in a DB query every time although the data does not exist.
Solution: The data that cannot be retrieved from the cache is also not retrieved in the database. At this time, the key-value pair can also be written as key-null
Nine, cache breakdown
Phenomenon: Cache breakdown means that there is no data in the cache but the data in the database (usually because the cache time expires). At this time, because there are so many concurrent users, and the data is not read in the read cache at the same time, the data is retrieved from the database at the same time. The database pressure increases instantly, causing excessive pressure.
solution:
1. Set hot data to never expire
2. Interface current limit and fuse, downgrade
3. Add mutex lock
4. Bloom filter
Ten, cache avalanche
Phenomenon: A large number of keys fail at the same time, and a large number of requests come in at the same time, causing traffic to be directly hit on the DB, making the DB unusable.
solution:
1. Set the key to never fail (hot data);
2. Stagger as much as possible when setting the key cache invalidation;
3. Use multi-level caching mechanism, such as using redsi and memcache cache at the same time, request ->redis->memcache->db;
4. Purchase a third-party Redis cloud server with high reliability;
Eleven, Redis hot key processing
1 Hot key discovery
Monitoring hot keys (the packet capture program captures the data of the redis listening port, and then discards the data in Kafka. Next, the flink streaming computing system consumes the data in Kafka and performs statistics)
Notify the system for processing
2 Solution
Local cache
Cluster (store this hot key in multiple redis)
Alibaba Cloud Redis has solved the hot key problem at the kernel level
- The hazards of hot keys
The traffic is concentrated, reaching the upper limit of the physical network card.
Too many requests, the cache fragmentation service was defeated.
DB breakdown, causing a business avalanche.
12. Reject big KEY
Cluster environment, big key will cause data migration to freeze
If the memory is reclaimed once when it is deleted, it will also freeze
When expanding, it will apply for more memory at one time, and it will also freeze
Note: If the Redis memory fluctuates greatly, it is likely to be caused by a large key. At this time, the large key needs to be located and optimized
Locating the big key can be done using scan or redis-cli commands
13. Redis is single-threaded, but why is Redis so fast?
1. Based on memory
2. Simple data structure and operation
3. Multiple I/O multiplexing model (non-blocking IO)
4. Single thread avoids unnecessary context switching and race conditions
14. Funnel current limit
Distributed current limit: redis-cell
Stand-alone: Google's guava package provides the RateLimiter class
There are three common algorithms for current limiting:
1. Time window algorithm
2. Leaky bucket algorithm
3. Token algorithm
15. GEO
Redis uses GeoHash algorithm to realize the function of querying nearby people;
The internal data structure is zset, and the original coordinates can be obtained through score restoration;
The data corresponding to a single key in a cluster environment should not exceed 1M. If it exceeds, the data size of the key needs to be split according to the corresponding business rules.
16. scan
Step by step through the cursor, compared to keys, will not block the thread
Provide limit parameters to control the number of returned results
