Article from
https://www.toutiao.com/i6805738743253697035
Tutorial learning
A full set of video tutorials for redis interview in 2020
Link: http://www.mano100.cn/thread-1588-1-1.html
Article body
Most programmers who write business, when using Redis in actual development, only use Set Value and Get Value, and lack an overall understanding of Redis. Here is a summary of common Redis problems to solve everyone's knowledge blind spots.
1. Why use Redis to use Redis
in a project? Mainly consider two perspectives: performance and concurrency. If it is just for distributed locks and other functions, there are other middleware Zookpeer etc. instead, it is not necessary to use Redis.
Performance:
As shown in the figure below, when we encounter SQL that needs to be executed for a long time and the results do not change frequently, it is particularly suitable to put the running results into the cache. In this way, subsequent requests are read from the cache, so that requests can be responded quickly.
Especially in the spike system, at the same time, almost everyone is on the spot, all placing orders... the same operation is performed ---checking data in the database.
According to different interaction effects, there is no fixed standard for response time. In an ideal state, our page jump needs to be resolved in an instant, and for in-page operations, it needs to be resolved in an instant.
Concurrency:
As shown in the figure below, in the case of large concurrency, all requests directly access the database, and the database will have a connection exception. At this time, you need to use Redis to do a buffering operation, so that the request first accesses Redis instead of directly accessing the database.
Frequently asked questions about using Redis
- Cache and database double write consistency problem
- Cache avalanche problem
- Cache breakdown problem
- Cache concurrency competition problem
2. Why is single-threaded Redis so fast?
This question is an investigation of Redis's internal mechanism. Many people don't know that Redis is a single-threaded work model.
The main reasons are the following three points:
- Pure memory operation
- Single-threaded operation avoids frequent context switching
- Adopts non-blocking I/O multiplexing mechanism
Let me talk about the I/O multiplexing mechanism carefully. Let's make an analogy: Xiao Ming opened a fast food restaurant in City A, responsible for fast food service in the same city. Because of funding constraints, Xiao Ming hired a group of delivery staff, and then Xiao Qu found that the funds were not enough, and only enough to buy a car to deliver the express.
Business mode one
Every time a customer places an order, Xiao Ming asks a delivery person to stare at it, and then asks him to drive to deliver it. Slowly, I discovered the following problems with this business method:
-
Time is spent robbing the car, most of the delivery staff are in an idle state and can only be delivered if the car is robbed.
-
With the increase in orders and more and more delivery staff, Xiao Ming found that the express store was getting more and more crowded and there was no way to hire new delivery staff.
-
Coordination between the delivery staff takes time.
Based on the above shortcomings, Xiao Ming learned from the pain and proposed the second business method.
Business mode two
Xiao Ming only hires one delivery person. When a customer places an order, Xiao Ming will mark it according to the place of delivery and place it in one place in turn. Finally, let the delivery staff drive the car in turn to deliver them, and return to pick up one after delivery. Comparing the above two operating methods, it is obvious that the second is more efficient.
In the above analogy:
- Each delivery person → each thread
- Every order → every Socket (I/O stream)
- The place where the order is delivered → the different status of the Socket
- Customer delivery request → request from the client
- Mingqu's business mode → code running on the server
- Number of cores of a car→CPU
So there are the following conclusions:
- The first mode of operation is the traditional concurrency model. Each I/O flow (order) has a new thread (deliverer) management.
- The second mode of operation is I/O multiplexing. There is only a single thread (a dispatcher), which manages multiple I/O flows by tracking the status of each I/O flow (the delivery location of each dispatcher).
The following analogy to the real Redis threading model, as shown in the figure:
Redis-client will generate Sockets with different event types when operating. On the server side, there is an I/O multiplexing program, which is placed in the queue. Then, the file event dispatcher goes to the queue in turn and forwards them to different event handlers.
3. Redis data types and usage scenarios
A qualified programmer will use all five types.
String is the
most common set/get operation, Value can be String or number. Generally do some complex counting function cache.
Hash
Here Value stores structured objects, and it is more convenient to manipulate one of the fields. When I do single sign-on, I use this data structure to store user information, using CookieId as the Key, and setting 30 minutes as the cache expiration time, which can simulate a session-like effect.
List
uses the data structure of List and can do simple message queue function. In addition, you can use the lrange command to do the Redis-based paging function, which has excellent performance and good user experience.
Set
because Set is a collection of unique values. So it can do the global de-duplication function. Our systems are generally deployed in clusters, and it is troublesome to use the Set that comes with the JVM. In addition, by using operations such as intersection, union, and difference, you can calculate common preferences, all preferences, and your own unique preferences.
Sorted Set
Sorted Set has an additional weight parameter Score, and the elements in the set can be arranged according to Score. It can be used as a ranking application, and take TOP N operation. Sorted Set can be used for delayed tasks.
4. Redis's expiration strategy and memory elimination mechanism
It can be seen from this whether Redis is used at home. For example, your Redis can only store 5G data, but if you write 10G, 5G data will be deleted. How to delete, have you thought about this question?
Positive solution: Redis uses a regular deletion + lazy deletion strategy.
Why not use the timing deletion strategy?
Timing deletion, use a timer to monitor the Key, and automatically delete it when it expires. Although the memory is released in time, it consumes CPU resources very much. Under large concurrent requests, the CPU will use time to process the request instead of deleting the Key, so this strategy is not adopted.
How does regular deletion + lazy deletion work?
Regular deletion, Redis checks every 100ms by default, and deletes expired keys. It should be noted that Redis does not check all keys once every 100ms, but randomly selects them for checking. If only the regular deletion strategy is adopted, many keys will not be deleted in time. Thus, lazy deletion comes in handy.
Is there no other problem with regular deletion + lazy deletion?
No, if you delete the key regularly, the key is not deleted. And you did not request the Key in time, which means that the lazy deletion did not take effect. In this way, Redis's memory will get higher and higher. Then the memory elimination mechanism should be used.
There is a line of configuration in redis.conf:
maxmemory-policy volatile-lru
This configuration is equipped with memory elimination strategy:
- noeviction: When the memory is not enough to hold the newly written data, the new write operation will report an error.
- allkeys-lru: When the memory is insufficient to accommodate the newly written data, remove the least recently used Key in the key space. (Recommended, the project is currently using this) (the most recently used algorithm)
- allkeys-random: When the memory is insufficient to accommodate the newly written data, a key is randomly removed from the key space. (Should no one use it, you don’t delete the key at least, and delete randomly)
- Volatile-lru: When the memory is insufficient to accommodate the newly written data, remove the least recently used Key from the key space with the expiration time set. This situation is generally used when Redis is used as both a cache and a persistent storage. (Not recommended)
- Volatile-random: When the memory is insufficient to accommodate the newly written data, a key is randomly removed from the key space with the expiration time set. (Still not recommended)
- Volatile-ttl: When the memory is insufficient to accommodate the newly written data, in the key space with the expiration time set, the key with an earlier expiration time will be removed first. (Not recommended)
5. Redis and database double-write consistency problem The
consistency problem can be further divided into final consistency and strong consistency. If the database and cache are double-written, there will inevitably be inconsistencies. The premise is that if there is a strong data consistency requirement, the cache cannot be placed. Everything we do can only guarantee final consistency.
In addition, the plan we have made can only reduce the probability of inconsistency. Therefore, data with strong consistency requirements cannot be cached. First, take the correct update strategy, update the database first, and then delete the cache. Second, because there may be a problem of failure to delete the cache, a compensation measure can be provided, such as using a message queue.
6. How to deal with the
two problems of cache penetration and cache avalanche , generally small and medium-sized traditional software companies are difficult to encounter. If there is a large concurrent project with a traffic of several million or so, these two issues must be deeply considered. Cache penetration, that is, hackers deliberately request data that does not exist in the cache, causing all requests to be sent to the database, and the database connection is abnormal.
Cache penetration solution:
-
With mutex locks, when the cache fails, the lock is obtained first, and then the database is requested after the lock is obtained. If the lock is not obtained, it will sleep for a period of time and try again.
-
The asynchronous update strategy is adopted, and it returns directly regardless of whether the Key gets the value. The value maintains a cache invalidation time. If the cache expires, a thread is asynchronously started to read the database and update the cache. Need to do cache warm-up (load the cache before starting the project) operation.
-
Provide an interception mechanism that can quickly determine whether the request is valid, for example, using Bloom filters to maintain a series of legal and valid keys internally. Quickly determine whether the Key carried in the request is legal and valid. If it is illegal, return directly.
Cache avalanche, that is, the cache fails in a large area at the same time. At this time, another wave of requests comes. As a result, the requests are all sent to the database, resulting in abnormal database connection.
Cache avalanche solution:
-
Add a random value to the invalidation time of the cache to avoid collective invalidation.
Mutex locks are used, but the throughput of this solution is significantly reduced. -
Double buffering. We have two caches, cache A and cache B. The expiration time of cache A is 20 minutes, and cache B has no expiration time. Do the cache warm-up operation yourself.
-
Then subdivide the following points: read the database from cache A, and return directly; if A has no data, read data directly from B, return directly, and start an update thread asynchronously, which updates both cache A and cache B at the same time.
7. How to solve the problem of Redis's concurrent competition Key
This problem is roughly that there are multiple subsystems to Set a Key at the same time. What should I pay attention to at this time? Everyone basically recommends using Redis transaction mechanism.
But I don't recommend using Redis' transaction mechanism. Because our production environment is basically a Redis cluster environment, we have done data sharding operations. When multiple Key operations are involved in a transaction, the multiple Keys may not be stored on the same redis-server. Therefore, Redis's transaction mechanism is very tasteless.
If the sequence is not required for this Key operation, in
this case, prepare a distributed lock, and everyone should grab the lock and just do the set operation when the lock is grabbed, which is relatively simple.
If you operate on this key, the order is required.
Assuming there is a key1, system A needs to set key1 to valueA, system B needs to set key1 to valueB, and system C needs to set key1 to valueC.
It is expected that the value of key1 will change in the order of valueA> valueB> valueC. In this case, we need to save a timestamp when writing data to the database.
Suppose the timestamp is as follows:
System A key 1 {valueA 3:00}
System B key 1 {valueB 3:05}
System C key 1 {valueC 3:10}
Then, suppose system B grabs the lock first and sets key1 to {valueB 3:05}. Next, system A grabs the lock and finds that the timestamp of its valueA is earlier than the timestamp in the cache, so the set operation is not performed, and so on. Other methods, such as using queues, can also change the set method into serial access.
8. Summary
Redis can be seen in major domestic companies, such as the familiar Sina, Ali, Tencent, Baidu, Meituan, Xiaomi, etc. Learning Redis, these aspects are especially important: Redis client, Redis advanced features, Redis persistence and common problems in development and operation, Redis replication principles and optimization strategies, Redis distributed solutions, etc.
Systematic learning PHP
Scan code attention: PHP self-study center, reply to the corresponding keywords, receive the following video tutorial
1 Vue2.5 core technology source code analysis
Keywords: 19082201
2 Analysis and in-depth interpretation of design pattern examples
Keywords: 20190714
3 Complete PHP advanced practical tutorials
Keywords: 20190625
4 Zero-distance contact with mysql
Keywords: 20190128
5 The actual combat of high-performance Linux server construction
Keywords: 20190622
6 Analysis of the underlying source code of
ThinkPHP5 Keywords: 20190621
7 Thinkphp plug-in development of WeChat system
Keywords: 201907282319
8 Laravel basic entry to the actual development of
WeChat mall Keywords: 08250045
9 PHP asynchronous communication framework Swoole actual combat
Keywords: 08250024
For more keywords of the tutorial, please pay attention to the articles and tutorials shared by the official account every day
