Click below to follow me, and then click... "Set as star" in the upper right corner to receive updates as soon as possible~~~
General business scenarios involve more reading and less writing. When there are too many client requests, the pressure on the database is increasing. Introducing cache to reduce the pressure on the database is an inevitable choice. Currently, the mainstream choice in the industry is to use Redis as the database. cache. However, after the introduction of cache, it has brought great challenges to the design of our system. Among them, the data consistency problem between cache and database is a very difficult problem. Today we will talk about what solutions we use in the project. Solve the problem of data consistency.
1
Redis usage scenarios
In our actual projects, we usually use applications that read more and write less. We generally use Redis to solve the pressure problem of reading the database. In other words, we use Redis as a cache to reduce the impact on the database caused by the client's frequent query of the database. pressure.
Therefore, in the project, the client directly queries the database to return data, instead of the client querying the Redis cache. If there is data in the cache, the data will be returned directly. If there is no data in the cache, the database will be queried. At the same time, the database data will be written to the Redis cache.
In the above usage scenarios, we generally cache hot data, such as hot-selling products on e-commerce platforms, user login information, hot news on news platforms, etc.
2
Causes of data consistency problems
When the data in the database changes, the introduction of Redis cache increases the difficulty of data operations. It is necessary to operate both the database and Redis. There are two options for operating Redis and database:
1. Operate Redis first, then operate the database
2. Operate the database first, then Redis.
No matter which of the above options we choose, we all hope that the data operations will either succeed or fail. We do not want to see a failure and a successful result, because this will cause data inconsistency.
Give an example:
Suppose a hot product data is cached in Redis. There is a product with key 1001 and the name is "Huawei mobile phone". The product name No. 1001 in the database is also a Huawei mobile phone. At this time, the merchant feels that the product name is Huawei mobile phone is a bit broad and needs to be precise. For a moment, change the name of product No. 1001 to its corresponding specific name "Huawei P40 Pro".
1. If we choose to operate Redis first and then operate the database, when the Redis operation is successful and the database operation fails, the name in Redis is changed to "Huawei P40 Pro", but the name of the database is still "Huawei Mobile Phone", resulting in Data inconsistency issues.
2. If we choose to operate the database first and then operate Redis, when the database operation is successful and the Redis operation fails, the name in the database is changed to "Huawei P40 Pro", but the name of Redis is still "Huawei Mobile Phone", resulting in Data inconsistency issues.
Since Redis and the database are two different middlewares, we cannot solve the data consistency problem well through transactions. Therefore, we can only make a trade-off between real-time data consistency and system performance, and choose an acceptable solution.
Because the database is a stable and persistent system and more reliable than Redis, we generally use the database data as the standard. The principle to solve this data consistency problem is: we can set an expiration time for the Redis cache data. When the Redis data When it expires, go to the database to query, and then write the database data into the Redis cache to ensure data consistency.
If an update operation occurs on the data within this expiration time range, and one update operation fails and one succeeds, inconsistency problems will occur. The expiration time is set too short, and the pressure on the database is still very high. The expiration time If the setting is too long, the problem of inconsistency will become prominent. Therefore, we need to continue to explore solutions to the data consistency problem based on the principle of taking the data of the database as the standard .
3
How to operate Redis when data is updated
When data is updated, we are all familiar with operating the database. Just update the database value directly. But what should we do when operating Redis?
1. Some people will say that this is not simple. When the data changes, the cache is directly updated.
2. Some people will also say that the cached data is deleted when the data changes. When the query request finds that there is no data in the cache, new data will be loaded from the database.
Both of the above methods are possible, so how should we choose?
If we choose to update Redis, we need to evaluate whether the cost of the update is high. For example, getting data to update to Redis requires associated queries on many tables, or query calls on multiple interfaces, and a lot of calculations to get the data. Don't use the solution to update Redis, because facing such a complicated update process, it is better to delete it directly. If the data to be updated can be obtained directly, then it is feasible to choose to update Redis.
Directly deleting the Redis cache data is a simpler solution, because after the cache data is deleted, it can only be read from the database. This can avoid inconsistency. Internet companies generally choose the solution of deleting Redis.
Below we use a picture to represent the optional solutions:
From the above figure, we can see that when the data changes, the processing flow of Redis and the database can be analyzed from the two aspects of Redis update or delete, and then from the two aspects of operating Redis first or operating the database first, as follows Let’s continue our in-depth analysis along this line of thinking.
4
Select Redis update plan
1. Update Redis first, then update the database
First of all, it is not recommended to choose this option.
Because the basic principle of our data consistency is based on the data in the database, if we update the cache first, there will be situations where the cache update is successful but the database update fails. At this time, the problem we face requires rolling back the successful cache update just now. operation, then you need to add a lot of judgment logic to the business code to handle this exception. You need to consider whether the data changes are insert, update, delete, etc., and execute different rollback plans according to different scenarios. This kind of rollback operation is more troublesome. The intrusion into our business code is also relatively large, so this option is not recommended.
Of course, if the business level can accept this kind of data inconsistency, abnormal situations do not need to be considered, and this option is also acceptable.
2. Update the database first, then update Redis
The solution of updating the database first and then updating Redis puts the database operation first. If the database operation fails, the client can just initiate it again.
If the database operation succeeds but the Redis operation fails, since the database data has been set successfully, our focus is to discuss how to successfully operate the Redis data. From the client level, the data he submitted already exists in our system. The problem at this time is how to solve it within our system.
For Redis operation failure, we have the following options, which can be weighed and selected according to the data consistency level required by the business.
1. Do not do any operation, wait for the cached data in Redis to expire, and automatically synchronize the latest data from the database. At this time, the most serious data inconsistency period is during the cache expiration period (consider the range of this expiration time) ; If there is a new update request within this time period, maybe the cache update will be successful this time.
2. If the data consistency requirement is relatively high, then after the Redis operation fails, we record the operation, process it asynchronously, and use the Redis data to compare with the database. If it is inconsistent, update the cache again to ensure that the cached data is consistent with the database data.
5
Select Redis deletion plan
Due to current data changes, the mainstream solution is to delete Redis, so we focus on analyzing this solution.
1. Update the database first, then delete Redis
If we successfully update the database but fail to delete Redis, then an old value is stored in Redis. That is, the failure to delete the cache causes the data in the cache and the database to be inconsistent.
To analyze the abnormal situation of failure to delete Redis, we generally have the following solutions:
1. We can retry the deletion. For example: we can send the deletion action to the message queue MQ, and the MQ consumer will delete the key and try the deletion operation unanimously to ensure that the cache deletion is successful. The flaw of this solution is that the cache is deleted. There is an intrusion into the business logic through code implementation.
2. Another solution is a completely asynchronous solution, which does not intrude on the business logic and deletes the cache by monitoring changes in the binlog. When we update the database, a binlog will be generated. We can use a service to monitor changes in the database binlog and delete the cache asynchronously. Alibaba's open source canal is a tool that can monitor the binlog. As long as the data changes, the Redis data can be deleted.
2. Delete Redis first, then update the database
If we succeed in deleting the Redis cache but fail to update the database, because we are based on the database, we can just check again. This solution seems ideal and there is no problem. In fact, data consistency problems may also occur under concurrency.
Let’s first look at the update and query process in a concurrent environment:
The above figure shows that Thread-1 is an update process, Thread-2 is a query process, and the CPU execution sequence is: Thread-1 deletes the cache successfully. At this time, Thread-2 obtains that the CPU executes the query cache and there is no data, and then queries the database to put the database The value is written into the cache, because Thread-1 has not yet been executed to update the database, so the value in the cache is an old value (old). Finally, the CPU executes the code for Thread-1 to successfully update the database, so the value of the database at this time is Adding new (new) creates the problem of inconsistent data rows.
To solve the above problems, we generally have the following options:
1. Can concurrent multi-threaded operations allow two steps in the same operation to be completed at the same time before the next thread can be executed? Let each thread queue up for operations. Some people say locking. If a service is deployed to multiple machines, it will become It becomes a distributed lock, or a distributed queue operates the database or Redis in sequence. The side effect is that the database was originally concurrent, but now it has become serial, so this solution does not seem feasible. Adding locks or The queuing execution scheme reduces system performance.
2. Another solution is to delay double deletion, that is, delete the cache first, then update the database, sleep for a period of time after updating the data, and then delete the cache again.
The pseudocode is as follows:
redes.del(key);
db.update(data);
Thread.sleep(2000);
redes.del(key);The above mentioned solutions are all solutions. There is no saying which solution is better or which solution is not feasible. The above solutions are all based on real-time consistency and system performance. According to different businesses, different data are consistent. Performance requirements, combined with system performance, comprehensive consideration, just choose a solution that suits your system.
In general, since our basic principle is to take the database as the standard, the solution we choose should put the operation of the database first, that is to say, we should operate the database first and then Redis. For scenarios with high concurrency, We can use message queues to reduce client request pressure on the database before operating the database.