10 Discussion on Update Strategy of Cache Double Write Consistency

What is Cache Double Write Consistency

• If there is data in redis: it needs to be the same as the value in the database
• If there is no data in redis: the value in the database should be the latest value

The cache is divided according to the operation, and there are two subdivisions

• read-only cache
• read and write cache
  • Synchronous write-through strategy: When writing the cache, it also writes to the database synchronously, and the data in the cache and the database are consistent;
  • For the read-write cache, to ensure that the data in the cache and the database are consistent, a synchronous write-through strategy must be adopted

Several Update Strategies for Database and Cache Consistency

• Listing error, early morning upgrade: single-threaded, such heavyweight data operations are best not multi-threaded
• Purpose: In short, we want to achieve final consistency!
  • Setting an expiration time for the cache is a solution to ensure eventual consistency.
  • We can set an expiration time for the data stored in the cache. All write operations are subject to the database, and we only do our best for the cache operation. That is to say, if the database is written successfully but the cache update fails, as long as the expiration time is reached, subsequent read requests will naturally read new values ​​from the database and then backfill the cache to achieve consistency. Remember to use the mysql database write library as the standard .
  • The above plan and subsequent implementation cases are the mainstream + mature approach after research, but considering the gaps in the business systems of various companies, it is not 100% absolutely correct, and there is no guarantee that it will be absolutely suitable for all situations. Please choose the appropriate method according to your own discretion. Your own best.

We discuss 3 update strategies

Update the database first, then update the cache

• First update the inventory of a product in mysql. The current inventory of the product is 100, and it is updated to 99.
• First update mysql to 99 success, and then update redis.
• At this time, assuming an exception occurs, updating redis fails, which causes the inventory in mysql to be 99 and the inventory in redis is still 100.
• The occurrence of the above will make the data in the database and cache redis inconsistent, and read dirty data

Delete the cache first, then update the database

abnormal problem

1. Thread A successfully deletes the data in redis first, and then updates mysql. At this time, mysql is being updated and it is not over yet. (such as network delay) B suddenly appears to read the cached data.
   
2. At this time, the data in redis is empty, and thread B reads it. First, read the data in redis (which has been deleted by thread A). Here are two questions:
   • B obtained the old value from mysql : the B thread found that there was no cache in redis (the cache was missing) and immediately went to mysql to read it, and the old value was read from the database.
   • B will write the obtained old value back to redis : return to the foreground after obtaining the old value data and write it back into redis (the old data just deleted by A thread is very likely to be written back again).
   • 
3. Thread A finishes updating mysql and finds that the cache in redis is dirty data
   • Two concurrent operations, one is an update operation and the other is a query operation. After the update operation of A deletes the cache, the query operation of B does not hit the cache. B first reads the old data and puts it in the cache, and then the update operation of A updates the database .
   • Therefore, the data in the cache is still old data, causing the data in the cache to be dirty, and it has been dirty like this

Summary process

1. Request A to perform a write operation. After deleting the cache, the work is in progress... Has A been completely updated?
2. Request B to start work, query redis and find that the cache does not exist
3. Request B to continue, go to the database query to get the old value in myslq
4. Request B to write the old value to the redis cache
5. Request A to write the new value into the mysql database.
   The above situation will lead to inconsistencies.
   

• If the database update fails, when thread B requests to access the cache again, it finds that there is no data in redis and the cache is missing. When reading mysql again, the old value is read from the database

solution

Review Ali's internal cache breakdown scheme

• Multiple threads query this data in the database at the same time, then we can use a mutex to lock it on the first request to query data.
• When the other threads reach this point, they wait until the first thread gets the data and then cache it if they can't get the lock.
  Later threads come in and find that there is already a cache, so they go directly to the cache.
  

Adopt delay double delete strategy

• The time of adding sleep is to allow thread B to read data from the database first, then write the actual data into the cache, and then thread A deletes it. Therefore, the time for thread A to sleep needs to be longer than the time for thread B to read data and then write it into the cache.
• In this way, when other threads read data, they will find that the cache is missing, so they will read the latest value from the database. Because this solution will delete the cache value after a delay for a period of time after the first deletion, we also call it "delayed double deletion".

Double deletion scheme interview questions

• How long should this delete be dormant?

  • The time for thread Asleep needs to be longer than the time for thread B to read data and then write it into the cache.
  • When the business program is running, count the operation time of the thread reading data and writing the cache, and self-assess the time-consuming of the data reading business logic of your own project, and make estimates based on this. Then, the sleep time for writing data can be increased by 100 milliseconds on the basis of the time-consuming business logic for reading data.
  • The purpose of doing this is to ensure that the read request ends, and the write request can delete the cache dirty data caused by the read request.

• The effect of the current demonstration is mysql stand-alone, what if the mysql master-slave read-write separation architecture?

  • Request A to perform a write operation and delete the cache
  • Request A to write data into the database,
  • Request B query cache found that the cache has no value
  • Request B to query from the slave library. At this time, the master-slave synchronization has not been completed, so the query is the old value
  • Request B to write the old value to the cache
  • The database completes the master-slave synchronization, and the slave database changes to a new value. The above situation is the cause of data inconsistency. Still use the double-deletion delay strategy .
  • However, the sleep time is modified to add several hundred ms on the basis of the delay time of master-slave synchronization

• What should I do if the throughput is reduced by this synchronous elimination strategy?

  • 

Update the database first, then delete the cache

abnormal problem

business guiding ideology

• Foreigners Papers
• The well-known social networking site facebook also proposed in the paper " Scaling Memcache at Facebook "
• Our above canal is also a similar idea: the above binlog subscription program has a ready-made middleware called canal in mysql, which can complete the function of subscribing to binlog logs.

solution

1. The cache value to be deleted or the database value to be updated can be temporarily stored in the message queue (for example, using Kafka/RabbitMQ, etc.).
2. When the program fails to delete the cached value or update the database value successfully, it can re-read these values ​​from the message queue, and then delete or update again.
3. If it can be successfully deleted or updated, we need to remove these values ​​from the message queue to avoid repeated operations. At this time, we can also ensure that the data in the database and the cache are consistent, otherwise we need to retry again
4. If there is still no success after a certain number of retries, we need to send an error message to the business layer and notify the operation and maintenance personnel.

Summarize

Option 2 and Option 3 use which one? What are the pros and cons

• In most business scenarios, we will use Redis as a read-only cache. If the positioning is a read-only cache, theoretically we can either delete the cache value first and then update the database, or update the database first and then delete the cache, but there is no perfect solution, and the principle of balancing the two evils tends to be the lesser
• My personal suggestion is to give priority to the scheme of updating the database first and then deleting the cache. The reasons are as follows:
  • Delete the cache value first and then update the database, which may cause the request to access the database due to missing cache, which will put pressure on the database and seriously cause mysql to be full.
  • If it is not easy to estimate the time for reading the database and writing the cache in the business application, then it is not easy to set the waiting time in the delayed double deletion.
• One more thing to add: If you use the scheme of updating the database first and then deleting the cache: if the business layer requires that consistent data must be read, then we need to temporarily store concurrent read requests on the Redis cache client when updating the database, etc. After the database is updated and the cached value is deleted, the data is read again to ensure data consistency.