Art~Delayed double deletion to ensure data consistency between Redis and Mysql

Delayed double delete

This problem is related to the data update of cached redis and master-slave mysql. In high concurrency, it is easy to cause data inconsistency between the cache and the database.

If there is inconsistency, it is very dangerous, such as our common limited-edition panic buying, it needs to be fast in response and reliable in data.

What do I mean by double deletion?

First of all, we know that the data cached in redis is used for reading, and writing data is generally written to mysql.

If the cache is deleted first, and there is no time to write MySQL, another thread will read and find that the cache is empty, then go to the database to read the data and write it into the cache. At this time, the cache is dirty data.

If you write to the library first, before deleting the cache, once the library writing thread hangs up without deleting the cache, the cache will always store the dirty data.

So deleting it once will definitely not work, so you have to double delete.

But what's the problem if there is no delay?

For example, I did a cache delete first. Due to high concurrency, a read request just read the dirty data in mysql. If this data is written to the cache immediately, then I can delete it when I delete it for the second time. This is dirty data, but if the second deletion occurs before writing to the cache, there will still be a dirty data problem.

Summary process

1. Delete cache first
2. Rewrite the database
3. Sleep for xx milliseconds (according to specific business time)
4. Delete cache again

How to determine xx milliseconds?

It is necessary to evaluate the time-consuming business logic of project read data to ensure that the read request ends, and the write request can delete the dirty cached data caused by the read request.

This strategy also considers the time consuming of redis and database master-slave synchronization. The final sleep time of writing data: add a few hundred ms to the time-consuming business logic of reading data. For example: sleep for 1 second.

Of course, there are some implementation ideas, but the effect is definitely not as good as delayed double deletion, but the development cost will be more friendly, such as the following implementation ideas.

Set cache expiration time

In theory, setting the cache expiration time is a solution to ensure eventual consistency.
All write operations are based on the DB. As long as the cache expiration time is reached, subsequent read requests will naturally read the new value from the DB, and then backfill the cache.

Combined with the double delete strategy + cache timeout setting, the worst case is that there is inconsistency in the data within the timeout period, and the write request time is increased.

After the database is written, the cache is successfully deleted again to ensure that the
above scheme has a disadvantage, that is, after the database is operated, the cache fails to be deleted due to various reasons. At this time, data inconsistency may occur.
A guaranteed retry plan must be provided.

• Scheme 1
  Specific process:

1. Update database data
2. The cache failed to delete due to various problems
3. Send the key to be deleted to the message queue
4. Consume the message by yourself and get the key that needs to be deleted
5. Continue to retry the delete operation until it succeeds

However, this scheme has a shortcoming, causing a lot of intrusion to the line of business code. So there is a second option.
In the second scenario, start a subscription program to subscribe to the binlog of the database to obtain the data that needs to be manipulated. In the application, start another program to obtain the information from the subscription program and delete the cache operation.

• Scheme 2
  Specific process:

1. Update database data
2. The database will write the operation information into the binlog log
3. The subscription program extracts the required data and key
4. Start another non-business code to get the information
5. Attempt to delete the cache operation and found that the deletion failed
6. Send this information to the message queue
7. Re-obtain the data from the message queue and retry the operation.