[Technical Solution] (Multi-level) cache architecture best practice

It's 3:30 in the morning, I'm too sleepy, I still need to make up for it tomorrow...

Because the project I have done recently involves caching, I will write an article related to caching for your reference. If you find any mistakes in the article, I hope you can correct me.

Cache involves a wide range, from CPU cache, to in-process cache, to out-of-process cache. In addition, it is already one o'clock in the morning, and I have to keep my few stumps of hair. This article will not write every detail, sorry.

About CPU cache

Here is a word about CPU cache, because the core idea of ​​cache is all those things, such as hit, elimination, consistency, etc.
I have rewritten some things about the CPU before, and only one picture is attached here.

ps: I heard that some manufacturer’s CPU has recently changed the three-level cache architecture and bus lock. If you have relevant resources, please send them to me. I’ll take a look, hhh～

About multi-level caching

The focus of this article is not on multi-level caching, because I have written an article about the detailed design of multi-level caching before.
Brief steps:

1. browser cache
2. Nginx reverse proxy, load OpenResty cluster
3. OpenResty is based on Nginx and Lua, and can implement Lua business coding. Its cache performance is very good, and JD Technology has done a pressure test comparison.
4. Query Redis if OpenResty cache misses
5. If the Redis cache misses, query the process cache
6. In order to ensure the data consistency between the cache and DB, you can also use Canal and DTS for data synchronization (based on Mysql's Binlog, and the master-slave principle, disguised as a slave)

About the second level cache

Best practice of second-level cache: Caffeine + Redis

1. Go to Caffeine first, if not hit, go to Redis
2. In order to ensure data consistency, Canal / DTS can be used for data synchronization
3. If the process caches Caffeine, just set a timing synchronization

Performance optimization:

1. Caffeine is used for process caching because of its underlying ConcurrentHashMap structure, which supports concurrency (the performance comparison report of each process cache will be published later)
2. For out-of-process caching, I usually choose Redis without thinking, based on its fault tolerance, multiple data structures, etc. (Comparative analysis with memcache, etc. will be published later)

There are also secondary cache frameworks on the market, such as J2Cache. The framework itself does not do extra work, mainly integrating common in-process cache and out-of-process cache.

If it is developed based on Spring, the Spring Cache framework designed based on AOP is adapted to commonly used caches. Its own annotations and strategies are naturally decoupled from the business, which is very good. However, how to integrate Redis requires special attention here! ! !

Because when integrating Redis, Spring Cache's clearing strategy uses the keys instruction when deleting the cache from Redis . The time complexity of the keys instruction is O(N). If the number of caches is large, it will cause obvious obstacles. Therefore, in the production environment Redis will disable this command, resulting in an error.

//keys 指令
byte[][] keys = Optional.ofNullable(connection.keys(pattern)).orElse(Collections.emptySet())
         .toArray(new byte[0][]);

 if (keys.length > 0) {
    
    
     statistics.incDeletesBy(name, keys.length);
     connection.del(keys);
 }

Therefore, we can rewrite DefaultRedisCacheWriter (the default Redis cache writer provided by spring cache, which internally encapsulates logic such as cache addition, deletion, modification, etc.)

Use the scan command instead of the keys command

//使用scan命令代替keys命令
Cursor<byte[]> cursor = connection.scan(new ScanOptions.ScanOptionsBuilder().match(new String(pattern)).count(1000).build());
Set<byte[]> byteSet = new HashSet<>();
while (cursor.hasNext()) {
    
    
    byteSet.add(cursor.next());
}

byte[][] keys = byteSet.toArray(new byte[0][]);

To be honest, don’t use multi-level cache and second-level cache for the sake of showing off skills. It may increase unnecessary development costs and unknown problems, and you must also evaluate the amount of data. Don’t engage in caching, causing an avalanche. That would really be a waste of money.

Word of wisdom: technologies that do not combine business are hooligans .

in-process cache

What are the benefits of in-process caching?

Compared with no cache, the advantage of in-process caching is that data reading no longer requires access to the backend, such as a database.

Compared with out-of-process caching (such as redis/memcache), in-process caching saves network overhead, so firstly, it saves intranet bandwidth, and secondly, the response delay will be lower.

What are the disadvantages of in-process caching?

If the data is cached in multiple nodes of the site and service, and the data is stored in multiple copies, it is difficult to guarantee consistency.

How to ensure the data consistency of the in-process cache?

1. Other nodes can be notified by a single node.
2. Other nodes can be notified through MQ.
3. In order to avoid coupling and reduce complexity, "real-time consistency" is simply abandoned. Each node starts a timer to regularly pull the latest data from the backend and update the memory cache. Dirty data will be read between nodes updating backend data and other nodes updating data through timers.

Why can't the in-process cache be used frequently?

The in-process caching of sites and services actually violates the stateless principle of layered architecture design

When can I use in-process caching?

1. Read-only data can be considered loaded into memory when the process starts. (implements InitializingBean)
2. For extremely high concurrency, in-process caching can be considered if the transparent transmission backend is under extreme pressure. (spike)
3. Data inconsistency services are allowed to a certain extent.

Incorrectness of passing data between services through the cache

1. For data pipeline scenarios, MQ is more suitable than cache;
2. Multiple services should not share a single cache instance, and should be split and decoupled vertically;
3. Service-oriented architecture should not bypass the service to read its backend cache/db, but should be accessed through the RPC interface.

Using caching without considering avalanche bugs

If the cache is down, all requests will be pushed to the database. If the capacity is not estimated in advance, the database may be overwhelmed (the database may not be up until the cache is restored), causing the system as a whole to be unserviceable.

The capacity should be estimated in advance. If the cache hangs up, the database can still carry it, so that the above solution can be implemented.

Otherwise, further design is required:
using a high-availability cache cluster (such as active and standby), after a cache instance hangs up, it can automatically fail over.
Using cache horizontal segmentation, after a cache instance is down, all the traffic will not be pressed to the database.

Incorrectness of multi-service shared cache instance

1. It may lead to key conflicts and flush out each other's data; (you can use namespace:key to make keys and isolate them)
2. The amount of data and throughput corresponding to different services are different, and sharing one instance can easily cause one service to squeeze out the hot data of another service;
3. Sharing an instance will lead to coupling between services, which is contrary to the design principle of "database, private cache" of the microservice architecture;

For example, in a monolithic architecture project I have done, Caffeine is used for caching, and each business will have a Caffeine instance.

The solution to the inconsistency between the cache and the database

1. master-slave synchronization;
2. Subscribe to the binlog of the slave library through the tool (DTS/cannal), where you can know the most accurate time when the data synchronization of the slave library is completed;
3. After executing the write operation from the library, initiate deletion to the cache again, and eliminate old data that may be written to the cache during this period;

Operate the cache first, or the database

1. For read requests, read the cache first, if there is no hit, read the database, and then set back to the cache
2. write request
   • Cache first, then database
   • Cache, use delete instead of set

Cache Aside Pattern Solution

For read requests:

(1) Read cache first, then read db;

(2) If cache hit, return the data directly;

(3) If the cache miss, then access the db and set the data back to the cache;

For write requests:

(1) Eliminate the cache instead of updating the cache;

(2) Operate the database first, and then eliminate the cache;

Why is the cache always eliminated, not modified

The cost of modification is too high, it is not a big problem to choose to eliminate without thinking

Cache-related cleanup strategies

FIFO (first in first out)
strategy, the data that enters the cache first will be cleared first when the cache space is insufficient (exceeding the maximum element limit) to make room for new data. The policy algorithm mainly compares the creation time of cache elements. This type of strategy can be selected in scenarios where data effectiveness is required, and priority is given to ensuring the availability of the latest data.

LFU (less frequently used)
least frequently used strategy, regardless of whether it has expired or not, is judged according to the number of times the elements are used, and the elements that are used less frequently are cleared to release space. The policy algorithm mainly compares the hitCount (number of hits) of elements. In the scenario of ensuring the validity of high-frequency data, this type of strategy can be selected.

LRU (least recently used)
The least recently used strategy, regardless of whether it has expired, according to the timestamp when the element was last used, clear the element with the farthest usage timestamp to free up space. The strategy algorithm mainly compares the time when the element was last used by get. It is more applicable in hot data scenarios, and priority is given to ensuring the validity of hot data.

In addition, there are some simple strategies such as:

Judging by the expiration time, clean up the element with the longest expiration time;
judging by the expiration time, clean up the element that will expire recently;
clean up randomly;
clean up according to the length of the keyword (or element content), etc.

Why choose Caffeine

The underlying data structure, the W-TinyLFU algorithm, and of course an authoritative performance comparison chart of each component, who would not want to use it well, right? (About the source code of Caffeine, I will write a separate article in another day)

Why choose Redis

There is no reason, just choose without thinking. Next week, I will write an article on the source code of Redis7, and you will understand.

Redis best application practice

1. Display the latest project list on the home page: Redis uses a memory-resident cache, which is very fast. LPUSH is used to insert a content ID, which is stored as a key in the head of the list. LTRIM is used to limit the number of items in the list to a maximum of 5000. If the amount of data that the user needs to retrieve exceeds this cache capacity, then the request needs to be sent to the database.
2. Deletion and filtering: If an article is deleted, it can be completely removed from the cache using LREM.
3. Leaderboards and related issues: The leaderboard is sorted by score. The ZADD command can directly realize this function, and the ZREVRANGE command can be used to obtain the top 100 users according to the score, and ZRANK can be used to obtain the user ranking, which is very direct and easy to operate.
4. Sort by user votes and time: Leaderboards, scores will change over time. The LPUSH and LTRIM commands are used in combination to add articles to a list. A background task is used to fetch the list and recalculate the order of the list, and the ZADD command is used to populate the resulting list in the new order. Lists can be retrieved very quickly, even for heavily loaded sites.
5. Expired item handling: use Unix time as the key to keep the list sorted by time. Retrieving current_time and time_to_live does the hard work of finding expired items. Another background task does a query with ZRANGE...WITHSCORES, removing expired entries.
6. Counting: There are many uses for various statistics, such as knowing when an IP address is blocked. The INCRBY command makes this easy, by atomically incrementing the count; GETSET is used to reset the counter; and the expires attribute is used to identify when a key should be deleted.
7. Specific Items at Specific Time: This is a visitor-specific problem and can be solved by using the SADD command for each page view. SADD does not add already existing members to a set.
8. Pub/Sub: Keeping user-to-data mappings in updates is a common task in systems. Redis's pub/sub functionality makes this easier using the SUBSCRIBE, UNSUBSCRIBE, and PUBLISH commands.
9. Queues: Queues are everywhere in current programming. In addition to push and pop type commands, Redis also has blocking queue commands, which allow a program to be added to the queue by another program during execution.