concept

Caching is an important concept in computer science. Imagine that a component needs to access external resources, it requests resources from external sources, receives and uses resources, and these steps take time. When a component needs resources again, it can request resources again, but this method is relatively inefficient in terms of time. Instead, the component can save the request result somewhere locally and use it again. Using local data is always faster than requesting external data. This strategy is the basic concept of caching. We can find examples of these in memory, CPU caches, and server caches such as Redis.

different use cases

Caching in web services is used to reduce the latency of data requests. The web service saves the execution result of the first query, and then uses it again when needed without accessing the database again. Depending on the characteristics of the data, the cache has different situations. It can include relatively static data, such as statistical data and calculation results, or data that changes frequently, such as comment areas or SNS.

The best case is to cache data that rarely changes. Taking monthly statistical data as an example, the data of the previous month will not change. If it is cached, it may not be necessary to query the database to obtain the data of the previous month.

stupid design

For rapidly changing data, it is best to be cautious with multiple servers. Take a look at the above design, take the comment area service as an example, consider the following scenario, user A has posted some comments, then A decides to delete the comments, and user B tries to reply to the comments. In some cases, A and B send requests to different servers. A's delete operation may not propagate to B's server cache. The result would be this: Cache A and Cache B have different data, the database doesn't know which is the real one, and the integrity of the data is broken.

better way

In this case, a single external cache can be used (as shown in the figure above), and multiple servers only access the unified cache.

limitation factor

A cache is faster than a database, but much smaller in size. This is because databases store data on the drive and caches store data in memory. They follow the same characteristics, but also have different characteristics. If the host stops working, all the data in the cache will be lost, but the data in the database will not be lost.

Since the cache is located in memory, the space is limited, and it is necessary to choose which data to cache. In CS class, we will hear words like LRU (Least Recently Used, Least Recently Used), LFU (Least Frequently Used, Least Frequently Used) and FIFO (First In First Out, First In, First Out). Which" standard is called the eviction policy.

design & implementation

need

Key-Value Storage: The cache has both the read function of input key and output value, and the write function of input key and value. These functions should complete in O(logN) time on average, where N is the number of keys.
LRU eviction strategy: Due to the limited cache space, if the cache is full, some data should be cleared, and the LRU algorithm is used to implement it.
TTL (Time To Live): Each key value has a time to live. If the TTL expires, the key value should be evicted.

API design

Key-value storage means that if a key is requested, the cache will return the value of those keys that exist, similar to the hash-map abstract data type. Take an application that provides the following API concepts as an example:

func Get(key string) (hit bool, value []byte)
func Put(key string, value []byte) (hit bool)

Get: API to read value by key. Returns the equivalent value if the provided key exists in the cache. If not present, hit=false is returned. With the LRU strategy, the key will be marked as recently used so that the key will not be evicted.
Put: API for writing values by keys. If the provided key exists, value will be replaced with the new value. A new key-value store will be created if it does not exist. Because this function can add data, its execution may cause an overflow. In this case, according to the LRU policy, the least recently used key value will be cleared. Newly added/modified keys will be marked as most recently used.

data structure

concept of design

We use two different data structures: hash-map and doubly linked list to realize the characteristics of key-value reading and writing and LRU strategy.

Hash-map: Hash-map is the most widely used key-value data structure. It is a ready-made data type in Go and can be map[<type>]<type>defined.
Doubly linked list: LRU cache can be implemented by doubly linked list.

Based on these two data structures, key-value features and LRU strategies can be provided at the same time. Referring to the above design concept diagram, the key of the hash-map will be a string key, and the value will be a pointer to the node of the linked list, and the node will store the value of the key.

If called by the user Get(), the cache application will search the key in the hash-map, follow the pointer to a node in the linked list, get the value, complete the LRU strategy, and return the value to the user.

Similarly, if called Put(), it will search the key in the hash-map, track the pointer and replace the value, complete the LRU strategy, or insert a new key into the hash-map, and insert a new node into the linked list.

concurrency control

Since caches are designed to support frequent access, there will be multiple accesses at the same time, and there is always the possibility of concurrency issues.

In this design, two different data structures exist and are not always in sync. During execution, there is a tiny time interval between the modification of the hash-map and the modification of the linked list, please see the following example.

Concurrency problem case

The trigger condition for this problem is: the current cache is full, and the least recently used key is 1. This means that if a new key is added, key 1 and the equivalent value will be cleared.
User A calls Put() with new key 101 . The hash-map checks the key, finds that 101 does not exist, decides to clear 1 and adds 101 to the cache.
Meanwhile, User B calls Put() with key 1. The hash-map confirms that key 1 exists, and decides to modify the value.
The call of A continues, deletes node 1 from the linked list, and deletes key 1 from the hash-map.
Immediately afterwards, the call of B tries to access the address of node 1, and finds that the address no longer exists, so a panic occurs and the application fails.

The easiest way to prevent this from happening is to use mutual exclusion (Mutex) , refer to the following code.

func (s *CStorage) Get(key string) (data []byte, hit bool) {
  s.mutex.Lock()
  defer s.mutex.Unlock()
  
  n, ok := s.table[key]
  if !ok {
    return nil, false
  }
  if n.ttl.Before(time.Now()) {
    s.evict(n)
    s.size--
    return nil, false
  }
  
  return n.data, true
}

This code is Get()the function definition. You can see that there is a mutex code in the first line, and a mutex unlock code of defer in the second line (defer is a Go keyword that defers line execution to the end of the function. ). These codes are applied to all other datastore access functions such as Put, Delete, Clear, etc.

By using a mutex, each execution will not be affected by other operations, ensuring the security of data access.

Time To Live

Currently TTL is implemented in a passive way, which means that if a data access function (Get, Put) is executed, it will check whether the TTL has expired and decide whether to delete it. This also means that even if the node has expired, it will still exist in the data structure.

This approach doesn't need to consume a lot of CPU time to iterate over all nodes periodically, but the cache will most likely hold expired values.

In most cases, there is no problem doing this, because the expired node is likely to be in the "least recently used" state. However, it would be nice to have a function to clear expired nodes through the data structure, so we use RemoveExpired()functions.

func (s *CStorage) RemoveExpired() int64 {
  var count int64 = 0
  for key, value := range s.table {
    if value.ttl.Before(time.Now()) {
      s.Delete(key)
      count++
    }
  }
  return count
}

This function will be called periodically to clear all expired nodes.

result

github.com: https://github.com/cocm1324/cstorage
pkg.go.dev: https://pkg.go.dev/github.com/cocm1324/cstorage

Implemented Go packages can be imported into other Go projects. In addition, I also made a separate cache application, providing gRPC API, details can be found in this repository [2].

in conclusion

This was a great opportunity to revisit the concept of caching, and we implemented caching in Go. Caching is a great tool for lowering the latency of components that are space-constrained but faster.

Implementing the actual cache module can be done with hash-map and doubly linked list. Concurrency issues are a bit tricky, so have to use mutexes. Additionally, we mix reactive and proactive approaches to delete expired data.

Link: https://www.jianshu.com/p/b9187e629a07

(Copyright belongs to the original author, infringement and deletion)

Go-based cache implementation