In this section we will make a non-blocking cache, a tool that can help us solve real-world problems where concurrent programs appear but no ready-made library can solve them. This problem is called a memoizing function. That is to say, we need to cache the return result of the function, so that when we call the function, we only need to calculate once, and then we only need to return the result of the calculation. Our solution would be a concurrency-safe design that would avoid locking the entire cache and causing all operations to contend for a lock.
We will use the httpGetBody function below as an example of the function we need to cache. This function will make an HTTP GET request and get the http response body. The overhead of calling this function itself is relatively large, so we try to avoid calling it repeatedly when it is not necessary.
func httpGetBody(url string) (interface{}, error) {
resp, err := http.Get(url)
if err != nil {
return nil, err
}
defer resp.Body.Close()
return ioutil.ReadAll(resp.Body)
}
The last line hides some details a bit. ReadAll will return two results, a []byte array and an error, but these two objects can be assigned to the interface{} and error types in the return statement of httpGetBody, so we can return the result like this without additional work now. We chose this return type in httpGetBody so that it could match the cache.
Here is the first "draft" of the cache we're going to design:
gopl.io/ch9/memo1
// Package memo provides a concurrency-unsafe
// memoization of a function of type Func.
package memo
// A Memo caches the results of calling a Func.
type Memo struct {
f Func
cache map[string]result
}
// Func is the type of the function to memoize.
type Func func(key string) (interface{}, error)
type result struct {
value interface{}
err error
}
func New(f Func) *Memo {
return &Memo{f: f, cache: make(map[string]result)}
}
// NOTE: not concurrency-safe!
func (memo *Memo) Get(key string) (interface{}, error) {
res, ok := memo.cache[key]
if !ok {
res.value, res.err = memo.f(key)
memo.cache[key] = res
}
return res.value, res.err
}
The Memo instance will record the function f (of type Func) that needs to be cached, and the cache content (which is a map of string to result mapping). Each result is simply the value pair returned by the function -- a value and an error value. Moving on we will show some variants of Memo, but all examples will follow these aspects above.
Below is an example using Memo. For each element of the incoming URL we call Get and print a log of the call delay and the size of the data returned:
m := memo.New(httpGetBody)
for url := range incomingURLs() {
start := time.Now()
value, err := m.Get(url)
if err != nil {
log.Print(err)
}
fmt.Printf("%s, %s, %d bytes\n",
url, time.Since(start), len(value.([]byte)))
}
We can use the test package (the subject of Chapter 11) to systematically characterize the effects of caching. From the test output below, we can see that the URL stream contains some repetitions, and although our first call to each URL (*Memo).Gettakes hundreds of milliseconds, the second one takes only 1 millisecond Return complete data.
$ go test -v gopl.io/ch9/memo1
=== RUN Test
https://golang.org, 175.026418ms, 7537 bytes
https://godoc.org, 172.686825ms, 6878 bytes
https://play.golang.org, 115.762377ms, 5767 bytes
http://gopl.io, 749.887242ms, 2856 bytes
https://golang.org, 721ns, 7537 bytes
https://godoc.org, 152ns, 6878 bytes
https://play.golang.org, 205ns, 5767 bytes
http://gopl.io, 326ns, 2856 bytes
--- PASS: Test (1.21s)
PASS
ok gopl.io/ch9/memo1 1.257s
This test does all the calls sequentially.
Since such independent HTTP requests can be well concurrent, we can change this test to a concurrent form. You can use sync.WaitGroup to wait for all requests to complete before returning.
m := memo.New(httpGetBody)
var n sync.WaitGroup
for url := range incomingURLs() {
n.Add(1)
go func(url string) {
start := time.Now()
value, err := m.Get(url)
if err != nil {
log.Print(err)
}
fmt.Printf("%s, %s, %d bytes\n",
url, time.Since(start), len(value.([]byte)))
n.Done()
}(url)
}
n.Wait()
The test ran faster this time, but unfortunately it seems that the test doesn't work every time. We noticed some unexpected cache misses, or hit the cache but returned the wrong value, or even crashed outright.
But what's worse is that sometimes the program still runs correctly (translation: the most crashing occasional bug), so we may not even realize that the program has a bug. But we can run the program with the -race flag, and the race detector (§9.6) will print a report like this:
$ go test -run=TestConcurrent -race -v gopl.io/ch9/memo1
=== RUN TestConcurrent
...
WARNING: DATA RACE
Write by goroutine 36:
runtime.mapassign1()
~/go/src/runtime/hashmap.go:411 +0x0
gopl.io/ch9/memo1.(*Memo).Get()
~/gobook2/src/gopl.io/ch9/memo1/memo.go:32 +0x205
...
Previous write by goroutine 35:
runtime.mapassign1()
~/go/src/runtime/hashmap.go:411 +0x0
gopl.io/ch9/memo1.(*Memo).Get()
~/gobook2/src/gopl.io/ch9/memo1/memo.go:32 +0x205
...
Found 1 data race(s)
FAIL gopl.io/ch9/memo1 2.393s
Line 32 of memo.go appears twice, indicating that two goroutines updated the cache map without synchronization intervention. This shows that Get is not concurrency safe and there is a data race.
28 func (memo *Memo) Get(key string) (interface{}, error) {
29 res, ok := memo.cache(key)
30 if !ok {
31 res.value, res.err = memo.f(key)
32 memo.cache[key] = res
33 }
34 return res.value, res.err
35 }
The easiest way to make cache concurrency safe is to use monitor-based synchronization. As long as you add a mutex to Memo, acquire the mutex lock at the beginning of Get, and release the lock when returning, you can make the cache operation happen in the critical section:
gopl.io/ch9/memo2
type Memo struct {
f Func
mu sync.Mutex // guards cache
cache map[string]result
}
// Get is concurrency-safe.
func (memo *Memo) Get(key string) (value interface{}, err error) {
memo.mu.Lock()
res, ok := memo.cache[key]
if !ok {
res.value, res.err = memo.f(key)
memo.cache[key] = res
}
memo.mu.Unlock()
return res.value, res.err
}
The tests still run concurrently, but this time the race checker is "silent". Unfortunately this little change to Memo makes us completely lose the performance benefits of concurrency. The lock is held during each call to f, and Get serializes I/O operations that could otherwise run in parallel. Our goal in this chapter is to implement a lock-free cache, not a function cache that serializes all requests as it is now.
In the next implementation of Get, the goroutine calling Get will acquire the lock twice: once in the lookup phase, and again in the update phase if the lookup returns nothing. In the middle of these two lock acquisitions, other goroutines are free to use the cache.
gopl.io/ch9/memo3
func (memo *Memo) Get(key string) (value interface{}, err error) {
memo.mu.Lock()
res, ok := memo.cache[key]
memo.mu.Unlock()
if !ok {
res.value, res.err = memo.f(key)
// Between the two critical sections, several goroutines
// may race to compute f(key) and update the map.
memo.mu.Lock()
memo.cache[key] = res
memo.mu.Unlock()
}
return res.value, res.err
}
These modifications resulted in a performance boost again, but some URLs were fetched twice. This happens when two or more goroutines call Get at the same time to request the same URL. Multiple goroutines query the cache together, find that there is no value, and then call the slow and slow function f together. After getting the result, the map will also be updated. One of the results obtained will overwrite the results of the other.
Ideally, redundant work should be avoided. And this "avoidance" work is generally called duplicate suppression (repeated suppression / avoidance). In the following version of Memo, each map element is a pointer to an entry. Each entry contains the content cache of the result of a call to function f. The difference from the previous one is that this entry also contains a channel called ready. After the entry's result is set, the channel is closed, so that it is safe to broadcast (§8.9) to other goroutines to read the entry's result.
gopl.io/ch9/memo4
type entry struct {
res result
ready chan struct{} // closed when res is ready
}
func New(f Func) *Memo {
return &Memo{f: f, cache: make(map[string]*entry)}
}
type Memo struct {
f Func
mu sync.Mutex // guards cache
cache map[string]*entry
}
func (memo *Memo) Get(key string) (value interface{}, err error) {
memo.mu.Lock()
e := memo.cache[key]
if e == nil {
// This is the first request for this key.
// This goroutine becomes responsible for computing
// the value and broadcasting the ready condition.
e = &entry{ready: make(chan struct{})}
memo.cache[key] = e
memo.mu.Unlock()
e.res.value, e.res.err = memo.f(key)
close(e.ready) // broadcast ready condition
} else {
// This is a repeat request for this key.
memo.mu.Unlock()
<-e.ready // wait for ready condition
}
return e.res.value, e.res.err
}
Now the Get function includes the following steps: acquiring the mutex to protect the shared variable cache map, querying whether the specified entry exists in the map, if not found, then allocate space to insert a new entry, and release the mutex. If there is an entry and its value has not been written (that is, other goroutines are calling the slow function f), the goroutine must wait for the value to be ready before reading the result of the entry. If you want to know if it is ready, you can read directly from the ready channel, because the read operation is blocked until the channel is closed.
If there is no entry, an entry that is not ready needs to be inserted into the map, and the currently calling goroutine needs to be responsible for calling the slow function, updating the entry, and broadcasting to all other goroutines that the entry is ready.
The e.res.value and e.res.err variables in the entry are shared among multiple goroutines. The goroutine that created the entry also sets the entry's value, and other goroutines will read the entry's value immediately after receiving the "ready" broadcast message. Although it will be accessed by multiple goroutines at the same time, it does not require a mutex. The closing of the ready channel must occur before other goroutines receive the broadcast event, so the first goroutine's write operations to these variables must occur before these read operations. No data races will occur.
In this way, a concurrent, non-repetitive, non-blocking cache is completed.
The above Memo implementation uses a mutex to protect the shared map variable when multiple goroutines call Get. Let’s compare this design with the aforementioned scheme of restricting the map variable to a single monitor goroutine, which needs to send a message when calling Get.
The declarations of Func, result and entry are the same as before:
// Func is the type of the function to memoize.
type Func func(key string) (interface{}, error)
// A result is the result of calling a Func.
type result struct {
value interface{}
err error
}
type entry struct {
res result
ready chan struct{} // closed when res is ready
}
However, the Memo type now contains a channel called requests that the caller of Get uses to communicate with the monitor goroutine. The element type in the requests channel is request. The caller of Get will fill in the two sets of keys in this structure, and actually use these two variables to cache the function. Another channel called response will be used to send the response. This channel will only return a single value.
gopl.io/ch9/memo5
// A request is a message requesting that the Func be applied to key.
type request struct {
key string
response chan<- result // the client wants a single result
}
type Memo struct{ requests chan request }
// New returns a memoization of f. Clients must subsequently call Close.
func New(f Func) *Memo {
memo := &Memo{requests: make(chan request)}
go memo.server(f)
return memo
}
func (memo *Memo) Get(key string) (interface{}, error) {
response := make(chan result)
memo.requests <- request{key, response}
res := <-response
return res.value, res.err
}
func (memo *Memo) Close() { close(memo.requests) }
The Get method above will create a response channel, put it into the request structure, send it to the monitor goroutine, and then receive it again immediately.
The cache variable is restricted to the monitor goroutine `(*Memo).server, as will be seen below. The monitor will read requests in a loop until the request channel is closed by the Close method. Every request will go to the cache and if no entry is found, a new entry will be created/inserted.
func (memo *Memo) server(f Func) {
cache := make(map[string]*entry)
for req := range memo.requests {
e := cache[req.key]
if e == nil {
// This is the first request for this key.
e = &entry{ready: make(chan struct{})}
cache[req.key] = e
go e.call(f, req.key) // call f(key)
}
go e.deliver(req.response)
}
}
func (e *entry) call(f Func, key string) {
// Evaluate the function.
e.res.value, e.res.err = f(key)
// Broadcast the ready condition.
close(e.ready)
}
func (e *entry) deliver(response chan<- result) {
// Wait for the ready condition.
<-e.ready
// Send the result to the client.
response <- e.res
}
Similar to the mutex-based version, the first request for a key is responsible for calling the function f and passing in the key, storing the result in the entry, and closing the ready channel to broadcast the entry's ready message. Use (*entry).callto do the above work.
Immediately after the request for the same key, it will find that there is an existing entry in the map, and then wait for the result to become ready, and send the result from the response to the client's goroutine. The above work is (*entry).deliverused to accomplish. Calls to the call and deliver methods must be made in their own goroutines to ensure that monitor goroutines are not blocked from processing new requests.
This example shows that whether we use locking or communication to create concurrent programs is feasible.
It's hard to say which of the two options above is better in a particular situation, but it's worth knowing about them. Sometimes switching from one way to another can make your code more concise. (Annotation: Isn't it good that golang respects communication concurrency?)
Exercise 9.3: Extend the Func type and (*Memo).Getmethod to allow the caller to provide an optional done channel with the ability to cancel the entire operation through that channel (§8.9). The result of a cancelled Func call should not be cached.