In the development process, map data structure is essential, in Golang, use the map more or less experience with other languages different experience, such as access to non-existent element will return a null value for its type, map how much is the size, why would report "can not the address of take" error, traversing the map of randomness, and so on.
This paper aims to study the underlying map implementation, in order to answer these doubts.
基于Golang 1.8.3
1. Data structure and memory management
hashmap definition is located in src / runtime / hashmap.go, first we look at the definition of hashmap and bucket:
type hmap struct {
count int // 元素的个数 flags uint8 // 状态标志 B uint8 // 可以最多容纳 6.5 * 2 ^ B 个元素，6.5为装载因子 noverflow uint16 // 溢出的个数 hash0 uint32 // 哈希种子 buckets unsafe.Pointer // 桶的地址 oldbuckets unsafe.Pointer // 旧桶的地址，用于扩容 nevacuate uintptr // 搬迁进度，小于nevacuate的已经搬迁 overflow *[2]*[]*bmap } Wherein, overflow is a pointer, pointing to a number of elements of array 2, the type of array is a pointer to a slice, the slice is a bucket element (BMAP) address, the buckets are balanced histograms; Why two ? Go map because when too many hash conflict, expansion of operations will take place, in order to move the amount of incomplete data, using an incremental move, [0] indicates an overflow bucket collections currently in use, [1] is in the expansion occurs, save the old overflow bucket collection; the meaning of existence is to prevent overflow overflow barrels gc.
// A bucket for a Go map.
type bmap struct {
// 每个元素hash值的高8位，如果tophash[0] < minTopHash，表示这个桶的搬迁状态 tophash [bucketCnt]uint8 // 接下来是8个key、8个value，但是我们不能直接看到；为了优化对齐，go采用了key放在一起，value放在一起的存储方式， // 再接下来是hash冲突发生时，下一个溢出桶的地址 } Tophash existence is to fast trial and error, after all, only eight, comparison will be a little faster.
As can be seen from the definition, in a manner different from the STL map red-black tree implementation, the HashTable golang implemented using conflict resolution method uses a link address. That is, using an array + linked list to achieve the map. In particular, for a key, a few of the more important is calculated as follows:
| key | hash | Hshtop | bucket index |
|---|---|---|---|
| key | hash := alg.hash(key, uintptr(h.hash0)) | top := uint8(hash >> (sys.PtrSize*8 - 8)) | bucket := hash & (uintptr(1)<<h.B - 1)，即 hash % 2^B |
For example, for B = 3, if hash (key) = 4 when, hashtop = 0, bucket = 4, when the hash (key) = 20 when, hashtop = 0, bucket = 4; in this example we will use the relocation process to.
Memory layout like this:
2. Create - makemap
Create map is relatively simple, after the parameter calibration, to find the right application B to the memory space of the tub, and then this structure is hmap to wear, and its initialization.
3. Access - mapaccess
For a given key, you can be found through the following operations if it exists
The method is defined as
// returns key, if not find, returns nil
func mapaccess1(t *maptype, h *hmap, key unsafe.Pointer) unsafe.Pointer // returns key and exist. if not find, returns nil, false func mapaccess2(t *maptype, h *hmap, key unsafe.Pointer) (unsafe.Pointer, bool) // returns both key and value. if not find, returns nil, nil func mapaccessK(t *maptype, h *hmap, key unsafe.Pointer) (unsafe.Pointer, unsafe.Pointer) In the case of visible not find the corresponding key, it will return nil
4. Assign - mapassign
As a logical key distribution space, roughly find similar; but adds write protection and expansion of operations; note that the allocation process and the removal process are not found in oldbuckets, this is because we must first expansion judge and operations; as follows:
扩容是整个hashmap的核心算法，我们放在第6部分重点研究。
Create a bucket overflow, and splice in the current tail barrel to achieve a similar operation linked list:
// 获取当前桶的溢出桶
func (b *bmap) overflow(t *maptype) *bmap { return *(**bmap)(add(unsafe.Pointer(b), uintptr(t.bucketsize)-sys.PtrSize)) } // 设置当前桶的溢出桶 func (h *hmap) setoverflow(t *maptype, b, ovf *bmap) { h.incrnoverflow() if t.bucket.kind&kindNoPointers != 0 { h.createOverflow() //重点，这里讲溢出桶append到overflow[0]的后面 *h.overflow[0] = append(*h.overflow[0], ovf) } *(**bmap)(add(unsafe.Pointer(b), uintptr(t.bucketsize)-sys.PtrSize)) = ovf } 5. Delete - mapdelete
Delete a key similar to the dispensing operation, since the memory structure is an array hashmap + chain, so the real key to delete only the corresponding slot set to empty, and does not reduce memory; follows:
6. Expansion - growWork
First, it is determined whether the logical expansion is
func (h *hmap) growing() bool { return h.oldbuckets != nil } When h.oldbuckets not nil it? Assign logical allocation, when no key position for use, and satisfies the test condition (load factor of> 6.5 or too overflow communication), the trigger logic hashGrow:
func hashGrow(t *maptype, h *hmap) { //判断是否需要sameSizeGrow，否则"真"扩 bigger := uint8(1) if !overLoadFactor(int64(h.count), h.B) { bigger = 0 h.flags |= sameSizeGrow } // 下面将buckets复制给oldbuckets oldbuckets := h.buckets newbuckets := newarray(t.bucket, 1<<(h.B+bigger)) flags := h.flags &^ (iterator | oldIterator) if h.flags&iterator != 0 { flags |= oldIterator } // 更新hmap的变量 h.B += bigger h.flags = flags h.oldbuckets = oldbuckets h.buckets = newbuckets h.nevacuate = 0 h.noverflow = 0 // 设置溢出桶 if h.overflow != nil { if h.overflow[1] != nil { throw("overflow is not nil") } // 交换溢出桶 h.overflow[1] = h.overflow[0] h.overflow[0] = nil } } OK, the following formal entry into the focus of expansion stage; in assign and delete operations will trigger expansion growWork:
func growWork(t *maptype, h *hmap, bucket uintptr) { // 搬迁旧桶，这样assign和delete都直接在新桶集合中进行 evacuate(t, h, bucket&h.oldbucketmask()) //再搬迁一次搬迁过程中的桶 if h.growing() { evacuate(t, h, h.nevacuate) } } 6.1 relocation process
In general, the new bucket array size is 2 times the original (in! SameSizeGrow () conditions), the first half of the new bucket array can "analogy" as the old barrel for a key, which fall after the relocation of the index it?
假设旧桶数组大小为2^B， 新桶数组大小为2*2^B，对于某个hash值X
若 X & (2^B) == 0，说明 X < 2^B，那么它将落入与旧桶集合相同的索引xi中；
否则，它将落入xi + 2^B中。
例如，对于旧B = 3时，hash1 = 4，hash2 = 20，其搬迁结果类似这样。
源码中有些变量的命名比较简单，容易扰乱思路，我们注明一下便于理解。
| 变量 | 释义 |
|---|---|
| x *bmap | 桶x表示与在旧桶时相同的位置，即位于新桶前半段 |
| y *bmap | 桶y表示与在旧桶时相同的位置+旧桶数组大小，即位于新桶后半段 |
| xi int | 桶x的slot索引 |
| yi int | 桶y的slot索引 |
| xk unsafe.Pointer | 索引xi对应的key地址 |
| yk unsafe.Pointer | 索引yi对应的key地址 |
| xv unsafe.Pointer | 索引xi对应的value地址 |
| yv unsafe.Pointer | 索引yi对应的value地址 |
搬迁过程如下：
6.2 扩容
和 slice 一样，在 map 的元素持续增长时，每个bucket极端情况下会有很多overflow，退化成链表，需要 rehash。一般扩容是在 h.count > loadFactor(2^B)。 负载因子一般是：容量 / bucket数量，golang 的负载因子 loadFactorNum / loadFactorDen = 6.5，为什么不选择1呢，像 Redis 的 dictentry，只能保存一组键值对，golang的话，一个bucket正常情况下可以保存8组键值对； 那为什么选择6.5这个值呢，作者给出了一组数据。
| loadFactor | %overflow | bytes/entry | hitprobe | missprobe |
|---|---|---|---|---|
| 4.00 | 2.13 | 20.77 | 3.00 | 4.00 |
| 4.50 | 4.05 | 17.30 | 3.25 | 4.50 |
| 5.00 | 6.85 | 14.77 | 3.50 | 5.00 |
| 5.50 | 10.55 | 12.94 | 3.75 | 5.50 |
| 6.00 | 15.27 | 11.67 | 4.00 | 6.00 |
| 6.50 | 20.90 | 10.79 | 4.25 | 6.50 |
| 7.00 | 27.14 | 10.15 | 4.50 | 7.00 |
| 7.50 | 34.03 | 9.73 | 4.75 | 7.50 |
| 8.00 | 41.10 | 9.40 | 5.00 | 8.00 |
loadFactor：负载因子；
%overflow：溢出率，有溢出 bucket 的占比；
bytes/entry：每个 key/value 对占用字节比；
hitprobe：找到一个存在的key平均查找个数；
missprobe：找到一个不存在的key平均查找个数；
通常在负载因子 > 6.5时，就是平均每个bucket存储的键值对 超过6.5个或者是overflow的数量 > 2 ^ 15时会发生扩容（迁移）。它分为两种情况：
第一种：由于map在不断的insert 和 delete 中，bucket中的键值存储不够均匀，内存利用率很低，需要进行迁移。（注：bucket数量不做增加）
第二种：真正的，因为负载因子过大引起的扩容，bucket 增加为原 bucket 的两倍
不论上述哪一种 rehash，都是调用 hashGrow 方法：
- 定义原 hmap 中指向 buckets 数组的指针
- 创建 bucket 数组并设置为 hmap 的 bucket 字段
- 将 extra 中的 oldoverflow 指向 overflow，overflow 指向 nil
- 如果正在 growing 的话，开始渐进式的迁移，在
growWork方法里是 bucket 中 key/value 的迁移 - 在全部迁移完成后，释放内存
7.建议
做两组试验，第一组是：提前分配好 map 的总容量后追加k/v；另一组是：初始化 0 容量的 map 后做追加
package main import "testing" var count int = 100_000 func addition(m map[int]int) map[int]int { for i := 0; i < count; i++ { m[i] = i } return m } func BenchmarkGrows(b *testing.B) { b.ResetTimer() for i := 0; i < b.N; i++ { m := make(map[int]int) addition(m) } } func BenchmarkNoGrows(b *testing.B) { b.ResetTimer() for i := 0; i < b.N; i++ { m := make(map[int]int, count) addition(m) } }
sh: go test -bench=. -run=none map_grow_test.go
goos: darwin
goarch: amd64
BenchmarkGrows-4 80 15825209 ns/op
BenchmarkNoGrows-4 160 7235485 ns/op
PASS
ok command-line-arguments 3.944s
提前定义容量的case平均执行时间比未定义容量的快了100% --- 扩容时的数据拷贝和重新哈希成本很高！
再看看内存的分配次数：
sh: go test -bench=. -benchmem -run=none map_grow_test.go
goos: darwin
goarch: amd64
BenchmarkGrows-4 98 11200304 ns/op 5766531 B/op 4004 allocs/op
BenchmarkNoGrows-4 172 9005691 ns/op 2829246 B/op 1679 allocs/op
PASS
ok command-line-arguments 3.366s
提前定义容量的case的内存操作次数要少1倍多。
两个方法执行相同的次数，GC的次数也会多出一倍
package main var count int = 100_000 func addition(m map[int]int) map[int]int { for i := 0; i < count; i++ { m[i] = i } return m } func main() { for i := 0; i < 4; i++ { println("round ",i ) n := make(map[int]int, count) addition(n) println("0 size map\n") m := make(map[int]int) addition(m) } }
go build -o growth map_grow.go && GODEBUG=gctrace=1 ./growth round 0 0 size map gc 1 @0.009s 0%: 0.008+0.11+0.014 ms clock, 0.035+0.041/0.012/0.11+0.056 ms cpu, 4->4->0 MB, 5 MB goal, 4 P scvg: 0 MB released scvg: inuse: 3, idle: 59, sys: 63, released: 59, consumed: 4 (MB) scvg: 0 MB released scvg: inuse: 2, idle: 61, sys: 63, released: 59, consumed: 4 (MB) scvg: inuse: 5, idle: 58, sys: 63, released: 58, consumed: 5 (MB) scvg: inuse: 5, idle: 58, sys: 63, released: 58, consumed: 5 (MB) scvg: inuse: 5, idle: 58, sys: 63, released: 58, consumed: 5 (MB) scvg: inuse: 5, idle: 58, sys: 63, released: 58, consumed: 5 (MB) gc 2 @0.014s 0%: 0.002+0.16+0.026 ms clock, 0.009+0.051/0.009/0.093+0.10 ms cpu, 5->5->3 MB, 6 MB goal, 4 P round 1 0 size map gc 3 @0.028s 0%: 0.002+0.10+0.017 ms clock, 0.011+0/0.015/0.10+0.070 ms cpu, 7->7->0 MB, 8 MB goal, 4 P scvg: 0 MB released scvg: inuse: 7, idle: 56, sys: 63, released: 55, consumed: 7 (MB) scvg: 0 MB released scvg: inuse: 2, idle: 61, sys: 63, released: 55, consumed: 7 (MB) scvg: 0 MB released scvg: inuse: 2, idle: 60, sys: 63, released: 55, consumed: 7 (MB) gc 4 @0.033s 0%: 0.002+0.18+0.011 ms clock, 0.011+0.074/0.011/0.15+0.046 ms cpu, 5->5->3 MB, 6 MB goal, 4 P round 2 0 size map gc 5 @0.047s 0%: 0.002+0.21+0.011 ms clock, 0.011+0.032/0.016/0.079+0.045 ms cpu, 7->7->0 MB, 8 MB goal, 4 P scvg: 0 MB released scvg: inuse: 7, idle: 56, sys: 63, released: 55, consumed: 8 (MB) scvg: 0 MB released scvg: inuse: 2, idle: 61, sys: 63, released: 55, consumed: 8 (MB) scvg: 0 MB released scvg: inuse: 5, idle: 58, sys: 63, released: 55, consumed: 8 (MB) gc 6 @0.052s 0%: 0.004+0.11+0.003 ms clock, 0.016+0.062/0.008/0.10+0.015 ms cpu, 5->5->3 MB, 6 MB goal, 4 P round 3 gc 7 @0.066s 0%: 0.002+0.12+0.033 ms clock, 0.011+0.047/0.043/0.052+0.13 ms cpu, 6->6->2 MB, 7 MB goal, 4 P scvg: 0 MB released scvg: inuse: 6, idle: 56, sys: 63, released: 55, consumed: 8 (MB) 0 size map scvg: 0 MB released scvg: inuse: 4, idle: 59, sys: 63, released: 55, consumed: 8 (MB) gc 8 @0.070s 0%: 0.002+0.075+0.004 ms clock, 0.010+0.060/0.007/0.061+0.019 ms cpu, 5->5->1 MB, 6 MB goal, 4 P scvg: 0 MB released scvg: inuse: 2, idle: 61, sys: 63, released: 55, consumed: 8 (MB) scvg: 0 MB released scvg: inuse: 2, idle: 61, sys: 63, released: 55, consumed: 7 (MB) scvg: inuse: 4, idle: 58, sys: 63, released: 55, consumed: 7 (MB) gc 9 @0.075s 0%: 0.003+0.16+0.004 ms clock, 0.012+0.11/0.009/0.10+0.019 ms cpu, 4->4->3 MB, 5 MB goal, 4 P
0长度的map每次都触发gc, 但定长的不会gc.
有个1千万kv的 map，测试在什么情况下会回收内存
package main import "runtime/debug" var count = 10_000_000 var dict = make(map[int]int, count) func addition() { for i := 0; i < count; i++ { dict[i] = i } } func clear() { for k := range dict { delete(dict, k) } } func main() { addition() println("delete map item") clear() debug.FreeOSMemory() println("delete map") dict = nil debug.FreeOSMemory() }
go build -o growth big_map.go && GODEBUG=gctrace=1 ./growth gc 1 @0.005s 0%: 0.007+0.12+0.012 ms clock, 0.028+0.039/0.014/0.23+0.048 ms cpu, 306->306->306 MB, 307 MB goal, 4 P gc 2 @1.469s 0%: 0.004+1.1+0.009 ms clock, 0.018+0/0.99/0.76+0.036 ms cpu, 307->307->306 MB, 612 MB goal, 4 P delete map item gc 3 @2.101s 0%: 0.003+0.18+0.037 ms clock, 0.012+0/0.077/0.068+0.14 ms cpu, 309->309->306 MB, 612 MB goal, 4 P (forced) forced scvg: 4 MB released forced scvg: inuse: 306, idle: 77, sys: 383, released: 77, consumed: 306 (MB) delete map gc 4 @2.102s 0%: 0.001+0.14+0.002 ms clock, 0.007+0/0.12/0.002+0.011 ms cpu, 306->306->0 MB, 612 MB goal, 4 P (forced) scvg: inuse: 306, idle: 77, sys: 383, released: 77, consumed: 306 (MB) forced scvg: 306 MB released forced scvg: inuse: 0, idle: 383, sys: 383, released: 383, consumed: 0 (MB)
删除了所有kv，堆大小（goal）并无变化
设置为nil，才会真正释放map内存。（本身每2分钟强制 runtime.GC()，每5分钟 scavenge 释放内存，其实不必太过纠结是否真正释放，未真正释放也是为了后面有可能的重用， 但有时需要真实释放时，清楚怎么做才能解决问题）
总结
通过分析，我们了解了map是由数组+链表实现的HashTable，其大小和B息息相关，同时也了解了map的创建、查询、分配、删除以及扩容搬迁原理。总的来说，Golang通过hashtop快速试错加快了查找过程，利用空间换时间的思想解决了扩容的问题，利用将8个key(8个value)依次放置减少了padding空间等等。