The realization principle of slice in Golang

Preface

Simply put, a slice is a simplified version of a dynamic array. Because the length of the dynamic array is not fixed, the length of the slice naturally cannot be part of the type. Although arrays are suitable for them, the types and operations of arrays are not flexible enough, so arrays are not used much in Go code. The slicing is widely used.

Slice source code analysis

Note: I will comment out the function of each method in the source code, you can refer to the comments for understanding.

Before looking at the source code, let's look at a simple example

func main() {
    
    
	slice:=make([]int,0) //第6行
	slice=append(slice,1)//第7行
}

Created by make a length of 0, the capacity of the slice 0, slice then to append additional elements by function
Next, we analyze the slice and create additional process
sinceappend() It is a built-in function, so we can only know by looking at its assembly languageappend() what happened

Built-in functions: Go language has some built-in functions that can be used without import operations

Use the following command to view the pseudo assembly code corresponding to the Go language program:

go tool compile -S slice.go

The go tool compile command is used to call the underlying command tool provided by the Go language, and the -S parameter indicates the output assembly format.

		...
        0x0028 00040 (slice.go:6)       PCDATA  $2, $1
        0x0028 00040 (slice.go:6)       PCDATA  $0, $0
        0x0028 00040 (slice.go:6)       LEAQ    type.int(SB), AX
        0x002f 00047 (slice.go:6)       PCDATA  $2, $0
        0x002f 00047 (slice.go:6)       MOVQ    AX, (SP)
        0x0033 00051 (slice.go:6)       XORPS   X0, X0
        0x0036 00054 (slice.go:6)       MOVUPS  X0, 8(SP)
        0x003b 00059 (slice.go:6)       CALL    runtime.makeslice(SB)
        0x0040 00064 (slice.go:6)       PCDATA  $2, $1
        0x0040 00064 (slice.go:6)       MOVQ    24(SP), AX
        0x0045 00069 (slice.go:7)       PCDATA  $2, $2
        0x0045 00069 (slice.go:7)       LEAQ    type.int(SB), CX
        0x004c 00076 (slice.go:7)       PCDATA  $2, $1
        0x004c 00076 (slice.go:7)       MOVQ    CX, (SP)
        0x0050 00080 (slice.go:7)       PCDATA  $2, $0
        0x0050 00080 (slice.go:7)       MOVQ    AX, 8(SP)
        0x0055 00085 (slice.go:7)       XORPS   X0, X0
        0x0058 00088 (slice.go:7)       MOVUPS  X0, 16(SP)
        0x005d 00093 (slice.go:7)       MOVQ    $1, 32(SP)
        0x0066 00102 (slice.go:7)       CALL    runtime.growslice(SB)
        0x006b 00107 (slice.go:7)       PCDATA  $2, $1
        0x006b 00107 (slice.go:7)       MOVQ    40(SP), AX
        0x0070 00112 (slice.go:7)       MOVQ    48(SP), CX
        0x0075 00117 (slice.go:7)       MOVQ    56(SP), DX
        0x007a 00122 (slice.go:7)       MOVQ    $1, (AX)
		...

Among them, we only need to pay attention to the CALL instruction to see what it calls.

 0x003b 00059 (slice.go:6)       CALL    runtime.makeslice(SB)

The sixth line of slice.go callsmakesliceFunction, thenmakesliceFunction is used to initialize slice

0x0066 00102 (slice.go:7)       CALL    runtime.growslice(SB)

Similarly,growsliceFunctions are the
makeslice for appending elements.
Next, let’s take a look at how slices implement these functions in the go SDK.

Initialize makeslice

runtime\slice.go

type slice struct {
    
    
	array unsafe.Pointer
	len   int
	cap   int
}

The definition of the underlying structure of slice is very intuitive, a pointer to the underlying array, the current length len and the cap of the current slice.
Take a look againmakeslicefunction

func makeslice(et *_type, len, cap int) unsafe.Pointer {
    
    
	// 获取需要申请的内存大小
	mem, overflow := math.MulUintptr(et.size, uintptr(cap))
	if overflow || mem > maxAlloc || len < 0 || len > cap {
    
    
		// NOTE: Produce a 'len out of range' error instead of a
		// 'cap out of range' error when someone does make([]T, bignumber).
		// 'cap out of range' is true too, but since the cap is only being
		// supplied implicitly, saying len is clearer.
		// See golang.org/issue/4085.
		mem, overflow := math.MulUintptr(et.size, uintptr(len))
		if overflow || mem > maxAlloc || len < 0 {
    
    
			panicmakeslicelen()
		}
		panicmakeslicecap()
	}
	// 分配内存 
  	// 小对象从当前P 的cache中空闲数据中分配
  	// 大的对象 (size > 32KB) 直接从heap中分配
	return mallocgc(mem, et, true)
}

It's so simple, there is nothing to say. The mallocgc function will find the appropriate memory on the corresponding memory block linked list according to the requested memory size for allocation, which is the set of tcmalloc modified by Go.

Data expansion growslice

func growslice(et *_type, old slice, cap int) slice {
    
    
	//扫描内存
	if raceenabled {
    
    
		callerpc := getcallerpc()
		racereadrangepc(old.array, uintptr(old.len*int(et.size)), callerpc, funcPC(growslice))
	}
	if msanenabled {
    
    
		msanread(old.array, uintptr(old.len*int(et.size)))
	}
	//容量判断
	if cap < old.cap {
    
    
		panic(errorString("growslice: cap out of range"))
	}
	//如果存储的类型空间为0，  比如说 []struct{}, 数据为空，长度不为空
	if et.size == 0 {
    
    
		// append should not create a slice with nil pointer but non-zero len.
		// We assume that append doesn't need to preserve old.array in this case.
		return slice{
    
    unsafe.Pointer(&zerobase), old.len, cap}
	}

	newcap := old.cap
	doublecap := newcap + newcap
	//如果新的容量大于旧的两倍，则直接扩容到新的容量
	if cap > doublecap {
    
    
		newcap = cap
	} else {
    
    
	//当新的容量不大于旧的两倍
		// 如果旧长度小于1024，那扩容到旧的两倍
		if old.len < 1024 {
    
    
			newcap = doublecap
		} else {
    
    
			//否则扩容到旧的1.25倍
			for 0 < newcap && newcap < cap {
    
    
				newcap += newcap / 4
			}
			//检查newCap是否溢出
			if newcap <= 0 {
    
    
				newcap = cap
			}
		}
	}

	var overflow bool
	var lenmem, newlenmem, capmem uintptr
	// 为了加速计算
    // 对于不同的slice元素大小，选择不同的计算方法
    // 获取需要申请的内存大小。
	switch {
    
    
	case et.size == 1:
		lenmem = uintptr(old.len)
		newlenmem = uintptr(cap)
		capmem = roundupsize(uintptr(newcap))
		overflow = uintptr(newcap) > maxAlloc
		newcap = int(capmem)
	case et.size == sys.PtrSize:
		lenmem = uintptr(old.len) * sys.PtrSize
		newlenmem = uintptr(cap) * sys.PtrSize
		capmem = roundupsize(uintptr(newcap) * sys.PtrSize)
		overflow = uintptr(newcap) > maxAlloc/sys.PtrSize
		newcap = int(capmem / sys.PtrSize)
	case isPowerOfTwo(et.size):
		//如果是二的倍数，用位移运算
		var shift uintptr
		if sys.PtrSize == 8 {
    
    
			// Mask shift for better code generation.
			shift = uintptr(sys.Ctz64(uint64(et.size))) & 63
		} else {
    
    
			shift = uintptr(sys.Ctz32(uint32(et.size))) & 31
		}
		lenmem = uintptr(old.len) << shift
		newlenmem = uintptr(cap) << shift
		capmem = roundupsize(uintptr(newcap) << shift)
		overflow = uintptr(newcap) > (maxAlloc >> shift)
		newcap = int(capmem >> shift)
	default:
		//默认用除法
		lenmem = uintptr(old.len) * et.size
		newlenmem = uintptr(cap) * et.size
		capmem, overflow = math.MulUintptr(et.size, uintptr(newcap))
		capmem = roundupsize(capmem)
		newcap = int(capmem / et.size)
	}

	// 判断是否会溢出
	if overflow || capmem > maxAlloc {
    
    
		panic(errorString("growslice: cap out of range"))
	}

	var p unsafe.Pointer
	if et.kind&kindNoPointers != 0 {
    
    
		//为新的切片开辟容量为capmem的地址空间
		p = mallocgc(capmem, nil, false)
		// 仅清除不会被覆盖的部分。
		memclrNoHeapPointers(add(p, newlenmem), capmem-newlenmem)
	} else {
    
    
		// 注意：不能使用rawmem（这样可以避免内存清零），因为GC可以扫描未初始化的内存。
		p = mallocgc(capmem, et, true)
		if writeBarrier.enabled {
    
    
			//因为我们知道目标切片p，所以仅将指针隐藏在old.array中
			//仅包含nil指针，因为在分配过程中已将其清除。
			bulkBarrierPreWriteSrcOnly(uintptr(p), uintptr(old.array), lenmem)
		}
	}
	//数据拷贝
	memmove(p, old.array, lenmem)

	return slice{
    
    p, old.len, newcap}
}

Expansion rules:

1. The new capacity is 2 times larger than the old one, directly expand to the new capacity
2. When the new capacity is not more than 2 times the old
   1. When the old length is less than 1024, expand to twice the old length
   2. Expanded to 1.25 times the old capacity

Slice expansion will inevitably lead to memory copy. If it is a performance-sensitive system, it is a better choice to allocate slices as early as possible.

Data copy slicecopy

func slicecopy(to, fm slice, width uintptr) int {
    
    
	//从slice长度为0或者到slice长度为0
	if fm.len == 0 || to.len == 0 {
    
    
		return 0
	}

	n := fm.len
	if to.len < n {
    
    
		n = to.len
	}
	//长度为0，直接返回
	if width == 0 {
    
    
		return n
	}
	//分析内存
	if raceenabled {
    
    
		callerpc := getcallerpc()
		pc := funcPC(slicecopy)
		racewriterangepc(to.array, uintptr(n*int(width)), callerpc, pc)
		racereadrangepc(fm.array, uintptr(n*int(width)), callerpc, pc)
	}
	if msanenabled {
    
    
		msanwrite(to.array, uintptr(n*int(width)))
		msanread(fm.array, uintptr(n*int(width)))
	}
	
	size := uintptr(n) * width
	if size == 1 {
    
     
		// 直接内存拷贝
		*(*byte)(to.array) = *(*byte)(fm.array) // known to be a byte pointer
	} else {
    
    
		memmove(to.array, fm.array, size)
	}
	return n
}

to sum up

• Try to set the initial capacity value for the slice to avoid expansion. Slice expansion will inevitably lead to memory copy
• A slice is a structure that holds the capacity, actual length, and address of the array
• Multiple slices may point to the same underlying array
• Slices are passed by value, and only pass by value in Go (pass by value)