Tencent recently released "Tencent R & D Big Data Report", the author found that the use of language in GO geese plant has risen to a position of TOP5,
We know that Tencent especially Docker container of this one, is at the forefront of major plant, especially the GO based on their language development DEVOPS blue whales platform, very high level.
By the author in the field experience to get started, GO language in concurrency is still quite good, the author on a "one article read through the GO language hash table" , and readers who chatted GO language, continue following the author's last topic, to report that the latest results.
About GO slice language
Slice ( Slice ) is a reference to an array of consecutive segments, so that the same slice is a reference type, and Python the list type comparison Similarly, the fragments may be entire array may also be some identification of the start and end of the index subset of items. Go languages contain an internal configuration of the address slices, slice characteristics compared to the maximum size (len) and capacity (CAP) and the array capacity is variable.
Interpretation of language codes GO
1. the append function additive element
built-in function append Go language () elements which can be added to the slice, but it should be noted that since the slice itself is variable length, thus using append () function to dynamically add sections element, slice will automatically "expansion", while the length of the new chips will also increase, but one thing to note, append returns a new slice object, rather than the original slice operation. In the following code, we first define a slice, and append manner by continuously increasing its elements, and observed the length and volume changes of a slice.
package main
import (
"fmt"
)
func main() {
var a []int //定义一个切片
fmt.Printf("len: %d cap: %d pointer: %p\n", len(a), cap(a), a)//此时切片长度和容量都是0，运行结果为len: 0 cap: 0 pointer: 0x0
a = append(a, 1) // 追加1个元素
fmt.Printf("len: %d cap: %d pointer: %p\n", len(a), cap(a), a)//注意此时a的地址已经发生变化为新的切片了，新切片长度和容量都为1运行结果为：len: 1 cap: 1 pointer: 0xc000072098
a = append(a, 2, 3, 4) // 追加多个元素
fmt.Printf("len: %d cap: %d pointer: %p\n", len(a), cap(a), a)//注意此时a的地址再次发生变化实际上又生成为新的切片了，新切片长度和容量都为4运行结果为：len: 4 cap: 4 pointer: 0xc000070160
a = append(a, 5) // 再追加一个元素
fmt.Printf("len: %d cap: %d pointer: %p\n", len(a), cap(a), a)//注意切片扩容策略是倍增方式容量由4变成8，而长度是5运行结果为：len: 4 cap: 4 pointer: 0xc000070160
}Slice can be observed when the expansion, its capacity (CAP) speed law is carried out in multiples of 2.
2. Remove the element sections
deleted at the beginning of slice N elements
using x = x [N:] way to remove N elements starting from the i-th element in the slice
specific code is as follows:
package main
import (
"fmt"
)
func main() {
var a = []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10} //使用原始定义法来声明并初始化一个切片
fmt.Println(a) //运行结果为[1 2 3 4 5 6 7 8 9 10]
a = a[1:] // 删除第1个元素
fmt.Println(a) //删头第1个元素后，运行结果为[2 3 4 5 6 7 8 9 10]
a = a[2:] // 删除前2个元素
fmt.Println(a) //删头前2个元素后，运行结果为[4 5 6 7 8 9 10]
}
3, in-depth understanding of the language in GO slice
code for the location of the slices in GOPATH \ src \ runtime \ slice.go, where for several key functions interpreted as follows:
1.slice structure defines
first slice is such a structure, he has a an array of data storage, and a length len cap configuration and capacity
type slice struct {
array unsafe.Pointer
len int
cap int
}
2. Create makeslice function slices
and create a slice function makeslice below, you can see the function will be pre-allocated for memory, if successful then formally allocate memory, he built a slice of makeslice function source code and comments as follows:
func makeslice(et *_type, len, cap int) slice {
mem, overflow := math.MulUintptr(et.size, uintptr(cap))//此函数计算et.size也就是每个元素所占空间的大小，并与容量cap相乘，其中mem既为所需要最大内存，overflow代表是否会造成溢出
if overflow || mem > maxAlloc || len < 0 || len > cap {//判断是否有溢出，长度为负数或者长度比容量大的情况，如存在 直接panic
// NOTE: Produce a 'len out of range' error instead of a
// 'cap out of range' error when someone does make([]T, bignumber).
// 'cap out of range' is true too, but since the cap is only being
// supplied implicitly, saying len is clearer.
// See golang.org/issue/4085.
mem, overflow := math.MulUintptr(et.size, uintptr(len))
if overflow || mem > maxAlloc || len < 0 {
panicmakeslicelen()
}
panicmakeslicecap()
}
return mallocgc(mem, et, true)// 如果错误检查成功，则分配内存，注意slice对象会被GC所自动清除。
}
3. growslice expansion function
can be seen by reading the source growslice function in which, in the expansion rules are shorter than 1024 has been used according to the double expansion manner, at greater than 1024, each of capacity expansion to the original 1.25 times, the new capacity is calculated for the completion of pre-allocation of memory, this is also consistent with the idea makeslice, then take over the old data is copied to a new slice of the slice by memmove (p, old.array, lenmem) manner. growlice function source code and comments are as follows:
func growslice(et *_type, old slice, cap int) slice {
// 单纯地扩容，不写数据
if et.size == 0 {
if cap < old.cap {
panic(errorString("growslice: cap out of range"))
}
// append should not create a slice with nil pointer but non-zero len.
// We assume that append doesn't need to preserve old.array in this case.
return slice{unsafe.Pointer(&zerobase), old.len, cap}
}
// 扩容规则 1.新的容量大于旧的2倍，直接扩容至新的容量
// 2.新的容量不大于旧的2倍，当旧的长度小于1024时，扩容至旧的2倍，否则扩容至旧的1.25倍
newcap := old.cap
doublecap := newcap + newcap
if cap > doublecap {
newcap = cap
} else {
if old.len < 1024 {
newcap = doublecap
} else {
for newcap < cap {
newcap += newcap / 4
}
}
}
// 跟据切片类型和容量计算要分配内存的大小
var overflow bool
var lenmem, newlenmem, capmem uintptr
switch {
case et.size == 1:
lenmem = uintptr(old.len)
newlenmem = uintptr(cap)
capmem = roundupsize(uintptr(newcap))
overflow = uintptr(newcap) > maxAlloc
newcap = int(capmem)
case et.size == sys.PtrSize:
lenmem = uintptr(old.len) * sys.PtrSize
newlenmem = uintptr(cap) * sys.PtrSize
capmem = roundupsize(uintptr(newcap) * sys.PtrSize)
overflow = uintptr(newcap) > maxAlloc/sys.PtrSize
newcap = int(capmem / sys.PtrSize)
case isPowerOfTwo(et.size):
var shift uintptr
if sys.PtrSize == 8 {
// Mask shift for better code generation.
shift = uintptr(sys.Ctz64(uint64(et.size))) & 63
} else {
shift = uintptr(sys.Ctz32(uint32(et.size))) & 31
}
lenmem = uintptr(old.len) << shift
newlenmem = uintptr(cap) << shift
capmem = roundupsize(uintptr(newcap) << shift)
overflow = uintptr(newcap) > (maxAlloc >> shift)
newcap = int(capmem >> shift)
default:
lenmem = uintptr(old.len) * et.size
newlenmem = uintptr(cap) * et.size
capmem, overflow = math.MulUintptr(et.size, uintptr(newcap))
capmem = roundupsize(capmem)
newcap = int(capmem / et.size)
}
// 异常情况，旧的容量比新的容量还大或者新的容量超过限制了
if cap < old.cap || uintptr(newcap) > maxSliceCap(et.size) {
panic(errorString("growslice: cap out of range"))
}
var p unsafe.Pointer
if et.kind&kindNoPointers != 0 {
// 为新的切片开辟容量为capmem的地址空间
p = mallocgc(capmem, nil, false)
// 将旧切片的数据搬到新切片开辟的地址中
memmove(p, old.array, lenmem)
// The append() that calls growslice is going to overwrite from old.len to cap (which will be the new length).
// Only clear the part that will not be overwritten.
// 清理下新切片中剩余地址，不能存放堆栈指针
// memclrNoHeapPointers clears n bytes starting at ptr.
//
// Usually you should use typedmemclr. memclrNoHeapPointers should be
// used only when the caller knows that *ptr contains no heap pointers
// because either:
//
// 1. *ptr is initialized memory and its type is pointer-free.
//
// 2. *ptr is uninitialized memory (e.g., memory that's being reused
// for a new allocation) and hence contains only "junk".
memclrNoHeapPointers(add(p, newlenmem), capmem-newlenmem)
} else {
// Note: can't use rawmem (which avoids zeroing of memory), because then GC can scan uninitialized memory.
p = mallocgc(capmem, et, true)
if !writeBarrier.enabled {
memmove(p, old.array, lenmem)
} else {
for i := uintptr(0); i < lenmem; i += et.size {
typedmemmove(et, add(p, i), add(old.array, i))
}
}
}
return slice{p, old.len, newcap}
}GO language sections relevant conclusions
So by reading the source code above we can also know that there are the following two conclusions:
1.append way to increase data elements, if the trigger slice expansion, it definitely became a freshman slice object and involve memory operations, so append operation must be careful.
2. It is recommended as far as possible to declare a slice through the make function, and as far as possible during the initial set up a reasonable capacity value, avoid frequent expansion sliced bring unnecessary overhead.
Up for the job is done, let's GO language since you can write a tool to delete the duplicate files.
