The following content is reproduced from https://mp.weixin.qq.com/s/8eJm5hjwKXya85VnT4y8Cw
Awa love to write original programs Zhang source Travels 2019-05-09
This article is the 14th chapter of the "Go Language Scheduler Source Code Scenario Analysis" series, and it is also the 4th subsection of Chapter 2.
In the previous section, we discussed the creation and initialization process of goroutine in detail by analyzing the creation of main goroutine. In this section, we will then analyze how the scheduler schedules the main goroutine to run on the CPU. The issues that need to be focused on in this section are:
-
How to save the scheduling information of g0?
-
What is the important role of the schedule function?
-
How does the gogo function switch from g0 to main goroutine?
Then continue to analyze the code in the previous section, return to rt0_go from newproc, and continue to execute mstart.
runtime/proc.go : 1153
func mstart () {
_g_ := getg() //_g_ = g0 //对于启动过程来说，g0的stack.lo早已完成初始化，所以onStack = false osStack := _g_.stack.lo == 0 if osStack {
// Initialize stack bounds from system stack. // Cgo may have left stack size in stack.hi. // minit may update the stack bounds. size := _g_.stack.hi if size == 0 {
size = 8192 * sys.StackGuardMultiplier } _g_.stack.hi = uintptr(noescape(unsafe.Pointer(&size))) _g_.stack.lo = _g_.stack.hi - size + 1024 } // Initialize stack guards so that we can start calling // both Go and C functions with stack growth prologues. _g_.stackguard0 = _g_.stack.lo + _StackGuard _g_.stackguard1 = _g_.stackguard0 mstart1() // Exit this thread. if GOOS == "windows" || GOOS == "solaris" || GOOS == "plan9" || GOOS == "darwin" || GOOS == "aix" {
// Window, Solaris, Darwin, AIX and Plan 9 always system-allocate // the stack, but put it in _g_.stack before mstart, // so the logic above hasn't set osStack yet. osStack = true } mexit(osStack)}The mstart function itself has nothing to say, it continues to call the mstart1 function.
runtime/proc.go : 1184
func mstart1 () {
_g_ := getg() //启动过程时 _g_ = m0的g0 if _g_ != _g_.m.g0 {
throw("bad runtime·mstart") } // Record the caller for use as the top of stack in mcall and // for terminating the thread. // We're never coming back to mstart1 after we call schedule, // so other calls can reuse the current frame. //getcallerpc()获取mstart1执行完的返回地址 //getcallersp()获取调用mstart1时的栈顶地址 save(getcallerpc(), getcallersp()) asminit() //在AMD64 Linux平台中，这个函数什么也没做，是个空函数 minit() //与信号相关的初始化，目前不需要关心 // Install signal handlers; after minit so that minit can // prepare the thread to be able to handle the signals. if _g_.m == &m0 { //启动时_g_.m是m0，所以会执行下面的mstartm0函数 mstartm0() //也是信号相关的初始化，现在我们不关注 } if fn := _g_.m.mstartfn; fn != nil { //初始化过程中fn == nil fn() } if _g_.m != &m0 {// m0已经绑定了allp[0]，不是m0的话还没有p，所以需要获取一个p acquirep(_g_.m.nextp.ptr()) _g_.m.nextp = 0 } //schedule函数永远不会返回 schedule()}mstart1 first calls the save function to save the scheduling information of g0. The save line of code is very important and is one of the key points for us to understand the scheduling loop . The first thing to note here is that the getcallerpc() in the code returns the return address that was pushed onto the stack by the call instruction when mstart calls mstart1, and the getcallersp() function returns the top address of the mstart function before calling mstart1. Next, you need to look at it. What important work does the save function do?
runtime/proc.go : 2733
// save updates getg().sched to refer to pc and sp so that a following
// gogo will restore pc and sp.//// save must not have write barriers because invoking a write barrier// can clobber getg().sched.////go:nosplit//go:nowritebarrierrecfunc save(pc, sp uintptr) {
_g_ := getg() _g_.sched.pc = pc //再次运行时的指令地址 _g_.sched.sp = sp //再次运行时到栈顶 _g_.sched.lr = 0 _g_.sched.ret = 0 _g_.sched.g = guintptr(unsafe.Pointer(_g_)) // We need to ensure ctxt is zero, but can't have a write // barrier here. However, it should always already be zero. // Assert that. if _g_.sched.ctxt != nil {
badctxt() }}It can be seen that the save function saves all the information related to scheduling, including the most important address of the next instruction of the currently running g and the top address of the stack, whether it is for g0 or other goroutines, this information is used in the scheduling process It is essential. We will see how the scheduler uses this information to complete the scheduling in the later scheduling analysis. The state of g0 after the code executes the save function is shown in the following figure:
As can be seen from the above figure, g0.sched.sp points to the return address after the mstart1 function is executed, and the address is stored in the stack frame of the mstart function; g0.sched.pc points to the mstart function to call the mstart1 function The following if statement.
Why does g0 have executed the mstart1 function and will continue to call other functions, but the pc and sp in the scheduling information of g0 have to be set in the mstart function? Is it necessary to continue execution from the if statement in the mstart function when switching to g0 next time? However, you can see from the mstart function that the thread will exit after the if statement! This may seem strange, but as the analysis progresses, we will see why this is done here.
Continue to analyze the code. After the save function is executed, return to mstart1 to continue some other initializations related to m. After completing these initializations, call the core function schedule() of the scheduling system to complete the scheduling of goroutine. The reason why it is said to be the core is that Every time a goroutine is scheduled, it starts with the schedule function.
runtime/proc.go : 2469
// One round of scheduler: find a runnable goroutine and execute it.
// Never returns.func schedule() {
_g_ := getg() //_g_ = 每个工作线程m对应的g0，初始化时是m0的g0 //...... var gp *g //...... if gp == nil {
// Check the global runnable queue once in a while to ensure fairness. // Otherwise two goroutines can completely occupy the local runqueue // by constantly respawning each other. //为了保证调度的公平性，每进行61次调度就需要优先从全局运行队列中获取goroutine， //因为如果只调度本地队列中的g，那么全局运行队列中的goroutine将得不到运行 if _g_.m.p.ptr().schedtick%61 == 0 && sched.runqsize > 0 {
lock(&sched.lock) //所有工作线程都能访问全局运行队列，所以需要加锁 gp = globrunqget(_g_.m.p.ptr(), 1) //从全局运行队列中获取1个goroutine unlock(&sched.lock) } } if gp == nil {
//从与m关联的p的本地运行队列中获取goroutine gp, inheritTime = runqget(_g_.m.p.ptr()) if gp != nil && _g_.m.spinning {
throw("schedule: spinning with local work") } } if gp == nil {
//如果从本地运行队列和全局运行队列都没有找到需要运行的goroutine， //则调用findrunnable函数从其它工作线程的运行队列中偷取，如果偷取不到，则当前工作线程进入睡眠， //直到获取到需要运行的goroutine之后findrunnable函数才会返回。 gp, inheritTime = findrunnable() // blocks until work is available } //跟启动无关的代码..... //当前运行的是runtime的代码，函数调用栈使用的是g0的栈空间 //调用execte切换到gp的代码和栈空间去运行 execute(gp, inheritTime) }The schedule function selects the next goroutine to be run from the global run queue and the local run queue of the current worker thread by calling the globrunqget() and runqget() functions. If there is no goroutine that needs to be run in these two queues, use findrunnalbe() The function steals the goroutine from the run queue of other p. Once it finds the next goroutine that needs to be run, it calls the excute function to switch from g0 to that goroutine to run. For our scenario, the previous startup process has created the first goroutine and put it into the local run queue of the current worker thread, so here will be the only one goroutine taken out through runqget, as for how to take it out. Yes, we will come back to analyze in detail the implementation process of the three functions globrunqget(), runqget() and findrunnable() when discussing scheduling strategies in Chapter 3. Now let’s analyze how the execute function is removed from the run queue. The found goroutine is scheduled to run on the CPU.
runtime/proc.go : 2136
// Schedules gp to run on the current M.
// If inheritTime is true, gp inherits the remaining time in the// current time slice. Otherwise, it starts a new time slice.// Never returns.//// Write barriers are allowed because this is called immediately after// acquiring a P in several places.////go:yeswritebarrierrecfunc execute(gp *g, inheritTime bool) {
_g_ := getg() //g0 //设置待运行g的状态为_Grunning casgstatus(gp, _Grunnable, _Grunning) //...... //把g和m关联起来 _g_.m.curg = gp gp.m = _g_.m //...... //gogo完成从g0到gp真正的切换 gogo(&gp.sched)}The first parameter gp of the execute function is the goroutine that needs to be scheduled to run. Here, first change the status of gp from _Grunnable to _Grunning, and then associate gp with m, so that you can find the current worker thread is executing through m Which goroutine and vice versa.
After completing the preparatory work before the gp operation, execute calls the gogo function to complete the switch from g0 to gp: the transfer of CPU execution rights and the switch of the stack .
The gogo function is also written in assembly language. The reason why you need to use assembly here is because the scheduling of goroutine involves switching between different execution streams. We have seen before when we discussed the operating system switching threads. The switching of the execution stream is essentially The above is the switching of CPU registers and function call stacks. However, high-level languages such as go or c cannot precisely control the modification of CPU registers. Therefore, high-level languages are powerless here and can only rely on assembly instructions to achieve their goals.
runtime/asm_amd64.s : 251
# func gogo (buf * gobuf)
# restore state from Gobuf; longjmpTEXT runtime·gogo(SB), NOSPLIT, $16-8 #buf = &gp.sched MOVQ buf+0(FP), BX # BX = buf #gobuf->g --> dx register MOVQ gobuf_g(BX), DX # DX = gp.sched.g #下面这行代码没有实质作用，检查gp.sched.g是否是nil，如果是nil进程会crash死掉 MOVQ 0(DX), CX # make sure g != nil get_tls(CX) #把要运行的g的指针放入线程本地存储，这样后面的代码就可以通过线程本地存储 #获取到当前正在执行的goroutine的g结构体对象，从而找到与之关联的m和p MOVQ DX, g(CX) #把CPU的SP寄存器设置为sched.sp，完成了栈的切换 MOVQ gobuf_sp(BX), SP # restore SP #下面三条同样是恢复调度上下文到CPU相关寄存器 MOVQ gobuf_ret(BX), AX MOVQ gobuf_ctxt(BX), DX MOVQ gobuf_bp(BX), BP #清空sched的值，因为我们已把相关值放入CPU对应的寄存器了，不再需要，这样做可以少gc的工作量 MOVQ $0, gobuf_sp(BX) # clear to help garbage collector MOVQ $0, gobuf_ret(BX) MOVQ $0, gobuf_ctxt(BX) MOVQ $0, gobuf_bp(BX) #把sched.pc值放入BX寄存器 MOVQ gobuf_pc(BX), BX #JMP把BX寄存器的包含的地址值放入CPU的IP寄存器，于是，CPU跳转到该地址继续执行指令， JMP BXThis assembly code of the gogo function is short and powerful. Although the author has made detailed comments in the code, in order to fully understand its working principle, we need to analyze these instructions one by one:
The execute function passes the address of the sched member of gp as an actual parameter (type parameter buf) when calling gogo. The parameter is located at the position pointed to by the FP register, so the first instruction
MOVQ buf+0(FP), BX # &gp.sched --> BX
Put the value of buf, which is the address of gp.sched, in the BX register, so that it is convenient for subsequent instructions to rely on the BX register to access the members of gp.sched. The sched member saves scheduling-related information. As we have seen in the previous section, this information has been set when the main goroutine is created.
Article 2 Directive
MOVQ gobuf_g(BX), DX # gp.sched.g --> DX
Read gp.sched.g into the DX register. Note that the source operand of this instruction is indirect addressing. If the reader is not familiar with indirect addressing, you can refer to the assembly language part of the preparatory knowledge.
Article 3 Directive
MOVQ 0(DX), CX # make sure g != nil
The function of gp.sched.g is to check whether gp.sched.g is nil. If it is a nil pointer, this instruction will cause the program to die. Readers may wonder why it should die because of this gp.sched. g is set by the go runtime code. It is reasonable to say that it is impossible to be nil. If it is nil, there must be a problem with the programming logic, so this bug needs to be exposed instead of hidden.
Article 4 and Article 5 Directives
get_tls(CX)
#把DX值也就是需要运行的goroutine的指针写入线程本地存储之中#运行这条指令之前，线程本地存储存放的是g0的地址MOVQ DX, g(CX)Write the value of the DX register, that is, gp.sched.g (this is a pointer to g) into the thread local storage, so that the following code can obtain the g structure of the currently executing goroutine through the thread local storage Object to find the m and p associated with it.
Article 6 Directive
MOVQ gobuf_sp(BX), SP # restore SP
Set the stack top register SP of the CPU to gp.sched.sp. This instruction completes the stack switch, switching from the g0 stack to the gp stack.
7th to 13th instructions
#The following three are also to restore the scheduling context to the CPU related registers
MOVQ gobuf_ret(BX), AX #系统调用的返回值放入AX寄存器 MOVQ gobuf_ctxt(BX), DX MOVQ gobuf_bp(BX), BP //清空gp.sched中不再需要的值，因为我们已把相关值放入CPU对应的寄存器了，不再需要，这样做可以少gc的工作量 MOVQ $0, gobuf_sp(BX) // clear to help garbage collector MOVQ $0, gobuf_ret(BX) MOVQ $0, gobuf_ctxt(BX) MOVQ $0, gobuf_bp(BX)One is to set the CPU-related registers according to other fields of gp.sched, you can see that the stack base address register BP of the CPU is restored here, and the other is to set the members that are no longer needed in gp.sched to 0, which can reduce the workload of gc. .
Article 14 Directive
MOVQ gobuf_pc(BX), BX
Read the value of gp.sched.pc into the BX register. This pc value is the address of the first instruction that the goroutine needs to execute immediately. For our scenario, it is now the first instruction of the runtime.main function. , Now the address of this instruction is placed in the BX register. Last instruction
JMP BX
The JMP BX instruction here puts the instruction address in the BX register into the rip register of the CPU, so the CPU will jump to this address and continue to execute the code belonging to the goroutine of gp, thus completing the switch of the goroutine.
Summarizing these 15 instructions, in fact, only two things are done:
-
Restore the members of gp.sched to the register completion state of the CPU and switch the stack;
-
Jump to the instruction address (runtime.main) pointed to by gp.sched.pc for execution.
Now it has switched from g0 to the goroutine of gp. For our scenario, gp is scheduled to run for the first time. Its entry function is runtime.main, so the CPU then starts to execute the runtime.main function:
runtime/proc.go : 109
// The main goroutine.
func main() {
g := getg() // g = main goroutine，不再是g0了 ...... // Max stack size is 1 GB on 64-bit, 250 MB on 32-bit. // Using decimal instead of binary GB and MB because // they look nicer in the stack overflow failure message. if sys.PtrSize == 8 { //64位系统上每个goroutine的栈最大可达1G maxstacksize = 1000000000 } else {
maxstacksize = 250000000 } // Allow newproc to start new Ms. mainStarted = true if GOARCH != "wasm" { // no threads on wasm yet, so no sysmon //现在执行的是main goroutine，所以使用的是main goroutine的栈，需要切换到g0栈去执行newm() systemstack(func() {
//创建监控线程，该线程独立于调度器，不需要跟p关联即可运行 newm(sysmon, nil) }) } ...... //调用runtime包的初始化函数，由编译器实现 runtime_init() // must be before defer // Record when the world started. runtimeInitTime = nanotime() gcenable() //开启垃圾回收器 ...... //main 包的初始化函数，也是由编译器实现，会递归的调用我们import进来的包的初始化函数 fn := main_init // make an indirect call, as the linker doesn't know the address of the main package when laying down the runtime fn() ...... //调用main.main函数 fn = main_main // make an indirect call, as the linker doesn't know the address of the main package when laying down the runtime fn() ...... //进入系统调用，退出进程，可以看出main goroutine并未返回，而是直接进入系统调用退出进程了 exit(0) //保护性代码，如果exit意外返回，下面的代码也会让该进程crash死掉 for {
var x *int32 *x = 0 }}The main workflow of runtime.main function is as follows:
-
Start a sysmon system monitoring thread, which is responsible for the monitoring of the gc, preemption scheduling, netpoll and other functions of the entire program. In the preemption scheduling chapter, we will continue to analyze how sysmon assists in completing the preemption scheduling of goroutine;
-
Perform the initialization of the runtime package;
-
Perform the initialization of the main package and all packages of the main package import;
-
Execute main.main function;
-
After returning from the main.main function, call the exit system call to exit the process;
It can be seen from the above process that after runtime.main executes the main function of the main package, it directly calls the exit system call to end the process, and it does not return to the function that called it (remember where the runtime.main was executed from? ?), in fact, runtime.main is the entry function of main goroutine. It is not called directly. Instead, it jumps directly with assembly code in the gogo function of the schedule()->execute()->gogo() call chain. Come here, so from this perspective, goroutine really shouldn't return, there is no place to return! But from the previous analysis, we learned that when creating a goroutine, a return address was already placed on its stack, which caused the goexit function to call the entry function of the goroutine. Why is this return address not used here? In fact, it is prepared for the non-main goroutine. After the execution of the non-main goroutine is completed, it will return to goexit to continue execution, and after the main goroutine is executed, the entire process ends. This is a difference between the main goroutine and other goroutines.
Summarize the process of switching from g0 to main goroutine:
-
Save the scheduling information of g0, mainly save the CPU stack top register SP to the g0.sched.sp member;
-
Call the schedule function to find the goroutine that needs to be run. In this scenario, we find the main goroutine;
-
Calling the gogo function first switches from the g0 stack to the main goroutine stack, and then takes the value of sched.pc from the g structure object of the main goroutine and uses the JMP instruction to jump to the address for execution;
-
After the main goroutine is executed, it directly calls the exit system call to exit the process.
In the next section, we will use examples to analyze the exit of non-main goroutines.
Finally, if you think this article is helpful to you, please help me click on the “Looking” at the bottom right corner of the article or forward it to the circle of friends, thank you very much!
