Record the stepping process of a dynamic library generated by calling go in C language

Record the stepping process of a dynamic library generated by calling go in C language

problem phenomenon

Due to some special reasons, it is necessary to call the so generated by the go language in the C language. It was smooth and everything worked well. Suddenly one day, I don’t know what happened, and a new program can’t run normally. The phenomenon I saw was that the program didn’t respond at all, similar to being stuck directly. Use gdb to view the current information of the process, and see the following call stack:

(gdb) source /root/go/src/runtime/runtime-gdb.py
Loading Go Runtime support.
(gdb) bt
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:520
#1 0x00007fff23c550f6 in runtime.futexsleep (addr=0xfffffffffffffe00, val=0, ns=140733793710979) at /usr/local/go/src/runtime/os_linux.go:44
#2 0x00007fff23c35667 in runtime.notetsleep_internal (n=0x7fff23dc66e8 <runtime.work+232>, ns=-1) at /usr/local/go/src/runtime/lock_futex.go:183
#3 0x00007fff23c35785 in runtime.notetsleepg (n=0x7fff23dc66e8 <runtime.work+232>, ns=-1) at /usr/local/go/src/runtime/lock_futex.go:237
#4 0x00007fff23c41338 in runtime.gcBgMarkStartWorkers () at /usr/local/go/src/runtime/mgc.go:1126
#5 0x00007fff23c3fd92 in runtime.gcStart (trigger=...) at /usr/local/go/src/runtime/mgc.go:637
#6 0x00007fff23c36e3d in runtime.mallocgc (size=565248, typ=0x7fff23cfd160, needzero=true) at /usr/local/go/src/runtime/malloc.go:1174
#7 0x00007fff23c6dfaa in runtime.growslice (et=, old=..., cap=) at /usr/local/go/src/runtime/slice.go:267
#8 0x00007fff23c918ef in regexp/syntax.(*compiler).inst (c=0xc000044bf8, op=) at /usr/local/go/src/regexp/syntax/compile.go:164
#9 regexp/syntax.(*compiler).rune (c=0xc000044bf8, r=[]int32 = {...}, flags=212) at /usr/local/go/src/regexp/syntax/compile.go:273
#10 0x00007fff23c90f1e in regexp/syntax.(*compiler).compile (c=0xc000044bf8, re=0xc000190000) at /usr/local/go/src/regexp/syntax/compile.go:101
#11 0x00007fff23c8f805 in regexp/syntax.Compile (re=0xc000190000) at /usr/local/go/src/regexp/syntax/compile.go:74
#12 0x00007fff23ca4a9a in regexp.compile (expr=,
mode=, longest=false) at /usr/local/go/src/regexp/regexp.go:178
#13 0x00007fff23ca5591 in regexp.Compile (expr="") at /usr/local/go/src/regexp/regexp.go:133
#14 regexp.MustCompile (str="") at /usr/local/go/src/regexp/regexp.go:309
#15 0x00007fff23ca8e0a in main.CheckPostAddress (buf=, bufLen=) at /root/codes/di_rechk/rechk.go:540
#16 0x00007fff23ca9a88 in _cgoexp_67df0785bef2_CheckPostAddress (a=0x7fffffffc740) at _cgo_gotypes.go:162
#17 0x00007fff23c2e21a in runtime.cgocallbackg1 (fn=0x7fff23ca9a60 <_cgoexp_67df0785bef2_CheckPostAddress>, frame=0x7fffffffc740, ctxt=0)
at /usr/local/go/src/runtime/cgocall.go:306
#18 0x00007fff23c2dee9 in runtime.cgocallbackg (fn=0x7fff23ca9a60 <_cgoexp_67df0785bef2_CheckPostAddress>, frame=0x7fffffffc740, ctxt=0)
at /usr/local/go/src/runtime/cgocall.go:232
#19 0x00007fff23c83791 in runtime.cgocallbackg (fn=0x7fff23ca9a60 <_cgoexp_67df0785bef2_CheckPostAddress>, frame=0x7fffffffc740, ctxt=0)
at :1
#20 0x00007fff23c813f3 in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:915
#21 0x00007fff23c81601 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1581
#22 0x0000000000000000 in ?? ()

(gdb) info goroutines

    17 syscall runtime.notetsleepg
    2 waiting runtime.gopark
    3 waiting runtime.gopark
    4 waiting runtime.gopark
    19 runnable runtime.gcBgMarkWorker

(gdb) goroutine 17 bt
#0 runtime.notetsleepg (n=0x7fff23dc66e8 <runtime.work+232>, ns=-1) at /usr/local/go/src/runtime/lock_futex.go:237
#1 0x00007fff23c41338 in runtime.gcBgMarkStartWorkers () at /usr/local/go/src/runtime/mgc.go:1126
#2 0x00007fff23c3fd92 in runtime.gcStart (trigger=...) at /usr/local/go/src/runtime/mgc.go:637
#3 0x00007fff23c36e3d in runtime.mallocgc (size=565248, typ=0x7fff23cfd160, needzero=true) at /usr/local/go/src/runtime/malloc.go:1174
#4 0x00007fff23c6dfaa in runtime.growslice (et=, old=..., cap=) at /usr/local/go/src/runtime/slice.go:267
#5 0x00007fff23c918ef in regexp/syntax.(*compiler).inst (c=0xc000044bf8, op=) at /usr/local/go/src/regexp/syntax/compile.go:164
#6 regexp/syntax.(*compiler).rune (c=0xc000044bf8, r=[]int32 = {...}, flags=212) at /usr/local/go/src/regexp/syntax/compile.go:273
#7 0x00007fff23c90f1e in regexp/syntax.(*compiler).compile (c=0xc000044bf8, re=0xc00022a000) at /usr/local/go/src/regexp/syntax/compile.go:101
#8 0x00007fff23c8f805 in regexp/syntax.Compile (re=0xc00022a000) at /usr/local/go/src/regexp/syntax/compile.go:74
#9 0x00007fff23ca4a9a in regexp.compile (expr=,
mode=, longest=false) at /usr/local/go/src/regexp/regexp.go:178
#10 0x00007fff23ca5591 in regexp.Compile (expr=""") at /usr/local/go/src/regexp/regexp.go:133
#11 regexp.MustCompile (str=""") at /usr/local/go/src/regexp/regexp.go:309
#12 0x00007fff23ca8e0a in main.CheckPostAddress (buf=, bufLen=) at /root/codes/di_rechk/rechk.go:540
#13 0x00007fff23ca9a88 in _cgoexp_67df0785bef2_CheckPostAddress (a=0x7fffffffc740) at _cgo_gotypes.go:162
#14 0x00007fff23c2e21a in runtime.cgocallbackg1 (fn=0x7fff23ca9a60 <_cgoexp_67df0785bef2_CheckPostAddress>, frame=0x7fffffffc740, ctxt=0)
at /usr/local/go/src/runtime/cgocall.go:306
#15 0x00007fff23c2dee9 in runtime.cgocallbackg (fn=0x7fff23ca9a60 <_cgoexp_67df0785bef2_CheckPostAddress>, frame=0x7fffffffc740, ctxt=0)
at /usr/local/go/src/runtime/cgocall.go:232
#16 0x00007fff23c83791 in runtime.cgocallbackg (fn=0x7fff23ca9a60 <_cgoexp_67df0785bef2_CheckPostAddress>, frame=0x7fffffffc740, ctxt=0)
at :1
#17 0x00007fff23c813f3 in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:915
#18 0x00007fff23c81601 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1581
#19 0x0000000000000000 in ?? ()

(gdb) goroutine 2 bt
#0 runtime.gopark (unlockf=, lock=, reason=, traceEv=, traceskip=)
at /usr/local/go/src/runtime/proc.go:367
#1 0x00007fff23c5b86d in runtime.goparkunlock (reason=, traceEv=, traceskip=, lock=)
at /usr/local/go/src/runtime/proc.go:372
#2 runtime.forcegchelper () at /usr/local/go/src/runtime/proc.go:306
#3 0x00007fff23c81601 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1581
#4 0x0000000000000000 in ?? ()
(gdb) goroutine 3 bt
#0 runtime.gopark (unlockf=, lock=, reason=, traceEv=, traceskip=)
at /usr/local/go/src/runtime/proc.go:367
#1 0x00007fff23c48ee8 in runtime.goparkunlock (reason=, traceEv=, traceskip=, lock=)
at /usr/local/go/src/runtime/proc.go:372
#2 runtime.bgsweep () at /usr/local/go/src/runtime/mgcsweep.go:163
#3 0x00007fff23c81601 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1581
#4 0x0000000000000000 in ?? ()
(gdb) goroutine 4 bt
#0 runtime.gopark (unlockf=, lock=, reason=, traceEv=, traceskip=)
at /usr/local/go/src/runtime/proc.go:367
#1 0x00007fff23c46fed in runtime.goparkunlock (reason=, traceEv=, traceskip=, lock=)
at /usr/local/go/src/runtime/proc.go:372
#2 runtime.bgscavenge () at /usr/local/go/src/runtime/mgcscavenge.go:265
#3 0x00007fff23c81601 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1581
#4 0x0000000000000000 in ?? ()
(gdb) goroutine 19 bt
#0 runtime.gcBgMarkWorker () at /usr/local/go/src/runtime/mgc.go:1166
#1 0x00007fff23c81601 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1581
#2 0x0000000000000000 in ?? ()

(gdb) info threads
Id Target Id Frame
1 Thread 0x7ffff7966b80 (LWP 2024707) "nginx" runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:520
2 Thread 0x7ffef1ffb700 (LWP 2024717) "nginx" 0x00007ffff7a483bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0,
    req=req@entry=0x7ffef1ffae80, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
3 Thread 0x7ffef27fc700 (LWP 2024718) "ZMQbg/Reaper" 0x00007ffff7a8a5ce in epoll_wait (epfd=42, events=events@entry=0x7ffef27fb200,
    maxevents=maxevents@entry=256, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
4 Thread 0x7ffef2ffd700 (LWP 2024719) "ZMQbg/IO/0" 0x00007ffff7a8a5ce in epoll_wait (epfd=44, events=events@entry=0x7ffef2ffc200,
    maxevents=maxevents@entry=256, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30

problem solving process

Since I'm not very familiar with go, I don't know what's wrong. The code of go itself is also relatively simple, without too complicated operations. Moreover, the so generated by the previous program calling these go codes can also be run as a whole. Looking up information from the Internet is also at a loss, and I don't know how to look it up. After searching for a while, finally saw a similar question in stackoverflow . According to this reason, it was modified, and it was modified.

Summary of the cause of the problem

As summarized in stackoverflow . The reason is that when the program starts, if the so generated by go is called, then go will call some functions to establish the go runtime environment and create some threads. In my new program, the specific place where the go function is called is called in the child process, and when the parent process forks the child process, the runtime threads created by go will not be copied to the child process, even if The so generated by re-dlopen go in the child process will not recreate those runtime threads, so those codes in go that involve some runtime program intervention (such as gc, etc.) cannot run normally. And my go code just triggered GC, which made go unable to run and was in an endless loop.
The solution is: run dlopen after the fork to open the so generated by go, so it will work.
After modification, the program running information is as follows:

(gdb) info threads
  Id   Target Id                                          Frame
* 1    Thread 0x7ffff7966b80 (LWP 2366880) "nginx"        0x00007ffff7a8a5ce in epoll_wait (epfd=31, events=0x555555d0af90, maxevents=512,
    timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  2    Thread 0x7fff481fd700 (LWP 2366893) "nginx"        0x00007ffff7a483bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0,
    req=req@entry=0x7fff481fce80, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
  3    Thread 0x7fff489fe700 (LWP 2366894) "ZMQbg/Reaper" 0x00007ffff7a8a5ce in epoll_wait (epfd=42, events=events@entry=0x7fff489fd200,
    maxevents=maxevents@entry=256, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  4    Thread 0x7fff491ff700 (LWP 2366895) "ZMQbg/IO/0"   0x00007ffff7a8a5ce in epoll_wait (epfd=44, events=events@entry=0x7fff491fe200,
    maxevents=maxevents@entry=256, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  5    Thread 0x7fff245cb700 (LWP 2368456) "nginx"        runtime.futex () at /root/go/src/runtime/sys_linux_amd64.s:560
  6    Thread 0x7fff20c34700 (LWP 2368457) "nginx"        runtime.futex () at /root/go/src/runtime/sys_linux_amd64.s:560
  7    Thread 0x7fff091a1700 (LWP 2368458) "nginx"        runtime.futex () at /root/go/src/runtime/sys_linux_amd64.s:560
  8    Thread 0x7fff089a0700 (LWP 2368459) "nginx"        runtime.futex () at /root/go/src/runtime/sys_linux_amd64.s:560
  9    Thread 0x7ffed7fff700 (LWP 2368460) "nginx"        runtime.futex () at /root/go/src/runtime/sys_linux_amd64.s:560
  10   Thread 0x7ffed77fe700 (LWP 2368461) "nginx"        runtime.futex () at /root/go/src/runtime/sys_linux_amd64.s:560
  11   Thread 0x7ffed6ffd700 (LWP 2368462) "nginx"        runtime.futex () at /root/go/src/runtime/sys_linux_amd64.s:560
  12   Thread 0x7ffed67fc700 (LWP 2368463) "nginx"        runtime.futex () at /root/go/src/runtime/sys_linux_amd64.s:560

Compared with the info threads information when there is a problem, it is found that there are several more runtime.futex threads, and these threads are the threads needed to run the go program. The reason why the previous program did not have a problem is because the previous program did not call fork, so this problem will not occur.

additional questions

When there is a problem, why does the go program fail to run when the GC is triggered, and is always in an endless loop? This needs to understand the garbage collection mechanism of go to know the reason. I am not familiar with go, so I just understand a little bit, and I can't explain it in detail.