Online problems caused by Go third-party libraries, how to debug and locate problems in the online environment, and summary of various problems in golang development.
01 Preface
When using Go language for web development, we often choose some excellent libraries to simplify the processing of HTTP requests. Among them, go-resty is a widely used HTTP client. It is precisely because of the use of the go-resty library that there is a topic that I want to share today: the problem and solution of go-resty when dealing with HTTP 307/308 redirection status codes.
02 HTTP status code
I believe everyone has heard of the HTTP 302 status code, but have you ever heard of the HTTP 307/308 status code? Let's look at the definitions of HTTP 302, 307 and 308 status codes and the differences between them:
-
HTTP 302:
Name: Found or Temporary Redirect.
Description: This is a temporary redirect, meaning that the requested resource has been temporarily moved to a new URL. This status code indicates that the original request method should be preserved for the new URL. But in practice, many clients, especially browsers, may convert POST to GET.
Application: In practical applications, 302 redirects are often used in scenarios such as URL short links and login status verification. -
HTTP 307:
Name: Temporary Redirect.
Description: Similar to HTTP 302, HTTP 307 also indicates that the resource has been temporarily moved to another URL. But the main difference from 302 is that HTTP 307 clearly stipulates that the client must not change the request method for redirection. For example, if the original request was a POST, then the redirected request must also be a POST.
Application: Since 307 is more specific, you should use 307 instead of 302 if you need to ensure that the request method is not changed when redirected. -
HTTP 308:
Name: Permanent Redirect.
Description: Similar to HTTP 301 (Permanent Redirect), HTTP 308 indicates that the requested resource has been permanently moved to a new URL. But the difference with 301 is that 308 clearly stipulates that the client must keep the method of the original request unchanged. For example, a POST request should still be a POST after a 308 redirect.
Application: 308 should be used when the resource has moved permanently and you want to ensure that the request method has not been changed.
Main difference:
Permanent vs. Temporary: 302 and 307 are temporary redirects, while 308 is permanent.
Retention of request method: 302 does not guarantee that the original request method will be preserved after redirection; 307 and 308 guarantee that the request method will not change after redirection.
03 Project background
Due to the multi-area deployment of the background service, more than two environmental data need to be connected. When the request reaches area A, it is found that the data is in area B, and HTTP 307 will be used for temporary redirection to jump to area B.
The normal request is as above. Let’s take a look at the problem caused by go-resty:
When multiple requests access area A at the same time, there is request A that needs 307 redirection, and request B also accesses area A at the same time. At this time, Due to the redirected request A caused by go-resty reusing the cache, the data of request B is obtained.
04 issue
The latest version of go-resty is tag: v2.7.0, which was released in November 2021, and in May 2022, someone raised this issue on github, fixed it, and finally merged it into master The main branch is up, but no new version has been released so far, which leads to this redirection problem online.
Fix redirect request body not matching source request body caused by bug reuse buffer
https://github.com/go-resty/resty/pull/568
Below, let's take a look at the code where mr has repaired.
You can see that it was originally directly through bytes.NewBuffer(r.bodyBuf.Bytes()) Returned, the way to fix it is to copy a new cache area through io.Copy to solve the problem of reuse.
The final submitted code also writes unit tests to verify that the results are accurate:
func TestPostRedirectWithBody(t *testing.T) {
ts := createPostServer(t)
defer ts.Close()
targetURL, _ := url.Parse(ts.URL)
t.Log("ts.URL:", ts.URL)
t.Log("targetURL.Host:", targetURL.Host)
c := dc()
wg := sync.WaitGroup{
}
for i := 0; i < 100; i++ {
wg.Add(1)
go func() {
defer wg.Done()
resp, err := c.R().
SetBody([]byte(strconv.Itoa(newRnd().Int()))).
Post(targetURL.String() + "/redirect-with-body")
assertError(t, err)
assertNotNil(t, resp)
}()
}
wg.Wait()
}
05 Summary
When 307 redirection requests appear in the online environment and the amount of concurrency increases sharply, the probability of collisions at the same time increases, and data strings appear, which is a very serious problem.
Summarize the common mistakes and problems in Golang programming:
unrecognized import path "golang.org/x/…
golang has established a mirror library on github, download the mirror library on github and put it under GOPATH
mkdir -p $GOPATH/src/golang.org/x
cd $GOPATH/src/golang.org/x
git clone https://github.com/golang/sync.git
git clone https://github.com/golang/crypto.git
git clone https://github.com/golang/sys.git
The go package management proxy URL cannot be accessed: proxy.golang.org
Solution:
go env -w GOPROXY=https://goproxy.cn
verifying module: invalid GOSUMDB: malformed verifier id
解决：
turn off package verification
go env -w GOSUMDB=off
设置go env -w GOPROXY=***warning: go env -w GOPROXY=… does not override conflicting OS environment variable
solution:
unset GOPROXY
Then reset it
Type Mismatch
In Golang, type matching is crucial. A classic mistake with Golang type matching is comparing two variables of different types. For example, compare a variable of type string with a variable of type integer. To avoid this error, you should always make sure that you are comparing variables of the same type. Or, when performing type conversion, it is necessary to ensure that the variable type meets the requirements.
Null pointer errors
Pointers to null values are very common errors in Golang. This usually arises in terms of resource management in the code, like opening and closing files, connecting to databases, etc. When dealing with pointer variables, you should always ensure that it points to a valid object. In addition, the use of pointer variables should be careful to avoid the situation that different pointers point to the same object during programming.
Memory Allocation Errors
Memory allocation is a necessary task for computer programs, but in Golang, this task can be problematic. The most common error is failing to free memory. When using the new() or make() functions in Golang, care should be taken to release memory to ensure that the program does not crash due to memory leaks.
Large-scale concurrent processing problem
In Golang, large-scale concurrent processing is a notable feature, but it also brings some problems. Error handling in concurrent programs is trickier than in serial programs because the developer needs to identify errors in multiple places of concurrent execution. When developing Golang concurrent programs, you should design a clear error management scheme and make full use of the resource management functions provided by Golang for concurrency control.
Deadlock
Deadlock is a common Golang error. Most deadlocks occur when a program fails to release a lock or acquires too many locks. Worse, the user may need to restart the program to resolve the deadlock. To avoid deadlocks, concurrency control techniques can be used to ensure that subroutines cannot access the same resource at the same time.
File handling issues
In Golang, file handling is an ongoing task. Common file handling errors include the file cannot be opened, read and write errors. To avoid these errors, you should choose a perfectly suitable file manipulation function and check that the file path is correct. Also, you should check the format of the file when parsing it to ensure that the data in the file has been converted correctly.
Out-of-bounds arrays and slices
Out-of-bounds access to arrays and slices is another possible bug in Golang. Since arrays and slices are indexed from 0, an out-of-bounds error may occur if the developer mistakenly writes a negative number, a positive number out of bounds, or an index other than an integer when indexing. To avoid this problem, developers can use loops to iterate over arrays and slices, and always ensure that the index values of arrays and slices are valid.
Summarize
The above are common mistakes in Golang. These mistakes can be overwhelming for newcomers in the learning process of Golang, but they can be easily avoided with enough patience and learning attitude and following best practices.
Recently, an online service suddenly stuck. It seems that the process has started, but the request interface responds that the connection cannot be made. There is no abnormal information in the error log. There has been no enrichment for this kind of online service stuck problem. experience to check. Looking at some people on the Internet, they use tools such as gdb and systemtap to debug online problems, and I am very envious.
I first used strace -p pid to look at the problem, and found that sometimes it was in wait futex, and sometimes there was a lot of normal information. According to my previous experience, when a wait futex appears, it usually waits for the channel to appear. If the wait futex is always on, it means that the process is stuck on a certain channel. But now the stuck process has not been in the wait futex all the time, indicating that it may not be stuck on a certain channel, at least there are other goroutines in normal activities.
The process is not stuck in the wait futex, but it cannot listen to the port to process the request, which means that it is either in the startup phase, and the deadlock is stuck before the port is listened to, or there is a problem with the part of the code that listens to the port.
In the past, I only used strace -p to judge whether there is a deadlock. I tried to use gdb to debug python services earlier, but I don’t know what tools to use for go services. In theory, strace and gdb can also be used. Some problems were found, but neither of these two tools can recognize goroutines, so it is difficult to debug.
Later, my colleagues introduced dlv, which is the tool delve, a debugging tool specially built for the go language, which can identify goroutines and so on. After searching the official documents and the respective introduction articles on the Internet, the writing is not very practical. Many articles are toy-level introductions, and many details are not explained in the actual production environment. But I can only bite the bullet and read a little introduction and try it out. However, this method of learning while solving problems is still very efficient. In the end, it is also relying on the dlv tool to locate the deadlock and the cause of the problem.
Installation:
refer to the official document, execute first
go install github.com/go-delve/delve/cmd/dlv@latest
This will install dlv to your GOPATH directory, which will generate a dlv executable file. Then upload this dlv to the server and it can be used.
Of course, you can also pull the code first, and then execute go build in the code directory to generate executable files.
attach and coredump:
After having the dlv, the first step is to make the dlv accessible to the stuck process, that is, use the attach command to attach the dlv to the stuck process.
First, use ps -ef | grep xxxx to view the pid of the stuck process, assuming the stuck process pid=1234, and then execute ./dlv attach 1234 to attach to the stuck process. If the process is started using a non-current user, sudo must be added.
After connecting to the process, no matter what happens, you must export the coredump as soon as possible, because once there is any problem that causes the process to die or the stuck state is resolved, you cannot continue to troubleshoot and debug. With coredump, you can always debug slowly.
In dlv, execute dump ~/sample.core to record the current state of the entire process. In this way, you can download the coredump file and debug the problem step by step on your own development machine without affecting online services.
To view the coredump file on your own development machine, you also need to provide a binary file that is consistent with the online running program. For example, if you are running version v0.22 online, you can either download the binary file directly from the Internet, or you can switch the code to v0.22 and compile it without changing the compilation parameters. It can only be used if the binary file is completely consistent with the corresponding coredump.
Assuming that the binary file name is: sample_bin, then execute dlv core sample_bin sample.core on the development machine to load it into the coredump file for troubleshooting.
The difference between attach and coredump is that attach is attached to a running instance, and can perform functions such as calling functions and setting breakpoints, while coredump can only statically view information such as stack frames and memory. Therefore, if there is a fault like stuck online, save the coredump first, and if possible, directly attach and debug online, which will make it easier to troubleshoot the problem.
Check:
To troubleshoot, a few commands are mainly used:
bt, check the stack frame
list, check the code grs run by the current stack frame
, check the list of all current goroutines
gr 4, switch to the fourth groutine
print xxx, and print the current stack The variable value args in the frame
, print the function parameter value
locals, print the local variable of the current stack frame
Summarize the process:
grs View goroutine list
Prioritize switching to gr 1
Execute the bt command to view stack frames, find business-related code
frame xxx, switch to the specified stack frame
list View the code run by the current stack frame
print xxx View variable values
Problem encountered:
During online debugging, I wanted to execute list to view the code of the current stack frame, and an error similar to Command failed: open /path/to/the/mainfile.go: no such file or directory appeared. Even if you download the coredump file and the binary to your own development machine for debugging, this problem will still occur, because the binary is compiled by another colleague, and the Go compilation will bring the full path of the code by default, and the GOPATH of my colleague and I are different. Same, how to solve this?
The ultimate solution, compile parameters plus trimpath: go build -gcflags=-trimpath= GOPATH − asmflags = − trimpath = GOPATH -asmflags=-trimpath=GOP A T H−asmflags=− t r im p a t h= GOPATH The code path compiled in this way will remove your own folder path and use the path relative to GOPATH. The coredump obtained by others on your compiled binary can also be debugged normally.
If this instruction is not added before compiling, it can be executed in dlv: config substitute-path otherPath mypath.
If the path is wrong, how to clear the wrong configuration? Just execute config substitute-path otherPath directly
Note that mypath must be filled with a large path, and using a path like ~/ will cause problems.
After the config is configured, if it needs to be permanent, you can execute config -save, so that the configuration will be saved in the ~/.config/dlv directory by default.
However, another problem is also difficult to solve. Generally, Go code and code blocks are not carried on the server, so when debugging on the server, the list command will not be used, and it is difficult to troubleshoot the problem. In the end, you can only use coredump to check and debug on the development machine?
In fact, you can also enable the debugging service, open it online, and use the dlv connection on the development machine to connect the service opened online, and then you can debug. But this is also quite troublesome.
Another problem is that when printing variables, if it is a string variable, it seems that only the first 64 characters are printed by default, which is obviously not enough. You can increase the length of the printed string by setting: config max-string-len 99999. At this time, you can execute config -save to save this configuration permanently.