table of Contents
0x04 execute command with timeout
Recently, there is a need to execute a command with Golang and set a certain timeout for the command. Starting from this, we started from executing a basic shell command and gradually studied black magic.
0x00 Preface
In this article, we assume that there is no pre-knowledge and can be eaten with confidence.
Just yesterday, the interview asked about the principles of Go-routine and the concept of CSP. The first diss is a wave of interview students, who feel very academic, and don’t know the meaning of asking a security algorithm job Engineer instead of a Researcher. A beginner user certainly doesn’t know the definition of CSP Concurrent mode . , And also asked "Why don't you study in depth if you like Golang" and other uncomfortable issues (I said at the time that I was new to ML and didn't study language features). Here is a brief study.
However, the article still uses this as the title. In addition to the requirements at the beginning, some interview tips about this part of knowledge are licking the dog in front of the work (x
0x01 First run an os/exec
We use the sample code of the official document to run a command and get the output.
func RunCommand(command string, args ...string) (stdout string) {
cmd := exec.Command(command, args...)
outBytes, err := cmd.Output()
if err != nil {
log.Fatal(err)
}
stdout = string(outBytes)
return
}
func main() {
log.Print(RunCommand("echo", "fuck"))
}Output 2018/11/21 22:27:00 fuck.
0x02 os/exec black magic
os/exev has some black magic,
We created a new bash script called echo_loop:
while true; do date; sleep 1; doneWhen we executed RunCommand("bash", "echo_loop.sh"), the program got stuck. Obviously this script is continuously outputting, and we hope to continuously obtain intermediate output. But we can't let him get stuck, otherwise how to detect timeout. We have some operations on the command object that cmd.Run()wait for the completion of the command, and cmd.Start()we can make the command not wait:
func RunCommand(command string, args ...string) (stdout string) {
cmd := exec.Command(command, args...)
err := cmd.Start()
if err != nil {
log.Fatal(err)
}
// stdout = string(outBytes)
return
}However, if you want to get the output at this time, you can pass it cmd.StdoutPipe()(note that this needs to be done Start()before, otherwise the output content will be exec: StdoutPipe after process started). After adding this we are stuck again:
func RunCommand(command string, args ...string) (stdout string) {
cmd := exec.Command(command, args...)
err := cmd.Start()
if err != nil {
log.Fatal(err)
}
out, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
defer out.Close()
outBytes, err := ioutil.ReadAll(out)
if err != nil {
log.Fatal(err)
}
stdout = string(outBytes)
return
}Of course, there is another one cmd.Wait(), which returns an error type, so you can get the output like this. Unfortunately it also gets stuck:
func RunCommand(command string, args ...string) (stdout string) {
cmd := exec.Command(command, args...)
err := cmd.Start()
if err != nil {
log.Fatal(err)
}
outBytes := cmd.Wait()
stdout = outBytes.Error()
return
}We naturally think of an operation, we need to execute a command not in the foreground (in order to control the timeout), and get its output. Fortunately, Golang implements this set of mechanisms.
0x03 Channel & Goroutine
We first look at a piece of code:
func sum(s []int, c chan int) {
sum := 0
for _, v := range s {
sum += v
}
time.Sleep(2 * time.Second)
c <- sum // 发送数据到 Channel
}
func main() {
s := []int{1, 2, 3, 4, 5, 6}
c := make(chan int)
log.Print("Waiting")
go sum(s[len(s)/2:], c) // 前半段
go sum(s[:len(s)/2], c) // 后半段
x, y := <-c, <-c // 分别阻塞地接收数据
log.Print("Done")
log.Print(x, y)
}
// Output:
// 2018/11/21 23:51:54 Waiting
// 2018/11/21 23:51:56 Done
// 2018/11/21 23:51:56 15 6Channel
Channel is a core type in Go. You can think of it as a pipeline through which the concurrent core unit can send or receive data for communication. The basic operation method is as follows:
ch := make(chan int) // 新建一个 Channel
ch <- v // 发送值 v 到 Channel ch中
v := <-ch // 从 Channel ch 中接收数据，并将数据赋值给 vA Channel can be set to one-way and two-way, the type of sending and receiving, and the capacity. For some knowledge about Channel, you can refer to the official document or this article . One problem worth noting is that during a receive operation, if there is no data in the Channel, the program will be blocked here. At this time, data should be written to it from a goroutine . The same access principle is that they are all blocking, and a goroutine is needed to read them.
Goroutine
Regarding Routine, there is actually no complicated calling method, just add an identifier before calling the function go, so we often call it goroutine. It can be seen as a thread we know well, but it is lighter in implementation. Golang implements a thread pool. The specific implementation will be mentioned later in the [0x05 Interview Tips] section.
Goroutine and Channel are also complementary. Because there is no similar thread.join()concept in Python . As mentioned before, Channel access is blocked. The default Channel size is 0, we call this unbuffered . The unbuffered Channel will suspend the current goroutine when fetching and storing messages. Therefore, we can complete the goroutine writing data to a Channel, after the writing, the Channel can read the data, and continue to execute the remaining part.
The default Channel size is 0, which is called unbuffered . We do not default to a size, it can buffer a. Trying to make an unbuffered Channel carry data is one of the important causes of deadlock. Giving the Channel a size can avoid it.
some problems
Q1: What happens if the written data has not been read? What if there is no data that has been read?
All deadlocks, the error message isall goroutines are asleep - deadlock
We add a sentence before printing Done c <- 0, and there will be a deadlock. No one reads this data.
We add a sentence before printing Done _ = <- c, there will be a deadlock. No data has been read.
Q2: What about adding these two sentences at the same time?
It is also a deadlock.
Key point: Operation data in a single goroutine must be deadlocked . When passing in 0 to c, a thread that has been joined in a thread pool is required to read. At this time, there is no other goroutine except for the main thread. As the error message says, all goroutines are asleep.
Q3: How can I deadlock?
good question. We need to know the principle of deadlock, which is the occurrence of circular reading and circular waiting, which makes an unbuffered Channel carry data. Here is an example of a typical loop waiting:
c, quit := make(chan int), make(chan int)
go func() {
c <- 1 // c 正在等待数据被取走，才能执行 quit
quit <- 0 // 楼上等待期间这里也在等
}()
x := <- quit // quit 在等待 goroutine 有人写
y := <- c // 由于一直没有 quit，c 中数据不可能被取走In layman's terms, the data stored in Channel one by one needs to be taken out one by one. If you really need it, you can't take it out one by one, just like the previous search, you can give the Channel a size. Written here c, quit := make(chan int, 1), make(chan int, 1), the above program will not deadlock.
Of course, there are tens of millions of bugs, and we are here to give a counter-example that does not deadlock and is not a bug. The reason is that before this goroutine runs, the main program exits:
c := make(chan int)
go func() {
c <- 1
}()
0x04 execute command with timeout
We seem to have solved the original problem: we need to execute a command when it is not in the foreground (it will cmd.Wait()start as a goroutine), and get its output (via Channel). Modify the original code to have these:
- In order to avoid the magical operation of closing the sequence between the pipeline when sending SigInt and Kill threads (StdoutPipe cannot get the content after the end, it will block waiting when it is not ended), refer to the processing method in this article , through the bytes buffer Come to accept.
- Use select to handle the operations of the commands that complete normal operation and end Kill.
- After sending the SigInt signal, some commands will output some information (such as ping), adding a 10ms delay before Kill.
The code is as follows, you can copy the test run directly:
package main
import (
"syscall"
"time"
"bytes"
"log"
"os/exec"
)
func RunCommandWithTimeout(timeout int, command string, args ...string) (stdout, stderr string, isKilled bool) {
var stdoutBuf, stderrBuf bytes.Buffer
cmd := exec.Command(command, args...)
cmd.Stdout = &stdoutBuf
cmd.Stderr = &stderrBuf
cmd.Start()
done := make(chan error)
go func() {
done <- cmd.Wait()
}()
after := time.After(time.Duration(timeout) * time.Millisecond)
select {
case <-after:
cmd.Process.Signal(syscall.SIGINT)
time.Sleep(10*time.Millisecond)
cmd.Process.Kill()
isKilled = true
case <-done:
isKilled = false
}
stdout = string(bytes.TrimSpace(stdoutBuf.Bytes())) // Remove \n
stderr = string(bytes.TrimSpace(stderrBuf.Bytes())) // Remove \n
return
}
func main() {
resultOut, resultErr, resultStat := RunCommandWithTimeout(3000, "ping", "baidu.com")
log.Print("Is Killed: ", resultStat)
log.Print("Res: \n===\n", resultOut, "\n===")
log.Print("Err: \n===\n", resultErr, "\n===")
}
// Output:
// 2018/11/22 01:47:44 Is Killed: true
// 2018/11/22 01:47:44 Res:
// ===
// PING baidu.com (220.181.57.216): 56 data bytes
// 64 bytes from 220.181.57.216: icmp_seq=0 ttl=53 time=44.607 ms
// 64 bytes from 220.181.57.216: icmp_seq=1 ttl=53 time=44.476 ms
// 64 bytes from 220.181.57.216: icmp_seq=2 ttl=53 time=45.519 ms
//
// --- baidu.com ping statistics ---
// 3 packets transmitted, 3 packets received, 0.0% packet loss
// round-trip min/avg/max/stddev = 44.476/44.867/45.519/0.464 ms
// ===
// 2018/11/22 01:47:44 Err:
// ===
//
// ===