Golang support smooth upgrade HTTP service

Some time ago with Golang doing a HTTP interface, due to the characteristics of compiled languages, modify the code needs to be recompiled executable file, close the old programs that are running, and start the new program. Access to a large amount of user-oriented products, closed the case can not be accessed in the process of restarting bound to appear, thus affecting the user experience.

Golang system using HTTP service package development, is unable to support a smooth upgrade (Graceful Restart), this article will explore how to solve the problem.

A smooth upgrade (Graceful Restart) of the general idea

In general, to achieve a smooth upgrade requires the following steps:

1. Replacing the old with the new executable file executable file (such as simply elegant restart, you can skip this step)
2. Send a specific signal (kill -SIGUSR2 $ pid) to the old running process by pid
3. Old running processes, after receiving the specified signal to child processes way to start the new executable file and start processing new requests
4. The old process is no longer accepting new requests waiting for service of process is not completed is completed, then the end of normal
5. New processes when the parent process exits, would be to adopt the init process, and continue to provide services

Two, Golang Socket network programming

Socket is a transport layer protocol TCP / IP encapsulation and applications programmer level. Function and structure is defined in Golang Socket relevant in net package, we start with a simple example to learn about Golang Socket network programming, key instructions written directly in the comments.

1, the server program server.go

package main

import (
	"fmt"
	"log"
	"net"
	"time"
)

func main() {
	// 监听8086端口
	listener, err := net.Listen("tcp", ":8086")
	if err != nil {
		log.Fatal(err)
	}
	defer listener.Close()

	for {
		// 循环接收客户端的连接，没有连接时会阻塞，出错则跳出循环
		conn, err := listener.Accept()
		if err != nil {
			fmt.Println(err)
			break
		}

		fmt.Println("[server] accept new connection.")

		// 启动一个goroutine 处理连接
		go handler(conn)
	}
}

func handler(conn net.Conn) {
	defer conn.Close()

	for {
		// 循环从连接中 读取请求内容，没有请求时会阻塞，出错则跳出循环
		request := make([]byte, 128)
		readLength, err := conn.Read(request)

		if err != nil {
			fmt.Println(err)
			break
		}

		if readLength == 0 {
			fmt.Println(err)
			break
		}

		// 控制台输出读取到的请求内容，并在请求内容前加上hello和时间后向客户端输出
		fmt.Println("[server] request from ", string(request))
		conn.Write([]byte("hello " + string(request) + ", time: " + time.Now().Format("2006-01-02 15:04:05")))
	}
}

2, the client program client.go

package main

import (
	"fmt"
	"log"
	"net"
	"os"
	"time"
)

func main() {

	// 从命令行中读取第二个参数作为名字，如果不存在第二个参数则报错退出
	if len(os.Args) != 2 {
		fmt.Fprintf(os.Stderr, "Usage: %s name ", os.Args[0])
		os.Exit(1)
	}
	name := os.Args[1]

	// 连接到服务端的8086端口
	conn, err := net.Dial("tcp", "127.0.0.1:8086")
	checkError(err)

	for {
		// 循环往连接中 写入名字
		_, err = conn.Write([]byte(name))
		checkError(err)

		// 循环从连接中 读取响应内容，没有响应时会阻塞
		response := make([]byte, 256)
		readLength, err := conn.Read(response)
		checkError(err)

		// 将读取响应内容输出到控制台，并sleep一秒
		if readLength > 0 {
			fmt.Println("[client] server response:", string(response))
			time.Sleep(1 * time.Second)
		}
	}
}

func checkError(err error) {
	if err != nil {
		log.Fatal("fatal error: " + err.Error())
	}
}

3, run the example

# 运行服务端程序
go run server.go

# 在另一个命令行窗口运行客户端程序
go run client.go "tabalt"

Three, Golang HTTP programming

HTTP transport layer protocol is based on TCP / IP application layer protocol. Golang implemented in the HTTP-related net / http package, and used directly as a function of net structure associated Socket package.

Let us start with a simple example to learn about Golang HTTP programming, key instructions written directly in the comments.

1, http service program http.go

package main

import (
	"log"
	"net/http"
	"os"
)

// 定义http请求的处理方法
func handlerHello(w http.ResponseWriter, r *http.Request) {
	w.Write([]byte("http hello on golang\n"))
}

func main() {

	// 注册http请求的处理方法
	http.HandleFunc("/hello", handlerHello)

	// 在8086端口启动http服务，会一直阻塞执行
	err := http.ListenAndServe("localhost:8086", nil)
	if err != nil {
		log.Println(err)
	}

	// http服务因故停止后 才会输出如下内容
	log.Println("Server on 8086 stopped")
	os.Exit(0)
}

2, running the sample program

# 运行HTTP服务程序
go run http.go

# 在另一个命令行窗口curl请求测试页面
curl http://localhost:8086/hello/

# 输出如下内容：
http hello on golang

Attaining Golang net / http packets Socket operation

From the simple example above, we see http Golang To start a service, you need only a simple three-step:

1. Http request processing method defined
2. Treatment of registered http request
3. In a port start HTTP service

The most crucial start http service, calling http.ListenAndServe () function to achieve. Here we find the implementation of the function:

func ListenAndServe(addr string, handler Handler) error {
	server := &Server{Addr: addr, Handler: handler}
	return server.ListenAndServe()
}

Here creates an object Server and call it ListenAndServe () method, we'll find realize ListenAndServe () method of the structure Server:

func (srv *Server) ListenAndServe() error {
	addr := srv.Addr
	if addr == "" {
		addr = ":http"
	}
	ln, err := net.Listen("tcp", addr)
	if err != nil {
		return err
	}
	return srv.Serve(tcpKeepAliveListener{ln.(*net.TCPListener)})
}

See from the code, here is listening tcp port and listener packed into a structure tcpKeepAliveListener, then call srv.Serve () method; we continue to track the realization Serve () method:

func (srv *Server) Serve(l net.Listener) error {
	defer l.Close()
	var tempDelay time.Duration // how long to sleep on accept failure
	for {
		rw, e := l.Accept()
		if e != nil {
			if ne, ok := e.(net.Error); ok && ne.Temporary() {
				if tempDelay == 0 {
					tempDelay = 5 * time.Millisecond
				} else {
					tempDelay *= 2
				}
				if max := 1 * time.Second; tempDelay > max {
					tempDelay = max
				}
				srv.logf("http: Accept error: %v; retrying in %v", e, tempDelay)
				time.Sleep(tempDelay)
				continue
			}
			return e
		}
		tempDelay = 0
		c, err := srv.newConn(rw)
		if err != nil {
			continue
		}
		c.setState(c.rwc, StateNew) // before Serve can return
		go c.serve()
	}
}

It can be seen in front of us, and sample code Socket programming, like cycling Accept connection from the port to listen, and if it returns an error net.Error and this is temporary, it will sleep a time to continue. If other errors will be returned to terminate the cycle. After a successful Accept a connection, call the method srv.newConn () to make the connection layer of the packaging, and finally start a process goroutine http request.

Five, Golang achieve a smooth upgrade (Graceful Restart) HTTP services

I created a new package gracehttp to achieve a smooth upgrade support (Graceful Restart) HTTP service, in order to write less code and reduce the cost, the new package as much as possible the use of net/httpimplementation packages and and net/httppackages consistent way to keep foreign . Now we look at gracehttpthe package supports a smooth upgrade (Graceful Restart) Golang HTTP services involved in the details of how to achieve.

1, Golang processed signal

Golang of os/signalpacket encapsulation processing on the signal. Consider a simple example of usage:

package main

import (
	"fmt"
	"os"
	"os/signal"
	"syscall"
)

func main() {

	signalChan := make(chan os.Signal)

	// 监听指定信号
	signal.Notify(
		signalChan,
		syscall.SIGHUP,
		syscall.SIGUSR2,
	)

	// 输出当前进程的pid
	fmt.Println("pid is: ", os.Getpid())

	// 处理信号
	for {
		sig := <-signalChan
		fmt.Println("get signal: ", sig)
	}
}

2, the child process to launch a new program, the same listening port

We can see in the code that implements ListenAndServe () method of the fourth part, the net/httppackage using a net.Listenfunction to listen for a certain port, but if a running program has been listening on a port, other programs can not go to listen to this port. The solution is the way to use the child process started, and passes the listening port file descriptor to the child process, the child process to achieve in listening to this port from the file descriptor.

Need the help of a specific implementation environment variable to distinguish between the normal process is started, or to child processes are started, the relevant code excerpt follows:

// 启动子进程执行新程序
func (this *Server) startNewProcess() error {

	listenerFd, err := this.listener.(*Listener).GetFd()
	if err != nil {
		return fmt.Errorf("failed to get socket file descriptor: %v", err)
	}

	path := os.Args[0]

	// 设置标识优雅重启的环境变量
	environList := []string{}
	for _, value := range os.Environ() {
		if value != GRACEFUL_ENVIRON_STRING {
			environList = append(environList, value)
		}
	}
	environList = append(environList, GRACEFUL_ENVIRON_STRING)

	execSpec := &syscall.ProcAttr{
		Env:   environList,
		Files: []uintptr{os.Stdin.Fd(), os.Stdout.Fd(), os.Stderr.Fd(), listenerFd},
	}

	fork, err := syscall.ForkExec(path, os.Args, execSpec)
	if err != nil {
		return fmt.Errorf("failed to forkexec: %v", err)
	}

	this.logf("start new process success, pid %d.", fork)

	return nil
}

func (this *Server) getNetTCPListener(addr string) (*net.TCPListener, error) {

	var ln net.Listener
	var err error

	if this.isGraceful {
		file := os.NewFile(3, "")
		ln, err = net.FileListener(file)
		if err != nil {
			err = fmt.Errorf("net.FileListener error: %v", err)
			return nil, err
		}
	} else {
		ln, err = net.Listen("tcp", addr)
		if err != nil {
			err = fmt.Errorf("net.Listen error: %v", err)
			return nil, err
		}
	}
	return ln.(*net.TCPListener), nil
}

3, the parent process waits for the connection request processing has been completed unfinished

This one is the most complex; first of all we need a counter upon successful Accept a connection counter by one count by 1 when the connection is closed, the counter is 0 normal parent can quit. Bag WaitGroup sync of Golang may well achieve this function.

To establish control and then close the connection, we need to go deep into net/httpServe Server package structure () method. Achieve relive Part IV Serve () method, you will find if you want to re-write a Serve () method is almost impossible, because this method in a good number of internal method calls can not be exported, rewrite Serve () method is almost to rewrite the entire net/httppackage.

Fortunately, we also found that the transfer ListenAndServe () method in a listener to Serve () method, and finally call the listener's Accept () method, which returns an example of a Conn's, eventually disconnected when calls Conn's Close () method, the structures and methods are exportable!

We can define their own structure and Conn Listener structure, the combination net/httppackage corresponding structure, and rewrite the Accept () and Close () method, implemented count, the relevant code excerpt connected as follows:

type Listener struct {
	*net.TCPListener

	waitGroup *sync.WaitGroup
}

func (this *Listener) Accept() (net.Conn, error) {

	tc, err := this.AcceptTCP()
	if err != nil {
		return nil, err
	}
	tc.SetKeepAlive(true)
	tc.SetKeepAlivePeriod(3 * time.Minute)

	this.waitGroup.Add(1)

	conn := &Connection{
		Conn:     tc,
		listener: this,
	}
	return conn, nil
}

func (this *Listener) Wait() {
	this.waitGroup.Wait()
}

type Connection struct {
	net.Conn
	listener *Listener

	closed bool
}

func (this *Connection) Close() error {

	if !this.closed {
		this.closed = true
		this.listener.waitGroup.Done()
	}

	return this.Conn.Close()
}

Usage 4, gracehttp package

gracehttp package has been applied to hundreds of millions every day PV project, also open to the GitHub: github.com/tabalt/gracehttp , very simple to use.

The following sample code, after the introduction of the package can modify a keyword, the http.ListenAndServe gracehttp.ListenAndServe can be changed.

package main

import (
    "fmt"
    "net/http"

    "github.com/tabalt/gracehttp"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, "hello world")
    })

    err := gracehttp.ListenAndServe(":8080", nil)
    if err != nil {
        fmt.Println(err)
    }
}

Test smooth upgrade (Graceful Restart) effect can be described with reference to the following page:
https://github.com/tabalt/gracehttp#demo