Go HTTP Programming (2): Anatomy of the underlying implementation of http.Client

The data structure of http.Client

We have already introduced, http.Get(), http.Post(), http.PostForm() and  http.Head() methods are actually  http.DefaultClient carried out on the basis of the call.

http.DefaultClient It is the   default implementation of the HTTP client provided by the net/http package:

// DefaultClient is the default Client and is used by Get, Head, and Post.
var DefaultClient = &Client{}

In fact, we can also implement it based on  http.Client a custom HTTP client. Before that, let's take a look at  Client the data structure of the type:

type Client struct {
    // Transport 用于指定单次 HTTP 请求响应的完整流程
    // 默认值是 DefaultTransport 
    Transport RoundTripper

    // CheckRedirect 用于定义重定向处理策略
    // 它是一个函数类型，接收 req 和 via 两个参数，分别表示即将发起的请求和已经发起的所有请求，最早的已发起请求在最前面
    // 如果不为空，客户端将在跟踪 HTTP 重定向前调用该函数
    // 如果返回错误，客户端将直接返回错误，不会再发起该请求
    // 如果为空，Client 将采用一种确认策略，会在 10 个连续请求后终止 
    CheckRedirect func(req *Request, via []*Request) error

    // Jar 用于指定请求和响应头中的 Cookie 
    // 如果该字段为空，则只有在请求中显式设置的 Cookie 才会被发送
    Jar CookieJar

    // 指定单次 HTTP 请求响应事务的超时时间
    // 未设置的话使用 Transport 的默认设置，为零的话表示不设置超时时间
    Timeout time.Duration
}

The  Transport field must implement the  http.RoundTripper interface, Transport specifying the complete flow of an HTTP transaction (request response). If not specified  Transport, http.DefaultTransport the default implementation will be used by  default. For example  http.DefaultClient , this is the case, and we will discuss http.DefaultTransport the underlying implementation in depth later  .

CheckRedirect Functions are used to define strategies for handling redirection. When  sending an HTTP request using the Get() or  Head()method provided by the HTTP default client  , if the response status code is  30x (such as  301, 302 etc.), the HTTP client will call this CheckRedirect function before following the redirect rule  .

Jar It can be used to set cookies in the HTTP client. The Jar type must implement an  http.CookieJar interface, which has predefined  SetCookies() and  Cookies() two methods. If there is no setting in the HTTP client  Jar, the cookie will be ignored and not sent to the client. In fact, we generally use  http.SetCookie() methods to set cookies.

Timeout The field is used to specify  Transport the timeout period, if not specified, Transport a custom setting is used  .

The underlying implementation of http.Transport

Below we will use  http.DefaultTransport the implementation to focus on   , it will be used when http.Transportthere is no explicit setting of the  Transportfield  DefaultTransport:

func (c *Client) transport() RoundTripper {
    if c.Transport != nil {
        return c.Transport
    }
    return DefaultTransport
}

DefaultTransport Yes  Transport , the default implementation, the corresponding initialization code is as follows:

var DefaultTransport RoundTripper = &Transport{
    Proxy: ProxyFromEnvironment,
    DialContext: (&net.Dialer{
        Timeout:   30 * time.Second,
        KeepAlive: 30 * time.Second,
        DualStack: true,
    }).DialContext,
    MaxIdleConns:          100,
    IdleConnTimeout:       90 * time.Second,
    TLSHandshakeTimeout:   10 * time.Second,
    ExpectContinueTimeout: 1 * time.Second,
}

Here only Transport part of the attributes are set  , and Transport the complete data structure of the type is as follows:

type Transport struct {
    ...
    
    // 定义 HTTP 代理策略
    Proxy func(*Request) (*url.URL, error)
    
    // 用于指定创建未加密 TCP 连接的上下文参数（通过 net.Dial()创建连接时使用）
    DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
    // 已废弃，使用 DialContext 替代
    Dial func(network, addr string) (net.Conn, error)
    // 创建加密 TCP 连接
    DialTLS func(network, addr string) (net.Conn, error)
    // 指定 tls.Client 所使用的 TLS 配置
    TLSClientConfig *tls.Config
    // TLS 握手超时时间
    TLSHandshakeTimeout time.Duration
    
    // 是否禁用 HTTP 长连接
    DisableKeepAlives bool
    // 是否对 HTTP 报文进行压缩传输（gzip）
    DisableCompression bool
    
    // 最大空闲连接数（支持长连接时有效）
    MaxIdleConns int
    // 单个服务（域名）最大空闲连接数
    MaxIdleConnsPerHost int
    // 单个服务（域名）最大连接数
    MaxConnsPerHost int
    // 空闲连接超时时间
    IdleConnTimeout time.Duration

    // 从客户端把请求完全提交给服务器到从服务器接收到响应报文头的超时时间
    ResponseHeaderTimeout time.Duration
    // 包含 "Expect: 100-continue" 请求头的情况下从客户端把请求完全提交给服务器到从服务器接收到响应报文头的超时时间
    ExpectContinueTimeout time.Duration
    
    ...        
}

Transport Look at DefaultTransport the settings in conjunction with the  data structure  :

• By  net.Dialer initializing the Dial context configuration, the default timeout period is set to 30 seconds;
• Via  MaxIdleConns connection 100 to specify the maximum number of idle, not explicitly set  MaxIdleConnsPerHost and  MaxConnsPerHost, MaxIdleConnsPerHost with a default value, by  http.DefaultMaxIdleConnsPerHost setting the corresponding default value is 2;
• By  IdleConnTimeout specifying the maximum idle connection time as 90 seconds, that is, when an idle connection is not reused for more than 90 seconds, it will be destroyed. The idle connection needs  DisableKeepAlives to  false be available, that is, it is valid in the HTTP long connection state (HTTP/1.1 and above) Support long connection, corresponding to the request header  Connection:keep-alive);
• By  TLSHandshakeTimeout specifying that a secure TCP connection based on TLS protocol is established, the timeout period of the handshake phase is 10 seconds;
• By  ExpectContinueTimeout specifying that the client wants to use a POST request to send a large message body to the server, it first Expect: 100-continue asks the server whether it is willing to receive the timeout period corresponding to this large message body by sending a request header included  . The default setting here is 1 second.

In addition, Transport the RoundTrip method implementation is included  , so the RoundTripper interface is implemented  . Let us look at  Transport the  RoundTrip implementation of the method.

Transport.RoundTrip() method implementation

First, let's take a look at  http.RoundTripper the specific definition of the interface:

type RoundTripper interface {
    RoundTrip(*Request) (*Response, error)
}

As you can see from the above code, the http.RoundTripper interface is very simple and only defines a RoundTrip method named  . RoundTrip() The method is used to perform an independent HTTP transaction, accept the incoming  *Request request value as a parameter and return the corresponding  *Response response value, as well as a  error value.

When implementing a specific  RoundTrip() method, you should not try to parse the HTTP response information in this function. If the response is successful, error the value must be  nil, regardless of the HTTP status code returned. If the response from the server cannot be successfully obtained, it error must be a non-zero value. Similarly, you should not try  RoundTrip() to deal with protocol-level details such as redirection, authentication, or cookies.

If it is not necessary, the RoundTrip() incoming request object ( *Request) should not be rewritten in the  method. The content of the request (such as URL and Header, etc.) must be RoundTrip() organized and initialized before being passed in  .

Any RoundTrip() type that implements a  method implements an  http.RoundTripper interface. http.Transport It is the RoundTrip() method that implements the  method and then implements the interface. At the bottom layer, Go  implements a single HTTP request response transaction through the  WHATWG Fetch API :

func (t *Transport) RoundTrip(req *Request) (*Response, error) {
	if useFakeNetwork() {
		return t.roundTrip(req)
	}

	ac := js.Global().Get("AbortController")
	if ac != js.Undefined() {
		// Some browsers that support WASM don't necessarily support
		// the AbortController. See
		// https://developer.mozilla.org/en-US/docs/Web/API/AbortController#Browser_compatibility.
		ac = ac.New()
	}

	opt := js.Global().Get("Object").New()
	// See https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch
	// for options available.
	opt.Set("method", req.Method)
	opt.Set("credentials", "same-origin")
	if h := req.Header.Get(jsFetchCreds); h != "" {
		opt.Set("credentials", h)
		req.Header.Del(jsFetchCreds)
	}
	if h := req.Header.Get(jsFetchMode); h != "" {
		opt.Set("mode", h)
		req.Header.Del(jsFetchMode)
	}
	if ac != js.Undefined() {
		opt.Set("signal", ac.Get("signal"))
	}
	headers := js.Global().Get("Headers").New()
	for key, values := range req.Header {
		for _, value := range values {
			headers.Call("append", key, value)
		}
	}
	opt.Set("headers", headers)

	if req.Body != nil {
		// TODO(johanbrandhorst): Stream request body when possible.
		// See https://bugs.chromium.org/p/chromium/issues/detail?id=688906 for Blink issue.
		// See https://bugzilla.mozilla.org/show_bug.cgi?id=1387483 for Firefox issue.
		// See https://github.com/web-platform-tests/wpt/issues/7693 for WHATWG tests issue.
		// See https://developer.mozilla.org/en-US/docs/Web/API/Streams_API for more details on the Streams API
		// and browser support.
		body, err := ioutil.ReadAll(req.Body)
		if err != nil {
			req.Body.Close() // RoundTrip must always close the body, including on errors.
			return nil, err
		}
		req.Body.Close()
		a := js.TypedArrayOf(body)
		defer a.Release()
		opt.Set("body", a)
	}
	respPromise := js.Global().Call("fetch", req.URL.String(), opt)
	var (
		respCh = make(chan *Response, 1)
		errCh  = make(chan error, 1)
	)
	success := js.FuncOf(func(this js.Value, args []js.Value) interface{} {
		result := args[0]
		header := Header{}
		// https://developer.mozilla.org/en-US/docs/Web/API/Headers/entries
		headersIt := result.Get("headers").Call("entries")
		for {
			n := headersIt.Call("next")
			if n.Get("done").Bool() {
				break
			}
			pair := n.Get("value")
			key, value := pair.Index(0).String(), pair.Index(1).String()
			ck := CanonicalHeaderKey(key)
			header[ck] = append(header[ck], value)
		}

		contentLength := int64(0)
		if cl, err := strconv.ParseInt(header.Get("Content-Length"), 10, 64); err == nil {
			contentLength = cl
		}

		b := result.Get("body")
		var body io.ReadCloser
		// The body is undefined when the browser does not support streaming response bodies (Firefox),
		// and null in certain error cases, i.e. when the request is blocked because of CORS settings.
		if b != js.Undefined() && b != js.Null() {
			body = &streamReader{stream: b.Call("getReader")}
		} else {
			// Fall back to using ArrayBuffer
			// https://developer.mozilla.org/en-US/docs/Web/API/Body/arrayBuffer
			body = &arrayReader{arrayPromise: result.Call("arrayBuffer")}
		}

		select {
		case respCh <- &Response{
			Status:        result.Get("status").String() + " " + StatusText(result.Get("status").Int()),
			StatusCode:    result.Get("status").Int(),
			Header:        header,
			ContentLength: contentLength,
			Body:          body,
			Request:       req,
		}:
		case <-req.Context().Done():
		}

		return nil
	})
	defer success.Release()
	failure := js.FuncOf(func(this js.Value, args []js.Value) interface{} {
		err := fmt.Errorf("net/http: fetch() failed: %s", args[0].String())
		select {
		case errCh <- err:
		case <-req.Context().Done():
		}
		return nil
	})
	defer failure.Release()
	respPromise.Call("then", success, failure)
	select {
	case <-req.Context().Done():
		if ac != js.Undefined() {
			// Abort the Fetch request
			ac.Call("abort")
		}
		return nil, req.Context().Err()
	case resp := <-respCh:
		return resp, nil
	case err := <-errCh:
		return nil, err
	}
}

Because http.RoundTripper the code that implements the  interface usually needs to be executed concurrently in multiple goroutines, we must ensure the thread safety of the implementation code.

The above are the  http.Client core components of the underlying implementation and their default implementations. The focus is on  http.Transport. It defines the complete process of an HTTP transaction. We can customize  Transport the HTTP client request through customization. Just understand it. In actual development , We generally only need to call a few methods provided in the previous tutorial , unless we need to do low-level development and customization, otherwise we will generally not involve these.