K8s the command is completed by the apiserver, kubelet, cri, docker and other components together, which is the most complex protocol switching and various related stream copy, let us look at the key to achieving that although the code more, but will not develop It should also be able to read, I wish you good luck
1. Basic concepts
There are many protocols associated with processing performed in K8s commands, let's look together these basic concepts related to protocol processing
1.1 Http Connection Agreement and Upgrade
HTTP / 1.1 is allowed on the same link protocol conversion is achieved by Header head in Connection with the Upgrade, it is simply allowed to communicate using other protocols on the establishment of a link via HTTP, which is k8s command the key to achieve agreement upgrade
1.2 Http protocol status code 101
In the HTTP protocol, in addition to our common HTTP1.1, also supports websocket / spdy other agreements, how that service and client complete different protocols over http switched it, first of all here is the first element 101 (Switching Protocal ) status code that the server tell the client we switched to Uprage up a protocol defined for communication (multiplexing current link)
1.3 SPDY protocol stream
Google SPDY protocol is developed TCP session layer protocol in the protocol SPDY the Http Request / Response called Stream, and supports the TCP link multiplexing between a plurality of simultaneously labeled by the stream Stream-id, it is simply supports a plurality of request response processing simultaneously in a single link, and independently of each other, K8S the command is carried out mainly through the stream of message passing
1.4 redirect file descriptor
In Linux execution process usually consists of three FD: standard input, standard output, standard error, k8s the corresponding FD command will be redirected in order to gain command of the vessel output, redirect it where? Of course we've mentioned above, the stream (because docker not familiar with, so this place does not guarantee the accuracy of Docker section)
1.5 http The Hijacker
Between client and server by status code 101, connection, upragde the like after completion of the conversion of the current link based on the current data transmission link is not in agreement prior to the http1.1, then we must turn to the corresponding http link forwarding to the corresponding protocol conversion, during the execution of the command k8s, we obtain the corresponding link request tcp and Response, http through the bottom of the interface to obtain Hijacker to continue to complete the requested
1.6 based on the flow of forwarding tcp Duikao
After tcp ReaderWriter acquired by the two bottom Hijacker, io.copy can be done directly by the two copies of the corresponding data stream, thereby eliminating the need for protocol conversion apiserver in place, but directly by tcp Duikao flow can be achieved and the results of the forwarding request
Probably these basic introduction, then we went to see the underlying concrete realization, we start from kubectl part to drill down
2.kubectl
Kubectl execute commands divided into two parts Pod legitimacy detection and command execution, testing the legality Pod Pod mainly to obtain the corresponding state detection is running, where we focus on the command execution section
2.1 Command execution core processes
The core command is divided into two steps: 1. Constructing Stream to link to link 2) by SPDY protocol
func (*DefaultRemoteExecutor) Execute(method string, url *url.URL, config *restclient.Config, stdin io.Reader, stdout, stderr io.Writer, tty bool, terminalSizeQueue remotecommand.TerminalSizeQueue) error {
exec, err := remotecommand.NewSPDYExecutor(config, method, url)
if err != nil {
return err
}
return exec.Stream(remotecommand.StreamOptions{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
Tty: tty,
TerminalSizeQueue: terminalSizeQueue,
})
}2.2 exec request construction
We can see this place spliced Url / pods / {namespace} / {podName} / exec actually corresponds to the pod subresource apiserver above interfaces, then we can see the end of the process the request apiserver
// 创建一个exec
req := restClient.Post().
Resource("pods").
Name(pod.Name).
Namespace(pod.Namespace).
SubResource("exec")
req.VersionedParams(&corev1.PodExecOptions{
Container: containerName,
Command: p.Command,
Stdin: p.Stdin,
Stdout: p.Out != nil,
Stderr: p.ErrOut != nil,
TTY: t.Raw,
}, scheme.ParameterCodec)
return p.Executor.Execute("POST", req.URL(), p.Config, p.In, p.Out, p.ErrOut, t.Raw, sizeQueue)2.3 ESTABLISHMENT Stream
Stream in the type of exec.Stream mainly through Headers transfer to be established, in consultation with the server end
// set up stdin stream
if p.Stdin != nil {
headers.Set(v1.StreamType, v1.StreamTypeStdin)
p.remoteStdin, err = conn.CreateStream(headers)
if err != nil {
return err
}
}
// set up stdout stream
if p.Stdout != nil {
headers.Set(v1.StreamType, v1.StreamTypeStdout)
p.remoteStdout, err = conn.CreateStream(headers)
if err != nil {
return err
}
}
// set up stderr stream
if p.Stderr != nil && !p.Tty {
headers.Set(v1.StreamType, v1.StreamTypeStderr)
p.remoteStderr, err = conn.CreateStream(headers)
if err != nil {
return err
}
}3.APIServer
APIServer play in the process of implementation of the order in the agency's role, which is responsible for requests between Kubectl and kubelet to be forwarded, forwarded this note is mainly based on tcp stream Duikao completed because the communication between the kubectl and kubelet, actually spdy agreement, let us look at the key to achieving it
3.1 Connection
Exec is SPDY request is first sent to the Connect Interface, Connection interface responsible with kubelet rear end of the link is established, and in response the result returned in the Connection interface, first obtains the Node information corresponding through Pod, and constructs Location i.e. Kubelet back-end address and transport links
func (r *ExecREST) Connect(ctx context.Context, name string, opts runtime.Object, responder rest.Responder) (http.Handler, error) {
execOpts, ok := opts.(*api.PodExecOptions)
if !ok {
return nil, fmt.Errorf("invalid options object: %#v", opts)
}
// 返回对应的地址，以及建立链接
location, transport, err := pod.ExecLocation(r.Store, r.KubeletConn, ctx, name, execOpts)
if err != nil {
return nil, err
}
return newThrottledUpgradeAwareProxyHandler(location, transport, false, true, true, responder), nil
}3.2 acquire the back-end service address
Acquiring address mainly constructed location information rear end, information is acquired to a corresponding node of the host and Port Information herein will be reported by kubelet, and assembling the final path to the pod, i.e., where the Path field / exec / {namespace} / {podName} / {containerName}
loc := &url.URL{
Scheme: nodeInfo.Scheme,
Host: net.JoinHostPort(nodeInfo.Hostname, nodeInfo.Port), // node的端口
Path: fmt.Sprintf("/%s/%s/%s/%s", path, pod.Namespace, pod.Name, container), // 路径
RawQuery: params.Encode(),
}3.3 protocol to enhance the handler initialization
Protocol mainly through UpgradeAwareHandler lift controllers, after receiving the request handler will first try to enhance the protocol, which is mainly a detection value Connection http header Upragde which is not implemented, the previous analysis kubelet can know from where It is certainly true
func newThrottledUpgradeAwareProxyHandler(location *url.URL, transport http.RoundTripper, wrapTransport, upgradeRequired, interceptRedirects bool, responder rest.Responder) *proxy.UpgradeAwareHandler {
handler := proxy.NewUpgradeAwareHandler(location, transport, wrapTransport, upgradeRequired, proxy.NewErrorResponder(responder))
handler.InterceptRedirects = interceptRedirects && utilfeature.DefaultFeatureGate.Enabled(genericfeatures.StreamingProxyRedirects)
handler.RequireSameHostRedirects = utilfeature.DefaultFeatureGate.Enabled(genericfeatures.ValidateProxyRedirects)
handler.MaxBytesPerSec = capabilities.Get().PerConnectionBandwidthLimitBytesPerSec
return handler
}
func (h *UpgradeAwareHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {
// 如果协议提升成功，则由该协议完成
if h.tryUpgrade(w, req) {
return
}
// 省略N多代码
}3.4 protocol upgrade process
Protocol processing logic to enhance more, here is divided into several sections to be successively explained, mainly HTTP acquisition request to start the link, and forwards and at the same time holding the two links, copying and TCP flows on the link
3.4.1 establish links with kubelet
The first step is to enhance agreement with kubelet backend establish a link, here will kubelet sent me a copy of the request and sent to the back-end kubelet, and will also get a link here http of an established and kubelet, conducted behind when need to use flow Duikao, note this fact http request response status code is 101, i.e., the actually constructed kubelet spdy a protocol handler to communicate
// 构建http请求
req, err := http.NewRequest(method, location.String(), body)
if err != nil {
return nil, nil, err
}
req.Header = header
// 发送请求建立链接
intermediateConn, err = dialer.Dial(req)
if err != nil {
return nil, nil, err
}
// Peek at the backend response.
rawResponse.Reset()
respReader := bufio.NewReader(io.TeeReader(
io.LimitReader(intermediateConn, maxResponseSize), // Don't read more than maxResponseSize bytes.
rawResponse)) // Save the raw response.
// 读取响应信息
resp, err := http.ReadResponse(respReader, nil)3.4.2 Request request Hijack
This request is actually spdy agreement, after obtaining a link to the bottom by Hijack, you need to first forward the request to the above kubelet thereby triggering kubelet send a request to establish a link behind Stream is Write here the results forwarded kubelet
requestHijackedConn, _, err := requestHijacker.Hijack()
// Forward raw response bytes back to client.
if len(rawResponse) > 0 {
klog.V(6).Infof("Writing %d bytes to hijacked connection", len(rawResponse))
if _, err = requestHijackedConn.Write(rawResponse); err != nil {
utilruntime.HandleError(fmt.Errorf("Error proxying response from backend to client: %v", err))
}
}3.4.3 two-way flow Duikao
After the above two steps, the two came to have apiserver http link, http protocol is not because it is not directly apiserver operation, but only using streaming Duikao way to forward requests and responses
// 双向拷贝链接
go func() {
var writer io.WriteCloser
if h.MaxBytesPerSec > 0 {
writer = flowrate.NewWriter(backendConn, h.MaxBytesPerSec)
} else {
writer = backendConn
}
_, err := io.Copy(writer, requestHijackedConn)
if err != nil && !strings.Contains(err.Error(), "use of closed network connection") {
klog.Errorf("Error proxying data from client to backend: %v", err)
}
close(writerComplete)
}()
go func() {
var reader io.ReadCloser
if h.MaxBytesPerSec > 0 {
reader = flowrate.NewReader(backendConn, h.MaxBytesPerSec)
} else {
reader = backendConn
}
_, err := io.Copy(requestHijackedConn, reader)
if err != nil && !strings.Contains(err.Error(), "use of closed network connection") {
klog.Errorf("Error proxying data from backend to client: %v", err)
}
close(readerComplete)
}()
4.kubelet
Execute commands on Kubelet mainly dependent on CRI.RuntimeService performed, kubelet only responsible for forwarding a corresponding request, and eventually build a proxy forwards subsequent requests Stream, to complete his mission
4.1 Run the main process
The main primary process is to obtain the command to be executed, then the corresponding Pod new detection and call host.GetExec returns a corresponding URL, and then subsequent requests will be completed by proxyStream, we began a thorough step by step
func (s *Server) getExec(request *restful.Request, response *restful.Response) {
// 获取执行命令
params := getExecRequestParams(request)
streamOpts, err := remotecommandserver.NewOptions(request.Request)
// 获取pod的信息
pod, ok := s.host.GetPodByName(params.podNamespace, params.podName)
podFullName := kubecontainer.GetPodFullName(pod)
url, err := s.host.GetExec(podFullName, params.podUID, params.containerName, params.cmd, *streamOpts)
proxyStream(response.ResponseWriter, request.Request, url)
}4.2 Exec return the results
host.GetExec eventually calls to the Exec runtimeService i.e. cri.RuntimeService execution request to the interface, the interface returns an address that is / exec / {token}, is not performed at this time and the actual command simply creates a command execution request nothing more
func (m *kubeGenericRuntimeManager) GetExec(id kubecontainer.ContainerID, cmd []string, stdin, stdout, stderr, tty bool) (*url.URL, error) {
// 省略请求构造
// 执行命令
resp, err := m.runtimeService.Exec(req)
return url.Parse(resp.Url)
}The final cri is actually calling the exec interface, we'll ignore what the specific interfaces returned, the remaining logic watching kubelet
func (c *runtimeServiceClient) Exec(ctx context.Context, in *ExecRequest, opts ...grpc.CallOption) (*ExecResponse, error) {
err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/Exec", in, out, opts...)
}
4.3 proxyStream
Here we can see is that we've seen before UpgradeAwareHandler, but this is the back end of the url exec implementation of the return of the url, and then the remaining part apiserver inside almost just like, for streaming http link between the two beat
We think about this place Request and Response, in fact, is linked with the corresponding apiserver kubelet established is spdy head on this link, remember this place, at this time to continue to build any connection with the back-end link, the rear end is actually a spdy protocol server, so far we are still the last part is the link returned in the end is what, who is the corresponding controller, be part of the next section cri
// proxyStream proxies stream to url.
func proxyStream(w http.ResponseWriter, r *http.Request, url *url.URL) {
// TODO(random-liu): Set MaxBytesPerSec to throttle the stream.
handler := proxy.NewUpgradeAwareHandler(url, nil /*transport*/, false /*wrapTransport*/, true /*upgradeRequired*/, &responder{})
handler.ServeHTTP(w, r)
}
5.CRI
CRI.RuntimeService ultimately responsible for command execution, command execution position is actually executed, which also involves a lot of protocol processing related to the operation, let us look at the key to achieving it
The registered 5.1 DockerRuntime
In the above, we call RuntimeService of Exec interface eventually found the following code in kubelet, create and launch a DockerServer
ds, err := dockershim.NewDockerService(kubeDeps.DockerClientConfig, crOptions.PodSandboxImage, streamingConfig,
dockerServer := dockerremote.NewDockerServer(remoteRuntimeEndpoint, ds)
if err := dockerServer.Start(); err != nil {
return err
}Where Start function inside, registered the following two RuntimeService, wrote grpc friends all know, this is actually a registered correspond to implement rpc interface that ultimately we call the interface is DockerService
runtimeapi.RegisterRuntimeServiceServer(s.server, s.service)
runtimeapi.RegisterImageServiceServer(s.server, s.service)5.2 DockerService realization of Exec
Exec final implementation can be found to actually called streamingServer GetExec interface returns a / exec / {token} Interface
func (ds *dockerService) Exec(_ context.Context, req *runtimeapi.ExecRequest) (*runtimeapi.ExecResponse, error) {
// 执行Exec请求
return ds.streamingServer.GetExec(req)
}We continue to track streamingServer GetExec interface can be seen as follows, eventually build a url = / exec / {token}, Note that the current is actually stored in the cache request Request
func (s *server) GetExec(req *runtimeapi.ExecRequest) (*runtimeapi.ExecResponse, error) {
// 生成token
token, err := s.cache.Insert(req)
return &runtimeapi.ExecResponse{
Url: s.buildURL("exec", token),
}, nil
}
Exec 5.3 build command execution parameters
First, let's get through the token cache before Request, and then request command via exec, building StreamOpts, and end up calling ServeExec execution, the next step is the most difficult to understand in part, the high-energy front
func (s *server) serveExec(req *restful.Request, resp *restful.Response) {
// 获取token
token := req.PathParameter("token")
// 缓存请求
cachedRequest, ok := s.cache.Consume(token)
// 构建exec参数s
exec, ok := cachedRequest.(*runtimeapi.ExecRequest)
streamOpts := &remotecommandserver.Options{
Stdin: exec.Stdin,
Stdout: exec.Stdout,
Stderr: exec.Stderr,
TTY: exec.Tty,
}
// 构建ServerExec执行请求
remotecommandserver.ServeExec(
resp.ResponseWriter,
req.Request,
s.runtime,
"", // unused: podName
"", // unusued: podUID
exec.ContainerId,
exec.Cmd,
streamOpts,
s.config.StreamIdleTimeout,
s.config.StreamCreationTimeout,
s.config.SupportedRemoteCommandProtocols)
}5.4 ServerExec
ServerExec on two key steps: 1) Create a stream 2) execution of the request, the more complex is mainly concentrated in part to create a stream, we note the parameters section ExecInContainer, passing the ctx related file descriptor obtained by creating a stream Stream, createStreams inside there are two protocols to achieve websocket and https, where we analyze the main https (we use the https protocol is used kubectl)
func ServeExec(w http.ResponseWriter, req *http.Request, executor Executor, podName string, uid types.UID, container string, cmd []string, streamOpts *Options, idleTimeout, streamCreationTimeout time.Duration, supportedProtocols []string) {
// 创建serveExec
ctx, ok := createStreams(req, w, streamOpts, supportedProtocols, idleTimeout, streamCreationTimeout)
defer ctx.conn.Close()
// 获取执行，这是一个阻塞的过程，err会获取当前的执行是否成功, 这里将ctx里面的信息，都传入进去，对应的其实就是各种流
err := executor.ExecInContainer(podName, uid, container, cmd, ctx.stdinStream, ctx.stdoutStream, ctx.stderrStream, ctx.tty, ctx.resizeChan, 0)
}5.5 Creating HTTPS Stream
Stream establishment I be summed up into the following steps: 1) be https handshake 2) protocol upgraded to spdy 3) waits for the establishment stream, we turn to see
1. Complete the https handshake
protocol, err := httpstream.Handshake(req, w, supportedStreamProtocols)2. Protocol upgrade
// 流管道
streamCh := make(chan streamAndReply)
upgrader := spdy.NewResponseUpgrader()
// 构建spdy链接
conn := upgrader.UpgradeResponse(w, req, func(stream httpstream.Stream, replySent <-chan struct{}) error {
// 当新请求建立之后，会追加到streamch
streamCh <- streamAndReply{Stream: stream, replySent: replySent}
return nil
})There is a key mechanism for the transfer and the callback function func streamch back pass, established here will create a Server after a link, and was introduced to a controller func is a callback function that every time after the establishment of a link, if the acquisition to the corresponding stream will be appended to StreamCh, the following is the most complex part of the network processing, too complicated, so it is still an open bar alone
5.6 spdy stream of establishment
Overall the process looks simple, is the first major protocol switching is performed according to the request, and then returns 101, constructed and SPDY request processing based on the current link, and then wait to be established kubectl Stream transmitted through apiserver, complete flow communication with each other stream of establishment
5.6.1 protocol to enhance the response
The first step will be to enhance the response protocol, where we pay attention to several key parts, spdy agreement, as well as 101 status code
// 协议
hijacker, ok := w.(http.Hijacker)
if !ok {
errorMsg := fmt.Sprintf("unable to upgrade: unable to hijack response")
http.Error(w, errorMsg, http.StatusInternalServerError)
return nil
}
w.Header().Add(httpstream.HeaderConnection, httpstream.HeaderUpgrade)
// sydy协议
w.Header().Add(httpstream.HeaderUpgrade, HeaderSpdy31)
w.WriteHeader(http.StatusSwitchingProtocols)
5.6.2 establish spdyServer
spdyConn, err := NewServerConnection(connWithBuf, newStreamHandler)The final will be responsible for establishing new links through newConnection
func NewServerConnection(conn net.Conn, newStreamHandler httpstream.NewStreamHandler) (httpstream.Connection, error) {
// 创建一个新的链接, 通过一个已经存在的网络链接
spdyConn, err := spdystream.NewConnection(conn, true)
return newConnection(spdyConn, newStreamHandler), nil
}Here we can see is to start a background server to process requests link
func newConnection(conn *spdystream.Connection, newStreamHandler httpstream.NewStreamHandler) httpstream.Connection {
c := &connection{conn: conn, newStreamHandler: newStreamHandler}
// 当建立链接后，进行syn请求创建流的时候，会调用newSpdyStream
go conn.Serve(c.newSpdyStream)
return c
}5.6.3 Serve
1. First starts goroutine responsible plurality of processing requests, the number of worker here is 5, the queue size is 20,
frameQueues := make([]*PriorityFrameQueue, FRAME_WORKERS)
for i := 0; i < FRAME_WORKERS; i++ {
frameQueues[i] = NewPriorityFrameQueue(QUEUE_SIZE)
// Ensure frame queue is drained when connection is closed
go func(frameQueue *PriorityFrameQueue) {
<-s.closeChan
frameQueue.Drain()
}(frameQueues[i])
wg.Add(1)
go func(frameQueue *PriorityFrameQueue) {
// let the WaitGroup know this worker is done
defer wg.Done()
s.frameHandler(frameQueue, newHandler)
}(frameQueues[i])
}2. monitor synStreamFrame, diverter frame, will hash to select a corresponding frameQueues streamID frame queues according to the
case *spdy.SynStreamFrame:
if s.checkStreamFrame(frame) {
priority = frame.Priority
partition = int(frame.StreamId % FRAME_WORKERS)
debugMessage("(%p) Add stream frame: %d ", s, frame.StreamId)
// 添加到对应的StreamId对应的frame里面
s.addStreamFrame(frame)
} else {
debugMessage("(%p) Rejected stream frame: %d ", s, frame.StreamId)
continue
// 最终会讲frame push到上面的优先级队列里面
frameQueues[partition].Push(readFrame, priority)3. Read the reading frame and to be transmitted to the stream by the above StreamCH newHandler
func (s *Connection) frameHandler(frameQueue *PriorityFrameQueue, newHandler StreamHandler) {
for {
popFrame := frameQueue.Pop()
if popFrame == nil {
return
}
var frameErr error
switch frame := popFrame.(type) {
case *spdy.SynStreamFrame:
frameErr = s.handleStreamFrame(frame, newHandler)
}
}The next section of the flow consumption
5.7 waiting establish Stream
Stream of waiting for the establishment of mainly achieved through Headers inside StreamType, there speaks the corresponding stdinStream and corresponding spdy inside the stream binding, other types, too
func (*v3ProtocolHandler) waitForStreams(streams <-chan streamAndReply, expectedStreams int, expired <-chan time.Time) (*context, error) {
ctx := &context{}
receivedStreams := 0
replyChan := make(chan struct{})
stop := make(chan struct{})
defer close(stop)
WaitForStreams:
for {
select {
case stream := <-streams:
streamType := stream.Headers().Get(api.StreamType)
switch streamType {
case api.StreamTypeError:
ctx.writeStatus = v1WriteStatusFunc(stream)
go waitStreamReply(stream.replySent, replyChan, stop)
case api.StreamTypeStdin:
ctx.stdinStream = stream
go waitStreamReply(stream.replySent, replyChan, stop)
case api.StreamTypeStdout:
ctx.stdoutStream = stream
go waitStreamReply(stream.replySent, replyChan, stop)
case api.StreamTypeStderr:
ctx.stderrStream = stream
go waitStreamReply(stream.replySent, replyChan, stop)
case api.StreamTypeResize:
ctx.resizeStream = stream
go waitStreamReply(stream.replySent, replyChan, stop)
default:
runtime.HandleError(fmt.Errorf("unexpected stream type: %q", streamType))
}
case <-replyChan:
receivedStreams++
if receivedStreams == expectedStreams {
break WaitForStreams
}
case <-expired:
// TODO find a way to return the error to the user. Maybe use a separate
// stream to report errors?
return nil, errors.New("timed out waiting for client to create streams")
}
}
return ctx, nil
}5.8 CRI adapter performs command
Trace invocation chain may be seen as the final call, eventually a point execHandler.ExecInContainer interface to execute commands in a vessel
func (a *criAdapter) ExecInContainer(podName string, podUID types.UID, container string, cmd []string, in io.Reader, out, err io.WriteCloser, tty bool, resize <-chan remotecommand.TerminalSize, timeout time.Duration) error {
// 执行command
return a.Runtime.Exec(container, cmd, in, out, err, tty, resize)
}
func (r *streamingRuntime) Exec(containerID string, cmd []string, in io.Reader, out, err io.WriteCloser, tty bool, resize <-chan remotecommand.TerminalSize) error {
// 执行容器
return r.exec(containerID, cmd, in, out, err, tty, resize, 0)
}
// Internal version of Exec adds a timeout.
func (r *streamingRuntime) exec(containerID string, cmd []string, in io.Reader, out, errw io.WriteCloser, tty bool, resize <-chan remotecommand.TerminalSize, timeout time.Duration) error {
// exechandler
return r.execHandler.ExecInContainer(r.client, container, cmd, in, out, errw, tty, resize, timeout)
}5.9 main command execution process
The main point of order flow is divided into two parts: 1) Create a mission exec 2) Start exec mission
func (*NativeExecHandler) ExecInContainer(client libdocker.Interface, container *dockertypes.ContainerJSON, cmd []string, stdin io.Reader, stdout, stderr io.WriteCloser, tty bool, resize <-chan remotecommand.TerminalSize, timeout time.Duration) error {
// 在容器中执行命令
done := make(chan struct{})
defer close(done)
// 执行命令
createOpts := dockertypes.ExecConfig{
Cmd: cmd,
AttachStdin: stdin != nil,
AttachStdout: stdout != nil,
AttachStderr: stderr != nil,
Tty: tty,
}
// 创建执行命令任务
execObj, err := client.CreateExec(container.ID, createOpts)
startOpts := dockertypes.ExecStartCheck{Detach: false, Tty: tty}
// 这里我们可以看到我们将前面获取到的stream的封装,都作为FD传入到容器的执行命令里面去了
streamOpts := libdocker.StreamOptions{
InputStream: stdin,
OutputStream: stdout,
ErrorStream: stderr,
RawTerminal: tty,
ExecStarted: execStarted,
}
// 执行命令
err = client.StartExec(execObj.ID, startOpts, streamOpts)
ticker := time.NewTicker(2 * time.Second)
defer ticker.Stop()
count := 0
for {
// 获取执行结果
inspect, err2 := client.InspectExec(execObj.ID)
if !inspect.Running {
if inspect.ExitCode != 0 {
err = &dockerExitError{inspect}
}
break
}
<-ticker.C
}
return err
}Docker's command interface calls
func (cli *Client) ContainerExecCreate(ctx context.Context, container string, config types.ExecConfig) (types.IDResponse, error) {
resp, err := cli.post(ctx, "/containers/"+container+"/exec", nil, config, nil)
return response, err
}5.10 command execution core implementation
Command execution core to achieve two main steps: 1) first sends execution request exec 2) and starts the corresponding exec acquisition result, complex or SPDY associated logic Stream
func (d *kubeDockerClient) StartExec(startExec string, opts dockertypes.ExecStartCheck, sopts StreamOptions) error {
// 启动执行命令, 获取结果
resp, err := d.client.ContainerExecAttach(ctx, startExec, dockertypes.ExecStartCheck{
Detach: opts.Detach,
Tty: opts.Tty,
})
// 将输入流拷贝到输出流， 这里会讲resp里面的结果拷贝到outputSTream里面
return d.holdHijackedConnection(sopts.RawTerminal || opts.Tty, sopts.InputStream, sopts.OutputStream, sopts.ErrorStream, resp)
}5.10.1 command execution request
cli.postHijacked(ctx, "/exec/"+execID+"/start", nil, config, headers)5.10.2 Send Request connection
Here HiHijackConn with similar functions previously introduced, the core is through the establishment of an http link, and then upgrade agreement, which is the underlying tcp conn link, and a link back to the corresponding current is set Keepliave 30s, we have this a two-way communication link based spdy
func (cli *Client) postHijacked(ctx context.Context, path string, query url.Values, body interface{}, headers map[string][]string) (types.HijackedResponse, error) {
conn, err := cli.setupHijackConn(ctx, req, "tcp")
return types.HijackedResponse{Conn: conn, Reader: bufio.NewReader(conn)}, err
}5.10.3 establish flow Duikao
At this point in kubelet we get to the top there and apiserver Stream Stream established to execute the command with the backend, then you only need to copy two streams direct transmission of data can be achieved
func (d *kubeDockerClient) holdHijackedConnection(tty bool, inputStream io.Reader, outputStream, errorStream io.Writer, resp dockertypes.HijackedResponse) error {
receiveStdout := make(chan error)
if outputStream != nil || errorStream != nil {
// 将响应结果拷贝到outputstream里面
go func() {
receiveStdout <- d.redirectResponseToOutputStream(tty, outputStream, errorStream, resp.Reader)
}()
}
stdinDone := make(chan struct{})
go func() {
if inputStream != nil {
io.Copy(resp.Conn, inputStream)
}
resp.CloseWrite()
close(stdinDone)
}()
return nil
}
5.10.4 detection execution status
After the completion of command execution occurs, it will conduct an inspection of the implementation status of every 2s bell, if it is found executed, it will exit
func (*NativeExecHandler) ExecInContainer(client libdocker.Interface, container *dockertypes.ContainerJSON, cmd []string, stdin io.Reader, stdout, stderr io.WriteCloser, tty bool, resize <-chan remotecommand.TerminalSize, timeout time.Duration) error {
ticker := time.NewTicker(2 * time.Second)
defer ticker.Stop()
count := 0
for {
// 获取执行结果
inspect, err2 := client.InspectExec(execObj.ID)
if err2 != nil {
return err2
}
if !inspect.Running {
if inspect.ExitCode != 0 {
err = &dockerExitError{inspect}
}
break
}
<-ticker.C
}
return err
}
func (cli *Client) ContainerExecInspect(ctx context.Context, execID string) (types.ContainerExecInspect, error) {
resp, err := cli.get(ctx, "/exec/"+execID+"/json", nil, nil)
return response, err
}6. Summary
The whole process of implementation of the order but it is still quite complex, mainly lies in that part of the network protocol switching, we can see that in fact throughout the process, are based on the SPDY protocol to carry out, and we can also see in that part of CRI.RuntimeService Stream processing of the request is actually more than goroutine concurrent admiration at the big cow designs, what is written in the wrong place, welcome to discuss, thank bigwigs can see here
kubernetes Study Notes Address: https://www.yuque.com/baxiaoshi/tyado3