Read the principle of Kubernetes APIServer in one article

Preface

The entire Kubernetes technology system is composed of declarative API and Controller, and kube-apiserver is the declarative api server of Kubernetes, and provides a bridge for the interaction of other components. Therefore, it is very important to deepen the understanding of kube-apiserver.

Overall component function

kube-apiserver, as the only entry for etcd operation of the entire Kubernetes cluster, is responsible for the authentication & authentication, verification and CRUD operations of Kubernetes resources, and provides RESTful APIs for other components to call:

kube-apiserver contains three APIServers:

• aggregatorServer : responsible for handling apiregistration.k8s.ioAPIService resource requests under the group, while the request from the user to intercept forwarded to the aggregated server (AA)
• kubeAPIServer : Responsible for some general processing of requests, including: authentication, authentication, and REST services of various built-in resources (pod, deployment, service and etc), etc.
• apiExtensionsServer : Responsible for the registration of CustomResourceDefinition (CRD) apiResources and apiVersions, and handles CRD and the corresponding CustomResource (CR) REST request (if the corresponding CR cannot be processed, it will return 404), which is also the last link of apiserver Delegation

It also includes bootstrap-controller, which is mainly responsible for the creation and management of Kubernetes default apiserver service.

The following will give an overview of the above components.

bootstrap-controller

• The apiserver bootstrap-controller creation & operation logic is in the k8s.io/kubernetes/pkg/master directory
• bootstrap-controller is mainly used to create and maintain internal kubernetes default apiserver service
• kubernetes default apiserver service spec.selector is empty. This is the biggest difference between the default apiserver service and other normal services. It indicates that the endpoints corresponding to this special service are not controlled by the endpoints controller, but directly managed by the kube-apiserver bootstrap-controller (maintained by this code, not by the pod selector)
• The main functions of bootstrap-controller are as follows:
  • Create default, kube-system and kube-public and kube-node-lease namespaces
  • Create & maintain kubernetes default apiserver service and corresponding endpoint
  • Provide inspection and repair functions based on Service ClusterIP ( --service-cluster-ip-rangespecified range)
  • Provide inspection and repair functions based on Service NodePort ( --service-node-port-rangespecified range)

// k8s.io/kubernetes/pkg/master/controller.go:142
// Start begins the core controller loops that must exist for bootstrapping
// a cluster.
func (c *Controller) Start() {
    if c.runner != nil {
        return
    }
    // Reconcile during first run removing itself until server is ready.
    endpointPorts := createEndpointPortSpec(c.PublicServicePort, "https", c.ExtraEndpointPorts)
    if err := c.EndpointReconciler.RemoveEndpoints(kubernetesServiceName, c.PublicIP, endpointPorts); err != nil {
        klog.Errorf("Unable to remove old endpoints from kubernetes service: %v", err)
    }
    repairClusterIPs := servicecontroller.NewRepair(c.ServiceClusterIPInterval, c.ServiceClient, c.EventClient, &c.ServiceClusterIPRange, c.ServiceClusterIPRegistry, &c.SecondaryServiceClusterIPRange, c.SecondaryServiceClusterIPRegistry)
    repairNodePorts := portallocatorcontroller.NewRepair(c.ServiceNodePortInterval, c.ServiceClient, c.EventClient, c.ServiceNodePortRange, c.ServiceNodePortRegistry)
    // run all of the controllers once prior to returning from Start.
    if err := repairClusterIPs.RunOnce(); err != nil {
        // If we fail to repair cluster IPs apiserver is useless. We should restart and retry.
        klog.Fatalf("Unable to perform initial IP allocation check: %v", err)
    }
    if err := repairNodePorts.RunOnce(); err != nil {
        // If we fail to repair node ports apiserver is useless. We should restart and retry.
        klog.Fatalf("Unable to perform initial service nodePort check: %v", err)
    }
    // 定期执行bootstrap controller主要的四个功能(reconciliation)  
    c.runner = async.NewRunner(c.RunKubernetesNamespaces, c.RunKubernetesService, repairClusterIPs.RunUntil, repairNodePorts.RunUntil)
    c.runner.Start()
}

For more code principle details, please refer to kubernetes-reading-notes .

kubeAPIServer

KubeAPIServer mainly provides operation requests for built-in API Resources, registers routing information for each API Resources in Kubernetes, and exposes RESTful API at the same time, so that services in and outside the cluster can operate resources in Kubernetes through RESTful API

In addition, kubeAPIServer is the core of the entire Kubernetes apiserver. The aggregatorServer and apiExtensionsServer that will be described below are all based on kubeAPIServer for extension (complements Kubernetes' ability to support user-defined resources)

The core function of kubeAPIServer is to add routes to Kubernetes built-in resources, as follows:

• Call m.InstallLegacyAPIto add a route to the core API Resources, in apiserver That is the /apibeginning of the resource;
• Call m.InstallAPIsthe extended API Resources added to the route, in apiserver That is the /apisbeginning of the resource;

// k8s.io/kubernetes/pkg/master/master.go:332
// New returns a new instance of Master from the given config.
// Certain config fields will be set to a default value if unset.
// Certain config fields must be specified, including:
//   KubeletClientConfig
func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*Master, error) {
    ...
    // 安装 LegacyAPI(core API)
    // install legacy rest storage
    if c.ExtraConfig.APIResourceConfigSource.VersionEnabled(apiv1.SchemeGroupVersion) {
        legacyRESTStorageProvider := corerest.LegacyRESTStorageProvider{
            StorageFactory:              c.ExtraConfig.StorageFactory,
            ProxyTransport:              c.ExtraConfig.ProxyTransport,
            KubeletClientConfig:         c.ExtraConfig.KubeletClientConfig,
            EventTTL:                    c.ExtraConfig.EventTTL,
            ServiceIPRange:              c.ExtraConfig.ServiceIPRange,
            SecondaryServiceIPRange:     c.ExtraConfig.SecondaryServiceIPRange,
            ServiceNodePortRange:        c.ExtraConfig.ServiceNodePortRange,
            LoopbackClientConfig:        c.GenericConfig.LoopbackClientConfig,
            ServiceAccountIssuer:        c.ExtraConfig.ServiceAccountIssuer,
            ServiceAccountMaxExpiration: c.ExtraConfig.ServiceAccountMaxExpiration,
            APIAudiences:                c.GenericConfig.Authentication.APIAudiences,
        }
        if err := m.InstallLegacyAPI(&c, c.GenericConfig.RESTOptionsGetter, legacyRESTStorageProvider); err != nil {
            return nil, err
        }
    }
    ...
    // 安装 APIs(named groups apis)
    if err := m.InstallAPIs(c.ExtraConfig.APIResourceConfigSource, c.GenericConfig.RESTOptionsGetter, restStorageProviders...); err != nil {
        return nil, err
    }
    ...
    return m, nil
}

The entire kubeAPIServer provides three types of API Resource interfaces:

• core group: mainly /api/v1at;
• named groups: its path is /apis/$GROUP/$VERSION;
• Some APIs of system status: such as /metrics, /versionetc.;

The URL and the API substantially in /apis/{group}/{version}/namespaces/{namespace}/resource/{name}composition, structure as shown below:

kubeAPIServer will create a corresponding RESTStorage for each API resource. The purpose of RESTStorage is to correspond the access path of each resource and its back-end storage operations: through the constructed REST Storage interface to determine which operations the resource can perform (such as : Create, update, etc.), and store its corresponding operations in action. Each operation corresponds to a standard REST method. For example, create corresponds to REST method as POST, and update corresponds to REST method as PUT. Finally, traverse sequentially according to the actions array, add a handler to each operation (handler corresponds to the relevant interface implemented by REST Storage), and register it to the route, and finally provide RESTful API externally, as follows:

// m.GenericAPIServer.InstallLegacyAPIGroup --> s.installAPIResources --> apiGroupVersion.InstallREST --> installer.Install --> a.registerResourceHandlers
// k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints/installer.go:181
func (a *APIInstaller) registerResourceHandlers(path string, storage rest.Storage, ws *restful.WebService) (*metav1.APIResource, error) {
    ...
    // 1、判断该 resource 实现了哪些 REST 操作接口，以此来判断其支持的 verbs 以便为其添加路由
    // what verbs are supported by the storage, used to know what verbs we support per path
    creater, isCreater := storage.(rest.Creater)
    namedCreater, isNamedCreater := storage.(rest.NamedCreater)
    lister, isLister := storage.(rest.Lister)
    getter, isGetter := storage.(rest.Getter)
    ...
    // 2、为 resource 添加对应的 actions(+根据是否支持 namespace)
    // Get the list of actions for the given scope.
    switch {
    case !namespaceScoped:
        // Handle non-namespace scoped resources like nodes.
        resourcePath := resource
        resourceParams := params
        itemPath := resourcePath + "/{name}"
        nameParams := append(params, nameParam)
        proxyParams := append(nameParams, pathParam)
        ...
        // Handler for standard REST verbs (GET, PUT, POST and DELETE).
        // Add actions at the resource path: /api/apiVersion/resource
        actions = appendIf(actions, action{"LIST", resourcePath, resourceParams, namer, false}, isLister)
        actions = appendIf(actions, action{"POST", resourcePath, resourceParams, namer, false}, isCreater)
        ...
    }
    ...
    // 3、从 rest.Storage 到 restful.Route 映射
    // 为每个操作添加对应的 handler
    for _, action := range actions {
        ...
        switch action.Verb {
        ...
        case "POST": // Create a resource.
            var handler restful.RouteFunction
            // 4、初始化 handler
            if isNamedCreater {
                handler = restfulCreateNamedResource(namedCreater, reqScope, admit)
            } else {
                handler = restfulCreateResource(creater, reqScope, admit)
            }
            handler = metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, handler)
            ...
            // 5、route 与 handler 进行绑定    
            route := ws.POST(action.Path).To(handler).
                Doc(doc).
                Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
                Operation("create"+namespaced+kind+strings.Title(subresource)+operationSuffix).
                Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
                Returns(http.StatusOK, "OK", producedObject).
                // TODO: in some cases, the API may return a v1.Status instead of the versioned object
                // but currently go-restful can't handle multiple different objects being returned.
                Returns(http.StatusCreated, "Created", producedObject).
                Returns(http.StatusAccepted, "Accepted", producedObject).
                Reads(defaultVersionedObject).
                Writes(producedObject)
            if err := AddObjectParams(ws, route, versionedCreateOptions); err != nil {
                return nil, err
            }
            addParams(route, action.Params)
            // 6、添加到路由中    
            routes = append(routes, route)
        case "DELETE": // Delete a resource.
        ...
        default:
            return nil, fmt.Errorf("unrecognized action verb: %s", action.Verb)
        }
        for _, route := range routes {
            route.Metadata(ROUTE_META_GVK, metav1.GroupVersionKind{
                Group:   reqScope.Kind.Group,
                Version: reqScope.Kind.Version,
                Kind:    reqScope.Kind.Kind,
            })
            route.Metadata(ROUTE_META_ACTION, strings.ToLower(action.Verb))
            ws.Route(route)
        }
        // Note: update GetAuthorizerAttributes() when adding a custom handler.
    }
    ...
}

The kubeAPIServer code structure is organized as follows:

1. apiserver整体启动逻辑 k8s.io/kubernetes/cmd/kube-apiserver
2. apiserver bootstrap-controller创建&运行逻辑 k8s.io/kubernetes/pkg/master
3. API Resource对应后端RESTStorage(based on genericregistry.Store)创建k8s.io/kubernetes/pkg/registry
4. aggregated-apiserver创建&处理逻辑 k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator
5. extensions-apiserver创建&处理逻辑 k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver
6. apiserver创建&运行 k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/server
7. 注册API Resource资源处理handler(InstallREST&Install®isterResourceHandlers) k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints
8. 创建存储后端(etcdv3) k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/storage
9. genericregistry.Store.CompleteWithOptions初始化 k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/registry

The call chain is organized as follows:

For more code principle details, please refer to kubernetes-reading-notes .

aggregatorServer

aggregatorServer is mainly used to process the second method Aggregated APIServer (AA), which extends Kubernetes API Resources, and proxy CR requests to AA:

Here combined with the aggregated apiserver example sample-apiserver given by the Kubernetes official , the summary principle is as follows:

• AggregatorServer associates with a Service through the APIServices object to forward the request, and its associated Service type further determines the form of request forwarding. AggregatorServer includes one GenericAPIServerand maintains its own state Controller. Which GenericAPIServermainly handles apiregistration.k8s.ioAPIService resource requests under the group, and the Controller includes:

  • apiserviceRegistrationController: Responsible for building a proxy based on the aggregated server service defined by APIService, and forwarding CR requests to the aggregated server on the backend
  • availableConditionController: Maintain the availability status of APIServices, including whether the referenced Service is available, etc.;
  • autoRegistrationController: Used to maintain a set of specific APIServices in the API;
  • crdRegistrationController: Responsible for automatically registering CRD GroupVersions to APIServices;
  • openAPIAggregationController: Synchronize the changes of APIServices resources to the provided OpenAPI document;

• The apiserviceRegistrationController is responsible for building a proxy based on the aggregated server service defined by APIService, and forwarding the request of CR to the aggregated server on the backend. There are two types of apiService: Local (Service is empty) and Service (Service is not empty). apiserviceRegistrationController is responsible for setting up proxy for these two types of apiService: Local type will be directly routed to kube-apiserver for processing; while Service type will set up proxy and convert the request into a request for aggregated Service (proxyPath := "/apis/" + apiService.Spec.Group + "/" + apiService.Spec.Version), and the requested load balancing policy is to prioritize local access to kube-apiserver (if service is kubernetes default apiserver service:443) => access via service ClusterIP:Port (Default) Or access by randomly selecting service endpoint backend:

  func (s *APIAggregator) AddAPIService(apiService *v1.APIService) error {
  ...
    proxyPath := "/apis/" + apiService.Spec.Group + "/" + apiService.Spec.Version
    // v1. is a special case for the legacy API.  It proxies to a wider set of endpoints.
    if apiService.Name == legacyAPIServiceName {
        proxyPath = "/api"
    }
    // register the proxy handler
    proxyHandler := &proxyHandler{
        localDelegate:   s.delegateHandler,
        proxyClientCert: s.proxyClientCert,
        proxyClientKey:  s.proxyClientKey,
        proxyTransport:  s.proxyTransport,
        serviceResolver: s.serviceResolver,
        egressSelector:  s.egressSelector,
    }
  ...
    s.proxyHandlers[apiService.Name] = proxyHandler
    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle(proxyPath, proxyHandler)
    s.GenericAPIServer.Handler.NonGoRestfulMux.UnlistedHandlePrefix(proxyPath+"/", proxyHandler)
  ...
    // it's time to register the group aggregation endpoint
    groupPath := "/apis/" + apiService.Spec.Group
    groupDiscoveryHandler := &apiGroupHandler{
        codecs:    aggregatorscheme.Codecs,
        groupName: apiService.Spec.Group,
        lister:    s.lister,
        delegate:  s.delegateHandler,
    }
    // aggregation is protected
    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle(groupPath, groupDiscoveryHandler)
    s.GenericAPIServer.Handler.NonGoRestfulMux.UnlistedHandle(groupPath+"/", groupDiscoveryHandler)
    s.handledGroups.Insert(apiService.Spec.Group)
    return nil
  }
  // k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:109
  func (r *proxyHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {
    // 加载roxyHandlingInfo处理请求  
    value := r.handlingInfo.Load()
    if value == nil {
        r.localDelegate.ServeHTTP(w, req)
        return
    }
    handlingInfo := value.(proxyHandlingInfo)
  ...
    // 判断APIService服务是否正常
    if !handlingInfo.serviceAvailable {
        proxyError(w, req, "service unavailable", http.StatusServiceUnavailable)
        return
    }
    // 将原始请求转化为对APIService的请求
    // write a new location based on the existing request pointed at the target service
    location := &url.URL{}
    location.Scheme = "https"
    rloc, err := r.serviceResolver.ResolveEndpoint(handlingInfo.serviceNamespace, handlingInfo.serviceName, handlingInfo.servicePort)
    if err != nil {
        klog.Errorf("error resolving %s/%s: %v", handlingInfo.serviceNamespace, handlingInfo.serviceName, err)
        proxyError(w, req, "service unavailable", http.StatusServiceUnavailable)
        return
    }
    location.Host = rloc.Host
    location.Path = req.URL.Path
    location.RawQuery = req.URL.Query().Encode()
    newReq, cancelFn := newRequestForProxy(location, req)
    defer cancelFn()
   ...
    proxyRoundTripper = transport.NewAuthProxyRoundTripper(user.GetName(), user.GetGroups(), user.GetExtra(), proxyRoundTripper)
    handler := proxy.NewUpgradeAwareHandler(location, proxyRoundTripper, true, upgrade, &responder{w: w})
    handler.ServeHTTP(w, newReq)
  }

  $ kubectl get APIService           
  NAME                                   SERVICE                      AVAILABLE   AGE
  ...
  v1.apps                                Local                        True        50d
  ...
  v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        50d
  ...

  # default APIServices
  $ kubectl get -o yaml APIService/v1.apps
  apiVersion: apiregistration.k8s.io/v1
  kind: APIService
  metadata:
  labels:
    kube-aggregator.kubernetes.io/automanaged: onstart
  name: v1.apps
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1.apps
  spec:
  group: apps
  groupPriorityMinimum: 17800
  version: v1
  versionPriority: 15
  status:
  conditions:
  - lastTransitionTime: "2020-10-20T10:39:48Z"
    message: Local APIServices are always available
    reason: Local
    status: "True"
    type: Available
  
  # aggregated server    
  $ kubectl get -o yaml APIService/v1beta1.metrics.k8s.io
  apiVersion: apiregistration.k8s.io/v1
  kind: APIService
  metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: v1beta1.metrics.k8s.io
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
    port: 443
  version: v1beta1
  versionPriority: 100
  status:
  conditions:
  - lastTransitionTime: "2020-12-05T00:50:48Z"
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available
  
  # CRD
  $ kubectl get -o yaml APIService/v1.duyanghao.example.com
  apiVersion: apiregistration.k8s.io/v1
  kind: APIService
  metadata:
  labels:
    kube-aggregator.kubernetes.io/automanaged: "true"
  name: v1.duyanghao.example.com
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1.duyanghao.example.com
  spec:
  group: duyanghao.example.com
  groupPriorityMinimum: 1000
  version: v1
  versionPriority: 100
  status:
  conditions:
  - lastTransitionTime: "2020-12-11T08:45:37Z"
    message: Local APIServices are always available
    reason: Local
    status: "True"
    type: Available

• During the creation of aggregatorServer, a default APIService list will be created based on all API resources defined by kube-apiserver. The name is $VERSION.$GROUP, these APIServices will have tags kube-aggregator.kubernetes.io/automanaged: onstart, for example: v1.apps apiService. autoRegistrationController creates and maintains the APIService in these lists, which is the Local apiService we see; for custom APIService (aggregated server), it will not be processed

• The aggregated server implements the CRUD API interface of CR (custom API resources), and can flexibly choose back-end storage. It can share etcd with core kube-apiserver, or deploy etcd database or other databases independently. The CR API path implemented by the aggregated server is: /apis/$GROUP/$VERSION, the specific sample apiserver is: /apis/wardle.example.com/v1alpha1, the following resource types are: flunders and fischers

• The aggregated server implements integration and interaction with core kube-apiserver by deploying APIService type resources, and service fields point to the corresponding aggregated server service

• The sample-apiserver directory structure is as follows, you can refer to writing your own aggregated server:

  staging/src/k8s.io/sample-apiserver
  ├── artifacts
  │   ├── example
  │   │   ├── apiservice.yaml
      ...
  ├── hack
  ├── main.go
  └── pkg
  ├── admission
  ├── apis
  ├── apiserver
  ├── cmd
  ├── generated
  │   ├── clientset
  │   │   └── versioned
              ...
  │   │       └── typed
  │   │           └── wardle
  │   │               ├── v1alpha1
  │   │               └── v1beta1
  │   ├── informers
  │   │   └── externalversions
  │   │       └── wardle
  │   │           ├── v1alpha1
  │   │           └── v1beta1
  │   ├── listers
  │   │   └── wardle
  │   │       ├── v1alpha1
  │   │       └── v1beta1
  └── registry

  • Among them, artifacts are used to deploy yaml examples
  • The hack directory stores automatic scripts (eg: update-codegen)
  • main.go is the start entry of the aggregated server; pkg/cmd is responsible for starting the specific logic of the aggregated server; pkg/apiserver is used for the initialization of the aggregated server and routing registration
  • pkg/apis is responsible for the definition of the structure of the relevant CR, which is automatically generated (update-codegen)
  • pkg/admission is responsible for the relevant code for admission
  • pkg/generated is responsible for generating clientset, informers, and listeners to access CR
  • The pkg/registry directory is responsible for CR-related RESTStorage implementation

For more code principle details, please refer to kubernetes-reading-notes .

apiExtensionsServer

apiExtensionsServer is mainly responsible for the registration of CustomResourceDefinition (CRD) apiResources and apiVersions, and also processes CRD and corresponding CustomResource (CR) REST requests (if the corresponding CR cannot be processed, it will return 404), and it is also the last link of apiserver Delegation

The principle is summarized as follows:

• Custom Resource, referred to as CR, is a Kubernetes custom resource type, which corresponds to various built-in resource types in Kubernetes, such as Pod, Service, etc. Using CR we can define any resource type we want

• CRD registers CR with Kubernetes in the form of yaml files to implement custom api-resources, which is the second way to extend Kubernetes API resources and is also a commonly used method

• APIExtensionServer is responsible for the registration of CustomResourceDefinition (CRD) apiResources and apiVersions. It also processes CRD and the corresponding CustomResource (CR) REST request (if the corresponding CR cannot be processed, it will return 404), which is also the last link of apiserver Delegation

• crdRegistrationControllerResponsible for automatically registering CRD GroupVersions to APIServices. The specific logic is: enumerate all CRDs, then construct an APIService according to the crd.Spec.Group and crd.Spec.Versions fields defined by the CRD, and add it to autoRegisterController.apiServicesToSync, and autoRegisterController will create and maintain it. This is why the corresponding APIService object will be generated after the CRD is created

• The controller and functions included in APIExtensionServer are as follows:

  • openapiController: The OpenAPI document changes crd resources to provide synchronization can be accessed through the /openapi/v2view;

  • crdController: Crd responsible for information and to register apiVersions apiResources, the information both through kubectl api-versionsand kubectl api-resourcessee;

  • kubectl api-versionsThe command returns the version information of all Kubernetes cluster resources (actually two requests were issued, one https://127.0.0.1:6443/apiand one respectively https://127.0.0.1:6443/apis, and the return results of the two requests were merged at the end)

  $ kubectl -v=8 api-versions 
  I1211 11:44:50.276446   22493 loader.go:375] Config loaded from file:  /root/.kube/config
  I1211 11:44:50.277005   22493 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s
  ...
  I1211 11:44:50.290265   22493 request.go:1068] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"x.x.x.x:6443"}]}
  I1211 11:44:50.293673   22493 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s
  ...
  I1211 11:44:50.298360   22493 request.go:1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1"," [truncated 4985 chars]
  apiextensions.k8s.io/v1
  apiextensions.k8s.io/v1beta1
  apiregistration.k8s.io/v1
  apiregistration.k8s.io/v1beta1
  apps/v1
  authentication.k8s.io/v1beta1
  ...
  storage.k8s.io/v1
  storage.k8s.io/v1beta1
  v1
  

  • kubectl api-resourcesThe command is to first obtain all API version information, and then call the interface for each API version to obtain all API resource types under that version

    $ kubectl -v=8 api-resources
    5077 loader.go:375] Config loaded from file:  /root/.kube/config
    I1211 15:19:47.593450   15077 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s
    I1211 15:19:47.602273   15077 request.go:1068] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"x.x.x.x:6443"}]}
    I1211 15:19:47.606279   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s
    I1211 15:19:47.610333   15077 request.go:1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1"," [truncated 4985 chars]
    I1211 15:19:47.614700   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/batch/v1?timeout=32s
    I1211 15:19:47.614804   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/authentication.k8s.io/v1?timeout=32s
    I1211 15:19:47.615687   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/auth.tkestack.io/v1?timeout=32s
    https://127.0.0.1:6443/apis/authentication.k8s.io/v1beta1?timeout=32s
    I1211 15:19:47.616794   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/coordination.k8s.io/v1?timeout=32s
    I1211 15:19:47.616863   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/apps/v1?timeout=32s
    ...
    NAME                              SHORTNAMES   APIGROUP                       NAMESPACED   KIND
    bindings                                                                      true         Binding
    endpoints                         ep                                          true         Endpoints
    events                            ev                                          true         Event
    limitranges                       limits                                      true         LimitRange
    namespaces                        ns                                          false        Namespace
    nodes                             no                                          false        Node
    ...

    • namingController: Check whether there is a naming conflict in crd obj, which can be checked in crd .status.conditions;

    • establishingController：Check whether crd is in normal state, you can check it in crd .status.conditions;

    • nonStructuralSchemaController：Check if the crd obj structure is normal, you can .status.conditionsview it in crd ;

    • apiApprovalController: Check whether crd complies with the Kubernetes API declaration policy, which can be checked in crd .status.conditions;

    • finalizingController: Similar to the function of finalizes, related to the deletion of CRs;

• The processing logic of CR CRUD APIServer is summarized as follows:

  • createAPIExtensionsServer=>NewCustomResourceDefinitionHandler=>crdHandler=>Register CR CRUD API interface:

  // New returns a new instance of CustomResourceDefinitions from the given config.
  func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*CustomResourceDefinitions, error) {
    ...
      crdHandler, err := NewCustomResourceDefinitionHandler(
        versionDiscoveryHandler,
          groupDiscoveryHandler,
        s.Informers.Apiextensions().V1().CustomResourceDefinitions(),
          delegateHandler,
        c.ExtraConfig.CRDRESTOptionsGetter,
          c.GenericConfig.AdmissionControl,
        establishingController,
          c.ExtraConfig.ServiceResolver,
        c.ExtraConfig.AuthResolverWrapper,
          c.ExtraConfig.MasterCount,
          s.GenericAPIServer.Authorizer,
          c.GenericConfig.RequestTimeout,
          time.Duration(c.GenericConfig.MinRequestTimeout)*time.Second,
          apiGroupInfo.StaticOpenAPISpec,
          c.GenericConfig.MaxRequestBodyBytes,
      )
      if err != nil {
          return nil, err
      }
      s.GenericAPIServer.Handler.NonGoRestfulMux.Handle("/apis", crdHandler)
      s.GenericAPIServer.Handler.NonGoRestfulMux.HandlePrefix("/apis/", crdHandler)
      ...
      return s, nil
  }
  

  • The crdHandler processing logic is as follows:

  • Analyze req (GET /apis/duyanghao.example.com/v1/namespaces/default/students), and obtain the corresponding CRD according to the group (duyanghao.example.com), version (v1), and resource field (students) in the request path Content(crd, err := r.crdLister.Get(crdName))

  • Obtain crdInfo through crd.UID and crd.Name, if not exist, create the corresponding crdInfo(crdInfo, err := r.getOrCreateServingInfoFor(crd.UID, crd.Name)). crdInfo contains the CRD definition and the CRD corresponding to the custom resource of the Custom Resource.REST storage

  • The customresource.REST storage is created by the Group (duyanghao.example.com), Version (v1), Kind (Student), Resource (students), etc. corresponding to CR. Since CR does not have a specific structure definition in the Kubernetes code, so here It will first initialize a generic structure Unstructured (used to save all types of Custom Resource), and perform SetGroupVersionKind operation on the structure (set the specific Custom Resource Type)

  • After obtaining the Unstructured structure from customresource.REST storage, it will be converted and returned

    // k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:223
    func (r *crdHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {
    ctx := req.Context()
    requestInfo, ok := apirequest.RequestInfoFrom(ctx)
    ...
    crdName := requestInfo.Resource + "." + requestInfo.APIGroup
    crd, err := r.crdLister.Get(crdName)
    ...
    crdInfo, err := r.getOrCreateServingInfoFor(crd.UID, crd.Name)
    verb := strings.ToUpper(requestInfo.Verb)
    resource := requestInfo.Resource
    subresource := requestInfo.Subresource
    scope := metrics.CleanScope(requestInfo)
    ...
    switch {
    case subresource == "status" && subresources != nil && subresources.Status != nil:
        handlerFunc = r.serveStatus(w, req, requestInfo, crdInfo, terminating, supportedTypes)
    case subresource == "scale" && subresources != nil && subresources.Scale != nil:
        handlerFunc = r.serveScale(w, req, requestInfo, crdInfo, terminating, supportedTypes)
    case len(subresource) == 0:
        handlerFunc = r.serveResource(w, req, requestInfo, crdInfo, terminating, supportedTypes)
    default:
        responsewriters.ErrorNegotiated(
            apierrors.NewNotFound(schema.GroupResource{Group: requestInfo.APIGroup, Resource: requestInfo.Resource}, requestInfo.Name),
            Codecs, schema.GroupVersion{Group: requestInfo.APIGroup, Version: requestInfo.APIVersion}, w, req,
        )
    }
    if handlerFunc != nil {
        handlerFunc = metrics.InstrumentHandlerFunc(verb, requestInfo.APIGroup, requestInfo.APIVersion, resource, subresource, scope, metrics.APIServerComponent, handlerFunc)
        handler := genericfilters.WithWaitGroup(handlerFunc, longRunningFilter, crdInfo.waitGroup)
        handler.ServeHTTP(w, req)
        return
    }
    }
    

For more code principle details, please refer to kubernetes-reading-notes .

Conclusion

This article summarizes the Kubernetes apiserver from the source code level, including: aggregatorServer, kubeAPIServer, apiExtensionsServer, and bootstrap-controller. By reading this article, you can have a general understanding of the internal principles of apiserver, and also help follow-up in-depth research

Refs

• kubernetes-reading-notes