Kubernetes API Server source code learning (1): API Server architecture design, API Server startup process, APIObject loading, Scheme details, GenericAPIServer

This article is based on source code learning of Kubernetes v1.22.4 version

1. API Server architecture design

API Server consists of 3 HTTP Servers:

• AggregatorServer: The exposed function is similar to a seven-layer load balancing, intercepting and forwarding requests from users to other servers, and is responsible for the Discovery function of the entire APIServer
• KubeAPIServer: Responsible for processing REST requests for Kubernetes built-in resources
• APIExtensionsServer: Responsible for processing REST requests for CustomResourceDefinition (CRD) and CustomResource (CR)

The processing sequence of the three HTTP Servers is shown in the figure above. When a user request comes in, it first determines whether the AggregatorServer can handle it, otherwise it will be sent to KubeAPIServer. If KubeAPIServer cannot handle it, it will be sent to APIExtensionsServer. APIExtensionsServer is the last link of Delegation. If the corresponding request cannot be processed If so, 404 will be returned

2. API Server startup process

The startup entry of API Server is in cmd/kube-apiserver/apiserver.gothe file, which contains a main entry function:

// cmd/kube-apiserver/apiserver.go
func main() {
    
    
	rand.Seed(time.Now().UnixNano())

	pflag.CommandLine.SetNormalizeFunc(cliflag.WordSepNormalizeFunc)

	// 初始化Cobra.Command对象
	command := app.NewAPIServerCommand()

	logs.InitLogs()
	defer logs.FlushLogs()

	// 执行命令
	if err := command.Execute(); err != nil {
    
    
		os.Exit(1)
	}
}

In the main function, app.NewAPIServerCommand()a Cobra Command object is obtained through a method, and then command.Execute()the method is called to execute the command. NewAPIServerCommand()The method code is as follows:

// cmd/kube-apiserver/app/server.go
func NewAPIServerCommand() *cobra.Command {
    
    
	// 获取默认的配置参数
	s := options.NewServerRunOptions()
	cmd := &cobra.Command{
    
    
		Use: "kube-apiserver",
		Long: `The Kubernetes API server validates and configures data
for the api objects which include pods, services, replicationcontrollers, and
others. The API Server services REST operations and provides the frontend to the
cluster's shared state through which all other components interact.`,

		// stop printing usage when the command errors
		SilenceUsage: true,
		PersistentPreRunE: func(*cobra.Command, []string) error {
    
    
			// silence client-go warnings.
			// kube-apiserver loopback clients should not log self-issued warnings.
			rest.SetDefaultWarningHandler(rest.NoWarnings{
    
    })
			return nil
		},
		RunE: func(cmd *cobra.Command, args []string) error {
    
    
			verflag.PrintAndExitIfRequested()
			fs := cmd.Flags()
			cliflag.PrintFlags(fs)

			err := checkNonZeroInsecurePort(fs)
			if err != nil {
    
    
				return err
			}
			// set default options
			completedOptions, err := Complete(s)
			if err != nil {
    
    
				return err
			}

			// validate options
			if errs := completedOptions.Validate(); len(errs) != 0 {
    
    
				return utilerrors.NewAggregate(errs)
			}

			// 启动API Server主逻辑
			return Run(completedOptions, genericapiserver.SetupSignalHandler())
		},
		Args: func(cmd *cobra.Command, args []string) error {
    
    
			for _, arg := range args {
    
    
				if len(arg) > 0 {
    
    
					return fmt.Errorf("%q does not take any arguments, got %q", cmd.CommandPath(), args)
				}
			}
			return nil
		},
	}

	fs := cmd.Flags()
	namedFlagSets := s.Flags()
	verflag.AddFlags(namedFlagSets.FlagSet("global"))
	globalflag.AddGlobalFlags(namedFlagSets.FlagSet("global"), cmd.Name())
	options.AddCustomGlobalFlags(namedFlagSets.FlagSet("generic"))
	for _, f := range namedFlagSets.FlagSets {
    
    
		fs.AddFlagSet(f)
	}

	cols, _, _ := term.TerminalSize(cmd.OutOrStdout())
	cliflag.SetUsageAndHelpFunc(cmd, namedFlagSets, cols)

	return cmd
}

Run()The method contains the main logic of starting the API Server. The code is as follows:

// cmd/kube-apiserver/app/server.go
func Run(completeOptions completedServerRunOptions, stopCh <-chan struct{
    
    }) error {
    
    
	// To help debugging, immediately log version
	klog.Infof("Version: %+v", version.Get())

	// 1)构建http server链,其中包含ApiServer要启动的三个server,以及为每个server注册对应资源的路由
	server, err := CreateServerChain(completeOptions, stopCh)
	if err != nil {
    
    
		return err
	}

	// 2)进行http server运行前的准备,如设置健康检查、存活检查和OpenAPI路由的注册工作等操作
	prepared, err := server.PrepareRun()
	if err != nil {
    
    
		return err
	}

	// 3)启动https server
	return prepared.Run(stopCh)
}

Run()The main logic of the method is as follows:

1. Build an HTTP Server chain, which includes the three servers to be started by the API Server, and the routes for registering corresponding resources for each server.
2. Prepare the HTTP Server before running, such as setting up health checks, survival checks, and registration of OpenAPI routes, etc.
3. Start HTTPS Server

3. Loading of APIObject

The content of API Server is APIObject, which provides the ability to operate APIObject externally through Restful services. So how does API Server establish services for each APIObject?

Let’s take a look at the calling logic between the APIObject loading process functions in KubeAPIServer as follows:

pkg/controlplane/instance.goThe method in New()contains two parts:

1. Loading of Legacy APIObject (under core APIGroup such as Pod) : call InstallLegacyAPI()the method, InstallLegacyAPI()and then call the method in the method InstallLegacyAPIGroup()to load the core APIGroupInfo object into the API Server
2. Loading of APIObject in built-in APIGroup (non-core) : Define the RESTStorageProvider of all built-in APIGroup, and call InstallAPIs()the method to pass in the RESTStorageProvider array. InstallAPIs()The method will traverse the RESTStorageProvider array to create APIGroupInfo objects, and then load all built-in APIGroupInfo objects into the API Server.

4. Detailed explanation of Scheme

1), Scheme functions and application scenarios

Scheme is a structure that contains data and methods for converting between internal and external Versions, and between GVK and Go Type.

• Conversion between GVK and Go Type: There are two maps inside Scheme, corresponding to GVK to Type and Type to GVK respectively; so that the two can find each other
• Default value of APIObject: APIObject has many attributes. When operating an Object, it is unlikely that the user will provide all attribute values. In addition, when the Object is converted from one Version to another, it may also be necessary to fill in fields that do not have corresponding relationships. value
• Convert between internal and external Versions: All external Versions will be converted to internal Versions, and the conversion function is recorded in Scheme.

2) Internal structure of Scheme

// staging/src/k8s.io/apimachinery/pkg/runtime/scheme.go
type Scheme struct {
    
    
	// versionMap allows one to figure out the go type of an object with
	// the given version and name.
	// gvk转换为go type
	gvkToType map[schema.GroupVersionKind]reflect.Type

	// typeToGroupVersion allows one to find metadata for a given go object.
	// The reflect.Type we index by should *not* be a pointer.
	// go type转换为gvk
	typeToGVK map[reflect.Type][]schema.GroupVersionKind

	// unversionedTypes are transformed without conversion in ConvertToVersion.
	// 不区分version的type
	unversionedTypes map[reflect.Type]schema.GroupVersionKind

	// unversionedKinds are the names of kinds that can be created in the context of any group
	// or version
	// TODO: resolve the status of unversioned types.
	// 不区分version的kind
	unversionedKinds map[string]reflect.Type

	// Map from version and resource to the corresponding func to convert
	// resource field labels in that version to internal version.
	fieldLabelConversionFuncs map[schema.GroupVersionKind]FieldLabelConversionFunc

	// defaulterFuncs is an array of interfaces to be called with an object to provide defaulting
	// the provided object must be a pointer.
	// 设置默认值的函数
	defaulterFuncs map[reflect.Type]func(interface{
    
    })

	// converter stores all registered conversion functions. It also has
	// default converting behavior.
	// 用于做转换
	converter *conversion.Converter

	// versionPriority is a map of groups to ordered lists of versions for those groups indicating the
	// default priorities of these versions as registered in the scheme
	// 记录version的优先级
	versionPriority map[string][]string

	// observedVersions keeps track of the order we've seen versions during type registration
	observedVersions []schema.GroupVersion

	// schemeName is the name of this scheme.  If you don't specify a name, the stack of the NewScheme caller will be used.
	// This is useful for error reporting to indicate the origin of the scheme.
	schemeName string
}

Important methods of Scheme:

• AddKnownTypes(gv schema.GroupVersion, types ...Object): Store the GVK of APIObject into Scheme
• AddConversionFunc(a, b interface{}, fn conversion.ConversionFunc): Register the conversion function into Scheme
• AddTypeDefaultingFunc(srcType Object, fn func(interface{})): Register the function that sets the default value into Scheme

3), Scheme registration process

The registration process of Scheme is triggered through the import and init mechanisms of the Go language.

// cmd/kube-apiserver/app/server.go
import (
  // ...
	"k8s.io/kubernetes/pkg/controlplane"
	// ...
)

cmd/kube-apiserver/app/server.goThis package is imported in the file k8s.io/kubernetes/pkg/controlplane, and there is a file under this package pkg/controlplane/import_known_versions.go. The code is as follows:

// pkg/controlplane/import_known_versions.go
package controlplane

import (
	// These imports are the API groups the API server will support.
	_ "k8s.io/kubernetes/pkg/apis/admission/install"
	_ "k8s.io/kubernetes/pkg/apis/admissionregistration/install"
	_ "k8s.io/kubernetes/pkg/apis/apiserverinternal/install"
	_ "k8s.io/kubernetes/pkg/apis/apps/install"
	_ "k8s.io/kubernetes/pkg/apis/authentication/install"
	_ "k8s.io/kubernetes/pkg/apis/authorization/install"
	_ "k8s.io/kubernetes/pkg/apis/autoscaling/install"
	_ "k8s.io/kubernetes/pkg/apis/batch/install"
	_ "k8s.io/kubernetes/pkg/apis/certificates/install"
	_ "k8s.io/kubernetes/pkg/apis/coordination/install"
	_ "k8s.io/kubernetes/pkg/apis/core/install"
	_ "k8s.io/kubernetes/pkg/apis/discovery/install"
	_ "k8s.io/kubernetes/pkg/apis/events/install"
	_ "k8s.io/kubernetes/pkg/apis/extensions/install"
	_ "k8s.io/kubernetes/pkg/apis/flowcontrol/install"
	_ "k8s.io/kubernetes/pkg/apis/imagepolicy/install"
	_ "k8s.io/kubernetes/pkg/apis/networking/install"
	_ "k8s.io/kubernetes/pkg/apis/node/install"
	_ "k8s.io/kubernetes/pkg/apis/policy/install"
	_ "k8s.io/kubernetes/pkg/apis/rbac/install"
	_ "k8s.io/kubernetes/pkg/apis/scheduling/install"
	_ "k8s.io/kubernetes/pkg/apis/storage/install"
)

pkg/controlplane/import_known_versions.goThe install package of each built-in API Group is imported in the file, which will trigger the registration of each built-in API Group. Taking the apps API Group as an example, the code is as follows:

// pkg/apis/apps/install/install.go
package install

import (
	"k8s.io/apimachinery/pkg/runtime"
	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
	"k8s.io/kubernetes/pkg/api/legacyscheme"
	"k8s.io/kubernetes/pkg/apis/apps"
	"k8s.io/kubernetes/pkg/apis/apps/v1"
	"k8s.io/kubernetes/pkg/apis/apps/v1beta1"
	"k8s.io/kubernetes/pkg/apis/apps/v1beta2"
)

func init() {
    
    
	Install(legacyscheme.Scheme)
}

// Install registers the API group and adds types to a scheme
func Install(scheme *runtime.Scheme) {
    
    
	// 内部版本注册
	utilruntime.Must(apps.AddToScheme(scheme))
	// 外部版本注册
	utilruntime.Must(v1beta1.AddToScheme(scheme))
	utilruntime.Must(v1beta2.AddToScheme(scheme))
	utilruntime.Must(v1.AddToScheme(scheme))
	utilruntime.Must(scheme.SetVersionPriority(v1.SchemeGroupVersion, v1beta2.SchemeGroupVersion, v1beta1.SchemeGroupVersion))
}

init()The method calls AddToScheme()the methods of each version inside and outside the apps API group, and puts the type and other information of the corresponding version into the same Scheme. For KubeAPIServer, this Scheme is legacyscheme.Schemea variable.

4), internal version registration

pkg/apis/apps/install/install.goThe method of the file Install()is first called apps.AddToScheme(scheme)to register the internal version. The code is as follows:

// pkg/apis/apps/register.go
package apps

import (
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
	"k8s.io/kubernetes/pkg/apis/autoscaling"
)

var (
	// SchemeBuilder stores functions to add things to a scheme.
	SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
	// AddToScheme applies all stored functions t oa scheme.
	AddToScheme = SchemeBuilder.AddToScheme
)

// GroupName is the group name use in this package
const GroupName = "apps"

// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{
    
    Group: GroupName, Version: runtime.APIVersionInternal}

// Kind takes an unqualified kind and returns a Group qualified GroupKind
func Kind(kind string) schema.GroupKind {
    
    
	return SchemeGroupVersion.WithKind(kind).GroupKind()
}

// Resource takes an unqualified resource and returns a Group qualified GroupResource
func Resource(resource string) schema.GroupResource {
    
    
	return SchemeGroupVersion.WithResource(resource).GroupResource()
}

// Adds the list of known types to the given scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
    
    
	// TODO this will get cleaned up with the scheme types are fixed
	scheme.AddKnownTypes(SchemeGroupVersion,
		&DaemonSet{
    
    },
		&DaemonSetList{
    
    },
		&Deployment{
    
    },
		&DeploymentList{
    
    },
		&DeploymentRollback{
    
    },
		&autoscaling.Scale{
    
    },
		&StatefulSet{
    
    },
		&StatefulSetList{
    
    },
		&ControllerRevision{
    
    },
		&ControllerRevisionList{
    
    },
		&ReplicaSet{
    
    },
		&ReplicaSetList{
    
    },
	)
	return nil
}

In this file, SchemeBuilder is first called and initialized, and this function runtime.NewSchemeBuilder(addKnownTypes)is passed in . The actual function reference called in the method of the file isaddKnownTypes()pkg/apis/apps/install/install.goinit()apps.AddToScheme()SchemeBuilder.AddToScheme

Let’s take a look at SchemeBuilder. The code is as follows:

// vendor/k8s.io/apimachinery/pkg/runtime/scheme_builder.go
// SchemeBuilder collects functions that add things to a scheme. It's to allow
// code to compile without explicitly referencing generated types. You should
// declare one in each package that will have generated deep copy or conversion
// functions.
type SchemeBuilder []func(*Scheme) error

// AddToScheme applies all the stored functions to the scheme. A non-nil error
// indicates that one function failed and the attempt was abandoned.
func (sb *SchemeBuilder) AddToScheme(s *Scheme) error {
    
    
	for _, f := range *sb {
    
    
		if err := f(s); err != nil {
    
    
			return err
		}
	}
	return nil
}

// Register adds a scheme setup function to the list.
func (sb *SchemeBuilder) Register(funcs ...func(*Scheme) error) {
    
    
	for _, f := range funcs {
    
    
		*sb = append(*sb, f)
	}
}

// NewSchemeBuilder calls Register for you.
func NewSchemeBuilder(funcs ...func(*Scheme) error) SchemeBuilder {
    
    
	var sb SchemeBuilder
	sb.Register(funcs...)
	return sb
}

SchemeBuilder is essentially a collection of function arrays in which the function input parameters are Scheme types and has two important methods:

• Register(): Add another function whose input parameter is Scheme type to the function array collection.
• AddToScheme(): Pass the Scheme object into each function in the function array collection, and then run it in sequence

The process of registering Scheme using SchemeBuilder is as follows:

Take the API Group internal version registration of apps as an example and look at this process:

1. The method init()in install.go calls the AddToScheme function in register.goInstall()apps.AddToScheme(scheme)
2. The AddToScheme function in register.go points toSchemeBuilder.AddToScheme
3. Defined in register.go , here a function SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)is added to the SchemeBuilder function array collectionaddKnownTypes()
4. The addKnownType method defined in register.go is called to scheme.AddKnownTypes()store the GVK of the APIObject in the apps API Group into Scheme.
5. When install.go calls the AddToScheme function in register.go, what is actually called is SchemeBuilder.AddToSchemethat the function will call all functions in the SchemeBuilder function array collection in sequence. The function added in register.go will also be called addKnownTypes(), and finally scheme.AddKnownTypes()the GVK of the APIObject will be stored in Scheme.

The internal version registration process is as follows:

5), external version registration

External version registration takes the v1beta1 version of the apps API Group as an example. It is called first in the method pkg/apis/apps/install/install.goof the file . The code is as follows:Install()v1beta1.AddToScheme(scheme)

// pkg/apis/apps/v1beta1/register.go
var (
	localSchemeBuilder = &appsv1beta1.SchemeBuilder
	AddToScheme        = localSchemeBuilder.AddToScheme
)

func init() {
    
    
	// We only register manually written functions here. The registration of the
	// generated functions takes place in the generated files. The separation
	// makes the code compile even when the generated files are missing.
	localSchemeBuilder.Register(addDefaultingFuncs, addConversionFuncs)
}

The difference from the internal version is that the localSchemeBuilder here points to appsv1beta1.SchemeBuilder, and two functions init()are passed in the method . The actual code defined in the file is as follows:addDefaultingFuncs()addConversionFuncs()appsv1beta1.SchemeBuildervendor/k8s.io/api/apps/v1beta1/register.go

// vendor/k8s.io/api/apps/v1beta1/register.go
var (
	// TODO: move SchemeBuilder with zz_generated.deepcopy.go to k8s.io/api.
	// localSchemeBuilder and AddToScheme will stay in k8s.io/kubernetes.
	SchemeBuilder      = runtime.NewSchemeBuilder(addKnownTypes)
	localSchemeBuilder = &SchemeBuilder
	AddToScheme        = localSchemeBuilder.AddToScheme
)

// Adds the list of known types to the given scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
    
    
	scheme.AddKnownTypes(SchemeGroupVersion,
		&Deployment{
    
    },
		&DeploymentList{
    
    },
		&DeploymentRollback{
    
    },
		&Scale{
    
    },
		&StatefulSet{
    
    },
		&StatefulSetList{
    
    },
		&ControllerRevision{
    
    },
		&ControllerRevisionList{
    
    },
	)
	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
	return nil
}

The external version registration process is as follows:

5、GenericAPIServer

1), Positioning of GenericAPIServer

1. Provides the infrastructure required to expose HTTP services

   What each internal server does is the same: providing Restful services to the outside world to operate APIObject. So everyone is consistent on the big framework: HTTP Restful service needs to be implemented, and everyone needs HTTP Server, so this can be provided centrally

   Each internal server will be connected to each other to form a processing chain, which also requires an entity to be responsible.

2. Unify various processing mechanisms

   For the same matter, different internal servers should adopt the same method, which is the most guaranteed in open source projects. For example, API Resource’s external exposure form and login authentication mechanism

3. avoid duplication

   A large number of implementations can be reused

Each internal server is built on GenericAPIServer and fills in its own content into GenericAPIServer.

The most important output of each GenericAPIServer is something called Director, which is essentially a combination of a mux and a go container. All HTTP Requests are ultimately processed by these Directors.

2）、go http和go restful

Use go http package to make HTTP Server:

• http.Server
• Mux: the route of the request, mapping the url to a handler
• HandleFunc (function signature required)

Use go-restful library to provide HTTP service:

• http.Server
• Container: equivalent to the WebService aggregation place, containing a mux, so http.Handler is implemented. http.Server is passed in as mux when generated
• Router: triple of url, http method and handler
• WebService: A group of routes constitutes a WebService, and the routes within a WebService have the same rootPath.

3) Assembly of GenericAPIServer

vendor/k8s.io/apiserver/pkg/server/config.goNew()GenericAPIServer is built in the method in the file , the code is as follows:

// vendor/k8s.io/apiserver/pkg/server/config.go
func (c completedConfig) New(name string, delegationTarget DelegationTarget) (*GenericAPIServer, error) {
    
    
	if c.Serializer == nil {
    
    
		return nil, fmt.Errorf("Genericapiserver.New() called with config.Serializer == nil")
	}
	if c.LoopbackClientConfig == nil {
    
    
		return nil, fmt.Errorf("Genericapiserver.New() called with config.LoopbackClientConfig == nil")
	}
	if c.EquivalentResourceRegistry == nil {
    
    
		return nil, fmt.Errorf("Genericapiserver.New() called with config.EquivalentResourceRegistry == nil")
	}

	// 构建handlerChain
	// config.BuildHandlerChainFunc的实现为DefaultBuildHandlerChain方法
	handlerChainBuilder := func(handler http.Handler) http.Handler {
    
    
		return c.BuildHandlerChainFunc(handler, c.Config)
	}
	// 构建NewAPIServerHandler
	apiServerHandler := NewAPIServerHandler(name, c.Serializer, handlerChainBuilder, delegationTarget.UnprotectedHandler())

	// 实例化GenericAPIServer
	s := &GenericAPIServer{
    
    
		discoveryAddresses:         c.DiscoveryAddresses,
		LoopbackClientConfig:       c.LoopbackClientConfig,
		legacyAPIGroupPrefixes:     c.LegacyAPIGroupPrefixes,
		admissionControl:           c.AdmissionControl,
		Serializer:                 c.Serializer,
		AuditBackend:               c.AuditBackend,
		Authorizer:                 c.Authorization.Authorizer,
		delegationTarget:           delegationTarget,
		EquivalentResourceRegistry: c.EquivalentResourceRegistry,
		HandlerChainWaitGroup:      c.HandlerChainWaitGroup,

		minRequestTimeout:     time.Duration(c.MinRequestTimeout) * time.Second,
		ShutdownTimeout:       c.RequestTimeout,
		ShutdownDelayDuration: c.ShutdownDelayDuration,
		SecureServingInfo:     c.SecureServing,
		ExternalAddress:       c.ExternalAddress,

		// 构建了http request的路由,连接了url和响应函数;同时包含了一个request需要经过的预处理函数
		Handler: apiServerHandler,

		listedPathProvider: apiServerHandler,

		openAPIConfig:           c.OpenAPIConfig,
		skipOpenAPIInstallation: c.SkipOpenAPIInstallation,

		// 这些hook集合在New方法接下来的代码中填充,包含自己定义的和delegationTarget上的
		postStartHooks:         map[string]postStartHookEntry{
    
    },
		preShutdownHooks:       map[string]preShutdownHookEntry{
    
    },
		disabledPostStartHooks: c.DisabledPostStartHooks,

		healthzChecks:    c.HealthzChecks,
		livezChecks:      c.LivezChecks,
		readyzChecks:     c.ReadyzChecks,
		livezGracePeriod: c.LivezGracePeriod,

		DiscoveryGroupManager: discovery.NewRootAPIsHandler(c.DiscoveryAddresses, c.Serializer),

		maxRequestBodyBytes: c.MaxRequestBodyBytes,
		livezClock:          clock.RealClock{
    
    },

		lifecycleSignals: c.lifecycleSignals,

		APIServerID:           c.APIServerID,
		StorageVersionManager: c.StorageVersionManager,

		Version: c.Version,
	}

	for {
    
    
		if c.JSONPatchMaxCopyBytes <= 0 {
    
    
			break
		}
		existing := atomic.LoadInt64(&jsonpatch.AccumulatedCopySizeLimit)
		if existing > 0 && existing < c.JSONPatchMaxCopyBytes {
    
    
			break
		}
		if atomic.CompareAndSwapInt64(&jsonpatch.AccumulatedCopySizeLimit, existing, c.JSONPatchMaxCopyBytes) {
    
    
			break
		}
	}

	// 处理钩子hook操作
	// first add poststarthooks from delegated targets
	for k, v := range delegationTarget.PostStartHooks() {
    
    
		s.postStartHooks[k] = v
	}

	for k, v := range delegationTarget.PreShutdownHooks() {
    
    
		s.preShutdownHooks[k] = v
	}

	// add poststarthooks that were preconfigured.  Using the add method will give us an error if the same name has already been registered.
	for name, preconfiguredPostStartHook := range c.PostStartHooks {
    
    
		if err := s.AddPostStartHook(name, preconfiguredPostStartHook.hook); err != nil {
    
    
			return nil, err
		}
	}

	genericApiServerHookName := "generic-apiserver-start-informers"
	if c.SharedInformerFactory != nil {
    
    
		if !s.isPostStartHookRegistered(genericApiServerHookName) {
    
    
			err := s.AddPostStartHook(genericApiServerHookName, func(context PostStartHookContext) error {
    
    
				c.SharedInformerFactory.Start(context.StopCh)
				return nil
			})
			if err != nil {
    
    
				return nil, err
			}
		}
		// TODO: Once we get rid of /healthz consider changing this to post-start-hook.
		err := s.AddReadyzChecks(healthz.NewInformerSyncHealthz(c.SharedInformerFactory))
		if err != nil {
    
    
			return nil, err
		}
	}

	const priorityAndFairnessConfigConsumerHookName = "priority-and-fairness-config-consumer"
	if s.isPostStartHookRegistered(priorityAndFairnessConfigConsumerHookName) {
    
    
	} else if c.FlowControl != nil {
    
    
		err := s.AddPostStartHook(priorityAndFairnessConfigConsumerHookName, func(context PostStartHookContext) error {
    
    
			go c.FlowControl.MaintainObservations(context.StopCh)
			go c.FlowControl.Run(context.StopCh)
			return nil
		})
		if err != nil {
    
    
			return nil, err
		}
		// TODO(yue9944882): plumb pre-shutdown-hook for request-management system?
	} else {
    
    
		klog.V(3).Infof("Not requested to run hook %s", priorityAndFairnessConfigConsumerHookName)
	}

	// Add PostStartHooks for maintaining the watermarks for the Priority-and-Fairness and the Max-in-Flight filters.
	if c.FlowControl != nil {
    
    
		const priorityAndFairnessFilterHookName = "priority-and-fairness-filter"
		if !s.isPostStartHookRegistered(priorityAndFairnessFilterHookName) {
    
    
			err := s.AddPostStartHook(priorityAndFairnessFilterHookName, func(context PostStartHookContext) error {
    
    
				genericfilters.StartPriorityAndFairnessWatermarkMaintenance(context.StopCh)
				return nil
			})
			if err != nil {
    
    
				return nil, err
			}
		}
	} else {
    
    
		const maxInFlightFilterHookName = "max-in-flight-filter"
		if !s.isPostStartHookRegistered(maxInFlightFilterHookName) {
    
    
			err := s.AddPostStartHook(maxInFlightFilterHookName, func(context PostStartHookContext) error {
    
    
				genericfilters.StartMaxInFlightWatermarkMaintenance(context.StopCh)
				return nil
			})
			if err != nil {
    
    
				return nil, err
			}
		}
	}

	for _, delegateCheck := range delegationTarget.HealthzChecks() {
    
    
		skip := false
		for _, existingCheck := range c.HealthzChecks {
    
    
			if existingCheck.Name() == delegateCheck.Name() {
    
    
				skip = true
				break
			}
		}
		if skip {
    
    
			continue
		}
		s.AddHealthChecks(delegateCheck)
	}

	s.listedPathProvider = routes.ListedPathProviders{
    
    s.listedPathProvider, delegationTarget}

	// 安装API相关参数
	installAPI(s, c.Config)

	// use the UnprotectedHandler from the delegation target to ensure that we don't attempt to double authenticator, authorize,
	// or some other part of the filter chain in delegation cases.
	if delegationTarget.UnprotectedHandler() == nil && c.EnableIndex {
    
    
		s.Handler.NonGoRestfulMux.NotFoundHandler(routes.IndexLister{
    
    
			StatusCode:   http.StatusNotFound,
			PathProvider: s.listedPathProvider,
		})
	}

	return s, nil
}

The important attributes in GenericAPIServer are as follows:

• Handler: Constructs the route of the HTTP Request, connecting the URL and response function; it also contains a preprocessing function that the Request needs to go through.
• postStartHooks and preShutdownHooks: These Hook collections New()are populated in the code following the method, including self-defined and delegationTarget

4), Construction of Request Handler

New()In the method, NewAPIServerHandler()the method is called to build the ApiServerHandler. The code is as follows:

// vendor/k8s.io/apiserver/pkg/server/handler.go
func NewAPIServerHandler(name string, s runtime.NegotiatedSerializer, handlerChainBuilder HandlerChainBuilderFn, notFoundHandler http.Handler) *APIServerHandler {
    
    
	nonGoRestfulMux := mux.NewPathRecorderMux(name)
	// 把下个server的director设为NotFoundHandler
	if notFoundHandler != nil {
    
    
		nonGoRestfulMux.NotFoundHandler(notFoundHandler)
	}

	gorestfulContainer := restful.NewContainer()
	gorestfulContainer.ServeMux = http.NewServeMux()
	gorestfulContainer.Router(restful.CurlyRouter{
    
    }) // e.g. for proxy/{kind}/{name}/{*}
	gorestfulContainer.RecoverHandler(func(panicReason interface{
    
    }, httpWriter http.ResponseWriter) {
    
    
		logStackOnRecover(s, panicReason, httpWriter)
	})
	gorestfulContainer.ServiceErrorHandler(func(serviceErr restful.ServiceError, request *restful.Request, response *restful.Response) {
    
    
		serviceErrorHandler(s, serviceErr, request, response)
	})

	director := director{
    
    
		name:               name,
		goRestfulContainer: gorestfulContainer,
		nonGoRestfulMux:    nonGoRestfulMux,
	}

	// 主要由director来处理http请求,用handlerChainBuilder加了一层
	return &APIServerHandler{
    
    
		FullHandlerChain:   handlerChainBuilder(director),
		GoRestfulContainer: gorestfulContainer,
		NonGoRestfulMux:    nonGoRestfulMux,
		Director:           director,
	}
}

APIServerHandler implements the http.Handler interface, and http.Server will be able to hand over the request to APIServerHandler for processing. The ServeHTTP method of APIServerHandler directly hands the request to FullHandlerChain. The code is as follows:

// vendor/k8s.io/apiserver/pkg/server/handler.go
func (a *APIServerHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    
    
	a.FullHandlerChain.ServeHTTP(w, r)
}

handlerChainBuilder is an encapsulation of the CompleteHandlerChainFunc function of completedConfig. The default implementation is as follows:

// vendor/k8s.io/apiserver/pkg/server/config.go
// 通过装饰器模式包装apiHandler,包含认证、鉴权等一系列http filter chain,要先过这些http filter chain才访问到apiHandler
func DefaultBuildHandlerChain(apiHandler http.Handler, c *Config) http.Handler {
    
    
	handler := filterlatency.TrackCompleted(apiHandler)
	handler = genericapifilters.WithAuthorization(handler, c.Authorization.Authorizer, c.Serializer)
	handler = filterlatency.TrackStarted(handler, "authorization")

	if c.FlowControl != nil {
    
    
		handler = filterlatency.TrackCompleted(handler)
		handler = genericfilters.WithPriorityAndFairness(handler, c.LongRunningFunc, c.FlowControl, c.RequestWidthEstimator)
		handler = filterlatency.TrackStarted(handler, "priorityandfairness")
	} else {
    
    
		handler = genericfilters.WithMaxInFlightLimit(handler, c.MaxRequestsInFlight, c.MaxMutatingRequestsInFlight, c.LongRunningFunc)
	}

	handler = filterlatency.TrackCompleted(handler)
	handler = genericapifilters.WithImpersonation(handler, c.Authorization.Authorizer, c.Serializer)
	handler = filterlatency.TrackStarted(handler, "impersonation")

	handler = filterlatency.TrackCompleted(handler)
	handler = genericapifilters.WithAudit(handler, c.AuditBackend, c.AuditPolicyChecker, c.LongRunningFunc)
	handler = filterlatency.TrackStarted(handler, "audit")

	failedHandler := genericapifilters.Unauthorized(c.Serializer)
	failedHandler = genericapifilters.WithFailedAuthenticationAudit(failedHandler, c.AuditBackend, c.AuditPolicyChecker)

	failedHandler = filterlatency.TrackCompleted(failedHandler)
	handler = filterlatency.TrackCompleted(handler)
	handler = genericapifilters.WithAuthentication(handler, c.Authentication.Authenticator, failedHandler, c.Authentication.APIAudiences)
	handler = filterlatency.TrackStarted(handler, "authentication")

	handler = genericfilters.WithCORS(handler, c.CorsAllowedOriginList, nil, nil, nil, "true")

	// WithTimeoutForNonLongRunningRequests will call the rest of the request handling in a go-routine with the
	// context with deadline. The go-routine can keep running, while the timeout logic will return a timeout to the client.
	handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, c.LongRunningFunc)

	handler = genericapifilters.WithRequestDeadline(handler, c.AuditBackend, c.AuditPolicyChecker,
		c.LongRunningFunc, c.Serializer, c.RequestTimeout)
	handler = genericfilters.WithWaitGroup(handler, c.LongRunningFunc, c.HandlerChainWaitGroup)
	if c.SecureServing != nil && !c.SecureServing.DisableHTTP2 && c.GoawayChance > 0 {
    
    
		handler = genericfilters.WithProbabilisticGoaway(handler, c.GoawayChance)
	}
	handler = genericapifilters.WithAuditAnnotations(handler, c.AuditBackend, c.AuditPolicyChecker)
	handler = genericapifilters.WithWarningRecorder(handler)
	handler = genericapifilters.WithCacheControl(handler)
	handler = genericfilters.WithHSTS(handler, c.HSTSDirectives)
	handler = genericfilters.WithHTTPLogging(handler)
	if utilfeature.DefaultFeatureGate.Enabled(genericfeatures.APIServerTracing) {
    
    
		handler = genericapifilters.WithTracing(handler, c.TracerProvider)
	}
	handler = genericapifilters.WithRequestInfo(handler, c.RequestInfoResolver)
	handler = genericapifilters.WithRequestReceivedTimestamp(handler)
	handler = genericfilters.WithPanicRecovery(handler, c.RequestInfoResolver)
	handler = genericapifilters.WithAuditID(handler)
	return handler
}

DefaultBuildHandlerChain()The method wraps the apiHandler through the decorator mode, including a series of http filter chains such as authentication and authentication. You must first go through these http filter chains before accessing the apiHandler.

5), the formation of Server Chain

// vendor/k8s.io/apiserver/pkg/server/config.go
func (c completedConfig) New(name string, delegationTarget DelegationTarget) (*GenericAPIServer, error) {
    
    
  // ...
  // 构建NewAPIServerHandler
	apiServerHandler := NewAPIServerHandler(name, c.Serializer, handlerChainBuilder, delegationTarget.UnprotectedHandler())

New()The notFoundHandler passed in when calling the method NewAPIServerHandler()isdelegationTarget.UnprotectedHandler()

// vendor/k8s.io/apiserver/pkg/server/handler.go
func NewAPIServerHandler(name string, s runtime.NegotiatedSerializer, handlerChainBuilder HandlerChainBuilderFn, notFoundHandler http.Handler) *APIServerHandler {
    
    
	nonGoRestfulMux := mux.NewPathRecorderMux(name)
	// 把下个server的director设为NotFoundHandler
	if notFoundHandler != nil {
    
    
		nonGoRestfulMux.NotFoundHandler(notFoundHandler)
	}

	gorestfulContainer := restful.NewContainer()
	gorestfulContainer.ServeMux = http.NewServeMux()
	gorestfulContainer.Router(restful.CurlyRouter{
    
    }) // e.g. for proxy/{kind}/{name}/{*}
	gorestfulContainer.RecoverHandler(func(panicReason interface{
    
    }, httpWriter http.ResponseWriter) {
    
    
		logStackOnRecover(s, panicReason, httpWriter)
	})
	gorestfulContainer.ServiceErrorHandler(func(serviceErr restful.ServiceError, request *restful.Request, response *restful.Response) {
    
    
		serviceErrorHandler(s, serviceErr, request, response)
	})

	director := director{
    
    
		name:               name,
		goRestfulContainer: gorestfulContainer,
		nonGoRestfulMux:    nonGoRestfulMux,
	}

	// 主要由director来处理http请求,用handlerChainBuilder加了一层
	return &APIServerHandler{
    
    
		FullHandlerChain:   handlerChainBuilder(director),
		GoRestfulContainer: gorestfulContainer,
		NonGoRestfulMux:    nonGoRestfulMux,
		Director:           director,
	}
}

NewAPIServerHandler()In the method, set the director of the next Server to NotFoundHandler

// vendor/k8s.io/apiserver/pkg/server/handler.go
func (a *APIServerHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    
    
  // 请求被转发给FullHandlerChain处理
	a.FullHandlerChain.ServeHTTP(w, r)
}

The request is forwarded to FullHandlerChain for processing, and finally to the director for processing.

// vendor/k8s.io/apiserver/pkg/server/handler.gos
func (d director) ServeHTTP(w http.ResponseWriter, req *http.Request) {
    
    
	path := req.URL.Path

	// check to see if our webservices want to claim this path
	for _, ws := range d.goRestfulContainer.RegisteredWebServices() {
    
    
		switch {
    
    
		case ws.RootPath() == "/apis":
			// if we are exactly /apis or /apis/, then we need special handling in loop.
			// normally these are passed to the nonGoRestfulMux, but if discovery is enabled, it will go directly.
			// We can't rely on a prefix match since /apis matches everything (see the big comment on Director above)
			if path == "/apis" || path == "/apis/" {
    
    
				klog.V(5).Infof("%v: %v %q satisfied by gorestful with webservice %v", d.name, req.Method, path, ws.RootPath())
				// don't use servemux here because gorestful servemuxes get messed up when removing webservices
				// TODO fix gorestful, remove TPRs, or stop using gorestful
				d.goRestfulContainer.Dispatch(w, req)
				return
			}

		case strings.HasPrefix(path, ws.RootPath()):
			// ensure an exact match or a path boundary match
			if len(path) == len(ws.RootPath()) || path[len(ws.RootPath())] == '/' {
    
    
				klog.V(5).Infof("%v: %v %q satisfied by gorestful with webservice %v", d.name, req.Method, path, ws.RootPath())
				// don't use servemux here because gorestful servemuxes get messed up when removing webservices
				// TODO fix gorestful, remove TPRs, or stop using gorestful
				d.goRestfulContainer.Dispatch(w, req)
				return
			}
		}
	}

	// if we didn't find a match, then we just skip gorestful altogether
	klog.V(5).Infof("%v: %v %q satisfied by nonGoRestful", d.name, req.Method, path)
	// 当director发现当前server没有处理该请求时,转给NotFoundHandler
	d.nonGoRestfulMux.ServeHTTP(w, req)
}

When the director finds that the current server does not process the request, it transfers to NotFoundHandler. That is, when the current Server cannot find the processing method corresponding to the Request, it is handed over to the delegationTarget. The delegationTarget of AggregatorServer is KubeAPIServer, and the delegationTarget of KubeAPIServer is APIExtensionsServer. This forms an HTTP Server chain.

reference:

Kubernetes source code development journey three: API Server source code analysis

Design pattern of Builder and its application in Kubernetes API Server