Graphical implementation of kubernetes resource expansion mechanism (below)

Yesterday we introduced the core key components of the resource plug-in mechanism in k8s. Today we will continue to look at how each component communicates and the key design behind event processing in k8s

1.PluginManager

PluginManager is an upper-level component that contains the key components in the previous article, coordinates its internal data flow, and also provides specific controllers for different plugins

1.1 Core data structures

The core structure is actually designed according to the data flow. First, a perception plug-in desiredStateOfWorldPopulator is needed to perceive the creation or deletion of back-end services, and then the perceived events are added to the desiredStateOfWorld desired state cache, and the reconciler is responsible for the bottom layer. Register and log off, and store the result in the actualStateOfWorld actual state cache

type pluginManager struct {
	// 插件感知
	desiredStateOfWorldPopulator *pluginwatcher.Watcher

    // 协调器插件
	reconciler reconciler.Reconciler

    // 实际状态缓存
	actualStateOfWorld cache.ActualStateOfWorld
	// 期望状态缓存
	desiredStateOfWorld cache.DesiredStateOfWorld
}

1.2 Initialization

During initialization, both dsw and asw will be handed over to the reconciler for event perception and updating the corresponding cache

func NewPluginManager(
	sockDir string,
	recorder record.EventRecorder) PluginManager {
	asw := cache.NewActualStateOfWorld()
	dsw := cache.NewDesiredStateOfWorld()
    // 这里会将期望状态缓存和实际状态缓存，都交给reconciler
	reconciler := reconciler.NewReconciler(
		operationexecutor.NewOperationExecutor(
			operationexecutor.NewOperationGenerator(
				recorder,
			),
		),
		loopSleepDuration,
		dsw,
		asw,
	)

	pm := &pluginManager{
        //  启动一个watcher并且存储dsw期望状态缓存，后续reconciler就可以通过dsw感知到新的状态了
		desiredStateOfWorldPopulator: pluginwatcher.NewWatcher(
			sockDir,
			dsw,
		),
		reconciler:          reconciler,
		desiredStateOfWorld: dsw,
		actualStateOfWorld:  asw,
	}
	return pm
}

1.3 Start the plugin manager

The plugin manager startup is actually to start the internal desiredStateOfWorldPopulator, which will talk about the events perceived by the watcher, and constantly modify its own internal cache so that the reconciler can continuously pass the desired state cache and make corresponding grpc calls to meet the desired state.

func (pm *pluginManager) Run(sourcesReady config.SourcesReady, stopCh <-chan struct{}) {
	defer runtime.HandleCrash()

    // 运行期望状态缓存，其实主要是通过watcher感知到的事件，修改自身的缓存
    // 后续reconciler会周期性的获取
	pm.desiredStateOfWorldPopulator.Start(stopCh)
	klog.V(2).Infof("The desired_state_of_world populator (plugin watcher) starts")

	klog.Infof("Starting Kubelet Plugin Manager")
    // 周期性的运行校证数据
	go pm.reconciler.Run(stopCh)

	metrics.Register(pm.actualStateOfWorld, pm.desiredStateOfWorld)
	<-stopCh
	klog.Infof("Shutting down Kubelet Plugin Manager")
}

1.4 Controller registration

The controller actually mainly refers to the reconciler by comparing the difference between the expected cache and the actual cache, and after generating the corresponding event, what is the subsequent processing flow for this type of plug-in, such as registration/offline specific grpc interface and corresponding Plugin type handling mechanism

func (pm *pluginManager) AddHandler(pluginType string, handler cache.PluginHandler) {
	pm.reconciler.AddHandler(pluginType, handler)
}

1.5 CSI and common equipment

There are currently two types of plug-in controllers registered in the kubelet, CSI and DEVICPLUGIn. You can also know the approximate meaning from the name.

	kl.pluginManager.AddHandler(pluginwatcherapi.CSIPlugin, plugincache.PluginHandler(csi.PluginHandler))
	kl.pluginManager.AddHandler(pluginwatcherapi.DevicePlugin, kl.containerManager.GetPluginRegistrationHandler())

2. PluginHandler

Here we only introduce one core implementation mechanism of DevicePlugin

2.1 Endpoint

Endpoint actually refers to a service that provides extended resources. In the reconciler mentioned earlier, the address of its corresponding grpc service will be obtained, and then grpc will be directly called for communication.

Endpoint needs to sense the change of the corresponding resource device, and at the same time notifies the corresponding device information and callback to the current

2.2 Manager

The Manager is mainly responsible for the implementation of the real Register/UnRegister in the backend. It creates an Endpoint for each Device internally and is responsible for collecting the information reported by the resource service provided by the backend, and finally sending the corresponding information to the kubelet. , and then the kubelet will pass the information to the APIServer when it is responsible for updating the node information

2.3 Checkpoint

The checkpoint mechanism is actually used in many systems. It is mainly used to periodically serialize and store the data in the memory to the local disk. During subsequent recovery, the previous data will be reloaded through the disk to realize the memory Fast recovery of resources

The overall implementation process of expanding resources is roughly like this. From how to perceive data, register resource services, obtain resource information of resource services, and finally report to kubelet, and land on the local disk at the same time, the overall process from perception to reporting of complete resources is realized. The main disadvantage of the detection of the resource is the description of the resource entity, which leads to a relatively large scalability limitation in resource allocation and resource reporting. For example, to achieve refined resource allocation expansion, it is not possible to achieve

k8s source code reading e-book address: https://www.yuque.com/baxiaoshi/tyado3

> WeChat ID: baxiaoshi2020 > Follow the bulletin number to read more source code analysis articles > For more articles, follow www.sreguide.com > This article is published by OpenWrite , a blog post multiple platform

{{o.name}}

{{m.name}}