Table of contents
(1) Processing logic in client-go
(2) Processing logic in controller-runtime
2. Use controller-runtime to develop operator projects
1. Usage scenarios
controller-runtime is a framework derived from the kubernetes controller model. A control loop is a non-terminating loop used to regulate the state of the system. A controller tracks at least one type of Kubernetes resource. These objects have a spec field representing the desired state. The resource's controller is responsible for ensuring that its current state is close to the desired state.
From it, we can conclude that the development of controller-runtime needs to follow the following points.
-
controller-runtime needs to continuously monitor the kubernetes object. (implemented through the list/watch mechanism)
-
The controller-runtime needs to provide certain mechanisms to maintain the status (actual state) of the Kubernetes object until it is consistent with the spec (expected state). (implemented by reconcile)
(1) Processing logic in client-go
In client-go, a Kubernetes object is encapsulated into a pair of informer types. The informer type includes Reflector, DeltaFIFO, Indexer, and EventHandler.
-
Reflector: API interface for calling Kubernetes. The informer will monitor the kubernetes object through two Kubernetes API interfaces. When list/watch fails, it will automatically re-list/watch
-
list interface: Get the full amount of kubernetes object data defined in the informer into the deltaFIFO through http requests
-
watch interface: start monitoring from the maximum resourceVersion stored locally. When a watch request is sent, the API server responds with a change stream. These changes itemize the results of operations (such as ADDED, MOD-IFIED, DELETED, BOOKMARK, ERROR) that occurred after the resourceVersion you specified as a watch request parameter.
-
DeltaFIFO: Stores each pending Kubernetes object. There are two data structures for storing kubernetes objects in DeltaFIFO, one is a one-dimensional array for storing object key values, and the other is a map structure for storing data streams.
-
queue: One-dimensional array, each kubernetes object corresponds to a key (namespace/name). The changed data that has been enqueued multiple times will be merged and stored, and when dequeued, the entire object will be processed.
-
items: Stores the change stream of kubernetes objects. (Added, Updated, Deleted, Replaced, Sync) When the informer is started, the data is continuously read from the deltaFIFO for processing.
-
Indexer: The Kubernetes object obtained from the DeltaFIFO is stored in the memory map.
-
EventHandler: The Kubernetes object obtained from DeltaFIFO is processed differently according to the change event type of the object. Generally, the key value is passed to the work queue. Then it is processed by custom controller.
(2) Processing logic in controller-runtime
Define the Informer according to the Kubernetes object to be monitored
Define EventHandler processing logic. Store the key value in the informer eventhandler into the work queue. The key value will be deduplicated in the work queue. To avoid concurrent processing of the same object in a work queue.
Take the key value from the work queue and process it in reconcile.
The custom structure implements the reconcile method. In this method, the state of the kubernetes object is coordinated to bring it closer to the desired state.
Introduction to the controller-runtime framework
The top layer of controller-runtime is encapsulated by manager. manager contains the following information
-
cluster: used to communicate with the kubernetes cluster. Contains the kubeconfig configuration information of kubernetes, the client information connected to the apiserver of kuberenetes, the kubernetes object scheme information registered to the operator, used to obtain the RESTMapper information of gvk and gvr, the indexer information stored by the informer, and used to read with kubernetes objects, Client information for writing and event publishing.
-
controller: used for informer configuration and startup, definition of reconcile
-
cache: Cache informer information. Contains the definition of informer and reading of indexer data.
-
webhook: used to publish three types of webhook services in kubernetes.
-
Election: active and standby configuration for operator
-
prometheus: data service for publishing prometheus
-
probe: defines the probe service of the operator, including two kinds of probe data: health probe and survival probe
2. Use controller-runtime to develop operator projects
(1) Generate framework code
The official recommendation is to use the kubebuilder plugin to automatically generate the controller-runtime framework.
Generate project framework: kubebuilder init --domain demo --plugins=go/v4-alpha
Generate the api corresponding to the crd object of kubernetes (the default object of Kubernetes can be ignored): kubebuilder create api --group webapp --version v1 --kind Guestbook
Generate webhook service code: kubebuilder create webhook --group webapp --version v1 --kind Guestbook --defaulting --programmatic-validation
The generated code directory structure is as follows
-
api: contains the api definition of crd, webhook file, and the required fields need to be supplemented in guestbook_types.go. Then generate the required files through the commands in the Makefile (the yaml file that can be deployed to the kubernetes cluster, the go file that implements the object method defined in client-go)
-
bin: The tool used when generating the file. controller-gen (generate crd associated files) and kustomize (generate deployment files for opertor projects)
-
config: the configuration file used when kustomize is executed.
-
controllers: controllers example file. Contains a custom Reconcile structure, implements the Reconcile method for coordinating kubenetes resources, and defines the informer method.
-
hack: the configuration file used when controller-gen is executed.
-
dockerignore Dockerfile: generate operator project image file
-
main.go: startup file
-
Makefile: Encapsulates some make commands. Contains commands for generating code, deploying operator project commands, etc.
-
PROJECT: Project configuration information file generated after executing the kubebuider command
(2) Define the crd field
Supplement guestbook_types.go file. In this file, two types of data need to be supplemented.
-
Fields in struct
The desired state field is defined in the spec. The actual status field is defined in status.
The json tag after the field is used to generate the corresponding field of the crd file in kubernetes. omitepty indicates whether the field is required or not.
-
kubebuilder-notes
Used to generate crd files and deepcopy files that implement runtime.Object defined in client-go.
The kubebuilder annotations on the official website show four usages of annotations:
- Validation (defined on the field, used to generate field validation for crd files)
- Additional Printer Columns (defined on the structure, corresponding to spec.ver-sions.additionalPrinterColumns in the crd file)
- Subresources (defined on the structure, corresponding to .spec.version.subresources of the crd file)
- Multiple Versions (defined on the structure, the specified version is defined in the multi-version, corresponding to the spec.versions.storage field definition in the crd file)
⚠️There can be no blank lines between comments defined on the structure and comments
(3) Generate crd file
Execute make manifest to generate WebhookConfiguration, ClusterRole, Custom-ResourceDefinition
- WebhookConfiguration: Generated according to the //+kubebuilder:webhook annotation, usually in the guestbook_webhook.go file
- ClusterRole: Generated according to the //+kubebuilder:rbac annotation, generally defined in the controllers/guestbook_controller.go file
- CustomResourceDefinition: Generated according to guestbook_types.go and groupversion_in-fo.go files.
Execute make generate to generate the zz_generated.deepcopy.go file
zz_generated.deepcopy.go file: implements the method in runtime.Object for type conversion in the client-go dependency package. Generated according to //+kubebuilder:object annotation.
(4) Initialize the manager
Initialize the manager in the main.go file.
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
Port: 9443,
HealthProbeBindAddress: probeAddr,
LeaderElection: enableLeaderElection,
LeaderElectionID: "aefd3536.demo",
})
Configure controllers
if err = (&controllers.GuestbookReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "Guestbook")
os.Exit(1)
}Configure webhooks
if err = (&webappv1.Guestbook{}).SetupWebhookWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create webhook", "webhook", "Guestbook")
os.Exit(1)
}configure probe
if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
setupLog.Error(err, "unable to set up health check")
os.Exit(1)
}
if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
setupLog.Error(err, "unable to set up ready check")
os.Exit(1)
}start-manager
if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
setupLog.Error(err, "problem running manager")
os.Exit(1)
}(5) Configure the controller
Two methods are defined in the guestbook_controller.go file.
func (r *GuestbookReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&webappv1.Guestbook{}).
WithEventFilter(predicate.Funcs{
CreateFunc: func(_ event.CreateEvent) bool {
return false
},
}).
WithOptions(controller.Options{MaxConcurrentReconciles: 2}).
Complete(r)
}
Called in the main.go file to define the informer and configure the informer into the manager
-
kubernetes object in informer
The framework provides three methods for defining object types:
- Watches(source.Source, handler.EventHandler, ...WatchesOption)
- For(client.Object,...ForOption)
- Owns(client.Object,...OwnsOption)
Among them, For and Owns are equivalent to Watches. The second parameter of For defaults to EnqueueRequestForObject. The second parameter of Owns defaults to EnqueueRequestForOwner
Method parameter description
- Source: the first parameter, the kubernetes object type
- EventHandler: The second parameter, the data taken from the DeltaFIFO, the operation performed before entering the work queue. EnqueueRequestForObject means to enter the work queue directly without any processing. EnqueueRequestForOwner needs to be used in conjunction with the For method. The object type referenced by ownerReference in the object in Owns must be the same as the object type defined in For, and the controller in ownerReference must be true.
- Predicate: The third parameter, the data taken from the work queue, the operation performed before reconcile processing. Predicates can be added to all objects through the WithEventFilter of the builder.
EventHandler and Predicate method description
Create: Called when a kubernetes object is added
Update: Called when the kubernetes object is updated
Delete: Called when the kubernetes object is deleted
Generic: Unknown operation. Change events for non-kubernetes clusters. Use it yourself in the operator
-
controller configuration
Configuration is defined via builder.WithOptions. The following configurations can be defined in the controller.
- MaxConcurrentReconciles: Data taken from the work queue can be processed using several coroutines. default one
- Reconciler: The coordinator needs to implement the Reconciler method. Defaults to Reconcile defined in the guest-book_controller.go file
- RateLimiter: The current limiting policy of the work queue. There are two current limiting strategies by default. Token Bucket Policy (BucketRateLimiter): The default number of tokens is 100. 10 tokens can be withdrawn per second. Backoff strategy (ItemExponentialFailureRateLimiter): When reconcile returns failure, re-enter the queue. The time to leave the team next time increases according to the exponential power of 2. Initial 5ms, maximum 1000s.
- LogConstructor: The log logr obtained from the context in the reconcile. The default is the log defined in mgr.
- CacheSyncTimeout: Informer synchronization timeout. When the informer performs list/watch, the informer synchronization is successful only when all the data called by the list interface is synchronized to the informer's indexer.
- RecoverPanic: Whether to automatically recover when reconcile is abnormal.
func (r *GuestbookReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)
// TODO(user): your logic here
return ctrl.Result{}, nil
}The Reconcile method is a custom logic. When implementing custom logic, you don't need to pay too much attention to the impact of the intermediate state, you just need to ensure that the current state should be as close as possible to the desired state. In the Result of the Reconcile return value, you can also define whether to re-enter the queue and the time for de-queuing.
(6) Configure webhook
The framework provides three webhooks:
Defaulter
func (r *Guestbook) Default() {
guestbooklog.Info("default", "name", r.Name)
// TODO(user): fill in your defaulting logic.
}Corresponds to the MutatingAdmissionWebhook object in kubernetes.
Validator
func (r *Guestbook) ValidateCreate() error {
guestbooklog.Info("validate create", "name", r.Name)
// TODO(user): fill in your validation logic upon object creation.
return nil
}
// ValidateUpdate implements webhook.Validator so a webhook will be registered for the type
func (r *Guestbook) ValidateUpdate(old runtime.Object) error {
guestbooklog.Info("validate update", "name", r.Name)
// TODO(user): fill in your validation logic upon object update.
return nil
}
// ValidateDelete implements webhook.Validator so a webhook will be registered for the type
func (r *Guestbook) ValidateDelete() error {
guestbooklog.Info("validate delete", "name", r.Name)
// TODO(user): fill in your validation logic upon object deletion.
return nil
}
Corresponds to the ValidatingWebhookConfiguration object in kuberenetes.
convertion.Webhook
Corresponds to the webhook defined in spec.convertion in crd
central version
// Hub 标注为中心版本
func (r *Guestbook) Hub() {
}other versions
func (r *Guestbook) ConvertTo(dst conversion.Hub) error {
// todo 当前版本向中心版本转换
return nil
}
func (r *Guestbook) ConvertFrom(src conversion.Hub) error {
// todo 中心版本向当前版本转换
return nil
}Then the above is the development and application of k8s operator in the project, I hope it will be helpful to you~
This is the end of this sharing. For more technical knowledge, please pay attention to the official account of "Digital China Cloud Base"
Copyright statement: The article is compiled and output by the team of Digital China Wuhan Cloud Base. Please indicate the source for reprinting.