We know that the data of various resource objects in Kubernetes (hereinafter referred to as "K8s") is submitted through the K8s API and persisted to the storage etcd (called K8s objects). The K8s object uses the K8s interface, and the kubelet client Use K8s capabilities by manipulating these objects.
Among them, kubectl is the command line tool we use the most. The official K8s summary of kubectl's technology for managing K8s objects is as follows:
Suppose we now want to create a Deployment object under the namespace named test to run the nginx container, and then change the number of copies of nginx to 5. The three methods are as follows:
-
Imperative command mode
Create a deployment object:
kubectl create deployment nginx-deployment --image nginx -n testNumber of renewal copies:
kubectl scale deployment nginx-deployment --replicas 5 -n test-
imperative object configuration
First create a configuration file definition, as shown in Figure 1-1:
Figure 1-1 ngin.yaml configuration
Create a deployment object:
kubectl create -f nginx.yamlChange the replicas in the configuration file from 2 to 5, and execute the command to change the number of replicas:
kubectl replace -f nginx.yaml-
Declarative object configuration
Using the configuration in Figure 1-1, create a deployment object:
kubectl apply -f nginx.yamlChange the replicas in the configuration file from 2 to 5, and execute the command to change the number of replicas:
kubectl apply -f nginx.yamlThe latter two ways of object configuration provide templates for creating new objects and are able to provide an audit trail associated with changes, commands cannot provide a source of record other than live content. Therefore, in the case of production projects, the way we use K8s objects is often to write corresponding .yamlfiles and hand them over to K8s (in the form of declarative API), rather than directly using imperative commands.
Our configuration file just declares the information of the K8s objects we want to create and the desired state we want them to achieve. This is an interactive way of declarative API. Unlike explicit instructions such as eating and sleeping (this method is also called: imperative API), the interaction of objects created using configuration feels similar to the communication mode between customers and implementers, because customers are often less clear than implementers The operations needed to achieve the goal, for example: build me a standard basketball court.
The declarative API is often equipped with the ability to combine multiple operations. We can lay the floor and install the large screen at the same time when building the stadium. The imperative API will be very complicated and inefficient in the case of concurrent access, and it is often achieved through locking. To ensure the predictability of the final result, the child must finish eating before going to bed, rather than eating while sleeping. And the declarative API naturally records the current and final state, making it easy to check the results. You must eat 3 bowls of rice every day and keep your weight at 50kg. The weight results of people of different weights eating three bowls of rice cannot be estimated, but different people can reach the goal of 50kg by adjusting their calorie intake. This is why the declarative API in K8s is so important, and the main body of the K8s declarative API description is the K8s object, as mentioned at the beginning, it is the medium and interface for us to operate K8s.
So how are K8s objects represented in the K8s API, how do we describe them with files, and how does it create a K8s object after .yamla file is submitted to K8s?.yaml
1. K8s object
K8s objects refer to the persistent entities in the K8s system, and K8s uses these entities to represent the running status of the entire cluster. In particular, they describe the following information:
-
Which containerized applications are running and on which nodes
-
resources that can be used by the application
-
The performance strategy of the application runtime, such as restart strategy, upgrade strategy and fault tolerance strategy.
K8s object operations (creation, modification, deletion) all need to use the K8s API. Whether you use the kubectl command line or use the client in the program, you will eventually call the K8s API.
Each K8s object contains two nested object fields spec (spec) and status (status). Among them, spec describes the expected state that K8s should reach after creation, such as how many copies are expected to be opened, and status describes the current state of the object, which is supervised and set by the K8s system and components and actively matches it with the spec. These two fields are also the most critical part of K8s declarative API implementation.
2. How K8s describes and manages API objects
When we create a K8s object, we must provide the spec field of the object to tell K8s the state we expect the object to achieve, and we must also provide some basic information about the object (such as the name). Regardless of whether we create objects through the kubectl command line or the client calls the K8s API, the API request must contain JSON-formatted information in the request body.
In most cases, we .yaml provide this information in the form of documents. When kubectl initiates an API request, it converts the information into JSON format. Take Figure 1-1 as an example, which shows the .yamlrequired fields and object specifications used to describe K8s objects. The following is a brief description of the required fields:
-
apiVersion - the K8s API version used to create the object
-
kind - the kind of object to create
-
metadata – some data to help uniquely represent the object, including a name string, UID and optional namespace
-
spec – the state you expect the object to be in
The precise format of spec is often different for different types of K8s objects. It is the subject of detailed descriptions of managed objects and will be persisted in etcd by K8s. If spec deleted, the object is also deleted from the system.
The complete resource path of an API object in etcd consists of three parts: Group (API group), Version (API version), and Resource (API resource type). The specific case is shown in Figure 1-2:
Figure 1-2 API structure of K8s
From the figure, we can see that the organization of API objects in K8s is progressive layer by layer. Here we will not delve into how to use these APIs to retrieve the required resources. Interested readers can refer to the following link address, here Here is the official detailed description:
https://kubernetes.io/zh/docs/reference/using-api/api-concepts/
Suppose now we want to create an CronJob object, and the beginning of its yaml file is as follows:
apiVersion: batch/v2alpha1kind: CronJob...
Among them, batch it means its API group (Group), v2alpha1 its API version (Version), CronJobthat is, conjobsa specific category under the API resource type (Resource) to which it belongs. The overall creation process is shown in Figure 1-3 below:
Figure 1-3 Persistence process
➤ Submission phase
When our yaml is submitted to the K8s cluster (whether through the program client or kubectl), the request will be converted into a POST, and then handed over to the K8s API Server for processing.
➤ Filtering and pre-processing stage
The API Server first filters the request, obtains the valid information, and then completes pre-processing tasks such as authorization, timeout processing, and auditing.
➤ Route lookup phase
The definition of CronJob is mainly searched by MUX and Routes routing. The specific process is roughly as follows:
-
Match Group. For non-core API objects such as CronJob, K8s will
/apissearch for its group in the hierarchy shown in Figure 1-2 (core objects, such as Pod and Node, belong to the /api level and their group is ""), According toyamlthe beginning information of , find it/apis/batch. -
Match the version number. Then, this path
v2alpha1is further matched according to the version number/apis/batch/ v2alpha1. In K8s, the same API object can have multiple versions. This is a means for users to manage coexisting multi-version APIs in K8s. -
Match resource type. Finally, match the resource type cronjobs. At this time, K8s can clearly know that the specific resource type it needs to create is CronJob.
➤ Create resource stage
After obtaining the type definition of CornJob, API Server will perform a Convert job: convert the yaml submitted by the user into a Super Version object, which is the complete set of fields of all versions of the API resource type, and then the user submits other versions of yaml, It can also be handled with this Super Version object.
Then, Admit processing is performed, mainly after the object is created, some public processing logic is injected, such as adding some labels Label. Finally, verify the validity of each field in this object.
➤ Define storage stages
If the Validate is passed, it will be saved by the API Server in a structure named Registry. It contains the definition of a valid K8s API object.
➤ Persistence phase
The API Server will reconvert the verified API objects from the Super Version version to the originally submitted version, then perform serialization operations, and finally call the etcd API to turn them into entity objects in K8s.
At this point, the object has been created and persisted, so how does it reach our desired state step by step? The spec and status information mentioned in the K8s object chapter is the basis for solving this problem. Based on this, K8s uses the controller mode to implement it.
3. The controller mode implements the sepc of the declarative API
K8s uses the controller mode to approach the spec state described in the declarative API, and the core of the control mode is the concept of "control loop". In K8s, the controller is a control loop that monitors the shared state of the cluster through the API server and makes changes in an attempt to transfer the current state (status) to the desired state (spec).
The actual state is often obtained through some components of K8s itself, such as calling the API periodically to obtain resource state information, or the container state and node state reported by the kubelet through the heartbeat. The desired state is often defined by the spec field of the yaml file and submitted to K8s etcd for persistence. Figure 1-4 roughly describes the process of the control loop in K8s:
Figure 1-4 Control loop
-
Compare spec and status, if they are the same, do not operate, continue to 1, otherwise output different points, go to 2;
-
The controller (Controller) issues control operations to the system according to different points, and the system executes the operation, and enters 3;
-
The system reports the current system status to the sensor (Sensor) or the sensor actively pulls the status, outputs the current status status, and returns 1.
The purpose is to keep the status close to the spec, so as to complete the realization of the yaml statement status. It can also be seen from the figure that Sensor and Controller play an extremely important role in the whole cycle.
In K8s, Sensor mainly consists of three components Reflector: , Informer, and Indexer.
The overall workflow of the Sensor is shown in Figure 1-5:
Figure 1-5 Sensor components
-
Reflector obtains the status of each resource object from the API Server through List and Watch operations. Among them, List is used to describe and return a collection of multiple resources, which is mainly used for the full update of system resources; Watch is used for the client to obtain the current state, and then subscribe to subsequent changes, and is mainly used for incremental updates of system resources;
-
After Reflector obtains new resource data, it will push a Delta record into the Delta queue. This record contains the information of the resource object itself and the type of resource object event. The Delta queue can ensure that the same object has only one record in the queue, thereby avoiding duplicate records when Reflector re-lists and watches.
-
The Informer component continuously pops up delta records from the Delta queue, and then hands resource objects to the indexer. The indexer records resources in a cache that uses namespaces as an index by default. The recording operation is thread-safe and the cache can be shared between Controller Manager or multiple Controllers. After that, the informer passes the event to the event callback function.
The main process of the Controller is shown in Figure 1-6:
Figure 1-6 controller component
The Controller (controller) component in the control loop is mainly composed of event processing functions and workers. The event processing functions determine whether to perform corresponding callback processing according to the controller logic. If event processing is required, it will put the namespace and resource name of the event-associated resource as an identifier into the work queue, which will be processed by a worker in the subsequent worker pool. The work queue will deduplicate the stored objects, thus Avoid situations where multiple wokers process the same resource.
When workers process resource objects, they generally need to use the name of the resource to retrieve the latest resource data. There are three situations in which these data will be used later:
-
Create or update resource objects
-
call other external services
-
Worker processing failed, re-add the name of the resource to the work queue for retry
Now if we need to expand the nginx replicas configured in Figure 1-1 from 2 to 3 for example, it is easy for readers to understand. The overall process is shown in Figure 1-7:
Figure 1-7 Overall flow chart of replica expansion
First, Reflector will watch to see that the Deployment has changed, and a record whose object is nginx-deployment and whose type is updated is inserted into the delta queue.
On the one hand, the Informer updates the new Deployment to the cache, and uses the namespace named test as an index. On the other hand, when the callback function of Update is called, the Deployment controller will insert the string test/nginx-deployment into the work queue after finding that the Deployment has changed, and a worker behind the work queue gets the test/nginx-deployment string from the work queue. Deployment is the key of the string, and the latest Deployment data is fetched from the cache.
By comparing the values in spec and status in the Deployment, the worker finds that the Deployment needs to be expanded, so the Deployment worker creates a Pod, and the Owner reference in the Pod points to nginx-deployment.
Then the Pod new event detected by Reflector Watch adds an additional deta record of the Add type to the delta queue.
On the one hand, the Informer stores the new Pod record in the cache through the Indexer, and on the other hand, it calls the Add callback function of the Deployment controller. The Add callback function finds the corresponding Deployment as nginx-deployment by checking the ownerReferences of pod3, and puts test/nginx The -deployment string is stuffed into the work queue.
After the Deployment woker gets the new work item, it fetches the new Deployment record from the cache and gets all its Pod information, and compares and finds that the Deployment status is not up-to-date (the number of associated pods has changed). At this time, the Deployment updates the status to make the spec and the status agree.
Summarize
-
Since the declarative API has natural characteristics such as recording the initial and final state, K8s adopts the API interaction method as the most critical part.
-
In the production environment, the management of K8s is often through the manager API object, and the API object is described through yaml. The spec and status are the most critical information, and K8s will describe a yaml through location and routing. The information is persisted to etcd as an entity.
-
The controller mode adopted by K8s is driven by a declarative API. It is the key to realize the K8s declarative API. The controller corresponding to the resource combines the sensor through the control loop to asynchronously approach the control system to the set final state.
-
Because the controller operates autonomously, automation and unattended operation of the overall system is possible.