--port int32 Default: 10250
	The port for the kubelet to serve on. (DEPRECATED: This parameter should be set via the config file specified by the kubelet's `--config` flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)

kubelet | Kubernetes https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
KubeletInfrastructureReference - Zachary's Cloud

Archive - Zachary's Cloudhttps://mazengxie.github.io/archive.html?tag=kubelet

kubelet architecture analysis

Kubelet is the component in the Kubernetes cluster that actually maintains the status of the container and does the specific "work". A kubelet service process runs on each node. It listens to port 10250 by default, receives and executes instructions from the master, and manages Pods and containers in Pods.

1. Main functions

Node management

Node management mainly includes node self-registration and node status update :

Kubelet can determine whether to register itself with the API Server by setting the startup parameter –register-node ;
If Kubelet does not choose the self-registration mode, the user needs to configure the Node resource information by himself, and also needs to inform the Kubelet of the location of the API Server on the cluster;
Kubelet registers node information through the API Server when it starts, and regularly sends new node messages to the API Server . After receiving the new messages, the API Server writes the information to etcd.

Self-registration of Nodes

When the kubelet flag --register-node is true (the default), the kubelet will attempt to register itself with the API server. This is the preferred pattern, used by most distros.

For self-registration, the kubelet is started with the following options:

--kubeconfig - Path to credentials to authenticate itself to the API server.
--cloud-provider - How to talk to a cloud provider to read metadata about itself.
--register-node - Automatically register with the API server.
--register-with-taints - Register the node with the given list of taints (comma separated <key>=<value>:<effect>).

No-op if register-node is false.
--node-ip - IP address of the node.
--node-labels - Labels to add when registering the node in the cluster (see label restrictions enforced by the NodeRestriction admission plugin).
--node-status-update-frequency - Specifies how often kubelet posts node status to master.

pod management

Container monitoring

health examination

2. Listening port

Kubelet listens to four ports by default, which are 10250, 10255, 10248, and 4194.

tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      3272/kubelet        
tcp6       0      0 :::10255                :::*                    LISTEN      3272/kubelet        
tcp6       0      0 :::4194                 :::*                    LISTEN      3272/kubelet        
tcp6       0      0 :::10250                :::*                    LISTEN      3272/kubelet

10250 (kubelet API): The port used by the kubelet server to communicate with the apiserver. It regularly requests the apiserver to obtain the tasks it should handle. Through this port, you can access and obtain node resources and status . kubectl checks the pod's logs and cmd commands, which are accessed through kubelet port 10250. If port 10250 is not opened locally:

View log

[devops@master cloudk8s]$ kubectl logs nginx-deployment-ddbc89dc5-7tkt5 
Error from server: Get https://192.168.0.116:10250/containerLogs/default/nginx-deployment-ddbc89dc5-7tkt5/nginx: dial tcp 192.168.0.116:10250: getsockopt: connection refused





[root@master ~]# kubectl get pod -n monitor -o wide
NAME                                 READY   STATUS    RESTARTS   AGE   IP              NODE     NOMINATED NODE   READINESS GATES
grafana-5d794f46d5-5nk4t             1/1     Running   2          25h   10.233.90.118   node1    <none>           <none>

[root@master ~]# kubectl logs grafana-5d794f46d5-5nk4t -n  monitor
t=2022-02-11T06:59:08+0000 lvl=info msg="Starting Grafana" logger=server version=7.1.0 commit=8101355285 branch=HEAD compiled=2020-07-16T11:04:17+0000
t=2022-02-11T06:59:08+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2022-02-11T06:59:08+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
...............................................


[root@node1 ~]# systemctl stop kubelet
[root@node1 ~]# systemctl status kubelet
?.kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           ?..10-kubeadm.conf
   Active: inactive (dead) since Fri 2022-02-11 15:02:44 CST; 7s ago
     Docs: http://kubernetes.io/docs/
  Process: 511 ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 511 (code=exited, status=0/SUCCESS)

Feb 11 15:02:32 node1 kubelet[511]: I0211 15:02:32.643574     511 topology_manager.go:221] [topologymanager] RemoveContainer - Container ID: 395478aba54...e31b5c3978
Feb 11 15:02:32 node1 kubelet[511]: E0211 15:02:32.645423     511 pod_workers.go:191] Error syncing pod 4646cd67-04f1-4e5d-ab5a-08b80f978ade ("hostnames-766c597cd...


[root@master ~]# kubectl logs grafana-5d794f46d5-5nk4t -n  monitor
Error from server: Get "https://192.168.100.6:10250/containerLogs/monitor/grafana-5d794f46d5-5nk4t/grafana": dial tcp 192.168.100.6:10250: connect: connection refused

Execute cmd command

[devops@master cloudk8s]$ kubectl exec -it nginx-deployment-ddbc89dc5-7tkt5 /bin/sh
Error from server: error dialing backend: dial tcp 192.168.0.116:10250: getsockopt: connection refused

10248 (health check port): Whether kubelet is working properly, specify the listening address and port through the kubelet startup parameters –healthz-port and –healthz-bind-address.

[root@node1 ~]# curl http://127.0.0.1:10248/healthz
ok


[root@master ~]# vim /var/lib/kubelet/config.yaml 
healthzPort: 10248
healthzBindAddress: 127.0.0.1

[root@master ~]# netstat -tpln | grep 10248
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      502/kubelet

4194 (cAdvisor monitoring): Kublet can obtain the environment information of the node and the status of the container running on the node through this port. Visit http://localhost:4194 to see the cAdvisor management interface, and pass the kubelet startup parameters – cadvisor-port can specify the port to start.

[root@node1 ~]# curl  http://127.0.0.1:4194/metrics

10255 (readonly API): Provides pod and node information. The interface is exposed in read-only form. Access to this port does not require authentication and authentication. Get the pod's interface, which is similar to the apiserver's http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName= interface

What needs special attention is that after Kubernetes version 1.11+, kubelet removed port 10255, and the metrics interface returned to port 10250.

[root@node1 ~]# curl  http://127.0.0.1:10255/pods

Node information interface, providing disk, network, CPU, memory and other information

[root@node1 ~]# curl http://127.0.0.1:10255/spec/

Prometheus service discovery 10250 port capture indicator monitoring kubelet

Since the node-exporter program is running on all three nodes here, if we collect the data together through a Service and configure it into Prometheus through static configuration, only one piece of data will be displayed. We have to add the indicator data ourselves. To filter the data of each node, is there a way for Prometheus to automatically discover the node-exporter program of our nodes and group them by nodes? Yes, it is the service discovery we mentioned before .

Under Kubernetes, Promethues currently supports five service discovery modes through integration with the Kubernetes API, namely: Node, Service, Pod, Endpoints, and Ingress.

We can easily obtain all node information in the current cluster through the kubectl command:

$ kubectl get nodes
NAME      STATUS    ROLES     AGE       VERSION
master    Ready     master    165d      v1.10.0
node02    Ready     <none>    85d       v1.10.0
node03    Ready     <none>    145d      v1.10.0

But if we want Prometheus to also obtain information about all nodes in the current cluster, we need to use Node's service discovery mode. Similarly, configure the following job tasks in the prometheus.yml file:

- job_name: 'kubernetes-nodes'
  kubernetes_sd_configs:
  - role: node

By specifying kubernetes_sd_configsthe mode node, Prometheus will automatically discover all node nodes from Kubernetes and use them as target instances for current job monitoring. The discovered node /metricsinterface is the default kubelet HTTP interface.

After the ConfigMap update of prometheus is completed, we also perform the reload operation to make the configuration take effect:

$ kubectl delete -f prome-cm.yaml
configmap "prometheus-config" deleted
$ kubectl create -f prome-cm.yaml
configmap "prometheus-config" created
# 隔一会儿再执行下面的 reload 操作
$ kubectl get svc -n kube-ops
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
prometheus      NodePort    10.102.74.90    <none>        9090:30358/TCP                   5d
......
$ curl -X POST "http://10.102.74.90:9090/-/reload"

After the configuration takes effect, we go to the prometheus dashboard to check whether Targets can capture data normally and access any node IP: 30358 :

prometheus nodes target

We can see that the above kubernetes-nodesjob task has automatically discovered our three node nodes, but failed when obtaining data, and an error message similar to the following appeared:

Get http://10.151.30.57:10250/metrics: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

This is because when prometheus discovers services in Node mode, the port accessed is 10250 by default , and now there is no /metricsindicator data under this port. Now the kubelet's read-only data interface is exposed through the 10255 port, so we should Let's replace the port here, but do we want to replace it with port 10255 ? No, because we are going to configure node-exporterthe node indicator data captured above, and did we specify it above , so a port 9100hostNetwork=true will be bound to each node , so we should replace 10250 here with 9100 , but how should it be replaced?

Here we need to use the capabilities provided by Prometheus relabel_configs. replaceRelabel can dynamically rewrite the value of the Label through the Metadata information of the Target instance before Prometheus collects data. In addition, we can also choose whether to collect or ignore the Target instance based on the Metadata information of the Target instance. For example, we can match __address__this Label here and then replace the port in it:

- job_name: 'kubernetes-nodes'
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - source_labels: [__address__]
    regex: '(.*):10250'
    replacement: '${1}:9100'
    target_label: __address__
    action: replace

Here is a regular expression to match, __address__then keep the host part and replace the port with 9100. Now we update the configuration file, perform the reload operation, and then look at the kubernetes-nodes job under the Targets path of the Prometheus Dashboard. Is the task normal?

prometheus nodes target2

We can see that it is normal now, but there is another problem. The label of the indicator data we collected only has the hostname of one node. This brings a lot of inconvenience when we perform monitoring group classification queries. If It would be great if we could also obtain the Label labels of the Node nodes in the cluster.

Here we can use labelmapthis attribute to add the Kubernetes Label as the Prometheus indicator label:

- job_name: 'kubernetes-nodes'
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - source_labels: [__address__]
    regex: '(.*):10250'
    replacement: '${1}:9100'
    target_label: __address__
    action: replace
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)

An action is added labelmap, and the regular expression is __meta_kubernetes_node_label_(.+)configured. What this means is that the matching data in the expression is also added to the Label tag of the indicator data.

For kubernetes_sd_configs the following tags are available: Available meta tags:

__meta_kubernetes_node_name: The name of the node object
_meta_kubernetes_node_label : each label in the node object
_meta_kubernetes_node_annotation : each annotation from the node object
_meta_kubernetes_node_address : The first address of each node address type (if present) *

For more information about kubernets_sd_configs, you can view the official documentation: kubernetes_sd_config

In addition, since kubelet also comes with some monitoring indicator data, such as the 10255 port we mentioned above, we also configure the monitoring tasks of kubelet here:

- job_name: 'kubernetes-nodes'
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - source_labels: [__address__]
    regex: '(.*):10250'
    replacement: '${1}:9100'
    target_label: __address__
    action: replace
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)

- job_name: 'kubernetes-kubelet'
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - source_labels: [__address__]
    regex: '(.*):10250'
    replacement: '${1}:10255'
    target_label: __address__
    action: replace
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)

It is particularly important to note that after Kubernetes version 1.11+, the kubelet removed the 10255 port, and the metrics interface returned to the 10250 port , so there is no need to replace the port here, but the https protocol needs to be used. So if you are using Kubernetes version 1.11+, you need to replace the above kubernetes-kubelet tasks with the following configuration :

- job_name: 'kubernetes-kubelet'
  kubernetes_sd_configs:
  - role: node
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)

Now let's update the configuration file, perform the reload operation to make the configuration take effect, and then access the Prometheus Dashboard to view the Targets path:

prometheus node targets

kubernetes-kubeletNow you can see that the two job tasks we added above kubernetes-nodeshave been configured successfully, and the Labels of both are consistent with the node labels of the cluster.

Now we can switch to the Graph path to view some collected indicator data, such as querying the node_load1 indicator:

prometheus nodes graph1

We can see that the node_load1 indicator data corresponding to the three node nodes have been queried. Similarly, we can also use PromQL statements to perform more complex aggregation query operations, and we can also aggregate indicator data based on our Labels tags. , for example, we only query the data of the node03 node here. You can use expressions node_load1{instance="node03"}to query:

prometheus nodes graph2

At this point, we have monitored the Kubernetes cluster nodes using Prometheus. In the next lesson, we will learn how to monitor resource objects such as Pod or Service.

The functions of each Kubelet port are 10250