Shang Silicon Valley Cloud Native Study Notes (173~XXX episodes)

List of notes:

• Shang Silicon Valley Cloud Native Study Notes (1-75 episodes)
• Shang Silicon Valley Cloud Native Study Notes (76~143 episodes)
• Shang Silicon Valley Cloud Native Study Notes (144~172 episodes)
• Shang Silicon Valley Cloud Native Study Notes (173~XXX episodes)

173. k8s scheduling principle - ResourceQuota quota limit_1

173.1. Background

A k8s cluster is often used by multiple teams. For example, our company is the general integrator, so our cluster is used by many third-party companies, and a namespace represents a third-party company. I used multithreading before. Call the event extraction service of another three-party company in the k8s cluster, and then directly blow up their services, causing the company's k8s clusters to be paralyzed, so it is very important to limit the resource quota of each namespace to avoid a single namespace A problem occurs that causes the entire cluster to go down

173.2. Working method

Official documentation: https://kubernetes.io/zh-cn/docs/concepts/policy/resource-quotas/

• Service object: served by the namespace
• Limit range: You can limit the memory, cpu, resource upper limit of each namespace
• Quantity: Multiple ResourceQuota objects can be created for a namespace
• Restriction method: When the resource exceeds the resource limit of ResourceQuota when creating or updating, an error will be reported
• Special restrictions on resource creation by cpu and memory: If we limit the minimum and maximum values ​​of cpu and memory, then the request value (request) and constraint value (limit) must be set when creating resources, otherwise the k8s system will reject the resource. resource creation

173.3. Restrict resource and limit

173.3.1. The yaml file of ResourceQuota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
  namespace: test # 本次使用test名称空间进行测试
spec:
  hard:
    requests.cpu: "1" # 限制cpu初始值是1
    requests.memory: 1Gi # 限制内存初始值是1Gi
    limits.cpu: "2" # 限制cpu最大值是2
    limits.memory: 2Gi # 限制内存最大值是2Gi

173.3.2. Test the yaml file of ResourceQuota

apiVersion: v1
kind: Pod
metadata:
  name: "nginx-resourcequota-pod-test"
  namespace: test
  labels:
    app: "nginx-resourcequota-pod-test"
spec:
  containers:
  - name: nginx-resourcequota-pod-test
    image: "nginx"
    resources:
      limits:
        cpu: 200m
        memory: 500Mi
      requests:
        cpu: 100m
        memory: 200Mi

173.3.3. Test results

Let's check ResourceQuotathe resource usage, as shown below:

173.4. Limit resource count

173.3.1. The yaml file of ResourceQuota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: object-counts
  namespace: test
spec:
  hard:
    configmaps: "10"
    persistentvolumeclaims: "4"
    pods: "4"
    replicationcontrollers: "20"
    secrets: "10"
    services: "10"

173.3.2. Test results

Let's check ResourceQuotathe resource usage, as shown below:

173.5. ResourceQuota Priority

173.5.1. The yaml file of ResourceQuota

It can be seen that this is a List collection, and the writing method and the writing method ---have the same effect

Then multiple ResourceQuotas are set in it, and the priority name is declared, that is, spec.scopeSelector.matchExpressions.valuesthe field. When declaring the Pod, you can spec.priorityClassNamespecify the priority name in

apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: ResourceQuota
  metadata:
    name: pods-high
    namespace: test
  spec:
    hard:
      cpu: "1000"
      memory: 200Gi
      pods: "10"
    scopeSelector:
      matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values: ["high"] # 优先级名称，下面会用到，帮助Pod确定使用哪个ResourceQuota
- apiVersion: v1
  kind: ResourceQuota
  metadata:
    name: pods-medium
    namespace: test
  spec:
    hard:
      cpu: "10"
      memory: 20Gi
      pods: "10"
    scopeSelector:
      matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values: ["medium"] # 优先级名称，下面会用到，帮助Pod确定使用哪个ResourceQuota
- apiVersion: v1
  kind: ResourceQuota
  metadata:
    name: pods-low
    namespace: test
  spec:
    hard:
      cpu: "5"
      memory: 10Gi
      pods: "10"
    scopeSelector:
      matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values: ["low"] # 优先级名称，下面会用到，帮助Pod确定使用哪个ResourceQuota

173.5.2. Test the yaml file of ResourceQuota

apiVersion: v1
kind: Pod
metadata:
  name: high-priority
  namespace: test
spec:
  containers:
  - name: high-priority
    image: nginx
    resources:
      requests:
        memory: "100Mi"
        cpu: "200m"
      limits:
        memory: "100Mi"
        cpu: "200m"
  priorityClassName: high # 设置优先级名称，和上面的spec.scopeSelector.matchExpressions.values的对应

174. k8s scheduling principle - LimitRange use_1

174.1. Background

If we set the sum in ResourceQuota , but do not set the sum when creating the Pod , according to the previous situation, an error will definitely be reported, but after we set it, we can use the default limitrequestlimitrequestlimitLimitRange

Another situation is that a Pod and Container apply for too many resources, crowding out other Pods and Containers. This situation is unacceptable, so it is also necessary to limit the resource request range of Pods and Containers.

174.2. Restricted scope

• Enforce minimum and maximum resource usage limits per Pod or Container within a namespace.
• Enforce the minimum and maximum storage size limits that each PersistentVolumeClaim can claim in a namespace.
• Implements control over the ratio of a resource's requested value to its restricted value in a namespace.
• Set the default application/limit value for computing resources in a namespace, and automatically inject it into multiple Containers at runtime.

174.3. Relationship between ResourceQuota and LimitRange

Both work together to limit

174.4. Set the default value of limit, the default value of request, and the value range of limit

174.4.1 Example of usage of LimitRange

apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-resource-constraint
  namespace: test
spec:
  limits: # 下面举例中都是cpu的限制，当然也可以换成memory，这就是对内存的限制了
 - default: # 此处定义默认限制值，也就是limit的默认值
      cpu: 500m
    defaultRequest: # 此处定义默认请求值，也就是request的默认值
      cpu: 500m
    max: # max 和 min 共同来定义限制范围，也就是limit的范围
      cpu: "1"
    min:
      cpu: 100m
    maxLimitRequestRatio: # limit /request <= 该ratio都是正常的
      cpu: 2
    type: Container # 添加对容器的限制，如果换成是Pod，那就是对Pod的限制了

174.4.2 Example of Pod usage

If we set spec.max.cpuand spec.min.cpu, but not set spec.default, if the value is not set when creating the Pod limit.cpu, then it will use spec.max.cputhe value

174.5. Several other usage examples

• How to configure the minimum and maximum CPU constraints per namespace
• How to configure the minimum and maximum memory constraints per namespace
• How to configure the default CPU request value and limit value for each namespace
• How to configure the default memory application value and limit value for each namespace
• How to configure the minimum and maximum storage usage per namespace
• Detailed example of configuring quotas per namespace

175. k8s scheduling principle - nodeSelector_1

175.1. Background

There are some resources that we only want to deploy on some nodes, so we can use labels to select nodes. We have used this when deploying ingress. After all, ingress only needs to be deployed on some nodes. In addition, I also encountered I have seen one thing, which is a master-slave architecture. Some information needed by jenkins only exists on the master node, but after jenkins starts, it will drift to the slave node. At that time, this situation was avoided, so we just When restarting jenkins, first suspend the scheduling of the slave node, and then deploy jenkins to the master node. It is a good choice to use nodeSelector in this case

175.2. Examples of use

apiVersion: v1
kind: Pod
metadata:
  name: "nginx-nodeselector-test"
  namespace: default
  labels:
    app: "nginx-nodeselector-test"
spec:
  containers:
  - name: nginx-nodeselector-test
    image: "nginx"
  nodeSelector: # 用node节点标签去选择部署的node
    kubernetes.io/os: linux # node节点中的标签，可以用kubectl get node --show-labels命令查看node节点的标签

176、Qingyun-Canceling the contract and operating according to the volume_1

Do not care

177. k8s scheduling principle - limitrange other setting items_1

# 174、k8s调度原理-LimitRange使用_1already mentioned in it

178. k8s scheduling principle - affinity and anti-affinity_1

178.1 Affinity and anti-affinity concepts

There must be a reason why pods are deployed to some machines or not to some machines. Affinity is to choose some machines you like, and anti-affinity is the opposite. The choice of machines can be based on these three points. These can be kubectl explain pod.spec.affinityfound via:

• nodeAffinity: Node affinity, that is to say, I will go to the tags of Node
• podAffinity: Pod affinity, that is to say, which Pods I will go to
• podAntiAffinity: Pod anti-affinity, that is to say, I will not go to any Pods

179. k8s scheduling principle - NodeAffinity_1

179.1. Two filter conditions of NodeAffinity

• pod.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution: Hard filtering, the condition must be met to be scheduled to the node; DuringSchedulingthe representative is valid during the scheduling period (scheduling is the role of the scheduler in k8s), and IgnoredDuringExecutionthe representative is ignored during the runtime
• pod.spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution: Soft scoring, which will be prioritized to nodes with high scores, and can be dynamically selected according to scores

179.2. The role of affinity strategy, anti-affinity strategy and topological partition

These methods are all to control the deployment node location of Pod

In order to allow Pods to be assigned to some nodes according to our wishes, or not to be assigned to nodes, that is to say, the deployment of Pods can be controlled by us

The role of these strategies is only valid during scheduling, and then invalid during runtime, which is DuringSchedulingIgnoredDuringExecutionthe origin of the above name

179.3. If none of the required affinity policies are met, what is the deployment status?

That is, it will not be deployed, but it will always try to wait for a node that meets the requirements to appear for deployment

179.4. Mandatory selection of node labels (requiredDuringSchedulingIgnoredDuringExecution)

apiVersion: v1
kind: Pod
metadata:
  name: "busybox-nodeaffinity"
  namespace: default
  labels:
    app: "busybox-nodeaffinity"
spec:
  containers:
  - name: busybox-nodeaffinity
    image: "busybox"
    command: ['sleep', '3600']
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms: # 节点选择
        - matchExpressions: # 使用标签进行选择
          - key: kubernetes.io/hostname # 标签名称
            operator: In # 操作符可以是In（存在于）, NotIn（不存在于）, Exists（存在，连values参数都不用写）, DoesNotExist（不存在，连values参数都不用写）. Gt（大于指定值，但是值两边也要加引号，比如：['100', '200']）, Lt（小于指定值，虽然是数字，但是值两边也要加引号，比如：['100', '200']）
            values: ['k8s-01'] # 标签键是kubernetes.io/hostname，值是k8s-01，并且是必须调度条件，那肯定会调度到k8s-01节点

The deployment effect is as follows:

179.5. Choose the preferred node label (preferredDuringSchedulingIgnoredDuringExecution)

apiVersion: v1
kind: Pod
metadata:
  name: "busybox-nodeaffinity-prefer"
  namespace: default
  labels:
    app: "busybox-nodeaffinity-prefer"
spec:
  containers:
  - name: busybox-nodeaffinity-prefer
    image: "busybox"
    command: ['sleep', '3600']
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference: # 节点选择
          matchExpressions: # 使用标签进行选择
          - key: kubernetes.io/hostname # 标签名称
            operator: In # 操作符可以是In, NotIn, Exists, DoesNotExist. Gt, Lt
            values: ['k8s-01'] # 标签键是kubernetes.io/hostname，值是k8s-01，并且是软性调度条件，其中k8s-02节点分数更高，那肯定会调度到k8s-02节点
        weight: 20 # 权重值范围0~100
      - preference: # 节点选择
          matchExpressions: # 使用标签进行选择
          - key: kubernetes.io/hostname # 标签名称
            operator: In # 操作符可以是In, NotIn, Exists, DoesNotExist. Gt, Lt
            values: ['k8s-02'] # 标签键是kubernetes.io/hostname，值是k8s-02，并且是软性调度条件，那肯定会调度到k8s-02节点
        weight: 80 # 权重值范围0~100

The deployment effect is as follows:

180. k8s scheduling principle - affinity and anti-affinity and topological keys_1

180.1, Pod affinity - label mandatory selection

illustrate:

The following is the Pod label affinity strategy, and then the topology key topologyKey is also mentioned, and then the function of the topology key is discussed. As long as the Pod label affinity is met, it is assumed that node A conforms, and then node B does not conform, but the topology of node B The value of the node label in the key is the same as that of topology A, so node B also conforms to Pod affinity

apiVersion: v1
kind: Pod
metadata:
  name: "busybox-podaffinity-require"
  namespace: default
  labels:
    app: "busybox-podaffinity-require"
spec:
  containers:
  - name: busybox-podaffinity-require
    image: "busybox"
    command: ['sleep', '3600']
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector: # 标签选择，还可以使用namespaceSelector（名称空间通过标签选择）、namespaces（名称空间通过名称选择）、topologyKey（使用拓扑键进行选择）
          matchExpressions: # 使用标签进行选择，还可以使用matchLabels
            - key: app # Pod的标签名称
              operator: In
              values: ['nginx'] # Pod的标签值，由于Pod标签app=nginx的只有在k8s-node2节点存在，所以本次肯定部署在k8s-node2节点
        topologyKey: "kubernetes.io/hostname" # node节点标签名称，只要和该pod所在的节点的该标签值相同，那就是同一个拓扑网络，那享受的亲和/反亲和待遇是一样的

180.2. Pod anti-affinity—avoid multiple Pods from being deployed on the same node by using the same label

illustrate:

Our purpose is to busyboxdeploy two Pod copies to different nodes. The label of the Pod is that app=busybox-podantiaffinityif Pod1 is deployed on Node A, then the second Pod cannot be deployed on Node A. After all, for Pod anti-parenting and principles

apiVersion: apps/v1
kind: Deployment
metadata:
  name:  busybox-podantiaffinity
  namespace: default
  labels:
    app:  busybox-podantiaffinity
spec:
  selector:
    matchLabels:
      app: busybox-podantiaffinity
  replicas: 2
  template:
    metadata:
      labels:
        app:  busybox-podantiaffinity
    spec:
      containers:
      - name:  busybox-podantiaffinity
        image:  "busybox"
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector: # 标签选择，还可以使用namespaceSelector、namespaces、topologyKey
                matchExpressions: # 使用标签进行选择，还可以使用matchLabels
                  - key: app # Pod的标签名称
                    operator: In
                    values: ['busybox-podantiaffinity'] # 本次部署的busybox的标签是app=busybox-podantiaffinity，我们本次的目的是避免两个容器部署到同一台机器上，所以我们禁止Pod部署到已经含有该标签的节点上
              topologyKey: "kubernetes.io/hostname" # node节点标签名称，只要和该pod所在的节点的该标签值相同，那就是同一个拓扑网络，那享受的亲和/反亲和待遇是一样的

The deployment results are as follows:

181. Principle of k8s scheduling - the stain of node_1

181.1. Why the master node cannot be scheduled by default

Because there are taints that cannot be scheduled by default on the mater node, we can view the effect through kuboard. The NodeSchedule in the lower right corner of the figure below is the taint, as follows:

181.2, stain type

• NoSchedule: Pods cannot be scheduled to this node, but it does not affect the use of existing Pods on the node. It is often used on the master node
• NoExecute: Pods cannot be scheduled to this node, and all Pods already on this node are expelled, commonly used in node maintenance
• PreferNoSchedule: try not to schedule, but not impossible to schedule

181.3. Add taint to nodes

The method of adding stains and adding labels is different, but stains are also added 节点, written as follows:

// 节点名称：可以通过hostname获取，比如k8s-01
// 污点键：随意取
// 污点值：随意取
// 污点效果：NoSchedule（不调度Pod到该节点）、NoExecute（不调度Pod到该节点，驱逐已有节点）、PreferNoSchedule（尽量不调度Pod到该节点）
kubectl taint node 节点名称 污点键=污点值:污点效果

For example:

kubectl taint nodes k8s-01 taint-key=taint-value:NoSchedule

181.4. Check the taint on the node

The command is as follows:

// 节点名称：可以通过hostname获取，比如k8s-01
kubectl describe node 节点名称

For example:

kubectl describe node k8s-01

We can find it directly in the result Taints, the effect is as follows:

181.5. Delete the taint on the node

// 1、最简单写法，例如：kubectl taint node k8s-01 taint-key-
kubectl taint node 节点名称 污点键-

// 2、简单写法，例如：kubectl taint node k8s-01 taint-key:NoSchedule-
kubectl taint node 节点名称 污点键:污点效果-

// 3、完整写法，例如：kubectl taint node k8s-01 taint-key=taint-value:key:NoSchedule-
kubectl taint node 节点名称 污点键=污点值:污点效果-

182. k8s scheduling principle - pod tolerance_1

182.1, Pod tolerance writing

Before the following script is executed, I have marked the two nodes k8s-01 and k8s-02 as unschedulable

182.1.1, full write

apiVersion: v1
kind: Pod
metadata:
  name: "busybox-pod-tolerations-01"
  namespace: default
  labels:
    app: "busybox-pod-tolerations-01"
spec:
  containers:
  - name: busybox-pod-tolerations-01
    image: "busybox"
    command: ['sleep', '3600']
  tolerations:
  - key: "node.kubernetes.io/unschedulable" # 污点键
    operator: "Equal" # 操作符，默认是Equal，其他可以选择的值是：Exists
    value: "" # 污点值
    effect: "NoSchedule" # 污点类型，可以选择的值是：NoSchedule（不调度Pod到该节点）、NoExecute（不调度Pod到该节点，驱逐已有节点）、PreferNoSchedule（尽量不调度Pod到该节点）

182.1.2, write only key and operator

apiVersion: v1
kind: Pod
metadata:
  name: "busybox-pod-tolerations-02"
  namespace: default
  labels:
    app: "busybox-pod-tolerations-02"
spec:
  containers:
  - name: busybox-pod-tolerations-02
    image: "busybox"
    command: ['sleep', '3600']
  tolerations:
  - key: "node.kubernetes.io/unschedulable" # 污点键
    operator: "Exists" # 操作符，默认是Equal，其他可以选择的值是：Exists

182.1.3, write-only key, operator, effect

apiVersion: v1
kind: Pod
metadata:
  name: "busybox-pod-tolerations-03"
  namespace: default
  labels:
    app: "busybox-pod-tolerations-03"
spec:
  containers:
  - name: busybox-pod-tolerations-03
    image: "busybox"
    command: ['sleep', '3600']
  tolerations:
  - key: "node.kubernetes.io/unschedulable" # 污点键
    operator: "Exists" # 操作符，默认是Equal，其他可以选择的值是：Exists
    effect: "NoSchedule" # 污点类型，可以选择的值是：NoSchedule（不调度Pod到该节点）、NoExecute（不调度Pod到该节点，驱逐已有节点）、PreferNoSchedule（尽量不调度Pod到该节点）

182.1.4. The tolerable taint type in the node is expulsion NoExecute

If the taint type that the Pod can tolerate is to expel NoExecute, it can also be deployed and will not be evicted

182.1.5, specify the time toleranceSeconds of the Pod tolerating the taint

The number of seconds to tolerate taint. If this second is exceeded, the Pod will be reassigned and rarely used. The full path is:pod.spec.tolerations.tolerationSeconds

By default, it is not set, that is, this time is unlimited

182.2. Common smudge writing, easy to troubleshoot

If the node is unavailable, we can see if there is a stain on the node. If so, then troubleshoot according to the stain type

183. k8s Scheduling Principle - Topology Partition Constraint_1

illustrate:

To put it bluntly, topological partition constraints are Podhow to select nodes during deployment. First, topologyKeydivide the partition according to the partition. It is possible that multiple nodes are in the same partition. This is supported, and then labelSelectorselect the appropriate Pod according to the partition. PodOnly partitions can be deployed, and then multiple partitions may meet the requirements. At this time, the maxSkewmaximum inclination depends on the maximum inclination of all nodes. As for whether the maximum inclination must be satisfied, it depends on the whenUnsatisfiable, if the value is DoNotSchedule, then you must ensure that the degree of inclination is equal to the degree of maximum inclination. If the value is ScheduleAnyway, it will be processed according to the optimal solution

effect:

• A more detailed strategy for specifying topological networks
• Used to plan and balance resources across the cluster

yaml file:

apiVersion: v1
kind: Pod
metadata:
  name: "nginx-topologyspreadconstraints"
  namespace: default
  labels:
    app: "nginx-topologyspreadconstraints"
spec:
  containers:
  - name: nginx-topologyspreadconstraints
    image: "nginx"
  topologySpreadConstraints:
  - topologyKey: "kubernetes.io/hostname" # 首先根据node的标签先划分分区，可能多个节点是同一个分区的
    labelSelector: # 根据Pod标签来选择适合的部署节点，也就说这个节点上有这个标签的Pod我才去部署
      matchLabels: # 直接匹配标签，还可以有另外一种写法是matchExpressions，这个是通过表达式中的标签去选择Pod
        app: "nginx" # 标签键和标签值
    maxSkew: 1 # 最大倾斜，这个最大倾斜说的部署完成之后所有分区中上述标签所属Pod数量的差值，如果有A和B两个区域，比如A区域中有两个标签app=nginx的Pod，然后B区域中没有一个，那么我们这个Pod的标签如果也是app=nginx的话，那Pod倾斜程度就是2-1=1，刚刚好
    whenUnsatisfiable: DoNotSchedule # 如果上面条件不满足的时候的做法，该值是默认值，也就是不满足最大倾斜就不调度。其他值是:ScheduleAnyway（按最优解处理）

184. k8s Scheduling Principle - Resource Scheduling Strategy_1

184.1. Determine the Pod allocation according to the runtime requests request information and the remaining resources of the node

We all know that the value of the sum can be set at the time Podof declaration .resourcesrequestslimits

If there is such a situation now, we have three containers in a Pod, and the values 1~3​​​​set in the containers requests.cpuare 200m, 300m, and 100m respectively, and now there are three available nodes with A~Ccpu remaining values ​​of 300m, 500m, and 700m respectively, then Where will this Pod choose? It will definitely choose node C. The reason is that only node C will meet the requirements. The resources required by the Pod to run are the total amount of all resources required. This is the key to choosing a node. You must understand that Pod is the smallest unit in K8s, not a container, so Pod must be deployed as a whole

184.2. Determine the Pod allocation according to the requests request information during initialization and the remaining resources of the node

Assume that multiple containers need to be started during the Pod startup. For example, the following is the startup of three containers. The cpu resources they need are 300m, 100m, 200m, and they must be started one by one, so as long as the maximum memory required by the container can be supported , that can be deployed on it. For example, A~Cthe cpu remaining values ​​of the three available nodes are 100m, 300m, , respectively 700m, so both node B and node C can deploy the Pod

184.3. How to use the commands of adding non-scheduling taints, deleting non-scheduling taints, and adding expulsion taints

• cordon (add non-scheduling taint, that is, taint NoSchedule, the full content of the taint is: node.kubernetes.io/unschedulableNoSchedule): the command is kubectl cordon 节点名称, for example:kubectl cordon k8s-01
• uncordon (remove unscheduled taints, that is, cancel NoScheduletaints): the command is kubectl uncordon k8s-01, for example:kubectl uncordon k8s-01
• drain (add expulsion taint, that is, taint NoExecute, never use): the command is kubectl drain k8s-01, for example:kubectl drain k8s-01

185. k8s security - role-based access control_1

185.1. RBAC Theory

The full name of RBAC is Role Based Access Controllerrole-based access control, which is to 用户》角色》权限complete the user's permission control

185.2. Authority control process

1. The user sends a request to modify cluster resources to the api-server of k8s with the token/certificate
2. k8s certification
3. k8s query user permissions
4. Use admission control to determine whether such an operation can be supported

185.3. Four kinds of K8s objects

• ServiceAccount: user, the password of each user exists in Secret, and there is a token in it which is the password and belongs to the namespace
• RoleBinding: Bind ServiceAccount and Role, which belong to the namespace
• Role: A namespace-based role that can operate resources under the namespace and belongs to the namespace
• ClusterRoleBinding: Bind ServiceAccount and ClusterRole, belonging to the namespace
• ClusterRole: A cluster-based role that can operate cluster resources

Description: ServiceAccount by RoleBindingbinding Role, by ClusterRoleBindingbindingClusterRole

186. How to write k8s security-role and clusterrole_1

186.1. Create ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-account # 用户名称

186.2. Create a Role (the role belongs to the namespace)

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: test-role
  namespace: default # Role属于名称空间
rules:
- apiGroups:
  - "" # 空字符串代表核心API组，一般都不用写
  resources:
  - pods # 资源类型名称，可以通过kubectl api-resources查看
  # resourceNames: # 很少使用，基本不用
  # - XXX # 上面指定资源类型名称，这里指定白名单，也就是说上面资源中白名单内的资源可以访问，但是上面资源中白名单外的资源都不能访问
  verbs:
  - get # 操作，可以通过kubectl api-resources -owide  | grep 资源名称的方式去查看，最后一个字段的值就是该值

186.3. Create a ClusterRole (the cluster role does not belong to the namespace)

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: test-clusterrole # clusterrole不属于任何名称空间
rules:
- apiGroups:
  - "" # 空字符串代表核心API组，一般都不用写
  resources:
  - namespaces # 资源类型名称，可以通过kubectl api-resources查看
  verbs:  # 操作，可以通过kubectl api-resources -owide  | grep 资源名称的方式去查看，最后一个字段的值就是该值
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups: # 可以写多个
  - "" # 空字符串代表核心API组，一般都不用写
  resources:
  - pods # 资源类型名称，可以通过kubectl api-resources查看
  verbs:  # 操作，可以通过kubectl api-resources -owide  | grep 资源名称的方式去查看，最后一个字段的值就是该值
  - create
  - delete

187. k8s security - why can dash operate the cluster_1

Because we k8s-dashboardset the user, role, and permission information at that time, these information worked, let's go directly to the kuboard page!

188. k8s security - ServiceAccount and ClusterRole actual combat_1

188.1. Create a RoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: test-rolebinding
  namespace: default
subjects:
- apiGroup: ""
  kind: ServiceAccount # 账户类型
  name: test-account # test-account是账户名称
roleRef:
  apiGroup: ""
  kind: Role # 角色类型
  name: test-role # test-role是角色名称

188.2. Creating a ClusterBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: test-clusterrolebinding
  namespace: default
subjects:
- apiGroup: ""
  kind: ServiceAccount # 账户类型
  name: test-account # test-account是账户名称
  namespace: default # 还必须写，不写还报错
roleRef:
  apiGroup: ""
  kind: ClusterRole # 集群角色类型
  name: test-clusterrole # test-clusterrole是集群角色名称

188.3. Log in with the token of the new user

We assume that we want to log in to k8sdashboard with a new account, then we need to obtain a login token through kuboard, as follows:

Just copy the token into the input box of k8sdashboard, as follows:

189. k8s Security-Pod ServiceAccount Precautions_1

189.1. Empty default service account

After the namespace is created, a default service account will be automatically created under the namespace, and this service account is empty, and each Pod will mount this account by default. If we want the Pod to have more permissions, The name of the service account can be specified when the Pod is created, that is pod.spec.serviceAccountName, the Pod has the permissions of the service account

189.2. Analyze the permission of nfs dynamic mount

It is best to directly look at the association relationship configuration files of users, roles, and permissions that are dynamically mounted by nfs. The location is: 85、k8s-集群创建完成》9.2、安装步骤（只用在nfs主节点执行）.

190. k8s security - how to develop a k8s visualization platform_1

The official API documentation is provided as follows:

We can test the api interface through Postman, as follows:

In addition, those who write java code can also integrate it through the java sdk development kit in the reference document , as follows:

191. k8s Security - Supplement_1

nothing to add

192. k8s ecology-helm application store_1

192.1. Install helm

It has already been introduced in " 85、k8s-集群创建完成, so I won't repeat it here10、安装helm

192.2. Related concepts

• Official website: https://helm.sh/zh/
• helm command line: https://helm.sh/zh/docs/intro/using_helm/
• Warehouse: https://artifacthub.io/
• Frames:
  

192.3. Common commands

• helm repo add warehouse name warehouse address: add a local warehouse, for example: the warehouse helm repo add bitnami https://charts.bitnami.com/bitnamiwill be bitnamiadded to this warehouse
• helm list: View all services deployed through helm
• helm search hub: Search for deployable services from the central warehouse
• helm search repo: Search from the warehouse you helm repo add added through the command, such asbitnami
• helm pull warehouse name/product name: for example helm pull bitnami/mysql, the product name can be queried from the central warehousehelm search repo , we can go to the warehouse first or check whether it exists in the warehouse through commands mysql; this downloads the compressed package of tgz, and then passes the compressed package tar -zxvf 压缩包名称进行解压, inside There are files that need to be executed
• helm install -f values.yaml helm service name ./: Install the mysql cluster through helm, for example:helm install -f values.yaml mysql-cluster ./
• helm uninstall service name: For example, helm uninstall mysql-clusterit can be uninstalled, and the service name can be helm listsearched, as follows:
  

192.4. Contents of the tgz compressed package downloaded by helm pull

• Chart.yaml: metadata description information
• templates: the yaml information used in k8s, many of the yaml files are variable names, from values.yamlthe
• values.yaml: Configuration information, k8smany yamlfile attribute values ​​​​come from here

192.5. Install a master-slave synchronous mysql cluster

192.5.1. Download the mysql product from the central warehouse and decompress it

The first download command ishelm pull bitnami/mysql

Then tar -zxvf XXX.tgzjust unzip it, and then enter mysqlthe directory where the decompression is completed

192.5.1. Modify vlaues.yaml

Modify the first service.type to NodePort:

values.yamlBy searching in , ClusterIPthen change it to NodePort, as follows:

Modify the second service.type to NodePort:

values.yamlBy searching in , ClusterIPthen change it to NodePort, as follows:

Modify the database password, for example, I changed it to 123456

values.yamlSearch in , rootPasswordand then change the password, as follows:

Modify the database mode to cluster mode:

values.yamlBy searching in standalone, and then modify the pattern replicationas follows:

After modifying the previous content, click the down arrow and slide down a little, and then modify the copy user and password, as follows:

Adjust the number of replicas in cluster mode:

values.yamlSearch in , replicaCountand then adjust the quantity to 2, as follows:

192.5.2. Executing commands

values.yamlExecute the following command in :

# values.yaml：是配置文件名称，这就是上面修改的那个
# mysql-cluster：这是helm应用的名称，也就是通过helm list查看到的名称
helm install -f values.yaml mysql-cluster ./

193. k8s small experiment - deploy mysql stateful service_1

193.1. Create a namespace

kubectl create ns mysql

193.2. Create ConfigMap

By kubectl apply -f XXX.yamlexecuting the following yaml file

apiVersion: v1
data:
  my.cnf: >
    [client]

    default-character-set=utf8


    [mysql]

    default-character-set=utf8


    [mysqld]

    # 设置client连接mysql时的字符集,防止乱码

    init_connect='SET NAMES utf8'

    init_connect='SET collation_connection = utf8_general_ci'


    # 数据库默认字符集

    character-set-server=utf8


    #数据库字符集对应一些排序等规则，注意要和character-set-server对应

    collation-server=utf8_general_ci


    # 跳过mysql程序起动时的字符参数设置 ，使用服务器端字符集设置

    skip-character-set-client-handshake


    #
    禁止MySQL对外部连接进行DNS解析，使用这一选项可以消除MySQL进行DNS解析的时间。但需要注意，如果开启该选项，则所有远程主机连接授权都要使用IP地址方式，否则MySQL将无法正常处理连接请求！

    skip-name-resolve
kind: ConfigMap
metadata:
  name: mysql-config
  namespace: mysql

Explanation: The purpose of creating a ConfigMapconfiguration is my.cnfto propose the configuration file, and the generated effect is as follows:

193.3. Create mysql Service

By kubectl apply -f XXX.yamlexecuting the following yaml file

apiVersion: v1
kind: Namespace
metadata:
  name: mysql # 创建名称空间
spec: {
    
    }

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql # 匹配下面spec.template中的labels中的键值对
  serviceName: mysql # 一般和service的名称一致，用来和service联合，可以用来做域名访问
  template:
    metadata:
      labels:
        app: mysql # 让上面spec.selector.matchLabels用来匹配Pod
    spec:
      containers:
        - env:
            - name: MYSQL_ROOT_PASSWORD # 密码
              value: "123456"
          image: 'mysql:5.7'
          livenessProbe: # 存活探针
            exec:
              command:
                - mysqladmin
                - '-uroot'
                - '-p${MYSQL_ROOT_PASSWORD}'
                - ping
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          name: mysql
          ports:
            - containerPort: 3306 # 容器端口
              name: client
              protocol: TCP
          readinessProbe: # 就绪探针
            exec:
              command:
                - mysqladmin
                - '-uroot'
                - '-p${MYSQL_ROOT_PASSWORD}'
                - ping
            failureThreshold: 3
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          volumeMounts: # 挂载声明
            - mountPath: /etc/mysql/conf.d/my.cnf # 配置文件
              name: conf
              subPath: my.cnf
            - mountPath: /var/lib/mysql # 数据目录
              name: data
            - mountPath: /etc/localtime # 本地时间
              name: localtime
              readOnly: true
      volumes:
        - configMap: # 配置文件使用configMap挂载
            name: mysql-config
          name: conf
        - hostPath: # 本地时间使用本地文件
            path: /etc/localtime
            type: File
          name: localtime
  volumeClaimTemplates: # 数据目录使用nfs动态挂载，下面的作用就是指定PVC
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: data # 和上面volumeMounts下面的name=data那个对应
      spec:
        accessModes:
          - ReadWriteMany # 多节点读写
        resources:
          requests:
            storage: 1Gi
        storageClassName: managed-nfs-storage # nfs存储类名称

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app: mysql
  name: mysql # 存储类名称
  namespace: mysql
spec:
  ports:
    - name: tcp
      port: 3306
      targetPort: 3306
      nodePort: 32306
      protocol: TCP
  selector:
    app: mysql # Pod选择器
  type: NodePort

194. MySQL is not master-slave synchronization by default, you need to set it yourself_1

It means that you need to configure the master-slave synchronization by yourself, but you don’t say the specific process, just don’t look at it

194~231, self-built cluster, ceph build

At present, I don't need to build a cluster by myself, and I need an external disk during the ceph building process. I don't meet the conditions here. Let's watch it later.

232~241, prometheus, harbor warehouse construction

You can just look at the operations in the k8s notes (continuously updated)二、安装其他组件》4、安装prometheus和grafana和5、安装harbor , but there is still a problem, let’s do this for now, and I will add it later, TODO

I won't summarize the following for the time being, and I will talk about it later