Explain in detail the whole process of MeterSphere configuring external Mysql5.7

Explain in detail the whole process of MeterSphere configuring external Mysql5.7

I bought cloud server from cnaaa.com.

Build a MySQL cluster with master-slave replication

Principle of local storage

Local storage is used here, that is to say, users hope that Kubernetes can directly use the local disk directory on the host without relying on remote storage services to provide persistent container Volume 1. We fix the storage
on a node, But when the Pod is scheduled, it floats around. How can the Pod be fixed on the PVC through the PVC?
2. Is it okay to add a nodeAffinity to this Pod?

Of course, but this destroys the developer's definition of resource objects in a disguised way, and the developer should not need to consider the details of scheduling all the time. Scheduling changes should be handed over to O&M. So in order to implement local storage, we adopted the method of delayed binding . The method is very simple. We all know that StorageClass is generally designed by operation and maintenance personnel. We only need to specify no-provisioner in StorageClass. This is because Local Persistent Volume does not yet support Dynamic Provisioning, so it cannot automatically create a corresponding PV when a user creates a PVC. At the same time, this StorageClass also defines a property of volumeBindingMode=WaitForFirstConsumer. It is a very important feature in Local Persistent Volume, namely: late binding.

Create a PV

First, pre-allocate several PVs on the Node (the IP of the Node node used in the experiment is node1) (it is not recommended to do this in production), and ensure that there is a corresponding directory on node1

ssh root@node1
mkdir -pv /data/svr/projects/{
    
    mysql,mysql2,mysql3}

vim 01-persistentVolume-1.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-mysql-pv
spec:
  capacity:
    storage: 15Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /data/svr/projects/mysql
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1

vim 01-persistentVolume-2.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-mysql-pv-2
spec:
  capacity:
    storage: 15Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /data/svr/projects/mysql2
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1

vim 01-persistentVolume-3.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-mysql-pv-3
spec:
  capacity:
    storage: 15Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /data/svr/projects/mysql3
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1

Remember, this is not recommended in production. I just manually pre-created it for experimental purposes. **The formal method should use Dynamic Provisioning through StorageClass instead of Static Provisioning mechanism to produce PV

[root@master mysql]# kubectl apply -f 01-persistentVolume-1.yaml 
persistentvolume/example-mysql-pv created
[root@master mysql]# kubectl apply -f 01-persistentVolume-2.yaml 
persistentvolume/example-mysql-pv-2 created
[root@master mysql]# kubectl apply -f 01-persistentVolume-3.yaml 
persistentvolume/example-mysql-pv-3 created
[root@master mysql]# kubectl get pv
NAME                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS    REASON   AGE
example-mysql-pv     15Gi       RWO            Delete           Available           local-storage            8s
example-mysql-pv-2   15Gi       RWO            Delete           Available           local-storage            6s
example-mysql-pv-3   15Gi       RWO            Delete           Available           local-storage            3s

02-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

[root@master mysql]# kubectl apply -f 02-storageclass.yaml 
storageclass.storage.k8s.io/local-storage created
[root@master mysql]# kubectl get sc
NAME            PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-storage   kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  5s

Create Namespace

03-mysql-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: mysql
  labels:
    app: mysql

[root@master mysql]# kubectl apply -f 03-mysql-namespace.yaml 
namespace/mysql unchanged
[root@master mysql]# kubectl get ns -n mysql
NAME              STATUS   AGE
mysql             Active   3d

Create a database configuration file configmap

Use ConfigMap to assign different configuration files to Master/Slave nodes

vim 04-mysql-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql
  namespace: mysql
  labels:
    app: mysql
data:
  master.cnf: |
    # Master配置
    [mysqld]
    log-bin=mysqllog
    skip-name-resolve
  slave.cnf: |
    # Slave配置
    [mysqld]
    super-read-only
    skip-name-resolve
    log-bin=mysql-bin
    replicate-ignore-db=mysql

[root@master mysql]# kubectl apply -f 04-mysql-configmap.yaml 
configmap/mysql created
[root@master mysql]# kubectl get cm -n mysql
NAME               DATA   AGE
mysql              2      5s

Create a MySQL password Secret

05-mysql-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: mysql-secret
  namespace: mysql
  labels:
    app: mysql
type: Opaque
data:
  password: MTIzNDU2 # echo -n "123456" | base64

[root@master mysql]# kubectl apply -f 05-mysql-secret.yaml 
secret/mysql-secret created
[root@master mysql]# kubectl get secret -n mysql
NAME                  TYPE                                  DATA   AGE
mysql-secret          Opaque                                1      18s

Use Service to provide read-write separation for MySQL

06-mysql-services.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: mysql
  labels:
    app: mysql
spec:
  ports:
  - name: mysql
    port: 3306
  clusterIP: None
  selector:
    app: mysql
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-read
  namespace: mysql
  labels:
    app: mysql
spec:
  ports:
  - name: mysql
    port: 3306
  selector:
    app: mysql

All user write requests must directly access the Master node in the form of DNS records, that is, the DNS record mysql-0.mysql.

All user read requests must access the automatically assigned DNS record and can be forwarded to any Master or Slave node, that is, the DNS record mysql-read.

[root@master mysql]# kubectl apply -f 06-mysql-services.yaml 
service/mysql created
service/mysql-read created
[root@master mysql]# kubectl get svc -n mysql
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
mysql        ClusterIP   None           <none>        3306/TCP   3s
mysql-read   ClusterIP   10.107.90.19   <none>        3306/TCP   3s

Create a MySQL cluster instance

Use StatefulSet to build MySQL master-slave cluster

07-mysql-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: mysql
  labels:
    app: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  serviceName: mysql
  replicas: 2
  template:
    metadata:
      labels:
        app: mysql
    spec:
      initContainers:
      - name: init-mysql
        image: mysql:5.7
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        command:
        - bash
        - "-c"
        - |
          set -ex
          # 从Pod的序号，生成server-id
          [[ $(hostname) =~ -([0-9]+)$ ]] || exit 1
          ordinal=${
    
    BASH_REMATCH[1]}
          echo [mysqld] > /mnt/conf.d/server-id.cnf
          # 由于server-id不能为0，因此给ID加100来避开它
          echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
          # 如果Pod的序号为0，说明它是Master节点，从ConfigMap里把Master的配置文件拷贝到/mnt/conf.d目录下
          # 否则，拷贝ConfigMap里的Slave的配置文件
          if [[ ${ordinal} -eq 0 ]]; then
            cp /mnt/config-map/master.cnf /mnt/conf.d
          else
            cp /mnt/config-map/slave.cnf /mnt/conf.d
          fi
        volumeMounts:
        - name: conf
          mountPath: /mnt/conf.d
        - name: config-map
          mountPath: /mnt/config-map
      - name: clone-mysql
        image: gcr.io/google-samples/xtrabackup:1.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        command:
        - bash
        - "-c"
        - |
          set -ex
          # 拷贝操作只需要在第一次启动时进行，所以数据已经存在则跳过
          [[ -d /var/lib/mysql/mysql ]] && exit 0
          # Master 节点（序号为 0）不需要这个操作
          [[ $(hostname) =~ -([0-9]+)$ ]] || exit 1
          ordinal=${
    
    BASH_REMATCH[1]}
          [[ $ordinal == 0 ]] && exit 0
          # 使用ncat指令，远程地从前一个节点拷贝数据到本地
          ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
          # 执行 --prepare，这样拷贝来的数据就可以用作恢复了
          xtrabackup --prepare --target-dir=/var/lib/mysql
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
      containers:
      - name: mysql
        image: mysql:5.7
        env:
 #        - name: MYSQL_ALLOW_EMPTY_PASSWORD
 #          value: "1"
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        ports:
        - name: mysql
          containerPort: 3306
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
        livenessProbe:
          exec:
            command: ["mysqladmin", "ping", "-uroot", "-p${MYSQL_ROOT_PASSWORD}"]
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command: ["mysqladmin", "ping", "-uroot", "-p${MYSQL_ROOT_PASSWORD}"]
          initialDelaySeconds: 5
          periodSeconds: 2
          timeoutSeconds: 1
      - name: xtrabackup
        image: gcr.io/google-samples/xtrabackup:1.0
        ports:
        - name: xtrabackup
          containerPort: 3307
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        command:
        - bash
        - "-c"
        - |
          set -ex
          cd /var/lib/mysql
          # 从备份信息文件里读取MASTER_LOG_FILE和MASTER_LOG_POS这2个字段的值，用来拼装集群初始化SQL
          if [[ -f xtrabackup_slave_info ]]; then
            # 如果xtrabackup_slave_info文件存在，说明这个备份数据来自于另一个Slave节点
            # 这种情况下，XtraBackup工具在备份的时候，就已经在这个文件里自动生成了“CHANGE MASTER TO”SQL语句
            # 所以，只需要把这个文件重命名为change_master_to.sql.in，后面直接使用即可
            mv xtrabackup_slave_info change_master_to.sql.in
            # 所以，也就用不着xtrabackup_binlog_info了
            rm -f xtrabackup_binlog_info
          elif [[ -f xtrabackup_binlog_info ]]; then
            # 如果只是存在xtrabackup_binlog_info文件，说明备份来自于Master节点，就需要解析这个备份信息文件，读取所需的两个字段的值
            [[ $(cat xtrabackup_binlog_info) =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
            rm xtrabackup_binlog_info
            # 把两个字段的值拼装成SQL，写入change_master_to.sql.in文件
            echo "CHANGE MASTER TO MASTER_LOG_FILE='${
     
     BASH_REMATCH[1]}',\
                  MASTER_LOG_POS=${
     
     BASH_REMATCH[2]}" > change_master_to.sql.in
          fi
          # 如果存在change_master_to.sql.in，就意味着需要做集群初始化工作
          if [[ -f change_master_to.sql.in ]]; then
            # 但一定要先等MySQL容器启动之后才能进行下一步连接MySQL的操作
            echo "Waiting for mysqld to be ready（accepting connections）"
            until mysql -h 127.0.0.1 -uroot -p${MYSQL_ROOT_PASSWORD} -e "SELECT 1"; do sleep 1; done
            echo "Initializing replication from clone position"
            # 将文件change_master_to.sql.in改个名字
            # 防止这个Container重启的时候，因为又找到了change_master_to.sql.in，从而重复执行一遍初始化流程
            mv change_master_to.sql.in change_master_to.sql.orig
            # 使用change_master_to.sql.orig的内容，也就是前面拼装的SQL，组成一个完整的初始化和启动Slave的SQL语句
            mysql -h 127.0.0.1 -uroot -p${MYSQL_ROOT_PASSWORD} << EOF
          $(< change_master_to.sql.orig),
            MASTER_HOST='mysql-0.mysql.mysql',
            MASTER_USER='root',
            MASTER_PASSWORD='${MYSQL_ROOT_PASSWORD}',
            MASTER_CONNECT_RETRY=10;
          START SLAVE;
          EOF
          fi
          # 使用ncat监听3307端口。
          # 它的作用是，在收到传输请求的时候，直接执行xtrabackup --backup命令，备份MySQL的数据并发送给请求者
          exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
            "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root --password=${MYSQL_ROOT_PASSWORD}"
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
      volumes:
      - name: conf
        emptyDir: {
    
    }
      - name: config-map
        configMap:
          name: mysql
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - "ReadWriteOnce"
      storageClassName: local-storage
      resources:
        requests:
          storage: 3Gi

The overall StatefulSet has two Replicas, one Master and one Slave, and then use the init-mysql initContainers to initialize the configuration file. Then use the initContainers of clone-mysql for data transmission; at the same time, use the sidecar container of xtrabackup for SQL initialization and data transmission.
Create a StatefulSet

kubectl apply -f 07-mysql-statefulset.yaml

It can be seen that after the StatefulSet starts successfully, there will be two Pods running. Next, we can try to initiate a request to this MySQL cluster and perform some SQL operations to verify that it is normal. The whole process will be very slow because of pulling mysql and a foreign image of gcr.io/google-samples/xtrabackup:1.0, but after creating mysql-0 and pulling it once, the subsequent creation of m>ysql-1 is relatively fast.
Finally, the container checks the running status of the pod

[root@master mysql]# kubectl get po -n mysql
NAME      READY   STATUS    RESTARTS   AGE
mysql-0   2/2     Running   0          2m7s
mysql-1   2/2     Running   0          116s

service verification

Verify master-slave status

[root@master mysql]# kubectl -n mysql exec mysql-1 -c mysql -- bash -c "mysql -uroot -p123456 -e 'show slave status \G'"
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: mysql-0.mysql.mysql
                  Master_User: root
                  Master_Port: 3306
                Connect_Retry: 10
              Master_Log_File: mysqllog.000003
          Read_Master_Log_Pos: 154
               Relay_Log_File: mysql-1-relay-bin.000002
                Relay_Log_Pos: 319
        Relay_Master_Log_File: mysqllog.000003
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: mysql0
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 154
              Relay_Log_Space: 528
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 100
                  Master_UUID: f8c7768a-6702-11ec-a648-c204e42fb62e
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
mysql: [Warning] Using a password on the command line interface can be insecure.

Next, we create databases and tables and insert databases through the Master container

[root@master mysql]# kubectl -n mysql exec mysql-0 -c mysql -- bash -c "mysql -uroot -p123456 -e 'create database test'"

[root@master mysql]# kubectl -n mysql exec mysql-0 -c mysql -- bash -c "mysql -uroot -p123456 -e 'use test;create table counter(c int);'"

[root@master mysql]# kubectl -n mysql exec mysql-0 -c mysql -- bash -c "mysql -uroot -p123456 -e 'use test;insert into counter values(123)'"

[root@master mysql]# kubectl -n mysql exec mysql-1 -c mysql -- bash -c "mysql -uroot -p123456 -e 'use test;select * from counter'"  
c
123

When you see the output, the master-slave synchronization is normal

extension node

[root@master mysql]# kubectl -n mysql scale statefulset mysql -—replicas=3
[root@master mysql]# kubectl get po -n mysql
NAME      READY   STATUS    RESTARTS   AGE
mysql-0   2/2     Running   0          11m
mysql-1   2/2     Running   0          11m
mysql-2   2/2     Running   0          20s

Check whether the expansion node data has been synchronized

kubectl -n mysql exec mysql-2 -c mysql -- bash -c "mysql -uroot -p123456 -e 'use test;select * from counter’"  
c
123