deploy mysql operator on k8s

By mysql-operator deployment of highly available mysql statefulset in kubernetes cluster.

Preparing the Environment

As used herein, the open source project operator mysql-operator with death only to support more than mysql 8.0.11 version, changed under the code, support for 5.7.0 or later, the project addresses , this deployment is mysql-5.7.26, the use of dockerhub mirror mysql / mysql-server: 5.7.26.

Code compilation

git clone download the program proceeds to code directory, execution sh hack/build.sh, the compiled code to obtain binary mysql-agent and mysql-operator, into the binary bin/linux_amd64, performing docker build -f docker/mysql-agent/Dockerfile -t $IMAGE_NAME_AGENT ., docker build -f docker/mysql-operator/Dockerfile -t $IMAGE_NAME_OPERATOR .construct the mirrors, mysql-operator operator generated image of a mirror image, MySQL- generating agent is a mirror image, when creating mysql service, and as sidecar mysql-server in the same container from the pod.

Deployment operator

First according to the document deployment Deployment mysql-operator of the document is to use the helm installation, do not want to install helm and tiller, you can only install a helm client, enter the code directory, then execute helm template --name mysql-operator mysql-operatoryaml file generation deployment need, and then directly execution kubectl apply -f mysql-operator.yamlcreate operator. The CRD yaml created the desired type of operator, the operator of the required Deployment and operator RBAC permissions.

# change directory into mysql-operator
cd mysql-operator
# generate mysql-operator.yaml
helm template --name mysql-operator mysql-operator > mysql-operator.yaml
# deploy on kubernetes
kubectl apply -f mysql-operator.yaml
# deployed.
[root@localhost]$ kubectl get deploy -n mysql-operator
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
mysql-operator   1/1     1            1           2d5h

Creating a cluster mysql

Herein cluster created for mysql 3 node, a node for the master, two for the slave, master node may read and write, slave node is read only, for use kubernetes Local PV persistent store.
First, create a node for each PV, Local PV needs to be defined nodeAffinity, node constraints created.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mypv0
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: mysql-storage
  local:
    path: /data/mysql-data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - 192.168.0.1

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mypv1
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: mysql-storage
  local:
    path: /data/mysql-data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - 192.168.0.2

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mypv2
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: mysql-storage
  local:
    path: /data/mysql-data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - 192.168.0.3

# create pv
kubectl create -f pv.yaml
# get presistence volume
[root@localhost]$ kubectl get pv
mypv-0               1Gi        RWO            Delete           Available                                                                     mysql-storage               4s
mypv-1               1Gi        RWO            Delete           Available                                                                     mysql-storage               4s
mypv-2               1Gi        RWO            Delete           Available                                                                     mysql-storage               4s

Next, we need to create in the namespace mysql to create corresponding RBAC permissions for mysql to create.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mysql-agent
  namespace: mysql2
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: mysql-agent
  namespace: mysql2
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mysql-agent
subjects:
- kind: ServiceAccount
  name: mysql-agent
  namespace: mysql2

If you need to customize the mysql password, you need to create a secret, password is required base64 encryption. the next linux echo -n 'password' | base64as password encryption.

apiVersion: v1
data:
  password: cm9vdA==
kind: Secret
metadata:
  labels:
    v1alpha1.mysql.oracle.com/cluster: mysql
  name: mysql-pv-root-password
  namespace: mysql2

kubectl apply -f rbac.yaml
kubectl apply -f secret.yaml

In the creation of the operator, we have created the following crd type of deployment is mysqlclusters types of resources needed to create a mysql cluster.

[root@localhost]$ kubectl get crd  | grep mysql
mysqlbackups.mysql.oracle.com                2019-05-14T02:51:11Z
mysqlbackupschedules.mysql.oracle.com        2019-05-14T02:51:11Z
mysqlclusters.mysql.oracle.com               2019-05-14T02:51:11Z
mysqlrestores.mysql.oracle.com               2019-05-14T02:51:11Z

The next operator to start creating a custom resource types (CRD) instances mysqlclusters.

apiVersion: mysql.oracle.com/v1alpha1
kind: Cluster
metadata:
  name: mysql
  namespace: mysql2
spec:  
  # 和mysql-server镜像版本的tag一直
  version: 5.7.26
  repository: 20.26.28.56/dcos/mysql-server
  # 节点数量
  members: 3
  # 指定 mysql 密码，和之前创建的secret名称一致
  rootPasswordSecret:
    name: mysql-pv-root-password
  resources:
    agent:
      limits:
        cpu: 500m
        memory: 200Mi
      requests:
        cpu: 300m
        memory: 100Mi
    server:
      limits:
        cpu: 1000m
        memory: 1000Mi
      requests:
        cpu: 500m
        memory: 500Mi
  volumeClaimTemplate:
    metadata:
      name: mysql-pv
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "mysql-storage"
      resources:
        requests:
          storage: 1Gi

kubectl apply -f mysql.yaml

After you do, you will see kubernetes in this namespace start pulling up statefulset mysql and will create a headless service.

[root@localhost]$ kubectl get all -n mysql2
NAME                                                      READY   STATUS    RESTARTS   AGE
pod/mysql-0                                               2/2     Running   0          8h
pod/mysql-1                                               2/2     Running   0          8h
pod/mysql-2                                               2/2     Running   0          8h


NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/mysql     ClusterIP   None            <none>        3306/TCP   21h


NAME                     READY   AGE
statefulset.apps/mysql   1/1     21h

Run the hack/cluster-status.shscript, the cluster will get the following information:

{
    "clusterName": "Cluster", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "mysql-0.mysql:3306", 
        "ssl": "DISABLED", 
        "status": "OK_NO_TOLERANCE", 
        "statusText": "Cluster is NOT tolerant to any failures. 2 members are not active", 
        "topology": {
            "mysql-0.mysql:3306": {
                "address": "mysql-0.mysql:3306", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }, 
            "mysql-1.mysql:3306": {
                "address": "mysql-1.mysql:3306", 
                "mode": "n/a", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }, 
            "mysql-2.mysql:3306": {
                "address": "mysql-2.mysql:3306", 
                "mode": "n/a", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "mysql-0.mysql:3306"
}

By DNS address mysql-0.mysql.mysql2.svc.cluster.local:3306can connect to the database to read and write. At this time, a multi-node cluster mysql have been deployed, but, outside of the cluster service can not access the database.

External access via haproxy-ingress

First, headless service only through the cluster DNS service access to external access, you also need to create an additional Service. In order to allow external access to the mysql-0 service, we create a ClusterIP type of service for mysql-0.

kind: Service
apiVersion: v1
metadata:
  name: mysql-0
  namespace: mysql2
spec:
  selector:
    # 通过 selector 将 pod 约束到 mysql-0
   statefulset.kubernetes.io/pod-name: mysql-0
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306

Next, we need to create an ingress-controller, paper choices are HAProxy-Ingress .
Since the mysql service over the TCP protocol communication, kubernetes ingress default port supports only http and https, haproxy-ingress provides port by means of configmap, configuring TCP service, you need to create a configmap, configmap the data in, key to HAProxy listening , value for the service to be forwarded services and ports.

apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-tcp
  namespace: mysql2
data:
  "3306": "mysql2/mysql-0:3306"

kubectl apply -f mysql-0.yaml
kubectl apply -f tcp-svc.yaml

Next, create ingress-controller,

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: haproxy-ingress
  name: haproxy-ingress-192.168.0.1-30080
  namespace: mysql2
spec:
  replicas: 1
  selector:
    matchLabels:
      run: haproxy-ingress
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        run: haproxy-ingress
    spec:
      tolerations:
      - key: app 
        operator: Equal 
        value: haproxy
        effect: NoSchedule
      serviceAccount: ingress-controller      
      nodeSelector:
        kubernetes.io/hostname: 192.168.0.1
      containers:
      - args:
        - --tcp-services-configmap=$(POD_NAMESPACE)/mysql-tcp
        - --default-backend-service=$(POD_NAMESPACE)/mysql
        - --default-ssl-certificate=$(POD_NAMESPACE)/tls-secret
        - --ingress-class=ha-mysql
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: jcmoraisjr/haproxy-ingress
        name: haproxy-ingress
        ports:
          # 和 configmap 中定义的端口对应
        - containerPort: 3306
          hostPort: 3306
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        - containerPort: 1936
          hostPort: 30081
          name: stat
          protocol: TCP

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-controller

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
      - create
      - update

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-controller
subjects:
  - kind: ServiceAccount
    name: ingress-controller
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: ingress-controller

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-controller
subjects:
  - kind: ServiceAccount
    name: ingress-controller
    namespace: mysql2
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: ingress-controller

kubectl apply -f ingress-controller.yaml
kubectl apply -f ingress-rbac.yaml -n mysql2

Finally, create ingress rules:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
    kubernetes.io/ingress.class: ha-mysql
  name: ha-mysql
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: mysql-0
          servicePort: 3306
        path: /

+ Can now be mapped port access to mysql cluster by haproxy of IP.

annex

The following is yaml file used above: