Deploy etcd cluster in Kubernetes cluster using Statefulset

Others 2022-04-20 09:31:10 views: 0

With the popularity of microservice architecture, Etcd appears more and more frequently in our field of vision as a basic platform for service discovery or partial storage. Therefore, the demand for rapid deployment of a highly available Etcd cluster is becoming stronger and stronger. This time, I will lead you to use the Statefulset feature of Kubernetes to quickly deploy an Etcd cluster.

What is Kubernetes?

Kubernetes is an open source platform for automated deployment, scaling, and O&M of container clusters.

With Kubernetes, you can respond quickly and efficiently to customer needs:

  • Deploy your applications quickly and without surprises.
  • Dynamically scale applications.
  • Release new features seamlessly.
  • Use only the resources you need to optimize hardware usage.

What is Etcd?

The purpose of Etcd is to provide a distributed key-value dynamic database and maintain a "Configuration Registry". One of the foundations of this Registry is Kubernetes cluster discovery and centralized configuration management. It is similar in some ways to Redis, the classic LDAP configuration backend, and the Windows registry.

Etcd's goals are:

  • Simple: Well-defined, user-facing API (JSON and gRPC)
  • Security: Automatic TLS and optional client certificate authentication
  • Fast: benchmarked 10,000 writes/sec
  • Reliable: use the Raft protocol as a distributed basis

The official already has Etcd-Operator, why should I deploy it in this way?

First look at the advantages:

  • The deployment method of Etcd-Operator requires both the Etcd version and the Kubernetes version, see the official documentation for details .
  • If the Etcd v2 Api is used, the data cannot be backed up. The official Etcd cluster data backup only supports the Etcd v3 version
  • Statefulset deploys Etcd data more reliably. If I use Etcd-Operator to deploy Etcd, unfortunately one day my Kubernetes cluster has a problem, causing all Etcd Pods to fail, then unfortunately, if the data is not backed up, I can only Pray God Bless, I pray that I don't Back the pot. Even if Etcd V3 Api is used and data backup is used, some data may be lost because the backup of Etcd-Operator is scheduled regularly.
  • The configuration of Statefulset is more flexible. For example, the configuration that requires affinity can be added directly in Statefulset.

Of course, no good business is perfect . Deploying Etcd using Statefulset also requires certain conditions:

  • Must be backed by reliable network storage.
  • Creating a cluster is slightly more cumbersome than Etcd-Operator.

Well, next, let's get to the point:

How to quickly deploy an Etcd cluster on Kubernetes.

First, create a Headless Service on Kubernetes

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: infra-etcd-cluster
    app: infra-etcd
  name: infra-etcd-cluster
  namespace: default
spec:
  clusterIP: None
  ports:
  - name: infra-etcd-cluster-2379
    port: 2379
    protocol: TCP
    targetPort: 2379
  - name: infra-etcd-cluster-2380
    port: 2380
    protocol: TCP
    targetPort: 2380
  selector:
    k8s-app: infra-etcd-cluster
    app: infra-etcd
  type: ClusterIP

The Service of the Headless type is created to facilitate the use of domain names to access Etcd nodes. 2379 and 2380 correspond to the Client Port and Peer Port of Etcd, respectively.

Next, let's create the Statefulset resource:

Prerequisite: Your cluster must create PVs in advance so that the Pods generated by Statefulset can use them. If you use StorageClass to manage PVs, you don't need to create them manually. Here has Ceph-RBD as an example.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    k8s-app: infra-etcd-cluster
    app: etcd
  name: infra-etcd-cluster
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      k8s-app: infra-etcd-cluster
      app: etcd
  serviceName: infra-etcd-cluster
  template:
    metadata:
      labels:
        k8s-app: infra-etcd-cluster
        app: etcd
      name: infra-etcd-cluster
    spec:
      containers:
      - command:
        - /bin/sh
        - -ec
        - |
          HOSTNAME=$(hostname)
          echo "etcd api version is ${ETCDAPI_VERSION}"

          eps() {
              EPS=""
              for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
                  EPS="${EPS}${EPS:+,}http://${SET_NAME}-${i}.${SET_NAME}.${CLUSTER_NAMESPACE}:2379"
              done
              echo ${EPS}
          }

          member_hash() {
              etcdctl member list | grep http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380 | cut -d':' -f1 | cut -d'[' -f1
          }

          initial_peers() {
                PEERS=""
                for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
                PEERS="${PEERS}${PEERS:+,}${SET_NAME}-${i}=http://${SET_NAME}-${i}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380"
                done
                echo ${PEERS}
          }

          # etcd-SET_ID
          SET_ID=${HOSTNAME##*-}
          # adding a new member to existing cluster (assuming all initial pods are available)
          if [ "${SET_ID}" -ge ${INITIAL_CLUSTER_SIZE} ]; then
              export ETCDCTL_ENDPOINTS=$(eps)

              # member already added?
              MEMBER_HASH=$(member_hash)
              if [ -n "${MEMBER_HASH}" ]; then
                  # the member hash exists but for some reason etcd failed
                  # as the datadir has not be created, we can remove the member
                  # and retrieve new hash
                  if [ "${ETCDAPI_VERSION}" -eq 3 ]; then
                      ETCDCTL_API=3 etcdctl --user=root:${ROOT_PASSWORD} member remove ${MEMBER_HASH}
                  else
                      etcdctl --username=root:${ROOT_PASSWORD} member remove ${MEMBER_HASH}
                  fi
              fi
              echo "Adding new member"
              rm -rf /var/run/etcd/*
              # ensure etcd dir exist
              mkdir -p /var/run/etcd/
              # sleep 60s wait endpoint become ready
              echo "sleep 60s wait endpoint become ready,sleeping..."
              sleep 60

              if [ "${ETCDAPI_VERSION}" -eq 3 ]; then
                  ETCDCTL_API=3 etcdctl --user=root:${ROOT_PASSWORD} member add ${HOSTNAME} --peer-urls=http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380 | grep "^ETCD_" > /var/run/etcd/new_member_envs
              else
                  etcdctl --username=root:${ROOT_PASSWORD} member add ${HOSTNAME} http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380 | grep "^ETCD_" > /var/run/etcd/new_member_envs
              fi
              
              

              if [ $? -ne 0 ]; then
                  echo "member add ${HOSTNAME} error."
                  rm -f /var/run/etcd/new_member_envs
                  exit 1
              fi

              cat /var/run/etcd/new_member_envs
              source /var/run/etcd/new_member_envs

              exec etcd --name ${HOSTNAME} \
                  --initial-advertise-peer-urls http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380 \
                  --listen-peer-urls http://0.0.0.0:2380 \
                  --listen-client-urls http://0.0.0.0:2379 \
                  --advertise-client-urls http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2379 \
                  --data-dir /var/run/etcd/default.etcd \
                  --initial-cluster ${ETCD_INITIAL_CLUSTER} \
                  --initial-cluster-state ${ETCD_INITIAL_CLUSTER_STATE}
          fi

          for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
              while true; do
                  echo "Waiting for ${SET_NAME}-${i}.${SET_NAME}.${CLUSTER_NAMESPACE} to come up"
                  ping -W 1 -c 1 ${SET_NAME}-${i}.${SET_NAME}.${CLUSTER_NAMESPACE} > /dev/null && break
                  sleep 1s
              done
          done

          echo "join member ${HOSTNAME}"
          # join member
          exec etcd --name ${HOSTNAME} \
              --initial-advertise-peer-urls http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380 \
              --listen-peer-urls http://0.0.0.0:2380 \
              --listen-client-urls http://0.0.0.0:2379 \
              --advertise-client-urls http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2379 \
              --initial-cluster-token etcd-cluster-1 \
              --data-dir /var/run/etcd/default.etcd \
              --initial-cluster $(initial_peers) \
              --initial-cluster-state new

        env:
        - name: INITIAL_CLUSTER_SIZE
          value: "3"
        - name: CLUSTER_NAMESPACE
          valueFrom: 
            fieldRef:
              fieldPath: metadata.namespace
        - name: ETCDAPI_VERSION
          value: "3"
        - name: ROOT_PASSWORD
          value: '@123#'
        - name: SET_NAME
          value: "infra-etcd-cluster"
        - name: GOMAXPROCS
          value: "4"
        image: gcr.io/etcd-development/etcd:v3.3.8
        imagePullPolicy: Always
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -ec
              - |
                HOSTNAME=$(hostname)

                member_hash() {
                    etcdctl member list | grep http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380 | cut -d':' -f1 | cut -d'[' -f1
                }

                eps() {
                    EPS=""
                    for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
                        EPS="${EPS}${EPS:+,}http://${SET_NAME}-${i}.${SET_NAME}.${CLUSTER_NAMESPACE}:2379"
                    done
                    echo ${EPS}
                }
                
                export ETCDCTL_ENDPOINTS=$(eps)

                SET_ID=${HOSTNAME##*-}
                # Removing member from cluster
                if [ "${SET_ID}" -ge ${INITIAL_CLUSTER_SIZE} ]; then
                    echo "Removing ${HOSTNAME} from etcd cluster"
                    if [ "${ETCDAPI_VERSION}" -eq 3 ]; then
                        ETCDCTL_API=3 etcdctl --user=root:${ROOT_PASSWORD} member remove $(member_hash)
                    else
                        etcdctl --username=root:${ROOT_PASSWORD} member remove $(member_hash)
                    fi
                    if [ $? -eq 0 ]; then
                        # Remove everything otherwise the cluster will no longer scale-up
                        rm -rf /var/run/etcd/*
                    fi
                fi
        name: infra-etcd-cluster
        ports:
        - containerPort: 2380
          name: peer
          protocol: TCP
        - containerPort: 2379
          name: client
          protocol: TCP
        resources:
          limits:
            cpu: "4"
            memory: 4Gi
          requests:
            cpu: "4"
            memory: 4Gi
        volumeMounts:
        - mountPath: /var/run/etcd
          name: datadir
  updateStrategy:
    type: OnDelete
  volumeClaimTemplates:
  - metadata:
      name: datadir
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      selector:
        matchLabels:
          k8s.cloud/storage-type: ceph-rbd

Note: SET_NAME must be the same as the Name of Statefulset

At this time, your Etcd can already pass through the internal

http://${SET_NAME}-${i}.${SET_NAME}.${CLUSTER_NAMESPACE}:2379

visited.

The last step: create a Client Service

This step can be omitted if your cluster networking scheme makes Pods accessible from outside the Kubernetes cluster, or if your Etcd cluster only needs to be accessible inside the Kubernetes cluster

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: infra-etcd-cluster-client
    app: infra-etcd
  name: infra-etcd-cluster-client
  namespace: default
spec:
  ports:
  - name: infra-etcd-cluster-2379
    port: 2379
    protocol: TCP
    targetPort: 2379
  selector:
    k8s-app: infra-etcd-cluster
    app: infra-etcd
  sessionAffinity: None
  type: NodePort

You're done! You can use NodePort to access the Etcd cluster smoothly.

Expand and shrink

###Extension

Just change the replicas in Statefulset. For example, I want to expand the number of clusters to 5.

kubectl scale --replicas=5 statefulset infra-etcd-cluster

shrink

Then one day I found five nodes was a waste for me and wanted to use three. OK, just execute the following command,

kubectl scale --replicas=3 statefulset infra-etcd-cluster

All the source code can be found in my Github . If you feel this article is useful to you, please click Star on Github. If you find any problems, you can submit a PR and contribute to open source together.

{{o.name}}
{{m.name}}

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=324188659&siteId=291194637
Recommended
The number of MaxKB GitHub Stars, an open source knowledge base question and answer system based on large language models, exceeded 5,000!
Ranking
Getting the basic concepts of ROS + catkin Profile
Spring Learning (2) --- Assembling Beans in the IoC Container
js to get the src attribute of the image regularly
STM32 is based on CubeIDE and HAL library basics entry study notes: Bluetooth WIFI STM32 connects to Alibaba Cloud
Short video learning - 3, pandas simple use of pivot_table
Print directory of project configuration log, output log
Understand the difference between vi and vim in Linux in 3 minutes
1 + x certificate Web front-end development HTML5 special exercises
mangodb save and insert the difference
7-1 Family tree processing (50 points) (binary tree solution)
Daily
More
2024-05-13(8)
2024-05-12(28)
2024-05-11(32)
2024-05-10(34)
2024-05-09(32)
2024-05-08(18)
2024-05-07(34)
2024-05-06(6)
2024-05-05(0)
2024-05-04(18)

Code World

© 2018 codetd.com