Operator refers to a class-based Kubernetes custom resource object (CRD) and the Controller (Controller) is native to expand cloud services, CRD define custom resource object is created for each operator and management, Controller contains the management objects operation and maintenance related logic code.
For the average user, if you want to deploy a high-availability cluster etcd in k8s cluster, not only to understand its associated configuration, but also require specific expertise to complete maintenance etcd arbitration, reconfigure the cluster members, to create a backup, processing disaster recovery and so tedious event.
And in this type of help to expand the service operator, we can use the straightforward YAML file (the same token reference Deployment) to declarative configuration, create and manage our etcd cluster, here we come together to understand the next etcd -operator this architecture services as well as some of the features it contains.
aims
Learn etcd-operator of architecture and CRD resource object
Etcd-operator deployment
Creating etcd cluster using etcd-operator
Etcd-operator-based backup and recovery etcd cluster
Services Architecture
etcd-operator is designed to expand based API Extension mechanism k8s, which is designed for users of a similar Deployment of the Controller, the Controller is used only specialized management etcd this service.
User default or carried out by kubectl or UI and API k8s of interaction, but more controllers (custom controller) a user-defined in this k8s cluster, operator controller services are run in the Pod k8s cluster , but the service also need to configure the required RBAC permissions (such as for Pod, Deployment, Volume, etc. to use the resources for CRUD operations), let's use a simple organization chart to elaborate:
etcd-operator custom resource object (CRD)
在k8s中，所有自定义的Controller和其自定义的资源对象（CRD）都必须满足k8s API的规范（参考下图）:
apiVersion描述了当前自定义资源对象的版本号Kind表示自定义资源对象的名称，用户可通过执行kubectl get $KIND_NAME来获取所创建的CRD对象Metadata继承了原生k8s的metadata，用于添加标签，Annotations等元数据Spec是用户可自定义设计的服务配置参数，如镜像版本号，节点数量，资源配置等等..Status包含了当前资源的的相关状态，每个operator controller可自定义status所包含的信息,一般会选择添加如conditions，updateTime和message等一类的信息。
下面先我们来了解一下etcd-operator所包含的几个自定义资源对象(CRDs):
1、EtcdCluster: etcdcluster用来描述用户自定义的etcd集群，可一键式部署和配置一个相关的etcd集群。
apiVersion: etcd.database.coreos.com/v1beta2
kind: EtcdCluster
metadata:
name: etcd-cluster
spec:
size: 3
version: 3.2.252、EtcdBackup: etcdbackup用来描述和管理一个etcd集群的备份，当前支持定期备份到云端存储，如AWS s3, Aliyun oss(oss当前需使用quay.io/coreos/etcd-operator:dev镜像)。
apiVersion: etcd.database.coreos.com/v1beta2
kind: EtcdBackup
metadata:
name: etcd-backup
spec:
etcdEndpoints: [<etcd-cluster-endpoints>]
storageType: OSS #options are S3/ABS/GCS/OSS
backupPolicy:
backupIntervalInSecond: 125
maxBackups: 4
oss:
#"<oss-bucket-name>/<path-to-backup-file>"
path: <full-oss-path>
ossSecret: <oss-secret>
# Details about regions and endpoints, see https://www.alibabacloud.com/help/doc-detail/31837.htm
endpoint: <endpoint> 3、EtcdRestore:etcdrestore用来帮助将etcdbackup服务所创建的备份恢复到一个指定的etcd的集群。
apiVersion: etcd.database.coreos.com/v1beta2
kind: EtcdRestore
metadata:
# name must be same to the spec.etcdCluster.name
name: example-etcd-cluster
spec:
etcdCluster:
name: example-etcd-cluster
backupStorageType: OSS
oss:
path: <full-oss-path>
ossSecret: <oss-secret>
endpoint: <endpoint>如何部署和使用etcd-operator
1、部署etcd-operator
在Rancher最新的stable v2.3.2 的版本中，用户可通过应用商店（Catalog）来一键式部署 etcd-operator v0.9.0版本，同时原生k8s也可下载rancher/charts到本地后通过helm install的方式进行部署。
1）（可选）部署etcd-operator时可选择同时创建一个etcd集群（此集群在etcd-operator被删除时会被一同移除），当然用户也可待etcd-operator部署完成通过kubectl apply -f myetcd.yaml来创建一个新的etcd集群。
2）部署时，如果用户选择启动Enable Clusterwide of etcd Operator这个选项，那么这个etcd-operator将作为集群层级对象来使用（否则为namespaced隔离），如果enable这个选项，那么在创建etcd集群时需添加以下注释才能创建创建:
kind: EtcdCluster
metadata:
name: etcd-cluster
# add this annotation when the clusterWide is enabled
annotations:
etcd.database.coreos.com/scope: clusterwide2、创建etcd集群
接下来我们就可以使用上述的CRD自定义资源对象对来创建和管理我们的etcd集群了。
2.1 手动创建etcd集群
cat <<EOF | kubectl apply -f -
apiVersion: etcd.database.coreos.com/v1beta2
kind: EtcdCluster
metadata:
name: "etcd-cluster"
spec:
size: 3 # 默认etcd节点数
version: "3.2.25" # etcd版本号
EOF2.2 部署后可通过CRD对象来查看我们创建的etcd集群和pod状态
$ kubectl get etcdcluster
NAME AGE
etcd-cluster 2m
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
etcd-cluster-g28f552vvx 1/1 Running 0 2m
etcd-cluster-lpftgqngl8 1/1 Running 0 2m
etcd-cluster-sdpcfrtv99 1/1 Running 0 2m2.3 可以往etcd集群任意的写入几条数据验证etcd集群是正常工作的（后续也可用来验证集群的备份和恢复功能）
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
etcd-cluster ClusterIP None <none> 2379/TCP,2380/TCP 17h
etcd-cluster-client ClusterIP 10.43.130.71 <none> 2379/TCP 17h
## write data
$ kubectl exec -it any-etcd-pod -- env "ETCDCTL_API=3" etcdctl --endpoints http://etcd-cluster-client:2379 put foo "Hello World"
## get data
$ kubectl exec -it any-etcd-pod -- env "ETCDCTL_API=3" etcdctl --endpoints http://etcd-cluster-client:2379 get foo
foo
Hello World3、基于operator备份etcd cluster
3.1 确认了etcd集群正常运行后，作为devops后面要考虑的就是如何创建etcd集群的自动化备份，下面以阿里云的OSS举例：
cat <<EOF | kubectl apply -f -
apiVersion: etcd.database.coreos.com/v1beta2
kind: EtcdBackup
metadata:
name: example-etcd-cluster-periodic-backup
spec:
etcdEndpoints: [http://etcd-cluster-client:2379] #内网可使用svc地址，外网可用NodePort或LB代理地址
storageType: OSS
backupPolicy:
backupIntervalInSecond: 120 #备份时间间隔
maxBackups: 4 #最大备份数
oss:
path: my-bucket/etcd.backup
ossSecret: oss-secret #需预先创建oss secret
endpoint: oss-cn-hangzhou.aliyuncs.com
EOF3.2 若OSS Secret不存在，用户可先手动创建，具体配置可参考如下：
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: oss-secret
type: Opaque
stringData:
accessKeyID: myAccessKey
accessKeySecret: mySecret
EOF3.3 待etcdbackup创建成功后，用户可以通过kubectl describe etcdbackup或查看etcd-backup controller日志来查看备份状态，如状态显示为Succeeded: true，可以前往oss查看具体的备份内容。
4、基于operator恢复etcd cluster
最后，假设我们要将etcd集群A的备份数据恢复到另一个新的etcd集群B，那么我们先手动创建一个名为etcd-cluster2的新集群(oss备份/恢复当前需使用quay.io/coreos/etcd-operator:dev镜像)。
cat <<EOF | kubectl apply -f -
apiVersion: etcd.database.coreos.com/v1beta2
kind: EtcdCluster
metadata:
name: "etcd-cluster2"
spec:
size: 3
version: "3.2.25"
EOF然后通过创建etcdresotre将备份数据恢复到etcd-cluster2集群
cat <<EOF | kubectl apply -f -
apiVersion: etcd.database.coreos.com/v1beta2
kind: EtcdRestore
metadata:
# name必须与下面的spec.etcdCluster.name保持一致
name: etcd-cluster2
spec:
etcdCluster:
name: etcd-cluster2
backupStorageType: OSS
oss:
path: my-bucket/etcd.backup_v1_2019-08-07-06:44:17
ossSecret: oss-secret
endpoint: oss-cn-hangzhou.aliyuncs.com
EOF待etcdresotre对象创建成功后，可以查看etcd-operator-restore的日志，大致内容如下：
$ kubectl logs -f etcd-operator-restore
...
time="2019-08-07T06:50:26Z" level=info msg="listening on 0.0.0.0:19999"
time="2019-08-07T06:50:26Z" level=info msg="starting restore controller" pkg=controller
time="2019-08-07T06:56:25Z" level=info msg="serving backup for restore CR etcd-cluster2"通过kubectl查看pod我们可以看到etcd-cluster2集群的etcd节点被删除重建：
NAME READY STATUS RESTARTS AGE
etcd-cluster2-5tq2d5bvpf 0/1 Terminating 0 93s
etcd-cluster2-kfgvc692pp 1/1 Terminating 0 101s
etcd-cluster2-xqkgz8chb8 0/1 Init:1/3 0 6s
etcd-cluster2-pf2qxgtg9d 1/1 Running 0 48s
etcd-cluster2-x92l9vpx97 1/1 Running 0 40s最后可通过etcdctl来验证之前的数据是否存在（需设置ETCDCTL_API=3）：
$ kubectl exec -it etcd-pod -- env "ETCDCTL_API=3" etcdctl --endpoints http://etcd-cluster2-client:2379 get foo
foo
Hello World小 结
Etcd作为当前非常流行的key-value分布式文件存储，它本身的强一致性和较优的性能可以为许多分布式计算解决分布式存储的需求，如果你的微服务和应用需要用到此类的数据库，不妨来试试Rancher Catalog应用中的etcd-operator吧，Just do it!
相关资料：
https://github.com/coreos/etcd-operator
https://coreos.com/blog/introducing-the-etcd-operator.html
https://github.com/rancher/charts/tree/master/charts/etcd-operator/v0.9.0