Etcd Practical Combat (1)-Deploying etcd Cluster

1 Overview

etcd is a highly available distributed key-value storage system and is an open source project developed by CoreOS (now part of Red Hat). It provides a simple interface to store and retrieve key-value pair data and implements distributed consistency using the Raft protocol. etcd is widely used in distributed systems such as Docker and Kubernetes to store configuration information, service discovery, leader election, etc.

2 etcd high availability cluster deployment

Government text:https://etcd.io/docs/v3.5/

安装包：https://github.com/etcd-io/etcd/releases/download

2.1 Server configuration

CPU name	IP address	cpu quantity	Memory quantity
etcd-1	10.220.43.206	2	4
etcd-2	10.220.43.207	2	4
etcd-3	10.220.43.208	2	4

2.2 Installation and deployment

2.2.1 Download and decompress

$ wget https://github.com/etcd-io/etcd/releases/download/v3.4.27/etcd-v3.4.27-linux-amd64.tar.gz
$ tar xvf etcd-v3.4.27-linux-amd64.tar.gz
$ mv etcd-v3.4.27-linux-amd64/etcd* /usr/local/bin/ ##复制etcd命令文件
$ mkdir -p /var/lib/etcd/   ##创建数据存放目录
$ mkdir -p /etc/etcd ##创建配置文件存放目录

Set etcd configuration files for three nodes, and the firewalls of the three nodes must allow ports 2379 and 2380.

2.2.2 Set cluster configuration

#etcd节点1
$ cat /etc/etcd/etcd.conf
ETCD_NAME=etcd-1
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="http://10.220.43.206:2380"
ETCD_LISTEN_CLIENT_URLS="http://127.0.0.1:2379,http://10.220.43.206:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.220.43.206:2380"
ETCD_INITIAL_CLUSTER="etcd-1=http://10.220.43.206:2380,etcd-2=http://10.220.43.207:2380,etcd-3=http://10.220.43.208:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="singless"
ETCD_ADVERTISE_CLIENT_URLS="http://10.220.43.206:2379"

#etcd节点2
$ cat /etc/etcd/etcd.conf
ETCD_NAME=etcd-2
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="http://10.220.43.207:2380"
ETCD_LISTEN_CLIENT_URLS="http://127.0.0.1:2379,http://10.220.43.207:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.220.43.207:2380"
ETCD_INITIAL_CLUSTER="etcd-2=http://10.220.43.206:2380,etcd-2=http://10.220.43.207:2380,etcd-3=http://10.220.43.208:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="singless"
ETCD_ADVERTISE_CLIENT_URLS="http://10.220.43.207:2379"

#etcd节点3
$ cat /etc/etcd/etcd.conf
ETCD_NAME=etcd-3
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="http://10.220.43.208:2380"
ETCD_LISTEN_CLIENT_URLS="http://127.0.0.1:2379,http://10.220.43.208:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.220.43.207:2380"
ETCD_INITIAL_CLUSTER="etcd-2=http://10.220.43.206:2380,etcd-2=http://10.220.43.207:2380,etcd-3=http://10.220.43.208:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="singless"
ETCD_ADVERTISE_CLIENT_URLS="http://10.220.43.208:2379"

parameter:

ETCD_NAME: Specify the name of the ETCD node. Each node should have a unique name within the cluster for identification and communication.
ETCD_DATA_DIR: Specifies the directory path where ETCD stores data. ETCD uses this directory to persist data.
ETCD_LISTEN_PEER_URLS: Specifies the URL where the ETCD node listens for peer node connections. Peer URLs are used for communication and data synchronization between nodes.
ETCD_LISTEN_CLIENT_URLS: Specifies the URL list for ETCD nodes to listen for client connections. The client URL is used to interact with the etcd cluster, such as performing read or write operations.
ETCD_INITIAL_ADVERTISE_PEER_URLS: Specifies the URL for the ETCD node to initially advertise peer node connections. When a new node joins the cluster, it announces its URL to other nodes so that other nodes can find and connect to it.
ETCD_INITIAL_CLUSTER: Specifies the initial member list of the ETCD cluster and its peer node connection URL. Each member consists of the name and peer connection URL, separated by commas. When starting a cluster, specify an initial member list for inserting new nodes.
ETCD_INITIAL_CLUSTER_STATE: Specifies the initial state of the ETCD cluster. Optional values are "new" and "existing". "new" means creating a new cluster, and "existing" means an existing cluster.
ETCD_INITIAL_CLUSTER_TOKEN: Specifies the initial token of the ETCD cluster. All members should start with the same token so that they can identify and join the same cluster.
ETCD_ADVERTISE_CLIENT_URLS: Specifies the connection URL that the ETCD node announces to the client. The client connection URL is used to expose the etcd node to an application or tool for read and write operations.

2.2.3 Configure system services

Configure systemd to manage etcd, the three nodes have the same configuration

$ cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
User=root
Type=notify
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd
Restart=on-failure
RestartSec=10s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target

2.2.4 Start the service

Start services on three nodes

$ systemctl enable etcd
$ systemctl start etcd

2.2.5 Exception problem handling

Error reported:

check file permission: directory "/var/lib/etcd" exist, but the permission is "drwxr-xr-x". The recommended permission is "-rwx------" to prevent possible unprivileged access to the data.

solution:

$ chmod 700 /var/lib/etcd

2.3 Verification

2.3.1 View cluster members

$ etcdctl member list
3fccfd9e945d331b, started, etcd-3, http://10.220.43.208:2380, http://10.220.43.208:2379, false
5d19435c38496e2c, started, etcd-2, http://10.220.43.207:2380, http://10.220.43.207:2379, false
9f23d9fd9d308b96, started, etcd-1, http://10.220.43.206:2380, http://10.220.43.206:2379, false

2.3.2 View leader node

$ etcdctl -w table endpoint status --endpoints=10.220.43.206:2379,10.220.43.207:2379,10.220.43.208:2379 
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.220.43.206:2379 | 9f23d9fd9d308b96 |  3.4.27 |   20 kB |      true |      false |        96 |          9 |                  9 |        |
| 10.220.43.207:2379 | 5d19435c38496e2c |  3.4.27 |   20 kB |     false |      false |        96 |          9 |                  9 |        |
| 10.220.43.208:2379 | 3fccfd9e945d331b |  3.4.27 |   16 kB |     false |      false |        96 |          9 |                  9 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

The results show that etcd-1 is the leader node.

2.3.3 Check the health status of cluster members

$ etcdctl -w table endpoint health --endpoints=10.220.43.206:2379,10.220.43.207:2379,10.220.43.208:2379 
+--------------------+--------+------------+-------+
|      ENDPOINT      | HEALTH |    TOOK    | ERROR |
+--------------------+--------+------------+-------+
| 10.220.43.206:2379 |   true | 1.740078ms |       |
| 10.220.43.207:2379 |   true | 2.115924ms |       |
| 10.220.43.208:2379 |   true | 2.174842ms |       |
+--------------------+--------+------------+-------+

3 How to deal with an exception on a certain node

Remove nodes using command
Delete the abnormal node data directory
Re-add the node to the cluster

3.1 Remove nodes

Assume that etcd3 has an exception.

etcd-1 node operation:

$ etcdctl member list
3fccfd9e945d331b, started, etcd-3, http://10.220.43.208:2380, http://10.220.43.208:2379, false
5d19435c38496e2c, started, etcd-2, http://10.220.43.207:2380, http://10.220.43.207:2379, false
9f23d9fd9d308b96, started, etcd-1, http://10.220.43.206:2380, http://10.220.43.206:2379, false
$ etcdctl member remove 3fccfd9e945d331b
Member 3fccfd9e945d331b removed from cluster cee4a9895463ca7d
$ etcdctl member list
5d19435c38496e2c, started, etcd-2, http://10.220.43.207:2380, http://10.220.43.207:2379, false
9f23d9fd9d308b96, started, etcd-1, http://10.220.43.206:2380, http://10.220.43.206:2379, false

3.2 Delete the data directory of the abnormal node

etcd-3 node operations:

$ rm -rf /var/lib/etcd
cat /etc/etcd/etcd.conf 
ETCD_NAME=etcd-3
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="http://10.220.43.208:2380"
ETCD_LISTEN_CLIENT_URLS="http://127.0.0.1:2379,http://10.220.43.208:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.220.43.208:2380"
ETCD_INITIAL_CLUSTER="etcd-1=http://10.220.43.206:2380,etcd-2=http://10.220.43.207:2380,etcd-3=http://10.220.43.208:2380"
ETCD_INITIAL_CLUSTER_STATE="existing" ##修改为existing
ETCD_INITIAL_CLUSTER_TOKEN="singless"
ETCD_ADVERTISE_CLIENT_URLS="http://10.220.43.208:2379"

3.3 Add nodes to the cluster

etcd-1 node operation:

$ etcdctl member add etcd-3 --peer-urls=http://10.220.43.208:2380
Member 765137d296bf17e7 added to cluster cee4a9895463ca7d

ETCD_NAME="etcd-3"
ETCD_INITIAL_CLUSTER="etcd-2=http://10.220.43.207:2380,etcd-3=http://10.220.43.208:2380,etcd-1=http://10.220.43.206:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.220.43.208:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
$ etcdctl member list
5d19435c38496e2c, started, etcd-2, http://10.220.43.207:2380, http://10.220.43.207:2379, false
765137d296bf17e7, unstarted, , http://10.220.43.208:2380, , false
9f23d9fd9d308b96, started, etcd-1, http://10.220.43.206:2380, http://10.220.43.206:2379, false

etcd-3 starts etcd.

$ systemctl start etcd

etcd-1 node operation:

$ etcdctl member list
5d19435c38496e2c, started, etcd-2, http://10.220.43.207:2380, http://10.220.43.207:2379, false
765137d296bf17e7, started, etcd-3, http://10.220.43.208:2380, http://10.220.43.208:2379, false
9f23d9fd9d308b96, started, etcd-1, http://10.220.43.206:2380, http://10.220.43.206:2379, false

4 etcdctl addition, deletion, modification and query operations

https://github.com/etcd-io/etcd/tree/main/etcdctl

etcdctl is a command-line tool for interacting with ETCD. It provides a series of commands and options for retrieving, setting, modifying, and deleting data in ETCD.

4.1 Data storage model of etcd

As introduced before, etcd is a distributed key-value storage system. etcd adopts a hierarchical spatial structure in the organization of keys, which is similar to the concept of directories in a file system. But there is no hierarchical relationship like a directory.

Similar to the directory (folder) structure in a file system, ETCD's namespace allows data to be organized in a hierarchical structure of keys, and each key can uniquely identify a node. The top-level node is called the root node, and child nodes can then be created under the root node, which in turn can contain more child nodes, and so on. This hierarchical structure can be used to classify, categorize and organize data.

For example, assume we have the following directory structure:

/singless/test
/singless/test/gender

In the above example, /singless/test and /singless/test/gender have the same index prefixes / and /singless as directories, but there is no hierarchical relationship like a directory. They can all be used as a key to store the corresponding value.

4.2 Additions and changes

Adding data and modifying data require the put command to operate.

$ etcdctl put /singless/test/gender male ##创建一个键/singless/test/gender，值为male
OK
$ etcdctl put /singless/test true ##创建一个键/singless/test，值为test
OK

4.3 Query operation

Query operations require the use of get parameters

$ etcdctl get /singless/test/gender ##根据具体的key查询对应的值
/singless/test/gender
male
$ etcdctl get --prefix / ##根据索引前缀查询，这里查询的是以/为前缀的key、value，所以能看到所有的键值对
/singless/test
true
/singless/test/gender
male
$ etcdctl put /singless/test false ##使用put修改键/singless/test的value
OK
$ etcdctl get /singless/test
/singless/test
false

4.4 Delete operation

Delete using del command

$ etcdctl del /singless/test ##可以指定key进行删除
1
$ etcdctl get --prefix /
/singless/test/gender
male
$ etcdctl del --prefix / ##也可以执行索引前缀进行删除
1
$ etcdctl get --prefix /