1 Etcd Introduction
Etcd is a distributed key-value storage developed by CoreOS based on the Raft protocol, which can be used for service discovery, shared configuration, and consistency assurance (such as database master selection, distributed locks, etc.).
1.1 etcd features
Simple: curl-accessible user API (HTTP + JSON)
Secure: optional SSL client certificate authentication
Fast: 1000 write operations per second per instance
Reliable: use Raft algorithm to guarantee consistency
1.2 etcd function
Basic key-value storage
monitoring mechanism
key expiration and renewal mechanism for monitoring and service discovery
atomic Compare And Swap and Compare And Delete for distributed locks and leader election
1.3 etcd write process
① When the client initiates a request to etcd, if etcd is not the state of the leader but a follower, the follower will forward the request to the leader; if it is the leader, it will pre-check it, check (quota, speed limit, authentication [Judging whether the request is legal], the package size [needs to be less than 1.5M, if it is too large, it will be rejected]).
② If the request itself is legal, the request will be forwarded to KVServer for processing.
③ The KVserver consistency module performs data processing. The consistency module is implemented based on the raft protocol. At this time, the data itself is in an unstable state.
④ When the leader processes the data in an unstable state, it will notify other followers to synchronize the data through rpc, and the leader itself will synchronize the data to the log module [wal log, wal log is transferred to the disk through fsync]. When other followers synchronize the data, they complete step 3 and synchronize the data to the log module, and the data of the follower consistency module becomes committed. After these are completed, the response body is returned to the leader through the last rpc.
⑤ The leader updates the MatchIndex after receiving confirmation from more than half of the cluster itself, and the data in the consistency module changes from unstable to committed. At this time, the state machine is written through the MVCC module [composed of treeIndex and BoltDB open source components], and the data is synchronized to treeIndex [the modified version [current version number], generations information [created version, current version number, all previous versions will be updated] version number]】. Then drop it to the disk through BoltDB. At this time, the consistency module data changes from committed to applied. [Here, if there is no strong or weak consistency requirement for the data, then the data is considered to have been synchronized in the committed state].
⑥ Synchronize the data to the MVCC module in the follower through heatbeat. Data consistency is finally completed. As shown below. [If the follower is several versions behind the leader, the leader will bring it to the follower through headbeat for synchronization].
2 etcd installation and deployment
2.1 Stand-alone version
2.1.1 Download the installation package
https://github.com/etcd-io/etcd/releases/tag/v3.5.4
2.1.2 Unzip and run
tar -zxvf etcd-v3.5.4-linux-amd64.tar.gz
./etcd
2.1.3 Verification
view information
./etcdctl member list --write-out=table
insert data
./etcdctl --endpoints=localhost:2379 put chen perfect
view inserted data
./etcdctl --endpoints=localhost:2379 get chen
2.2 Cluster deployment
2.2.1 Environment preparation
| the host | ip |
|---|---|
| linux1 | 192.168.159.200 |
| linux2 | 192.168.159.201 |
| linux3 | 192.168.159.202 |
2.2.2 Writing configuration files
Add a new conf directory under the /root/software/etcd-v3.5.4-linux-amd64 directory, and add an etcd-conf.yml configuration file under the conf directory
# 节点名字
name: linux1
# 数据存储目录
data-dir: /root/etcd/etcd-v3.5.4-linux-amd64/data
# 该节点与客户端通信时监听的地址列表
listen-client-urls: http://192.168.233.61:2379
# 广播给集群中其他成员自己的客户端地址列表
advertise-client-urls: http://192.168.233.61:2379
# 该节点与其他节点通信时所监听的地址列表，多个地址使用逗号隔开，其格式可以划分为scheme://IP:PORT，这里的scheme可以是http、https
listen-peer-urls: http://192.168.233.61:2380
# 该成员节点在整个集群中的通信地址列表，这个地址用来传输集群数据的地址。因此这个地址必须是可以连接集群中所有的成员的。
initial-advertise-peer-urls: http://192.168.233.61:2380
# 启动集群使, 使用静态连接方法, 定义每个 member 主机名 endpoint
initial-cluster: linux1=http://192.168.233.61:2380,linux2=http://192.168.233.62:2380,linux3=http://192.168.233.63:2380
# 用于标记集群唯一性的 token，每个节点要一样
initial-cluster-token: etcd-cluster-token
# 初始化集群状态，new表示新建
initial-cluster-state: new
The content of the linux1 configuration file is as follows:
name: linux1
data-dir: /root/software/etcd-v3.5.4-linux-amd64/data
listen-client-urls: http://192.168.159.200:2379
advertise-client-urls: http://192.168.159.200:2379
listen-peer-urls: http://192.168.159.200:2380
initial-advertise-peer-urls: http://192.168.159.200:2380
initial-cluster: linux1=http://192.168.159.200:2380,linux2=http://192.168.159.201:2380,linux3=http://192.168.159.202:2380
initial-cluster-token: etcd-cluster-token
initial-cluster-state: new
The content of the linux2 configuration file is as follows:
name: linux2
data-dir: /root/software/etcd-v3.5.4-linux-amd64/data
listen-client-urls: http://192.168.159.201:2379
advertise-client-urls: http://192.168.159.201:2379
listen-peer-urls: http://192.168.159.201:2380
initial-advertise-peer-urls: http://192.168.159.201:2380
initial-cluster: linux1=http://192.168.159.200:2380,linux2=http://192.168.159.201:2380,linux3=http://192.168.159.202:2380
initial-cluster-token: etcd-cluster-token
initial-cluster-state: new
The content of the linux3 configuration file is as follows:
name: linux3
data-dir: /root/software/etcd-v3.5.4-linux-amd64/data
listen-client-urls: http://192.168.159.202:2379
advertise-client-urls: http://192.168.159.202:2379
listen-peer-urls: http://192.168.159.202:2380
initial-advertise-peer-urls: http://192.168.159.202:2380
initial-cluster: linux1=http://192.168.159.200:2380,linux2=http://192.168.159.201:2380,linux3=http://192.168.159.202:2380
initial-cluster-token: etcd-cluster-token
initial-cluster-state: new
2.2.3 Start etcd cluster
Execute the following command on each node
./etcd --config-file /root/software/etcd-v3.5.4-linux-amd64/conf/etcd-conf.yml
2.2.4 View cluster member information
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 member list --write-out=table
2.2.5 Check the cluster status
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 endpoint status --write-out=table
2.2.6 View cluster health information
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 endpoint health --write-out=table
3 etcd cluster reduction and reduction
3.1 Capacity reduction
3.1.1 Insert data
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 put k1 v1
Check if the insertion is successful
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 get k1
3.1.2 Perform capacity reduction
# 查看减容节点的MEMBER_ID
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 member list --write-out=table
# 删除MEMBER_ID
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 member remove ${MEMBER_ID}
View cluster member status
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379 endpoint status --write-out=table
View cluster health status
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379 endpoint health --write-out=table
3.1.3 Data Validation
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379 get k1
3.2 Cluster Expansion
3.2.1 Writing configuration files for expansion nodes
Note: --initial-cluster-state existing
name: linux3
data-dir: /root/software/etcd-v3.5.4-linux-amd64/data
listen-client-urls: http://192.168.159.202:2379
advertise-client-urls: http://192.168.159.202:2379
listen-peer-urls: http://192.168.159.202:2380
initial-advertise-peer-urls: http://192.168.159.202:2380
initial-cluster: linux1=http://192.168.159.200:2380,linux2=http://192.168.159.201:2380,linux3=http://192.168.159.202:2380
initial-cluster-token: etcd-cluster-token
initial-cluster-state: existing
3.2.2 Execute capacity expansion
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379 member add linux3 --peer-urls=http://192.168.159.202:2380
3.2.3 Expand the node to start etcd service
./etcd --config-file /root/software/etcd-v3.5.4-linux-amd64/conf/etcd-conf.yml
3.2.4 View cluster member information
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 member list --write-out=table
Note: After reducing the capacity of 202 nodes, to re-expand the capacity back to the cluster, you need to delete the previous node storage directory on the 202 nodes.
3.2.5 Check the cluster status and health information
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 endpoint status --write-out=table
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 endpoint health --write-out=table
3.2.6 Data Validation
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 get k1
4 Data Backup and Restoration
4.1 Backup data
4.1.1 Insert data
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 put test1 value1
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 put test2 value2
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 put test3 value3
4.1.2 Start backup data
./etcdctl --endpoints=http://192.168.159.200:2379 snapshot save /opt/bakup.db
4.1.3 View data volume
./etcdctl --write-out=table --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 snapshot status /opt/bakup.db
4.2 Data recovery
4.2.1 Delete data
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 del test1
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 del test2
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http://192.168.159.202:2379 del test3
4.2.2 Check whether the data is deleted
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 get test1
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 get test2
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 get test3
4.2.3 Data import
Copy the backup file bakup.db to all nodes in the cluster, and execute the following commands on each node.
linux1
./etcdctl snapshot restore /opt/bakup.db \
--data-dir="/root/software/etcd-v3.5.4-linux-amd64/data2" \
--name linux1 \
--initial-cluster "linux1=http://192.168.159.200:2380,linux2=http://192.168.159.201:2380,linux3=http://192.168.159.202:2380" \
--initial-cluster-token etcd-cluster-token \
--initial-advertise-peer-urls "http://192.168.159.200:2380"
linux2
./etcdctl snapshot restore /opt/bakup.db \
--data-dir="/root/software/etcd-v3.5.4-linux-amd64/data2" \
--name linux2 \
--initial-cluster "linux1=http://192.168.159.200:2380,linux2=http://192.168.159.201:2380,linux3=http://192.168.159.202:2380" \
--initial-cluster-token etcd-cluster-token \
--initial-advertise-peer-urls "http://192.168.159.201:2380"
linux3
./etcdctl snapshot restore /opt/bakup.db \
--data-dir="/root/software/etcd-v3.5.4-linux-amd64/data2" \
--name linux3 \
--initial-cluster "linux1=http://192.168.159.200:2380,linux2=http://192.168.159.201:2380,linux3=http://192.168.159.202:2380" \
--initial-cluster-token etcd-cluster-token \
--initial-advertise-peer-urls "http://192.168.159.202:2380"
4.2.4 Stop the etcd service of all nodes in the cluster
4.2.5 Start a new directory to start etcd service
The data storage directory in the configuration file etcd-conf.yml needs to be modified
4.2.6 Data Validation
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 get test1
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 get test2
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 get test3
5 common operations
5.1 Insert data and get data
# 插入数据
etcdctl --endpoints=$ENDPOINTS put web1 value1
etcdctl --endpoints=$ENDPOINTS put web2 value2
etcdctl --endpoints=$ENDPOINTS put web3 value3
# 根据前缀获取数据
etcdctl --endpoints=$ENDPOINTS get web --prefix
example:
# 插入数据
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put k1 v1
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put k2 v2
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put k3 v3
# 获取单条数据
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 get k1
# 根据前缀获取数据
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 get k --prefix
5.2 Delete data
etcdctl --endpoints=$ENDPOINTS put key myvalue
etcdctl --endpoints=$ENDPOINTS del key
etcdctl --endpoints=$ENDPOINTS put k1 value1
etcdctl --endpoints=$ENDPOINTS put k2 value2
etcdctl --endpoints=$ENDPOINTS del k --prefix
example:
# 插入数据
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put key myvalue
# 删除数据
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 del key
# 获取数据
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 get key
# 插入2条数据
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put k1 value1
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put k2 value2
# 根据前缀删除数据
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 del k --prefix
# 获取数据
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 get k1
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 get k2
5.3 Transaction Operation
etcdctl --endpoints=$ENDPOINTS put user1 bad
etcdctl --endpoints=$ENDPOINTS txn --interactive
compares:
value("user1") = "bad"
success requests (get, put, delete):
del user1
failure requests (get, put, delete):
put user1 good
example:
# 插入数据
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put user1 bad
# 开启事务
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 txn --interactive
compares:
value("user1") = "bad"
success requests (get, put, del):
del user1
failure requests (get, put, del):
put user1 good
SUCCESS
5.4 Monitoring a value
etcdctl --endpoints=$ENDPOINTS watch stock1
etcdctl --endpoints=$ENDPOINTS put stock1 1000
etcdctl --endpoints=$ENDPOINTS watch stock --prefix
etcdctl --endpoints=$ENDPOINTS put stock1 10
etcdctl --endpoints=$ENDPOINTS put stock2 20
example:
# 设置stock1的值，默认值为0
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put stock1 0
# 监控key为stock1的值
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 watch stock1
# 修改stock1的值
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put stock1 1000
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put stock1 1001
Prefix value monitoring:
# 前缀值监控
# 设置chen1的值，默认值为0
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put chen1 0
# 设置chen2的值，默认为1
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put chen2 1
# 监控前缀为chen的值
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 watch chen --prefix
# 修改chen1和chen2的值
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put chen1 10
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put chen2 20
5.5 Create lease
grant Creates leases
keep-alive Keeps leases alive (renew)
list List all active leases
revoke Revokes leases
timetolive Get lease information
etcdctl --endpoints=$ENDPOINTS lease grant 300
# lease 2be7547fbc6a5afa granted with TTL(300s)
etcdctl --endpoints=$ENDPOINTS put sample value --lease=2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS get sample
etcdctl --endpoints=$ENDPOINTS lease keep-alive 2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS lease revoke 2be7547fbc6a5afa
# or after 300 seconds
etcdctl --endpoints=$ENDPOINTS get sample
example:
# 创建租约
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 lease grant 300
# lease 5ece82ff7769d712 granted with TTL(300s)
# 给某个值赋予租约
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 put sample value --lease=5ece82ff7769d712
# 查看租约剩余时间
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 lease timetolive 5ece82ff7769d712
# 租约时间到后无法查询其值
You can check its value normally during the validity period of the lease
. After the lease time expires, you can’t check its value normally.
If you renew the lease after the deadline, an error will be reported, indicating that the lease has expired and cannot be renewed.
# 创建租约
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 lease grant 100
# lease 455182ff7785601a granted with TTL(300s)
# 查看租约剩余时间
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 lease timetolive 455182ff7785601a
# 续租约
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 lease keep-alive 455182ff7785601a
# 查看租约剩余时间
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 lease timetolive 455182ff7785601a
# 删除租约
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 lease revoke 455182ff7785601a
# 获取值
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 get sample
5.6 Create a lock
tcdctl --endpoints=$ENDPOINTS lock mutex1
# another client with the same name blocks
etcdctl --endpoints=$ENDPOINTS lock mutex1
example:
# 创建一个锁
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 lock mutex1
# 其他节点也去访问这个锁住的节点没有返回
./etcdctl --endpoints=http://192.168.159.200:2379,http://192.168.159.201:2379,http:192.168.159.202:2379 lock mutex1
We shut down node 1 and find that node 2 acquires the lock
