In the previous article , we introduced the architecture and deployment scenarios of K3s in detail, and provided a good direction for those who have not yet understood K3s. So, in this article we will explore how to configure a 3-node etcd cluster, which will be used in a highly available, multi-node K3s cluster.
etcd is one of the most popular open source projects in the cloud native ecosystem. It is a project incubated by the Cloud Native Computing Foundation (CNCF) and has now become a core component of the Kubernetes infrastructure.
At the end of this tutorial, you will complete the deployment of a 3-node etcd cluster with TLS enabled as an external data storage for a highly available K3s cluster with multiple masters.
First, make sure you have 3 Linux hosts with static IP addresses. In my experimental environment, I run 4 Intel NUC mini computers. These 4 computers are running Ubuntu 18.04 with IP addresses ranging from 10.0.0.60 to 10.0.0.63. We will install etcd on hosts with IP addresses 10.0.0.60, 10.0.0.61, and 10.0.0.62. You must replace these IP addresses with your own set of addresses when you practice on your own.
Download etcd binary file
On each Linux host, run the following command to download and install the latest version of the binary file:
ETCD_VER=v3.4.10
DOWNLOAD_URL=https://storage.googleapis.com/etcd
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
chmod +x /tmp/etcd-download-test/etcd
chmod +x /tmp/etcd-download-test/etcdctl
#Verify the downloads
/tmp/etcd-download-test/etcd --version
/tmp/etcd-download-test/etcdctl version
#Move them to the bin folder
sudo mv /tmp/etcd-download-test/etcd /usr/local/bin
sudo mv /tmp/etcd-download-test/etcdctl /usr/local/bin
Generate and distribute certificates
We will use Cloudflare's cfssl tool to generate certificates and keys. If you use a Mac as your workstation, you can install it using Homebrew.
brew install cfssl
Create a directory named certs, and run the following command to generate the CA certificate and server certificate and key combination for each host.
mkdir certs && cd certs
First, create a CA certificate, which will be used by all etcd servers and clients.
echo '{"CN":"CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - | cfssljson -bare ca -
echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","server auth","client auth"]}}}' > ca-config.json
This will generate three files- ca-key.pem, ca.pemand ca.csr.
Next, we will generate certificates and keys for the first node
export NAME=node-1
export ADDRESS=10.0.0.60,$NAME
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME
Repeat the above steps for the next two nodes.
export NAME=node-2
export ADDRESS=10.0.0.61,$NAME
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME
export NAME=node-3
export ADDRESS=10.0.0.62,$NAME
Don't forget to replace the IP address and node name with your own combination.
At this point, we have generated certificates and keys for the CA and all three nodes.
Now it is time to start distributing certificates to each node of the cluster.
Run the following command, replace the username and IP address, and copy the certificate to the corresponding node.
HOST=10.0.0.60
USER=ubuntu
scp ca.pem $USER@$HOST:etcd-ca.crt
scp node-1.pem $USER@$HOST:server.crt
scp node-1-key.pem $USER@$HOST:server.key
SSH into each node and run the following command to move the certificate to the appropriate directory.
HOST=10.0.0.60
USER=ubuntu
ssh $USER@$HOST
sudo mkdir -p /etc/etcd
sudo mv * /etc/etcd
sudo chmod 600 /etc/etcd/server.key
We have completed the generation and distribution of certificates on each node. Next, we will create configuration files and Systemd unit files for each node.
Configure and start etcd cluster
On node 1, create a file named etcd.conf in the etc/etcd directory with the following content:
ETCD_NAME=node-1
ETCD_LISTEN_PEER_URLS="https://10.0.0.60:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.60:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER="node-1=https://10.0.0.60:2380,node-2=https://10.0.0.61:2380,node-3=https://10.0.0.62:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.60:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.60:2379"
ETCD_TRUSTED_CA_FILE="/etc/etcd/etcd-ca.crt"
ETCD_CERT_FILE="/etc/etcd/server.crt"
ETCD_KEY_FILE="/etc/etcd/server.key"
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/etcd-ca.crt"
ETCD_PEER_KEY_FILE="/etc/etcd/server.key"
ETCD_PEER_CERT_FILE="/etc/etcd/server.crt"
ETCD_DATA_DIR="/var/lib/etcd"
The file for node 2 uses the following content:
ETCD_NAME=node-2
ETCD_LISTEN_PEER_URLS="https://10.0.0.61:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.61:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER="node-1=https://10.0.0.60:2380,node-2=https://10.0.0.61:2380,node-3=https://10.0.0.62:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.61:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.61:2379"
ETCD_TRUSTED_CA_FILE="/etc/etcd/etcd-ca.crt"
ETCD_CERT_FILE="/etc/etcd/server.crt"
ETCD_KEY_FILE="/etc/etcd/server.key"
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/etcd-ca.crt"
ETCD_PEER_KEY_FILE="/etc/etcd/server.key"
ETCD_PEER_CERT_FILE="/etc/etcd/server.crt"
ETCD_DATA_DIR="/var/lib/etcd"
Finally, create a configuration file for node 3.
ETCD_NAME=node-3
ETCD_LISTEN_PEER_URLS="https://10.0.0.62:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.62:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER="node-1=https://10.0.0.60:2380,node-2=https://10.0.0.61:2380,node-3=https://10.0.0.62:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.62:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.62:2379"
ETCD_TRUSTED_CA_FILE="/etc/etcd/etcd-ca.crt"
ETCD_CERT_FILE="/etc/etcd/server.crt"
ETCD_KEY_FILE="/etc/etcd/server.key"
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/etcd-ca.crt"
ETCD_PEER_KEY_FILE="/etc/etcd/server.key"
ETCD_PEER_CERT_FILE="/etc/etcd/server.crt"
ETCD_DATA_DIR="/var/lib/etcd"
Please don't forget to change your network dedicated IP address.
After the configuration is complete, we can create systemd unit files on each node.
Create a file etcd.service at /lib/system/systemd with the following content:
[Unit]
Description=etcd key-value store
Documentation=https://github.com/etcd-io/etcd
After=network.target
[Service]
Type=notify
EnvironmentFile=/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd
Restart=always
RestartSec=10s
LimitNOFILE=40000
[Install]
WantedBy=multi-user.target
Since the configuration of each node has been moved to a dedicated file ( /etc/etcd/etcd.conf), the unit files of all nodes remain unchanged.
Now we are ready to start the service. Run the following command on each node to start the etcd cluster:
sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcd
Make sure that the etcd service has been started and there are no errors in its operation.
sudo systemctl status etcd
Test and verify the cluster
SSH into one of the nodes and connect to the cluster through etcd CLI.
etcdctl --endpoints https://10.0.0.60:2379 --cert /etc/etcd/server.crt --cacert /etc/etcd/etcd-ca.crt --key /etc/etcd/server.key put foo bar
We insert a key in the etcd database. Let us see if we can get it back.
etcdctl --endpoints https://10.0.0.60:2379 --cert /etc/etcd/server.crt --cacert /etc/etcd/etcd-ca.crt --key /etc/etcd/server.key get foo
Next, let us use the API endpoint to check the health of the cluster.
curl --cacert /etc/etcd/etcd-ca.crt --cert /etc/etcd/server.crt --key /etc/etcd/server.key https://10.0.0.60:2379/health
Finally, let us ensure that all nodes participate in the cluster.
etcdctl --endpoints https://10.0.0.60:2379 --cert /etc/etcd/server.crt --cacert /etc/etcd/etcd-ca.crt --key /etc/etcd/server.key member list
Congratulations! Now you have a secure, distributed and highly available etcd cluster, which is ready for the production-level K3s cluster environment.
In the next article, I will show you in detail how to install and configure a 4-node K3s cluster with a highly available control plane. Stay tuned~!