Article Directory
- 1. Background introduction
- 2. Open up the intranet of cross-cloud providers
- 3. Build a K8S cluster
- 4. Container clusters add workload nodes across cloud providers
- 5. Summary
1. Background introduction
As cloud computing service providers such as Amazon Cloud, Alibaba Cloud, Huawei Cloud, and Tencent Cloud become more secure, stable, and cheaper, more and more companies or individuals are beginning to try or use cloud computing service providers. The IaaS service replaces the infrastructure resources in the self-built IDC computer room. Faced with the marketing routines of various cloud computing vendors, many companies don’t know which cloud provider’s service is more suitable. If they are afraid of being kidnapped, it is easy to get in and out of the car, and it is free for the first year, and there is no discount for the second year; I am afraid that if I go to Ali this year, Huawei will be cheaper next year, and Tencent will be more favorable in the year after, so service migration will be difficult. How to switch business between different cloud providers at will? In this way, you can choose the resources of which cloud you want to use.
The following is an experiment to try a method to achieve the above demands, that is, to build a k8s container cluster platform, deploy workload nodes on cloud hosts of different cloud providers, and register to the k8s cluster. Realize a set of container clusters to manage the workload nodes of multiple cloud providers, and dynamically distribute containerized deployment services to cloud providers with higher cost performance according to the preferential policies of cloud providers.
As shown in the figure above, 3 nodes are deployed on Huawei Cloud, and 3 nodes are deployed on Alibaba Cloud. Among them, k8s-master1 and k8s-master2 on Huawei Cloud are the management services of the k8s cluster, and k8s-node1 is the workload node of the k8s cluster. The k8s-node2, k8s-node3, and k8s-node4 on Alibaba Cloud are the workload nodes of the k8s cluster.
2. Open up the intranet of cross-cloud providers
By establishing a GRE tunnel, establish a GRE tunnel between Huawei Cloud k8s-master2 and Alibaba Cloud k8s-node2 nodes, and open up the internal network communication between Huawei Cloud and Alibaba Cloud. At the same time, use iptables to set traffic forwarding rules. For example, k8s-master2 forwards the traffic of hosts on Alibaba Cloud to k8s-master1 and k8s-node1 by setting iptables rules. Similarly, k8s-node2 forwards traffic from hosts on Huawei Cloud to The host forwards the traffic of k8s-node3 and k8s-node4, so as to realize intranet communication between all hosts on Huawei Cloud and Alibaba Cloud.
For the method of establishing a GRE tunnel, please refer to Building a GRE Tunnel to Connect Cloud Host Intranets of Different Cloud Providers . This article mainly introduces the method of building a GRE tunnel between two cloud providers to realize the intercommunication between the two intranets. Since the route added by using the route tool will not be saved persistently (the server will be lost after restarting), this chapter will introduce how to build a GRE tunnel and save it persistently.
2.1 Alibaba Cloud Environment Installation and Deployment
2.1.1 Load the driver module
Execute the following command on the Alibaba Cloud k8s-node2 node
cat > /etc/sysctl.d/gre.conf <<EOF
net.ipv4.ip_forward=1
EOF
2.1.2 Add GRE tunnel startup script
Execute the following command on the Alibaba Cloud k8s-node2 node
cat > /etc/init.d/gre.sh <<EOF
#!/bin/bash
ip tunnel del tunnel999
ip tunnel add tunnel999 mode gre remote 112.124.59.21 local 172.26.32.235
ip link set tunnel999 up mtu 1476
ip addr add 192.168.100.2 peer 192.168.100.1/32 dev tunnel999
ip route add 192.168.0.0/24 dev tunnel999
EOF
chmod +x /etc/init.d/gre.sh
cat > /usr/lib/systemd/system/gre.service <<EOF
[Unit]
Description=GRE Service
After=network.target
[Service]
Type=oneshot
User=root
ExecStart=/etc/init.d/gre.sh
[Install]
WantedBy=multi-user.target
EOF
systemctl enable gre
systemctl start gre
2.1.3 Configure iptalbes rules
Execute the following command on the Alibaba Cloud k8s-node2 node
iptables -t nat -A POSTROUTING -o eth0 -s 192.168.100.1 -j MASQUERADE
iptables -A FORWARD -s 192.168.100.1 -o eth0 -j ACCEPT
iptables -t nat -A POSTROUTING -o tunnel999 -s 192.168.0.0/24 -j MASQUERADE
iptables -A FORWARD -s 192.168.0.0/24 -o tunnel999 -j ACCEPT
2.1.4 Configure routing on other nodes
Execute the following command on Alibaba Cloud k8s-node3 and k8s-node4 nodes to add the CIDR route of Huawei Cloud intranet to the intranet IP of k8s-node2 node.
route add -net 192.168.0.0/24 gw 172.26.32.235
After the addition is complete, k8s-node3 and k8s-node4 cannot access the internal network IP of the HUAWEI CLOUD host for the time being. After the installation and deployment on HUAWEI CLOUD is completed, they can access the internal IP of the host on HUAWEI CLOUD.
2.2 Installation and deployment on HUAWEI CLOUD
2.2.1 Load the driver module
Execute the following command on the HUAWEI CLOUD k8s-master2 node
cat > /etc/sysctl.d/gre.conf <<EOF
net.ipv4.ip_forward=1
EOF
2.2.2 Add GRE tunnel startup script
Execute the following command on the HUAWEI CLOUD k8s-master2 node
cat > /etc/init.d/gre.sh <<EOF
#!/bin/bash
ip tunnel del tunnel999
ip tunnel add tunnel999 mode gre remote 114.116.84.123 local 192.168.0.200
ip link set tunnel999 up mtu 1476
ip addr add 192.168.100.1 peer 192.168.100.2/32 dev tunnel999
ip route add 172.26.32.0/24 dev tunnel999
EOF
chmod +x /etc/init.d/gre.sh
cat > /usr/lib/systemd/system/gre.service <<EOF
[Unit]
Description=GRE Service
After=network.target
[Service]
Type=oneshot
User=root
ExecStart=/etc/init.d/gre.sh
[Install]
WantedBy=multi-user.target
EOF
systemctl enable gre
systemctl start gre
2.2.3 Configure iptalbes rules
Execute the following command on the HUAWEI CLOUD k8s-master2 node
iptables -t nat -A POSTROUTING -o eth0 -s 192.168.100.2 -j MASQUERADE
iptables -A FORWARD -s 192.168.100.2 -o eth0 -j ACCEPT
iptables -t nat -A POSTROUTING -o tunnel999 -s 172.26.32.0/24 -j MASQUERADE
iptables -A FORWARD -s 172.26.32.0/24 -o tunnel999 -j ACCEPT
2.2.4 Configure routing on other nodes
Execute the following command on the HUAWEI CLOUD k8s-master1 and k8s-node1 nodes to add the Alibaba Cloud internal network CIDR route to the internal network IP of the k8s-master2 node.
route add -net 172.26.32.0/24 gw 192.168.0.200
After adding, you can access the intranet IPs of all cloud hosts on Alibaba Cloud on k8s-node1 and k8s-master1 nodes, and all cloud hosts on Alibaba Cloud can access the intranet IPs of all cloud hosts on Huawei Cloud.
3. Build a K8S cluster
First, install and deploy the master node of the kubernetes cluster on HUAWEI CLOUD. For specific steps, please refer to: Deploy and install the kubernetes cluster .
4. Container clusters add workload nodes across cloud providers
Install the kubernetes worker node configuration on the Alibaba Cloud host. It mainly involves: Linux kernel upgrade , Containerd container installation, Linux parameter optimization, Kubelet and Kube-Proxy component deployment. The following operations take Alibaba Cloud host 172.26.32.235 as an example.
4.1 Install Containerd container service
- Install containerd service online
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install containerd
- Export containerd default configuration
containerd config default > /etc/containerd/config.toml
- Edit the /etc/containerd/config.toml file
sandbox_image = "registry.k8s.io/pause:3.6"
替换成
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = false
替换成
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
It is very important to add a domestic mirror warehouse, otherwise the mirror download will fail and the service in the Pod will fail to start.
[plugins."io.containerd.grpc.v1.cri".registry]
......
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
在containerd 配置文件中找到上边内容，并在此处添加下边两行, 注意缩进，下边两行内容与上边一行有2个空格的缩进，下边两行内容之间也存在2个空格的缩进。
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["http://hub-mirror.c.163.com","https://docker.mirrors.ustc.edu.cn","https://registry.docker-cn.com"]
- Restart the containerd container runtime service
systemctl enable containerd
systemctl restart containerd
4.2 Linux parameter optimization
For Linux parameter optimization, please refer to K8S installation process 7: Kubernetes node configuration adjustment
4.3 Installation of Kubernetes Worker node components
4.3.1 Get the installation package
Obtain the kubernetes installation package from the node that has been installed and deployed on HUAWEI CLOUD. For the detailed process, please refer to: K8S installation process nine: Kubernetes Worker node installation
- Obtain the installation package and ssl certificate.
scp [email protected]:/opt/kubernetes.tar.gz /opt/
mkdir -p /etc/kubernetes/ssl
scp [email protected]:/etc/kubernetes/ssl/* /etc/kubernetes/ssl/
cd /opt
tar -xvf kubernetes.tar.gz
mkdir /opt/kubernetes/manifests
4.3.2 kubelet creates systemctl to start the service
cat > /usr/lib/systemd/system/kubelet.service <<EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/server/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
4.3.3 Start kubelet
systemctl enable kubelet
systemctl start kubelet
- View the startup status of the kubelet service
systemctl status kubelet
4.3.4 Create systemctl startup service for kube-proxy
cat > /usr/lib/systemd/system/kube-proxy.service <<EOF
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/server/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
4.3.5 Start the kube-proxy service
systemctl enable kube-proxy
systemctl start kube-proxy
- View the startup status of the kube-proxy service
systemctl status kube-proxy
So far, the construction of container clusters across different cloud providers is completed. At this time, the workload nodes in the container cluster cover Huawei Cloud and Alibaba Cloud. To add cloud hosts of other cloud providers later, the process is similar to the above.
5. Summary
- The workload nodes in the container cluster span multiple cloud providers, and the mutual service calls between different cloud providers actually use public network traffic. If the public network bandwidth resources are insufficient, there will be performance bottlenecks in cross-cloud service calls.
- To deploy container clusters across cloud providers, it is recommended that the services running in the workloads of different cloud providers be independent, that is, to reduce the mutual calls between services on different cloud providers.
- When it comes to the use of middleware, it is necessary to consider ensuring access performance when accessing middleware across cloud providers.