Kubesphere multi-cluster management enables simultaneous application deployment of multiple Kubernetes clusters

Article directory

测试kubesphere多集群管理功能，至少需要两套kubesphere集群环境。

1. Kubesphere cluster deployment

1. Prepare the environment

To execute the following command, all nodes in the cluster need to execute it.

yum install socat
yum install conntrack
yum install ebtables
yum install ipset
export KKZONE=cn

2. Download the kubesphere installation tool KubeKey

Here I first download the installation tool, decompress it on the deployed operating system, and then have the kk command after decompression. If you do not have execution permission, execute the following command.

chmod 755 kk

Insert image description here

3. Prepare cluster deployment configuration file

建议安装的时候将配置文件保留下来，后续集群升级可能会编辑配置文件。

Use the following command to generate the cluster configuration file first.

./kk create config --with-kubesphere v3.3.2 --with-kubernetes v1.22.12 -f config-kubesphere.yaml

Just modify the configuration file according to your needs. If you only need to install the simplest cluster, you only need to configure the hosts and roleGroups parts.


spec:
  hosts:
  - {
    
    name: master, address: 192.168.0.2, internalAddress: 192.168.0.2, user: ubuntu, password: Testing123}
  - {
    
    name: node1, address: 192.168.0.3, internalAddress: 192.168.0.3, user: ubuntu, password: Testing123}
  - {
    
    name: node2, address: 192.168.0.4, internalAddress: 192.168.0.4, user: ubuntu, password: Testing123}
  roleGroups:
    etcd:
    - master
    control-plane:
    - master
    worker:
    - node1
    - node2
  controlPlaneEndpoint:
    domain: lb.kubesphere.local
    address: ""
    port: 6443

4. Install the cluster

./kk create cluster -f config-kubesphere.yaml

The entire installation process may take 10 to 20 minutes, depending on your computer and network environment.
During the installation process, you can view the installation progress log through the following command

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

5. Verify installation

After the installation is complete, you will see the following:

#####################################################
###              Welcome to KubeSphere!           ###
#####################################################

Console: http://192.168.0.2:30880
Account: admin
Password: P@88w0rd

NOTES：
  1. After you log into the console, please check the
     monitoring status of service components in
     the "Cluster Management". If any service is not
     ready, please wait patiently until all components
     are up and running.
  2. Please change the default password after login.

#####################################################
https://kubesphere.io             20xx-xx-xx xx:xx:xx
#####################################################

You can now access KubeSphere's web console via <NodeIP:30880 using the default account and password (admin/P@88w0rd).

2. Configure multiple clusters

1. Open the nodeport port of the cluster tower service

This needs to be done in the management cluster.

Insert image description here
As shown in the figure, the port of the Tower service is mapped to nodeport, and the mapped port will be used below.

2. Modify the main cluster (for management)

Log in to the console using the admin account, then enter the "Custom Resource Definition" menu on the cluster management page, enter the keyword ClusterConfiguration, and then go to its details page.
Insert image description here
Edit the YAML file of ks-installer, set the value of clusterRole to host, and configure proxyPublishAddress to http://<IP>:31989.

  multicluster:
    clusterRole: host
    proxyPublishAddress: 'http://192.168.54.55:31989'

Click OK. This will take some time. You can view the progress log of the cluster update through the following command.

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

After the cluster is updated, log in again and you will find that there is a cluster management function menu on the page.
Insert image description here

3. Obtain the jwtSecret of the control cluster

kubectl -n kubesphere-system get cm kubesphere-config -o yaml | grep -v "apiVersion" | grep jwtSecret

Insert image description here

4. Configure the cluster to be added

Log in to the console using the admin account, then enter the "Customized Resource Definition" function menu on the cluster management page, enter the keyword ClusterConfiguration, and then go to its details page. Edit the YAML file of ks-installer, set jwtSecret, and set the value of clusterRole to member.

spec:
  alerting:
    enabled: false
  auditing:
    enabled: false
  authentication:
    jwtSecret: UC8rmYlHpDn4dhIMf9c1T2lxguv58zlt
 ........
 ........  //此处略去一万字
 ........
  multicluster:
    clusterRole: member

3. Import the cluster

On the Import Cluster page, enter the basic information of the cluster to be imported. You can also click the edit mode in the upper right corner to view and edit basic information in YAML format. After editing is complete, click Next.

In Connection method, select Cluster connection proxy, and then click Create. The YAML configuration file generated by the master cluster for agent deployment (Deployment) is displayed on the console.

Create an agent.yaml file in the member cluster as directed, and then copy and paste the agent deployment into the file. Execute kubectl create -f agent.yaml on the node and wait for the agent to be up and running. Please ensure that the proxy address is reachable by the member cluster.
Insert image description here
Fill in the cluster name, label and provider optional. Select Cluster Connection Broker.

When you see this page, follow the page prompts and perform the above operations in the cluster you are about to join. After a short wait (I tested it in less than 1 minute), the cluster will be successfully added. Refresh the main cluster page and you will see the joined cluster. cluster.

Insert image description here

Summarize

At this point, the multi-cluster test has been verified. In the subsequent creation of the enterprise workspace, multiple clusters can be used to create the workload. The creation of the workload can also be built in a multi-cluster environment. If it is built in a multi-cluster environment, it is equivalent to A workload will be created separately in a multi-cluster environment.