A precondition and environment settings
1, set up environment
- Operating System Version: SLES15 SP1, without having to install swap
- Kernel version: 4.12.14-197.18-default
- Kubernetes version: v1.15.2
- VMware Workstation 14
2, virtualized environment to build and install the systems Reference:
- SUSE Storage6 environment to build a detailed step - Win10 + VMware WorkStation
- SUSE Linux Enterprise 15 SP1 Installation
3, the network environment
- Network management: band access, container management platform
- External Network: Internet access, easy to download the image from the image repository architecture diagram using the VMnet8, access the Internet through NAT
- Storage Networking: docking of Public SUSE Storage6 distributed storage network, providing container to use as a durable storage
4, CaaSP4 environment Chart
Second, the installation deployment
Installation system initialization
1, edit the Hosts file
# vim /etc/hosts 172.200.50.70 master01.example.com master01 172.200.50.71 master02.example.com master02 172.200.50.72 master03.example.com master03 172.200.50.73 worker01.example.com worker01 172.200.50.74 worker02.example.com worker02 172.200.50.75 worker03.example.com worker03 172.200.50.76 worker04.example.com worker04
2, add warehouse
## Pool zypper ar http://172.200.50.19/repo/SUSE/Products/SUSE-CAASP/4.0/x86_64/product/ SUSE-CAASP-4.0-Pool zypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Module-Containers/15-SP1/x86_64/product/ SLE-Module-Containers-15-SP1-Pool zypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Product-SLES/15-SP1/x86_64/product/ SLE-Product-SLES15-SP1-Pool zypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Module-Basesystem/15-SP1/x86_64/product/ SLE-Module-Basesystem-SLES15-SP1-Pool zypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Module-Server-Applications/15-SP1/x86_64/product/ SLE-Module-Server-Applications-SLES15-SP1-Pool zypper ar http://172.200.50.19/repo/SUSE/Products/SLE-Module-Legacy/15-SP1/x86_64/product/ SLE-Module-Legacy-SLES15-SP1-Pool ## Update zypper ar http://172.200.50.19/repo/SUSE/Updates/SLE-Product-SLES/15-SP1/x86_64/update/ SLE-Product-SLES15-SP1-Updates zypper ar http://172.200.50.19/repo/SUSE/Updates/SLE-Module-Basesystem/15-SP1/x86_64/update/ SLE-Module-Basesystem-SLES15-SP1-Upadates zypper ar http://172.200.50.19/repo/SUSE/Updates/SLE-Module-Server-Applications/15-SP1/x86_64/update/ SLE-Module-Server-Applications-SLES15-SP1-Upadates zypper ar http://172.200.50.19/repo/SUSE/Updates/SLE-Module-Legacy/15-SP1/x86_64/update/ SLE-Module-Legacy-SLES15-SP1-Updates zypper ar http://172.200.50.19/repo/SUSE/Updates/SUSE-CAASP/4.0/x86_64/update/ SUSE-CAASP-4.0-Updates zypper ar http://172.200.50.19/repo/SUSE/Updates/SLE-Module-Containers/15-SP1/x86_64/update/ SLE-Module-Containers-15-SP1-Updates
3, the basic software installation
# zypper in -y -t pattern yast2_basis base # zypper in -y net-tools vim man sudo tuned irqbalance # zypper in -y ethtool rsyslog iputils less supportutils-plugin-ses # zypper in -y net-tools-deprecated tree wget
4, close IPV6 (all nodes and admin) and closed swap
# vim /etc/sysctl.conf net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1
5, create and configure after.local file (optional)
Note: The system has been configured to use swap, please turn off select and execute scripts at boot
# touch /etc/init.d/after.local # chmod 744 /etc/init.d/after.local
cat >> /etc/init.d/after.local << EOF #! /bin/sh # # Copyright (c) 2010 SuSE LINUX Products GmbH, Germany. All rights reserved. # # Author: Werner Fink, 2010 # # /etc/init.d/after.local # # script with local commands to be executed from init after all scripts # of a runlevel have been executed. # # Here you should add things, that should happen directly after # runlevel has been reached. # swapoff -a EOF
6, the implementation after.local, close swap
# source /etc/init.d/after.local
CaaSP4 Cluster Setup
1, the mounting assembly (all nodes)
# zypper -n in -t pattern SUSE-CaaSP-Management
2 , SSH trust, and Agent-enabled SSH
(1) master01 host-generated key pair and copy the public key to the host woker01 and woker02
# ssh-keygen # cd .ssh/ # ssh-copy-id master01 # ssh-copy-id worker01 # ssh-copy-id worker02
(2) master01 host, start ssh-agent
# eval `ssh-agent` $SHELL Agent pid 13701
# ps -ef | grep ssh-agent root 13701 1 0 09:35 ? 00:00:00 ssh-agent root 13719 13702 0 09:35 pts/0 00:00:00 grep --color=auto ssh-agent
(3) adding to the private key in ssh-agent
# ssh-add /root/.ssh/id_rsa Identity added: /root/.ssh/id_rsa (/root/.ssh/id_rsa)
# ssh-add -l 2048 SHA256:7sn4CfA7qk7/q38CA6Oz3tgvVShY9+fpOXZRdjAM1MM root@master01 (RSA)
3 , the initialization Master node
Syntax: skuba cluster init --control-plane < LB IP / FQDN> my-cluster
# skuba cluster init --control-plane master01 my-cluster [init] configuration files written to /root/.ssh/my-cluster
My-cluster generating initialization directory, and the directory profile having kubeconfig
# ll /root/.ssh/my-cluster/ total 4 drwx------ 8 root root 78 Sep 15 13:31 addons -rw------- 1 root root 804 Sep 15 13:31 kubeadm-init.conf drwx------ 2 root root 62 Sep 15 13:31 kubeadm-join.conf.d
4, cluster Booststrap
# cd /root/.ssh/my-cluster/ # skuba node bootstrap --target master01 master01 -v3
-V3 the use of parameters, the terminal will generate installation details until the last displayed successfully bootstrapped
I0915 13:58:55.780337 4730 files.go:29] uploading to remote file "/tmp/gangway.d/gangway.yaml" with contents
I0915 13:58:55.867266 4730 ssh.go:167] running command: "kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f /tmp/gangway.d"
I0915 13:58:56.136908 4730 ssh.go:190] stdout | configmap/oidc-gangway-config created
I0915 13:58:56.143730 4730 ssh.go:190] stdout | deployment.apps/oidc-gangway created
I0915 13:58:56.201805 4730 ssh.go:190] stdout | service/oidc-gangway created
I0915 13:58:56.213596 4730 ssh.go:190] stdout | serviceaccount/oidc-gangway created
I0915 13:58:56.213946 4730 ssh.go:167] running command: "rm -rf /tmp/gangway.d"
I0915 13:58:56.255647 4730 states.go:40] === state gangway.deploy applied successfully ===
[bootstrap] successfully bootstrapped node "master01" with Kubernetes: "1.15.2" <=== 安装成功
5、加入 worker 节点
语法:
skuba node join --role <master/worker> --user <user-name> --sudo --target <IP/FQDN> <node-name>
- master01 nodes and perform with the root account:
# cd ~/.ssh/my-cluster # skuba node join --role worker --target worker01 worker01 -v3 # skuba node join --role worker --target worker02 worker02 -v3
I0915 14:42:41.768349 46412 states.go:35] === applying state skuba-update.start === I0915 14:42:41.769311 46412 ssh.go:167] running command: "systemctl enable --now skuba-update.timer" I0915 14:42:41.828002 46412 ssh.go:190] stderr | Created symlink /etc/systemd/system/timers.target.wants/skuba-update.timer → /usr/lib/systemd/system/skuba-update.timer. I0915 14:42:42.096214 46412 states.go:40] === state skuba-update.start applied successfully === [join] node successfully joined the cluster
Check the cluster status
1, Master node
(1) Master node downloaded image
master01:~ # crictl images IMAGE TAG IMAGE ID SIZE registry.suse.com/caasp/v4/cilium-init 1.5.3 726a5a2fd9f8e 121MB registry.suse.com/caasp/v4/cilium 1.5.3 15b448bea31e7 889MB registry.suse.com/caasp/v4/coredns 1.3.1 2f747b3021d99 163MB registry.suse.com/caasp/v4/etcd 3.3.11 b90b8b149824d 223MB registry.suse.com/caasp/v4/hyperkube v1.15.2 e19d9ad312c9d 672MB registry.suse.com/caasp/v4/kured 1.2.0 9d0512cb475d8 358MB registry.suse.com/caasp/v4/pause 3.1 247351923b047 936kB
(2) Master nodes running the container
# crictl ps -a CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID 98cc214ea02e6 e19d9ad312c9d698740a587857fdebc77c17bd7fc05214a74658b0b6f779c597 3 minutes ago Running kube-proxy 0 21617c8adaa8d adb894ff5bd1e b90b8b149824d799c958d04fd16df356ccf254944467e5666e3abb7e6789ffa5 3 minutes ago Running etcd 0 0629bbad3a0f8 cfbab432bd3dd e19d9ad312c9d698740a587857fdebc77c17bd7fc05214a74658b0b6f779c597 3 minutes ago Running kube-scheduler 0 d463bbd4dbefd 1cf5a0829a1ff e19d9ad312c9d698740a587857fdebc77c17bd7fc05214a74658b0b6f779c597 3 minutes ago Running kube-controller-manager 0 7c045016665d2 597c1feedc3dd e19d9ad312c9d698740a587857fdebc77c17bd7fc05214a74658b0b6f779c597 3 minutes ago Running kube-apiserver 0 8f258d290806a
(3) Master nodes running pods
# crictl pods POD ID CREATED STATE NAME NAMESPACE ATTEMPT e897316a364c7 4 minutes ago Ready cilium-shddp kube-system 0 21617c8adaa8d 5 minutes ago Ready kube-proxy-2f57p kube-system 0 0629bbad3a0f8 6 minutes ago Ready etcd-master01 kube-system 0 d463bbd4dbefd 6 minutes ago Ready kube-scheduler-master01 kube-system 0 7c045016665d2 6 minutes ago Ready kube-controller-manager-master01 kube-system 0 8f258d290806a 6 minutes ago Ready kube-apiserver-master01 kube-system 0
2, woker node
(1) worker node download image
worker01:~ # crictl images IMAGE TAG IMAGE ID SIZE registry.suse.com/caasp/v4/caasp-dex 2.16.0 32d64343dba35 146MB registry.suse.com/caasp/v4/cilium-init 1.5.3 726a5a2fd9f8e 121MB registry.suse.com/caasp/v4/cilium-operator 1.5.3 918bd75777bb1 196MB registry.suse.com/caasp/v4/cilium 1.5.3 15b448bea31e7 889MB registry.suse.com/caasp/v4/gangway 3.1.0 227070f1f75d6 136MB registry.suse.com/caasp/v4/hyperkube v1.15.2 e19d9ad312c9d 672MB 1.2.0 9d0512cb475d8 358MB Registry.suse.com/caasp/v4/kured Registry.suse.com/caasp/v4/pause 3.1 247351923b047 936kB
(2) worker nodes running the container
worker01:~ # crictl ps -a CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID b55f388e47cda 32d64343dba35eed1b365322f3f60ffe2dc42b7b327c7cc2870c881a2efdbf1b 37 minutes ago Running oidc-dex 1 4a536a9499080 a0c58bf84de32 registry.suse.com/caasp/v4/caasp-dex@sha256:0542dfb58a97e2de8d25128b673aaa8c1e2dc2680b0db31872eb47b8fe6ce409 37 minutes ago Exited oidc-dex 0 4a536a9499080 fd585e82ec6eb registry.suse.com/caasp/v4/gangway@sha256:85922202aaa071e5e13084c58c1375ae7029af77bfe7d99d63e22e98ca574bc3 37 minutes ago Running oidc-gangway 0 c865bd3ac446c cdfbabee16375 registry.suse.com/caasp/v4/kured@sha256:52d0bb956a4a012131436263cdc8126f86042af1d6f0c93bd461d229743f5bf3 38 minutes ago Running kured 0 27af5c286c721 7f9d9d80f52ed registry.suse.com/caasp/v4/cilium-operator@sha256:02ba730698cba66dbd71d66b68aa3fd9ce9a78ece55e246bd837a0a444ebaaed 38 minutes ago Running cilium-operator 0 9a90ff209bb50 427c05258e7b8 registry.suse.com/caasp/v4/hyperkube@sha256:6a7baa891d90d68bc856d321dcf4cbdd2b86b25204043276f44b4f81a270a515 39 minutes ago Running kube-proxy 0 ed62ccc311d4e c5ac07f3c68ea 15b448bea31e7ff401acac76f6518986be200667c00869582b3cd3beb30dd69b 39 minutes ago Running cilium-agent 1 54c628d7909f9 4b432bb3a085f 15b448bea31e7ff401acac76f6518986be200667c00869582b3cd3beb30dd69b 41 minutes ago Exited cilium-agent 0 54c628d7909f9 27be548f473f5 registry.suse.com/caasp/v4/cilium-init@sha256:77d7101f0af12c3f6f1f1fd2062a0efc93d058b8358b3b9f2a02a69ba321dff4 41 minutes ago Exited clean-cilium-state 0 54c628d7909f9 c393915545030 15b448bea31e7ff401acac76f6518986be200667c00869582b3cd3beb30dd69b 42 minutes ago Exited install-cni-bin 0 54c628d7909f9 8c0a5fd0fa598 15b448bea31e7ff401acac76f6518986be200667c00869582b3cd3beb30dd69b 42 minutes ago Exited install-cni-conf 0 54c628d7909f9
(3) Worker nodes running pods
# crictl pods POD ID CREATED STATE NAME NAMESPACE ATTEMPT c865bd3ac446c 40 minutes ago Ready oidc-gangway-7b7fbbdbdf-s94h5 kube-system 0 9a90ff209bb50 40 minutes ago Ready cilium-operator-7d6ddddbf5-8clw8 kube-system 0 27af5c286c721 40 minutes ago Ready kured-n2zz4 kube-system 0 4a536a9499080 40 minutes ago Ready oidc-dex-55fc689dc-g6rbq kube-system 0 54c628d7909f9 43 minutes ago Ready cilium-4fp22 kube-system 0 ed62ccc311d4e About an hour ago Ready kube-proxy-fwt2t kube-system 0
3, CaaSP4 cluster
# cd /root/.ssh/my-cluster/ # skuba cluster status NAME OS-IMAGE KERNEL-VERSION KUBELET-VERSION CONTAINER-RUNTIME HAS-UPDATES HAS-DISRUPTIVE-UPDATES master01 SUSE Linux Enterprise Server 15 SP1 4.12.14-197.18-default v1.15.2 cri-o://1.15.0 <none> <none> worker01 SUSE Linux Enterprise Server 15 SP1 4.12.14-197.18-default v1.15.2 cri-o://1.15.0 <none> <none> worker02 SUSE Linux Enterprise Server 15 SP1 4.12.14-197.18-default v1.15.2 cri-o://1.15.0 <none> <none>
CaaSP4 cluster using kubectl
1, the installation package (optional)
Zypper # in Kubernetes-Client (installed by default)
2, the establishment of flexible connections
# ln -s /root/.ssh/my-cluster/admin.conf ~/.kube/config
3, the cluster information
# kubectl cluster-info Kubernetes master is running at https://master01:6443 KubeDNS is running at https://master01:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
4, cluster node information
# kubectl get nodes # kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION master01 Ready master 72m v1.15.2 worker01 Ready <none> 29m v1.15.2 worker02 Ready <none> 28m v1.15.2
5, Pods status of the cluster running
# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system cilium-dph24 1/1 Running 0 28m kube-system cilium-fjs6c 1/1 Running 0 28m kube-system cilium-operator-7d6ddddbf5-xd6p9 1/1 Running 0 72m kube-system cilium-s5l4k 1/1 Running 0 28m kube-system coredns-69c4947958-25rsk 1/1 Running 4 72m kube-system coredns-69c4947958-ntn7w 1/1 Running 6 72m kube-system etcd-master01 1/1 Running 0 71m kube-system kube-apiserver-master01 1/1 Running 0 71m kube-system kube-controller-manager-master01 1/1 Running 0 71m kube-system kube-proxy-4hk7q 1/1 Running 0 29m kube-system kube-proxy-8jgd7 1/1 Running 0 72m kube-system kube-proxy-mwrzx 1/1 Running 0 28m kube-system kube-scheduler-master01 1/1 Running 0 71m kube-system kured-26h25 1/1 Running 0 26m kube-system kured-hvhdm 1/1 Running 0 28m kube-system kured-tpgvh 1/1 Running 0 70m kube-system oidc-dex-55fc689dc-pxgv2 1/1 Running 1 72m kube-system oidc-gangway-7b7fbbdbdf-jvfdk 1/1 Running 0 72m