記事ディレクトリ
kubernetes リリースは にアップグレードv1.27.4されました。関連する v1.27.4 アップグレード内容については、こちらを参照してください。アップグレードするのが待ちきれません。
1. 始める前に
- スナップショット
- 業務アプリケーションのバックアップデータ
アップグレードするクラスター情報を準備しますkubernetes。
$ kubectl get node
NAME STATUS ROLES AGE VERSION
kube-master01 Ready control-plane 42h v1.27.3
kube-node01 Ready <none> 42h v1.27.3
kube-prom01 Ready <none> 42h v1.27.3
2. 注意事項
- kubelet のマイナー バージョン アップグレードを実行する場合は、まずアップグレードするノードを空にする必要があります。コントロール プレーン ノードでは、CoreDNS ポッドまたはその他の重要なワークロードを実行している可能性があります。詳細については、 「ノードのドレイン」を参照してください。
- コンテナ仕様のハッシュ値が変更されたため、アップグレード後にすべてのコンテナが再起動されます。
- kubelet のアップグレード後に kubelet サービスが正常に再起動したかどうかを確認するには、 を実行する
systemctl status kubeletか、journalctl -xeu kubeletでサービス ログを表示できます。 - クラスターの再構成には推奨され
kubeadm upgradeない--configフラグとkubeadm 構成 APIタイプ。予期しない結果が生じる可能性があります。kubeadm クラスターの再構成の手順に従います。
3.マスター
3.1 最初のノードにログインし、kubeadm ツールのみをアップグレードする
$dnf list kubeadm kubelet kubectl --showduplicates | sort -r |grep 1.27.4
kubelet.x86_64 1.27.4-0 kubernetes
kubectl.x86_64 1.27.4-0 kubernetes
kubeadm.x86_64 1.27.4-0 kubernetes
$ dnf check-update kubeadm kubectl kubelet
Last metadata expiration check: 2:04:02 ago on Thu 20 Jul 2023 09:34:52 AM CST.
kubeadm.x86_64 1.27.4-0 kubernetes
kubectl.x86_64 1.27.4-0 kubernetes
kubelet.x86_64 1.27.4-0 kubernetes
$ dnf update -y kubeadm
出力：
[root@kube-prom01 ~]# dnf update -y kubeadm-1.27.4-0
Last metadata expiration check: 1:16:03 ago on Thu 20 Jul 2023 11:14:46 AM CST.
Dependencies resolved.
====================================================================================================================================================================================================
Package Architecture Version Repository Size
====================================================================================================================================================================================================
Upgrading:
kubeadm x86_64 1.27.4-0 kubernetes 11 M
Transaction Summary
====================================================================================================================================================================================================
Upgrade 1 Package
Total download size: 11 M
Downloading Packages:
e9bba51c897d8e465298724f44da6e457097f87aaac71b18fd6539b9e3503995-kubeadm-1.27.4-0.x86_64.rpm 9.2 MB/s | 11 MB 00:01
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 9.2 MB/s | 11 MB 00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: kubeadm-1.27.4-0.x86_64 1/1
Upgrading : kubeadm-1.27.4-0.x86_64 1/2
Cleanup : kubeadm-1.27.3-0.x86_64 2/2
Running scriptlet: kubeadm-1.27.3-0.x86_64 2/2
Verifying : kubeadm-1.27.4-0.x86_64 1/2
Verifying : kubeadm-1.27.3-0.x86_64 2/2
Upgraded:
kubeadm-1.27.4-0.x86_64
Complete!
$ kubeadm version
kubeadm version: &version.Info{
Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:19:40Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
3.2 アップグレード計画の確認
このコマンドは、クラスターがアップグレードできるかどうかを確認し、アップグレードできるバージョンを取得します。また、コンポーネントの構成のバージョンステータスを示す表も表示されます。
[root@kube-master01 ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.27.3
[upgrade/versions] kubeadm version: v1.27.4
[upgrade/versions] Target version: v1.27.4
[upgrade/versions] Latest version in the v1.27 series: v1.27.4
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 3 x v1.27.3 v1.27.4
Upgrade to the latest version in the v1.27 series:
COMPONENT CURRENT TARGET
kube-apiserver v1.27.3 v1.27.4
kube-controller-manager v1.27.3 v1.27.4
kube-scheduler v1.27.3 v1.27.4
kube-proxy v1.27.3 v1.27.4
CoreDNS v1.10.1 v1.10.1
etcd 3.5.7-0 3.5.7-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.27.4
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________
3.3 コントロールプレーンノードのドレイン
$ kubectl drain kube-master01 --ignore-daemonsets --delete-emptydir-data
node/kube-master01 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-q87zt
evicting pod kube-system/coredns-5d78c9869d-lz7h6
evicting pod kube-system/coredns-5d78c9869d-lwvp5
pod/coredns-5d78c9869d-lwvp5 evicted
pod/coredns-5d78c9869d-lz7h6 evicted
node/kube-master01 drained
3.4 kubeadm のアップグレード
注:
kubeadm upgradeこのノードで管理される証明書も自動的に更新されます。証明書の更新をオプトアウトするには、フラグ --certificate-renewal=false を使用できます。詳細については、『証明書管理ガイド』を参照してください。
[root@kube-master01 ~]# kubeadm upgrade apply v1.27.4
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.27.4"
[upgrade/versions] Cluster version: v1.27.3
[upgrade/versions] kubeadm version: v1.27.4
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.27.4" (timeout: 5m0s)...
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests4157307796"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-07-20-12-04-33/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-07-20-12-04-33/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-07-20-12-04-33/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1926833644/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.27.4". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
3.5 コントロールプレーンノードの遮断を解除する
$ kubectl uncordon kube-master01
node/kube-master01 uncordoned
3.6 kubelet と kubectl のアップグレード
#centos、rocky、rhel：
dnf update -y kubelet-1.27.4-0 kubectl-1.27.4-0 --disableexcludes=kubernetes
#ubuntu：
$ apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.27.4-0 kubectl=1.27.4-0 && \
apt-mark hold kubelet kubectl
出力：
[root@kube-master01 ~]# dnf update -y kubelet-1.27.4-0 kubectl-1.27.4-0 --disableexcludes=kubernetes
Last metadata expiration check: 2:44:01 ago on Thu 20 Jul 2023 09:34:52 AM CST.
Dependencies resolved.
====================================================================================================================================================================================================
Package Architecture Version Repository Size
====================================================================================================================================================================================================
Upgrading:
kubectl x86_64 1.27.4-0 kubernetes 11 M
kubelet x86_64 1.27.4-0 kubernetes 20 M
Transaction Summary
====================================================================================================================================================================================================
Upgrade 2 Packages
Total download size: 31 M
Downloading Packages:
(1/2): 28f442261f1306377aa2704f9f87117d27850ca00f5c26130080a57ccdb38c9d-kubectl-1.27.4-0.x86_64.rpm 1.3 MB/s | 11 MB 00:08
(2/2): 49e46174a716325c333a575df9c990b0e237616e7c78537580d7e14204eca1d0-kubelet-1.27.4-0.x86_64.rpm 1.9 MB/s | 20 MB 00:10
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 3.0 MB/s | 31 MB 00:10
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: kubelet-1.27.4-0.x86_64 1/1
Upgrading : kubelet-1.27.4-0.x86_64 1/4
Upgrading : kubectl-1.27.4-0.x86_64 2/4
Cleanup : kubectl-1.27.3-0.x86_64 3/4
Cleanup : kubelet-1.27.3-0.x86_64 4/4
Running scriptlet: kubelet-1.27.3-0.x86_64 4/4
Verifying : kubectl-1.27.4-0.x86_64 1/4
Verifying : kubectl-1.27.3-0.x86_64 2/4
Verifying : kubelet-1.27.4-0.x86_64 3/4
Verifying : kubelet-1.27.3-0.x86_64 4/4
Upgraded:
kubectl-1.27.4-0.x86_64 kubelet-1.27.4-0.x86_64
Complete!
kubeletを再起動します
sudo systemctl daemon-reload && sudo systemctl restart kubelet && systemctl status kubelet
注: kubeadm アップグレード プランに手動でアップグレードする必要があるコンポーネント構成が示されている場合、ユーザーは
--configコマンドライン フラグを使用して、置換構成を含む構成ファイルを指定する必要がありますkubeadm upgrade apply。これを行わないと、kubeadm upgrade applyエラーが発生して終了し、アップグレードは実行されません。
CNI プロバイダー プラグインを手動でアップグレードする:
Container Network Interface (CNI) プロバイダーには、独自のアップグレード手順がある場合があります。プラグインページをチェックしてCNI プロバイダーを見つけ、追加のアップグレード手順が必要かどうかを確認してください。
CNI プロバイダーが DaemonSet として実行されている場合、他のコントロール プレーン ノードではこの手順は必要ありません。
3.7 アップグレード計画を他のマスターノードに適用する
control-planeノードが 3 つの場合
$ kubectl drain kube-master02 --ignore-daemonsets --delete-emptydir-data
$ ssh [email protected]
$ dnf update -y kubeadm-1.27.4-0 --disableexcludes=kubernetes
$ kubeadm upgrade node experimental-control-plane
$ kubectl uncordon kube-master02
#centos、rocky、rhel：
dnf update -y kubelet-1.27.4 kubectl-1.27.4 --disableexcludes=kubernetes
#ubuntu：
$ apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.27.4-0 kubectl=1.27.4-0 && \
apt-mark hold kubelet kubectl
#重启 kubelet
sudo systemctl daemon-reload && sudo systemctl restart kubelet && systemctl status kubelet
4. 労働者
4.1 最初のワーカーノードで kubeadm をアップグレードする
$ ssh [email protected]
#ubuntu：
$ apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.27.4-00 && apt-mark hold kubeadm
#centos、rocky、rhel：
dnf list kubeadm kubelet kubectl --showduplicates | sort -r |grep 1.27.4
dnf update -y kubeadm-1.27.4-0
4.2 マスターノードにログインし、最初のワーカーノードをドレインします
$ ssh [email protected]
$ kubectl drain <node-to-drain> --ignore-daemonsets
$ kubectl drain kube-prom01 --ignore-daemonsets
出力：
[root@kube-master01 ~]# kubectl drain kube-prom01 --ignore-daemonsets
node/kube-prom01 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-rb7k2
node/kube-prom01 drained
4.3 ワーカーノードのkubelet構成をアップグレードする
$ ssh [email protected]
$ kubeadm upgrade node
或者
$ kubeadm upgrade node config --kubelet-version v1.27.4
出力：
[root@kube-prom01 ~]# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Skipping prepull. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config3537245876/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
4.4 kubelet と kubectl のアップグレード
```bash
#centos、rocky、rhel：
dnf update -y kubelet-1.27.4 kubectl-1.27.4 --disableexcludes=kubernetes
#ubuntu：
$ apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.27.4-0 kubectl=1.27.4-0 && \
apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet && systemctl status kubelet
4.5 ワーカーノードの遮断を解除する
ノードをスケジュール可能としてマークして、ノードをオンラインに戻します:
(マスター アクション)
kubectl uncordon <node-to-uncordon>
kubectl uncordon kube-prom01
注: 他のワーカー ノードのアップグレード手順は上記と同じです。
4.6 クラスターの確認
[root@kube-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-master01 Ready control-plane 43h v1.27.4
kube-node01 Ready <none> 43h v1.27.4
kube-prom01 Ready <none> 43h v1.27.4
[root@kube-master01 ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5d78c9869d-d6tjw 1/1 Running 0 46m
kube-system coredns-5d78c9869d-l6qsw 1/1 Running 0 46m
kube-system etcd-kube-master01 1/1 Running 1 (138m ago) 43h
kube-system kube-apiserver-kube-master01 1/1 Running 0 43m
kube-system kube-controller-manager-kube-master01 1/1 Running 0 43m
kube-system kube-proxy-862db 1/1 Running 0 42m
kube-system kube-proxy-gnpjp 1/1 Running 0 7m50s
kube-system kube-proxy-k7vpf 1/1 Running 0 48s
kube-system kube-scheduler-kube-master01 1/1 Running 0 42m
[root@kube-master01 ~]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{
Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:20:54Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{
Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:14:49Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
[root@kube-master01 ~]# kubelet --version
Kubernetes v1.27.4
[root@kube-master01 ~]# kubeadm version
kubeadm version: &version.Info{
Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:19:40Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
参考：