k8s cluster master node migration docker default path

When installing the k8s cluster, the data directory of docker was not changed, and the default path /var/lib/docker was used

The resulting problem is that as the use of mirroring increases, the use of /var/lib/docker becomes larger and larger. By default, the disk space under the /var path is very small, and it is usually necessary to mount a new data directory/ data, the disk is larger.

Adjust the data directory of docker.

echo "停止之前 检查docker状态"
sudo systemctl status docker.service | grep Active
echo "stop docker"
sudo systemctl stop docker.service
echo "停止之前，检查kubelet状态"
sudo systemctl status kubelet | grep Active
sudo systemctl stop kubelet
echo "都停止完后，检查状态"
sudo systemctl status docker.service | grep Active
sudo systemctl status kubelet | grep Active

The result of the final execution

 # sudo systemctl status kubelet | grep Active
   Active: inactive (dead) since Fri 2023-03-24 14:56:21 CST; 61ms ago
#  sudo systemctl status kubelet | grep Active
   Active: inactive (dead) since Fri 2023-03-24 14:56:21 CST; 28s ago

Manually move the original docker directory, so as to keep the existing image file, otherwise, you have to find the net image by yourself

# sudo mv docker /data/
mv: 无法删除 'docker/containers/02e3da8176b78187965bc0f309975b082eaaf2c4518027de1f41b7787f0bb18d/mounts/shm': 设备或资源忙

Modify the /etc/docker/daemon.json file and add the data path data-root

# cat /etc/docker/daemon.json
{
  "exec-opts": [
    "native.cgroupdriver=systemd"
  ],
  "insecure-registries": [
    "10.221.236.232:5000",
    "0.0.0.0/0",
    "127.0.0.1"
  ],
  "data-root": "/data/docker",
  "log-opts": {
    "max-file": "3",
    "max-size": "5m"
  },
  "registry-mirrors": [
    "https://mirrors.tuna.tsinghua.edu.cn",
    "https://registry.docker-cn.com"
  ]
}

Restart docker, kubelet

# sudo systemctl daemon-reload 
# sudo systemctl restart docker.service
# sudo systemctl restart docker.service 
# sudo systemctl status docker.service         
● docker.service - Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-03-24 15:04:41 CST; 19s ago
     Docs: https://docs.docker.com
 Main PID: 18682 (dockerd)
    Tasks: 238
   Memory: 230.2M
   CGroup: /system.slice/docker.service
           ├─18682 /usr/bin/dockerd

# sudo systemctl status kubelet.service        
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Fri 2023-03-24 15:04:52 CST; 15s ago
     Docs: http://kubernetes.io/docs/
 Main PID: 18950 (kubelet)
    Tasks: 16
   Memory: 118.6M
   CGroup: /system.slice/kubelet.service
           └─18950 /usr/local/bin/kubelet --bootstrap-kubeconfig=

df -h View, the docker directory has moved to the new directory

 

On another master host, check the pod status, and the new pods are also normal.

 also encountered problems

failed to listen on 0.0.0.0:6443: listen tcp 0.0.0.0:6443: bind: address alr

 How can it be so complicated, kill the original process and restart it, it will be normal.

$ netstat -lutpn | grep 6443
tcp6       0      0 :::6443                 :::*                    LISTEN      11395/some-service

$ kill 11395

$ service kubelet restart