Kubernetes docking GlusterFS disk expansion practice

foreword

knowledge points

Rating: entry level
Use Heketi Topology to expand disk capacity
Use Heketi CLI to expand disk capacity

Actual server configuration (architecture 1:1 reproduces the small-scale production environment, the configuration is slightly different)

CPU name	IP	CPU	Memory	system disk	data disk	use
ks-master-0	192.168.9.91	2	4	50	100	KubeSphere/k8s-master
ks-master-1	192.168.9.92	2	4	50	100	KubeSphere/k8s-master
ks-master-2	192.168.9.93	2	4	50	100	KubeSphere/k8s-master
ks-worker-0	192.168.9.95	2	4	50	100	k8s-worker/CI
ks-worker-1	192.168.9.96	2	4	50	100	k8s-worker
ks-worker-2	192.168.9.97	2	4	50	100	k8s-worker
storage-0	192.168.9.81	2	4	50	100+50+50	ElasticSearch/GlusterFS/Ceph/Longhorn/NFS/
storage-1	192.168.9.82	2	4	50	100+50+50	ElasticSearch/GlusterFS/Ceph/Longhorn
storage-2	192.168.9.83	2	4	50	100+50+50	ElasticSearch/GlusterFS/Ceph/Longhorn
registry	192.168.9.80	2	4	50	200	Sonatype Nexus 3
total	10	20	40	500	1100+

The actual combat environment involves software version information

OS: openEuler 22.03 LTS SP2 x86_64
KubeSphere: 3.3.2
Kubernetes：v1.24.12
Containerd：1.6.4
KubeKey: v3.0.8
GlusterFS：10.0-8
Heketi: v10.4.0

Introduction

In the previous practical course, we have learned how to install and deploy GlusterFS, Heketi and Kubernetes on openEuler 22.03 LTS SP2in-tree storage driver and use the mode to connect to GlusterFS as the back-end storage of the cluster.

Today we are going to simulate a scenario that will inevitably be encountered in the production environment. After the business goes online for a period of time, the GlusterFS data disk is full. What should I do if I need to expand the capacity?

There are two solutions for the expansion of GlusterFS data volumes based on Heketi management:

Adjust the existing Topology configuration file and reload
Use Heketi CLI to directly expand capacity (simple, recommended)

Prerequisites for actual combat simulation:

On the basis of the existing GlusterFS 100G data disk, two additional 50G disks are added to simulate two data volume expansion schemes.
In order to simulate the actual combat effect, the existing 100G space is consumed by 95G in advance.

The actual combat process in this article has nothing to do with the operating system, and all related operations are applicable to Heketi + GlusterFS storage clusters deployed on other operating systems.

Insufficient Disk Space Fault Simulation

Create a new PVC

Edit pvc resource filevi pvc-test-95g.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-data-95g
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: glusterfs
  resources:
    requests:
      storage: 95Gi

Execute the create command

 kubectl apply -f pvc-test-95g.yaml

 # 执行命令不会报错，但是 pvc 状态会处于 Pending 状态

View error message

View Heketi service log error information

# 执行命令（没有独立的日志文件，日志直接输出到了 messages 中）
tail -f /var/log/messages


# 输出结果如下（只截取了完整的一段，后面一直循环输出相同的错误信息）
[root@ks-storage-0 heketi]# tail -f /var/log/messages

Aug 16 15:29:32 ks-storage-0 heketi[34102]: [heketi] INFO 2023/08/16 15:29:32 Allocating brick set #0
Aug 16 15:29:32 ks-storage-0 heketi[34102]: [heketi] INFO 2023/08/16 15:29:32 Allocating brick set #0
Aug 16 15:29:32 ks-storage-0 heketi[34102]: [heketi] INFO 2023/08/16 15:29:32 Allocating brick set #0
Aug 16 15:29:32 ks-storage-0 heketi[34102]: [heketi] INFO 2023/08/16 15:29:32 Allocating brick set #1
Aug 16 15:29:32 ks-storage-0 heketi[34102]: [heketi] INFO 2023/08/16 15:29:32 Allocating brick set #0
Aug 16 15:29:32 ks-storage-0 heketi[34102]: [heketi] INFO 2023/08/16 15:29:32 Allocating brick set #1
Aug 16 15:29:32 ks-storage-0 heketi[34102]: [heketi] INFO 2023/08/16 15:29:32 Allocating brick set #2
Aug 16 15:29:32 ks-storage-0 heketi[34102]: [heketi] INFO 2023/08/16 15:29:32 Allocating brick set #3
Aug 16 15:29:32 ks-storage-0 heketi[34102]: [heketi] ERROR 2023/08/16 15:29:32 heketi/apps/glusterfs/volume_entry_allocate.go:37:glusterfs.(*VolumeEntry).allocBricksInCluster: Minimum brick size limit reached.  Out of space.
Aug 16 15:29:32 ks-storage-0 heketi[34102]: [heketi] ERROR 2023/08/16 15:29:32 heketi/apps/glusterfs/operations_manage.go:220:glusterfs.AsyncHttpOperation: Create Volume Build Failed: No space
Aug 16 15:29:32 ks-storage-0 heketi[34102]: [negroni] 2023-08-16T15:29:32+08:00 | 500 | #011 4.508081ms | 192.168.9.81:18080 | POST /volumes

Through the above simulation demonstration, we have learned how to judge that the data volume is full when using Glusterfs as the back-end storage in the K8s cluster.

After creation, the status is Pending
Hekiti has the keyword Create Volume Build Failed: No space in the error log

When the disk space of the GlusterFS storage cluster is allocated and no new data volumes can be created, we need to add new hard disks to the storage cluster to expand the capacity of the storage cluster as O&M.

Use Heketi to expand GlusterFS data volume

Please note that in order to fully demonstrate the expansion process in this article, the output results will be fully recorded when the relevant commands are executed. The consequence of this is that this article will be slightly lengthy, so you can read it selectively when reading this article.

View existing Topology information

# 执行命令
heketi-cli topology info

# 正常的输出结果如下
[root@ks-storage-0 heketi]# heketi-cli topology info

Cluster Id: 9ad37206ce6575b5133179ba7c6e0935

    File:  true
    Block: true

    Volumes:

        Name: vol_75c90b8463d73a7fd9187a8ca22ff91f
        Size: 95
        Id: 75c90b8463d73a7fd9187a8ca22ff91f
        Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
        Mount: 192.168.9.81:vol_75c90b8463d73a7fd9187a8ca22ff91f
        Mount Options: backup-volfile-servers=192.168.9.82,192.168.9.83
        Durability Type: replicate
        Replica: 3
        Snapshot: Enabled
        Snapshot Factor: 1.00

                Bricks:
                        Id: 37006636e1fe713a395755e8d34f6f20
                        Path: /var/lib/heketi/mounts/vg_8fd529a668d5c19dfc37450b755230cd/brick_37006636e1fe713a395755e8d34f6f20/brick
                        Size (GiB): 95
                        Node: 5e99fe0cd727b8066f200bad5524c544
                        Device: 8fd529a668d5c19dfc37450b755230cd

                        Id: 3dca27f98e1c20aa092c159226ddbe4d
                        Path: /var/lib/heketi/mounts/vg_51ad0981f8fed73002f5a7f2dd0d65c5/brick_3dca27f98e1c20aa092c159226ddbe4d/brick
                        Size (GiB): 95
                        Node: 7bb26eb30c1c61456b5ae8d805c01cf1
                        Device: 51ad0981f8fed73002f5a7f2dd0d65c5

                        Id: 7ac64e137d803cccd4b9fcaaed4be8ad
                        Path: /var/lib/heketi/mounts/vg_9af38756fe916fced666fcd3de786c19/brick_7ac64e137d803cccd4b9fcaaed4be8ad/brick
                        Size (GiB): 95
                        Node: 0108350a9d13578febbfd0502f8077ff
                        Device: 9af38756fe916fced666fcd3de786c19



    Nodes:

        Node Id: 0108350a9d13578febbfd0502f8077ff
        State: online
        Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
        Zone: 1
        Management Hostnames: 192.168.9.81
        Storage Hostnames: 192.168.9.81
        Devices:
                Id:9af38756fe916fced666fcd3de786c19   State:online    Size (GiB):99      Used (GiB):95      Free (GiB):4
                        Known Paths: /dev/disk/by-path/pci-0000:01:02.0-scsi-0:0:0:1 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 /dev/sdb

                        Bricks:
                                Id:7ac64e137d803cccd4b9fcaaed4be8ad   Size (GiB):95      Path: /var/lib/heketi/mounts/vg_9af38756fe916fced666fcd3de786c19/brick_7ac64e137d803cccd4b9fcaaed4be8ad/brick

        Node Id: 5e99fe0cd727b8066f200bad5524c544
        State: online
        Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
        Zone: 1
        Management Hostnames: 192.168.9.82
        Storage Hostnames: 192.168.9.82
        Devices:
                Id:8fd529a668d5c19dfc37450b755230cd   State:online    Size (GiB):99      Used (GiB):95      Free (GiB):4
                        Known Paths: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 /dev/disk/by-path/pci-0000:01:02.0-scsi-0:0:0:1 /dev/sdb

                        Bricks:
                                Id:37006636e1fe713a395755e8d34f6f20   Size (GiB):95      Path: /var/lib/heketi/mounts/vg_8fd529a668d5c19dfc37450b755230cd/brick_37006636e1fe713a395755e8d34f6f20/brick

        Node Id: 7bb26eb30c1c61456b5ae8d805c01cf1
        State: online
        Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
        Zone: 1
        Management Hostnames: 192.168.9.83
        Storage Hostnames: 192.168.9.83
        Devices:
                Id:51ad0981f8fed73002f5a7f2dd0d65c5   State:online    Size (GiB):99      Used (GiB):95      Free (GiB):4
                        Known Paths: /dev/disk/by-path/pci-0000:01:02.0-scsi-0:0:0:1 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 /dev/sdb

                        Bricks:
                                Id:3dca27f98e1c20aa092c159226ddbe4d   Size (GiB):95      Path: /var/lib/heketi/mounts/vg_51ad0981f8fed73002f5a7f2dd0d65c5/brick_3dca27f98e1c20aa092c159226ddbe4d/brick

View existing Node information

View node node list

# 执行命令
heketi-cli node list

# 正常的输出结果如下
[root@ks-storage-0 heketi]# heketi-cli node list
Id:0108350a9d13578febbfd0502f8077ff     Cluster:9ad37206ce6575b5133179ba7c6e0935
Id:5e99fe0cd727b8066f200bad5524c544     Cluster:9ad37206ce6575b5133179ba7c6e0935
Id:7bb26eb30c1c61456b5ae8d805c01cf1     Cluster:9ad37206ce6575b5133179ba7c6e0935

View node details

Take storage-0node as an example, view Node details.

# 执行命令
heketi-cli node info xxxxxx

# 正常的输出结果如下
[root@ks-storage-0 heketi]# heketi-cli node info 0108350a9d13578febbfd0502f8077ff
Node Id: 0108350a9d13578febbfd0502f8077ff
State: online
Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
Zone: 1
Management Hostname: 192.168.9.81
Storage Hostname: 192.168.9.81
Devices:
Id:9af38756fe916fced666fcd3de786c19   Name:/dev/sdb            State:online    Size (GiB):99      Used (GiB):95      Free (GiB):4       Bricks:1

View existing VG information

Take storage-0the node as an example, check the allocated VG information (the system VG information is deleted in the output result).

# 简单查看
[root@ks-storage-0 heketi]# vgs
  VG                                  #PV #LV #SN Attr   VSize   VFree
  vg_9af38756fe916fced666fcd3de786c19   1   2   0 wz--n-  99.87g <3.92g

# 查看详细信息
[root@ks-storage-0 heketi]# vgdisplay vg_9af38756fe916fced666fcd3de786c19
  --- Volume group ---
  VG Name               vg_9af38756fe916fced666fcd3de786c19
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  187
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               99.87 GiB
  PE Size               4.00 MiB
  Total PE              25567
  Alloc PE / Size       24564 / 95.95 GiB
  Free  PE / Size       1003 / <3.92 GiB
  VG UUID               jrxfIv-Fnjq-IYF8-aubc-t2y0-zwUp-YxjkDC

View existing LV information

Take storage-0the node as an example, check the allocated LV information (the system LV information is deleted in the output result).

# 简单查看
[root@ks-storage-0 heketi]# lvs
  LV                                     VG                                  Attr       LSize   Pool                                Origin Data%  Meta%  Move Log Cpy%Sync Convert
  brick_7ac64e137d803cccd4b9fcaaed4be8ad vg_9af38756fe916fced666fcd3de786c19 Vwi-aotz--  95.00g tp_3c68ad0d0752d41ede13afdc3db9637b        0.05
  tp_3c68ad0d0752d41ede13afdc3db9637b    vg_9af38756fe916fced666fcd3de786c19 twi-aotz--  95.00g                                            0.05   3.31

# 查看详细信息
[root@ks-storage-0 heketi]# lvdisplay
  --- Logical volume ---
  LV Name                tp_3c68ad0d0752d41ede13afdc3db9637b
  VG Name                vg_9af38756fe916fced666fcd3de786c19
  LV UUID                Aho32F-tBTa-VTTp-VfwY-qRbm-WUxu-puj4kv
  LV Write Access        read/write (activated read only)
  LV Creation host, time ks-storage-0, 2023-08-16 15:21:06 +0800
  LV Pool metadata       tp_3c68ad0d0752d41ede13afdc3db9637b_tmeta
  LV Pool data           tp_3c68ad0d0752d41ede13afdc3db9637b_tdata
  LV Status              available
  # open                 0
  LV Size                95.00 GiB
  Allocated pool data    0.05%
  Allocated metadata     3.31%
  Current LE             24320
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:5

  --- Logical volume ---
  LV Path                /dev/vg_9af38756fe916fced666fcd3de786c19/brick_7ac64e137d803cccd4b9fcaaed4be8ad
  LV Name                brick_7ac64e137d803cccd4b9fcaaed4be8ad
  VG Name                vg_9af38756fe916fced666fcd3de786c19
  LV UUID                VGTOMk-d07E-XWhw-Omzz-Pc1t-WwEH-Wh0EuY
  LV Write Access        read/write
  LV Creation host, time ks-storage-0, 2023-08-16 15:21:10 +0800
  LV Pool name           tp_3c68ad0d0752d41ede13afdc3db9637b
  LV Status              available
  # open                 1
  LV Size                95.00 GiB
  Mapped size            0.05%
  Current LE             24320
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192

Note: Heketi uses LVM storage pools to create LV volumes, and two LVs are seen in all output results. brick_The one at the beginning is the actual available LV.

Adjusting the topology configuration file for the expansion plan

Prerequisites

Drive letter for capacity expansion: /dev/sdc
Expansion capacity: 50G

View existing topology configuration files

cat /etc/heketi/topology.json

{
    "clusters": [
        {
            "nodes": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "192.168.9.81"
                            ],
                            "storage": [
                                "192.168.9.81"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/sdb"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "192.168.9.82"
                            ],
                            "storage": [
                                "192.168.9.82"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/sdb"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "192.168.9.83"
                            ],
                            "storage": [
                                "192.168.9.83"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/sdb"
                    ]
                }
            ]
        }
    ]
}

Modify the Topology file

To edit an existing topology.json, vi /etc/heketi/topology.json.

Add /dev/sdc under the configuration of each node's devices , pay attention to the punctuation configuration after /dev/sdb .

The modified topology.jsonfile is as follows:

{
    "clusters": [
        {
            "nodes": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "192.168.9.81"
                            ],
                            "storage": [
                                "192.168.9.81"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/sdb",
                        "/dev/sdc"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "192.168.9.82"
                            ],
                            "storage": [
                                "192.168.9.82"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/sdb",
                        "/dev/sdc"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "192.168.9.83"
                            ],
                            "storage": [
                                "192.168.9.83"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/sdb",
                        "/dev/sdc"
                    ]
                }
            ]
        }
    ]
}

Reload Topology

# 执行命令
heketi-cli topology load --json=/etc/heketi/topology.json

# 正常的输出结果如下
[root@ks-storage-0 heketi]# heketi-cli topology load --json=/etc/heketi/topology.json
        Found node 192.168.9.81 on cluster 9ad37206ce6575b5133179ba7c6e0935
                Found device /dev/sdb
                Adding device /dev/sdc ... OK
        Found node 192.168.9.82 on cluster 9ad37206ce6575b5133179ba7c6e0935
                Found device /dev/sdb
                Adding device /dev/sdc ... OK
        Found node 192.168.9.83 on cluster 9ad37206ce6575b5133179ba7c6e0935
                Found device /dev/sdb
                Adding device /dev/sdc ... OK

View updated Topology information

# 执行命令
heketi-cli topology info

# 正常的输出结果如下
[root@ks-storage-0 heketi]# heketi-cli topology info

Cluster Id: 9ad37206ce6575b5133179ba7c6e0935

    File:  true
    Block: true

    Volumes:

        Name: vol_75c90b8463d73a7fd9187a8ca22ff91f
        Size: 95
        Id: 75c90b8463d73a7fd9187a8ca22ff91f
        Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
        Mount: 192.168.9.81:vol_75c90b8463d73a7fd9187a8ca22ff91f
        Mount Options: backup-volfile-servers=192.168.9.82,192.168.9.83
        Durability Type: replicate
        Replica: 3
        Snapshot: Enabled
        Snapshot Factor: 1.00

                Bricks:
                        Id: 37006636e1fe713a395755e8d34f6f20
                        Path: /var/lib/heketi/mounts/vg_8fd529a668d5c19dfc37450b755230cd/brick_37006636e1fe713a395755e8d34f6f20/brick
                        Size (GiB): 95
                        Node: 5e99fe0cd727b8066f200bad5524c544
                        Device: 8fd529a668d5c19dfc37450b755230cd

                        Id: 3dca27f98e1c20aa092c159226ddbe4d
                        Path: /var/lib/heketi/mounts/vg_51ad0981f8fed73002f5a7f2dd0d65c5/brick_3dca27f98e1c20aa092c159226ddbe4d/brick
                        Size (GiB): 95
                        Node: 7bb26eb30c1c61456b5ae8d805c01cf1
                        Device: 51ad0981f8fed73002f5a7f2dd0d65c5

                        Id: 7ac64e137d803cccd4b9fcaaed4be8ad
                        Path: /var/lib/heketi/mounts/vg_9af38756fe916fced666fcd3de786c19/brick_7ac64e137d803cccd4b9fcaaed4be8ad/brick
                        Size (GiB): 95
                        Node: 0108350a9d13578febbfd0502f8077ff
                        Device: 9af38756fe916fced666fcd3de786c19



    Nodes:

        Node Id: 0108350a9d13578febbfd0502f8077ff
        State: online
        Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
        Zone: 1
        Management Hostnames: 192.168.9.81
        Storage Hostnames: 192.168.9.81
        Devices:
                Id:9af38756fe916fced666fcd3de786c19   State:online    Size (GiB):99      Used (GiB):95      Free (GiB):4
                        Known Paths: /dev/disk/by-path/pci-0000:01:02.0-scsi-0:0:0:1 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 /dev/sdb

                        Bricks:
                                Id:7ac64e137d803cccd4b9fcaaed4be8ad   Size (GiB):95      Path: /var/lib/heketi/mounts/vg_9af38756fe916fced666fcd3de786c19/brick_7ac64e137d803cccd4b9fcaaed4be8ad/brick
                Id:ab5f766ddc779449db2bf45bb165fbff   State:online    Size (GiB):49      Used (GiB):0       Free (GiB):49
                        Known Paths: /dev/disk/by-path/pci-0000:01:03.0-scsi-0:0:0:2 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2 /dev/sdc

                        Bricks:

        Node Id: 5e99fe0cd727b8066f200bad5524c544
        State: online
        Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
        Zone: 1
        Management Hostnames: 192.168.9.82
        Storage Hostnames: 192.168.9.82
        Devices:
                Id:8fd529a668d5c19dfc37450b755230cd   State:online    Size (GiB):99      Used (GiB):95      Free (GiB):4
                        Known Paths: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 /dev/disk/by-path/pci-0000:01:02.0-scsi-0:0:0:1 /dev/sdb

                        Bricks:
                                Id:37006636e1fe713a395755e8d34f6f20   Size (GiB):95      Path: /var/lib/heketi/mounts/vg_8fd529a668d5c19dfc37450b755230cd/brick_37006636e1fe713a395755e8d34f6f20/brick
                Id:b648c995486b0e785f78a8b674d8b590   State:online    Size (GiB):49      Used (GiB):0       Free (GiB):49
                        Known Paths: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2 /dev/disk/by-path/pci-0000:01:03.0-scsi-0:0:0:2 /dev/sdc

                        Bricks:

        Node Id: 7bb26eb30c1c61456b5ae8d805c01cf1
        State: online
        Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
        Zone: 1
        Management Hostnames: 192.168.9.83
        Storage Hostnames: 192.168.9.83
        Devices:
                Id:51ad0981f8fed73002f5a7f2dd0d65c5   State:online    Size (GiB):99      Used (GiB):95      Free (GiB):4
                        Known Paths: /dev/disk/by-path/pci-0000:01:02.0-scsi-0:0:0:1 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 /dev/sdb

                        Bricks:
                                Id:3dca27f98e1c20aa092c159226ddbe4d   Size (GiB):95      Path: /var/lib/heketi/mounts/vg_51ad0981f8fed73002f5a7f2dd0d65c5/brick_3dca27f98e1c20aa092c159226ddbe4d/brick
                Id:9b39c4e288d4a1783d204d2033444c00   State:online    Size (GiB):49      Used (GiB):0       Free (GiB):49
                        Known Paths: /dev/disk/by-path/pci-0000:01:03.0-scsi-0:0:0:2 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2 /dev/sdc

                        Bricks:

View updated Node information

Take storage-0the node as an example, check the detailed information of the updated Node ( focus on the Devices information ).

# 执行命令
heketi-cli node info xxxxxx

# 正常的输出结果如下

[root@ks-storage-0 heketi]# heketi-cli node info 0108350a9d13578febbfd0502f8077ff
Node Id: 0108350a9d13578febbfd0502f8077ff
State: online
Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
Zone: 1
Management Hostname: 192.168.9.81
Storage Hostname: 192.168.9.81
Devices:
Id:9af38756fe916fced666fcd3de786c19   Name:/dev/sdb            State:online    Size (GiB):99      Used (GiB):95      Free (GiB):4       Bricks:1
Id:ab5f766ddc779449db2bf45bb165fbff   Name:/dev/sdc            State:online    Size (GiB):49      Used (GiB):0       Free (GiB):49      Bricks:0

View updated VG information

Take storage-0node as an example, check the updated VG information (the system VG information is deleted in the output).

[root@ks-storage-0 heketi]# vgs
  VG                                  #PV #LV #SN Attr   VSize   VFree
  vg_9af38756fe916fced666fcd3de786c19   1   2   0 wz--n-  99.87g <3.92g
  vg_ab5f766ddc779449db2bf45bb165fbff   1   0   0 wz--n-  49.87g 49.87g

Create a test PVC

On k8s-master-0the node, execute the relevant commands below.

Edit pvc resource filevi pvc-test-45g.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-data-45g
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: glusterfs
  resources:
    requests:
      storage: 45Gi

Execute the create command

 kubectl apply -f pvc-test-45g.yaml

View creation results

NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
test-data-45g   Bound    pvc-19343e73-6b14-40ca-b65b-356d38d16bb0   45Gi       RWO            glusterfs      17s   Filesystem
test-data-95g   Bound    pvc-2461f639-1634-4085-af2f-b526a3800217   95Gi       RWO            glusterfs      42h   Filesystem

View the newly created Volume

view volume list

[root@ks-storage-0 heketi]# heketi-cli volume list
Id:75c90b8463d73a7fd9187a8ca22ff91f    Cluster:9ad37206ce6575b5133179ba7c6e0935    Name:vol_75c90b8463d73a7fd9187a8ca22ff91f
Id:ebd76f343b04f89ed4166c8f1ece0361    Cluster:9ad37206ce6575b5133179ba7c6e0935    Name:vol_ebd76f343b04f89ed4166c8f1ece0361

View information about the newly created volume

[root@ks-storage-0 heketi]# heketi-cli volume info ebd76f343b04f89ed4166c8f1ece0361
Name: vol_ebd76f343b04f89ed4166c8f1ece0361
Size: 45
Volume Id: ebd76f343b04f89ed4166c8f1ece0361
Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
Mount: 192.168.9.81:vol_ebd76f343b04f89ed4166c8f1ece0361
Mount Options: backup-volfile-servers=192.168.9.82,192.168.9.83
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distribute Count: 1
Replica Count: 3
Snapshot Factor: 1.00

View newly created LV information

Take storage-0the node as an example, check the newly allocated LV information (the system LV information is deleted in the output result).

[root@ks-storage-0 heketi]# lvs
  LV                                     VG                                  Attr       LSize   Pool                                Origin Data%  Meta%  Move Log Cpy%Sync Convert
  brick_7ac64e137d803cccd4b9fcaaed4be8ad vg_9af38756fe916fced666fcd3de786c19 Vwi-aotz--  95.00g tp_3c68ad0d0752d41ede13afdc3db9637b        0.05
  tp_3c68ad0d0752d41ede13afdc3db9637b    vg_9af38756fe916fced666fcd3de786c19 twi-aotz--  95.00g                                            0.05   3.31
  brick_27e193590ccdb5fba287fb66d5473074 vg_ab5f766ddc779449db2bf45bb165fbff Vwi-aotz--  45.00g tp_7bdcf1e2c3aab06cb25906f017ae1b08        0.06
  tp_7bdcf1e2c3aab06cb25906f017ae1b08    vg_ab5f766ddc779449db2bf45bb165fbff twi-aotz--  45.00g                                            0.06   6.94

So far, we have demonstrated the whole process of Heketi expanding the disk capacity and verifying the test through the Topology configuration file.

Heketi-CLI Direct Expansion of Capacity Expansion Solution

Prerequisites

Drive letter for capacity expansion: /dev/sdd
Expansion capacity: 50G

View Node information

View the Node list and get the Node ID

[root@ks-storage-0 heketi]# heketi-cli node list
Id:0108350a9d13578febbfd0502f8077ff     Cluster:9ad37206ce6575b5133179ba7c6e0935
Id:5e99fe0cd727b8066f200bad5524c544     Cluster:9ad37206ce6575b5133179ba7c6e0935
Id:7bb26eb30c1c61456b5ae8d805c01cf1     Cluster:9ad37206ce6575b5133179ba7c6e0935

View the detailed information of the Node and the information of the existing Devices (take storage-0 as an example).

[root@ks-storage-0 heketi]# heketi-cli node info 0108350a9d13578febbfd0502f8077ff
Node Id: 0108350a9d13578febbfd0502f8077ff
State: online
Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
Zone: 1
Management Hostname: 192.168.9.81
Storage Hostname: 192.168.9.81
Devices:
Id:9af38756fe916fced666fcd3de786c19   Name:/dev/sdb            State:online    Size (GiB):99      Used (GiB):95      Free (GiB):4       Bricks:1
Id:ab5f766ddc779449db2bf45bb165fbff   Name:/dev/sdc            State:online    Size (GiB):49      Used (GiB):45      Free (GiB):4       Bricks:1

Add a new Device

The newly added disk is displayed as /dev/sdd in the system , and each Node needs to execute the command to add Device.

# 执行的命令
heketi-cli device add --name /dev/sdd --node xxxxxx

# 实际的输出结果如下
[root@ks-storage-0 heketi]# heketi-cli device add --name /dev/sdd --node 0108350a9d13578febbfd0502f8077ff
Device added successfully

[root@ks-storage-0 heketi]# heketi-cli device add --name /dev/sdd --node 5e99fe0cd727b8066f200bad5524c544
Device added successfully

[root@ks-storage-0 heketi]# heketi-cli device add --name /dev/sdd --node 7bb26eb30c1c61456b5ae8d805c01cf1
Device added successfully

View updated Node information

Take storage-0the node as an example, check the updated Node information ( focus on the Devices information ).

[root@ks-storage-0 heketi]# heketi-cli node info 0108350a9d13578febbfd0502f8077ff
Node Id: 0108350a9d13578febbfd0502f8077ff
State: online
Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
Zone: 1
Management Hostname: 192.168.9.81
Storage Hostname: 192.168.9.81
Devices:
Id:9af38756fe916fced666fcd3de786c19   Name:/dev/sdb            State:online    Size (GiB):99      Used (GiB):95      Free (GiB):4       Bricks:1
Id:ab5f766ddc779449db2bf45bb165fbff   Name:/dev/sdc            State:online    Size (GiB):49      Used (GiB):45      Free (GiB):4       Bricks:1
Id:c189451c573814e05ebd83d46ab9a0af   Name:/dev/sdd            State:online    Size (GiB):49      Used (GiB):0       Free (GiB):49      Bricks:0

View updated Topology information

[root@ks-storage-0 heketi]# heketi-cli topology info

Cluster Id: 9ad37206ce6575b5133179ba7c6e0935

    File:  true
    Block: true

    Volumes:

        Name: vol_75c90b8463d73a7fd9187a8ca22ff91f
        Size: 95
        Id: 75c90b8463d73a7fd9187a8ca22ff91f
        Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
        Mount: 192.168.9.81:vol_75c90b8463d73a7fd9187a8ca22ff91f
        Mount Options: backup-volfile-servers=192.168.9.82,192.168.9.83
        Durability Type: replicate
        Replica: 3
        Snapshot: Enabled
        Snapshot Factor: 1.00

                Bricks:
                        Id: 37006636e1fe713a395755e8d34f6f20
                        Path: /var/lib/heketi/mounts/vg_8fd529a668d5c19dfc37450b755230cd/brick_37006636e1fe713a395755e8d34f6f20/brick
                        Size (GiB): 95
                        Node: 5e99fe0cd727b8066f200bad5524c544
                        Device: 8fd529a668d5c19dfc37450b755230cd

                        Id: 3dca27f98e1c20aa092c159226ddbe4d
                        Path: /var/lib/heketi/mounts/vg_51ad0981f8fed73002f5a7f2dd0d65c5/brick_3dca27f98e1c20aa092c159226ddbe4d/brick
                        Size (GiB): 95
                        Node: 7bb26eb30c1c61456b5ae8d805c01cf1
                        Device: 51ad0981f8fed73002f5a7f2dd0d65c5

                        Id: 7ac64e137d803cccd4b9fcaaed4be8ad
                        Path: /var/lib/heketi/mounts/vg_9af38756fe916fced666fcd3de786c19/brick_7ac64e137d803cccd4b9fcaaed4be8ad/brick
                        Size (GiB): 95
                        Node: 0108350a9d13578febbfd0502f8077ff
                        Device: 9af38756fe916fced666fcd3de786c19


        Name: vol_ebd76f343b04f89ed4166c8f1ece0361
        Size: 45
        Id: ebd76f343b04f89ed4166c8f1ece0361
        Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
        Mount: 192.168.9.81:vol_ebd76f343b04f89ed4166c8f1ece0361
        Mount Options: backup-volfile-servers=192.168.9.82,192.168.9.83
        Durability Type: replicate
        Replica: 3
        Snapshot: Enabled
        Snapshot Factor: 1.00

                Bricks:
                        Id: 27e193590ccdb5fba287fb66d5473074
                        Path: /var/lib/heketi/mounts/vg_ab5f766ddc779449db2bf45bb165fbff/brick_27e193590ccdb5fba287fb66d5473074/brick
                        Size (GiB): 45
                        Node: 0108350a9d13578febbfd0502f8077ff
                        Device: ab5f766ddc779449db2bf45bb165fbff

                        Id: 4fab639b551e573c61141508d75bf605
                        Path: /var/lib/heketi/mounts/vg_9b39c4e288d4a1783d204d2033444c00/brick_4fab639b551e573c61141508d75bf605/brick
                        Size (GiB): 45
                        Node: 7bb26eb30c1c61456b5ae8d805c01cf1
                        Device: 9b39c4e288d4a1783d204d2033444c00

                        Id: 8eba3fb2253452999a1ec60f647dcf03
                        Path: /var/lib/heketi/mounts/vg_b648c995486b0e785f78a8b674d8b590/brick_8eba3fb2253452999a1ec60f647dcf03/brick
                        Size (GiB): 45
                        Node: 5e99fe0cd727b8066f200bad5524c544
                        Device: b648c995486b0e785f78a8b674d8b590



    Nodes:

        Node Id: 0108350a9d13578febbfd0502f8077ff
        State: online
        Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
        Zone: 1
        Management Hostnames: 192.168.9.81
        Storage Hostnames: 192.168.9.81
        Devices:
                Id:9af38756fe916fced666fcd3de786c19   State:online    Size (GiB):99      Used (GiB):95      Free (GiB):4
                        Known Paths: /dev/disk/by-path/pci-0000:01:02.0-scsi-0:0:0:1 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 /dev/sdb

                        Bricks:
                                Id:7ac64e137d803cccd4b9fcaaed4be8ad   Size (GiB):95      Path: /var/lib/heketi/mounts/vg_9af38756fe916fced666fcd3de786c19/brick_7ac64e137d803cccd4b9fcaaed4be8ad/brick
                Id:ab5f766ddc779449db2bf45bb165fbff   State:online    Size (GiB):49      Used (GiB):45      Free (GiB):4
                        Known Paths: /dev/disk/by-path/pci-0000:01:03.0-scsi-0:0:0:2 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2 /dev/sdc

                        Bricks:
                                Id:27e193590ccdb5fba287fb66d5473074   Size (GiB):45      Path: /var/lib/heketi/mounts/vg_ab5f766ddc779449db2bf45bb165fbff/brick_27e193590ccdb5fba287fb66d5473074/brick
                Id:c189451c573814e05ebd83d46ab9a0af   State:online    Size (GiB):49      Used (GiB):0       Free (GiB):49
                        Known Paths: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi3 /dev/disk/by-path/pci-0000:01:04.0-scsi-0:0:0:3 /dev/sdd

                        Bricks:

        Node Id: 5e99fe0cd727b8066f200bad5524c544
        State: online
        Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
        Zone: 1
        Management Hostnames: 192.168.9.82
        Storage Hostnames: 192.168.9.82
        Devices:
                Id:5cd245e9826c0bfa46bef0c0d41ed0ed   State:online    Size (GiB):49      Used (GiB):0       Free (GiB):49
                        Known Paths: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi3 /dev/disk/by-path/pci-0000:01:04.0-scsi-0:0:0:3 /dev/sdd

                        Bricks:
                Id:8fd529a668d5c19dfc37450b755230cd   State:online    Size (GiB):99      Used (GiB):95      Free (GiB):4
                        Known Paths: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 /dev/disk/by-path/pci-0000:01:02.0-scsi-0:0:0:1 /dev/sdb

                        Bricks:
                                Id:37006636e1fe713a395755e8d34f6f20   Size (GiB):95      Path: /var/lib/heketi/mounts/vg_8fd529a668d5c19dfc37450b755230cd/brick_37006636e1fe713a395755e8d34f6f20/brick
                Id:b648c995486b0e785f78a8b674d8b590   State:online    Size (GiB):49      Used (GiB):45      Free (GiB):4
                        Known Paths: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2 /dev/disk/by-path/pci-0000:01:03.0-scsi-0:0:0:2 /dev/sdc

                        Bricks:
                                Id:8eba3fb2253452999a1ec60f647dcf03   Size (GiB):45      Path: /var/lib/heketi/mounts/vg_b648c995486b0e785f78a8b674d8b590/brick_8eba3fb2253452999a1ec60f647dcf03/brick

        Node Id: 7bb26eb30c1c61456b5ae8d805c01cf1
        State: online
        Cluster Id: 9ad37206ce6575b5133179ba7c6e0935
        Zone: 1
        Management Hostnames: 192.168.9.83
        Storage Hostnames: 192.168.9.83
        Devices:
                Id:51ad0981f8fed73002f5a7f2dd0d65c5   State:online    Size (GiB):99      Used (GiB):95      Free (GiB):4
                        Known Paths: /dev/disk/by-path/pci-0000:01:02.0-scsi-0:0:0:1 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 /dev/sdb

                        Bricks:
                                Id:3dca27f98e1c20aa092c159226ddbe4d   Size (GiB):95      Path: /var/lib/heketi/mounts/vg_51ad0981f8fed73002f5a7f2dd0d65c5/brick_3dca27f98e1c20aa092c159226ddbe4d/brick
                Id:6656246eafefffaea49399444989eab1   State:online    Size (GiB):49      Used (GiB):0       Free (GiB):49
                        Known Paths: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi3 /dev/disk/by-path/pci-0000:01:04.0-scsi-0:0:0:3 /dev/sdd

                        Bricks:
                Id:9b39c4e288d4a1783d204d2033444c00   State:online    Size (GiB):49      Used (GiB):45      Free (GiB):4
                        Known Paths: /dev/disk/by-path/pci-0000:01:03.0-scsi-0:0:0:2 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2 /dev/sdc

                        Bricks:
                                Id:4fab639b551e573c61141508d75bf605   Size (GiB):45      Path: /var/lib/heketi/mounts/vg_9b39c4e288d4a1783d204d2033444c00/brick_4fab639b551e573c61141508d75bf605/brick

Note: Focus on viewing Devices related information.

View updated VG information

Take storage-0node as an example, check the updated VG information (the system VG information is deleted in the output).

[root@ks-storage-0 heketi]# vgs
  VG                                  #PV #LV #SN Attr   VSize   VFree
  openeuler                             1   2   0 wz--n- <19.00g     0
  vg_9af38756fe916fced666fcd3de786c19   1   2   0 wz--n-  99.87g <3.92g
  vg_ab5f766ddc779449db2bf45bb165fbff   1   2   0 wz--n-  49.87g <4.42g
  vg_c189451c573814e05ebd83d46ab9a0af   1   0   0 wz--n-  49.87g 49.87g

In order to save space, the process of creating PVC verification and viewing is omitted here. Readers can refer to the previous operation to verify the test by themselves.

So far, we have demonstrated the whole process of disk expansion and verification testing through Heketi-CLI.

common problem

Question 1

error message

[root@ks-master-0 k8s-yaml]# kubectl apply -f pvc-test-10g.yaml
The PersistentVolumeClaim "test-data-10G" is invalid: metadata.name: Invalid value: "test-data-10G": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9] "a-z0-9")?(\.[a-z0-9]([-a-z0-9]*[a-z0-9] "a-z0-9")?)*')

solution

When creating pvc, the metadata.name defined in the yaml file uses uppercase letters test-data-10G, just change it to lowercase test-data-10g .

question 2

error message

The PersistentVolumeClaim "test-data-10g" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value

solution

This is my own operation error. I created a PVC named test-data-10g before , and then based on the original configuration file, I changed the storage value to a smaller value before executing the creation action, which caused the above error.

question 3

error message

[root@ks-storage-0 heketi]# heketi-cli topology load --json=/etc/heketi/topology.json
        Found node 192.168.9.81 on cluster 9ad37206ce6575b5133179ba7c6e0935
                Found device /dev/sdb
                Adding device /dev/sdc ... Unable to add device: Initializing device /dev/sdc failed (already initialized or contains data?):   No device found for /dev/sdc.
        Found node 192.168.9.82 on cluster 9ad37206ce6575b5133179ba7c6e0935
                Found device /dev/sdb
                Adding device /dev/sdc ... Unable to add device: Initializing device /dev/sdc failed (already initialized or contains data?):   No device found for /dev/sdc.
        Found node 192.168.9.83 on cluster 9ad37206ce6575b5133179ba7c6e0935
                Found device /dev/sdb
                Adding device /dev/sdc ... Unable to add device: Initializing device /dev/sdc failed (already initialized or contains data?):   No device found for /dev/sdc.

solution

This is my own operation error, and I have executed the reload command before adding the disk /dev/sdc .

Summarize

This article introduces the GlusterFS storage cluster managed by Hekiti in detail. When the data disk space allocation is full and the data volume cannot be created, the operation and maintenance personnel should add a new physical disk and add two solutions to the existing storage cluster. .

Adjusting the topology configuration file for the expansion plan
Heketi-CLI Direct Expansion of Capacity Expansion Solution

This article comes from a real case in the production environment, and all operations have been verified in practice. However, data is priceless and capacity expansion is risky, so operations need to be cautious.

This article is published by OpenWrite, a multi-post platform for blogging !