2.3. Btrfs file system installation method
2.3.1. About Btrfs
Btrfs is a copy-on-write (COW, copy on write) file system. Through the verification of data and metadata, snapshots, built-in RAID and self-healing functions can be realized. Starting from Proxmox VE 7.0, Proxmox VE introduced the Btrfs file system as an optional file system and root file system.
According to the "pve-admin-guide-7" user manual officially provided by Proxmox VE (see the chapter on storage types on page 112 for details), the storage types supported by Proxmox VE 7.0 are shown in Table 1.
| options | PVE type | storage type | whether to share | Whether snapshot | Is it stable |
|---|---|---|---|---|---|
| Table of contents | dir | document | no | No 1 | yes |
| LVM | lvm | piece | No 2 | no | yes |
| LVM-thin | lvmthin | piece | no | yes | yes |
| BTRFS | brtfs | document | no | yes | technology preview |
| NFS | nfs | document | yes | No 1 | yes |
| CIFS | cifs | document | yes | No 1 | yes |
| GlusterFS | glusterfs | document | yes | No 1 | yes |
| iSCSI/kernel | iscsi | piece | yes | no | yes |
| iSCSI/libiscsi | iscsidirect | piece | yes | no | yes |
| CephFS | cephfs | document | yes | yes | yes |
| Ceph/RBD | rbd | piece | yes | yes | yes |
| ZFS over iSCSI | zfs | piece | yes | yes | yes |
| ZFS(local) | zfspool | document | no | yes | yes |
| Proxmox Backup | pbs | file, block | yes | not applicable | yes |
1: On file system-based storage, snapshots can be implemented by using qcow2 format virtual disks.
2: LVM can be configured on iSCSI storage to obtain shared LVM storage.
Table 1 officially provided by Proxmox VE shows that Btrfs is not yet able to achieve the stable performance required by the production level. Proxmox VE officially recommends "technical preview" in terms of "whether it is stable", which is actually equivalent to saying that Btrfs is a test version or a trial version.
Compared with the ext4 and xfs file systems, the Btrfs file system can provide better performance (Btrfs does not use logs). One of the main advantages of using Btrfs is that it supports copy-on-write and writable snapshots, which is useful for virtual machine deployment and Cloning is very useful. In the file system, it also supports transparent compression, universal checksum and unified management of multiple devices. Btrfs also supports efficient XATTR, merging of small files, integrated volume management known on SSDs, and support for online fsck features. However, despite so many advantages and features, Btrfs is not ready for application and production systems so far, but for test scenarios, Btrfs is a good choice.
The advantages of Btrfs are as follows:
- The setup of Btrfs is almost the same as the traditional ext4 setup;
- snapshots and clones;
- Data compression at the file system level;
- copy-on-write;
- RAID0, RAID1 and RAID10；
- prevent data corruption;
- self-healing;
- Linux native support;
The warning of Btrfs is as follows:
- The RAID5 and RAID6 codes of Btrfs have security risks, and it is strongly recommended not to use RAID5 and RAID6 in Btrfs.
2.3.2. Btrfs installation
Step 1: Select btrfs(RAID1)
When installing with the Proxmox VE 7.0 installer, select Btrfs as the root file system, and also need to select the RAID level, as shown in Figure 1.
In terms of supporting soft RAID features, Btrfs is similar to ZFS, and also provides multiple levels of soft RAID. Similarly, it does not support any hard RAID, because this may cause data loss.
Compared with ZFS in terms of soft RAID levels, Btrfs lacks the RAID-Z function. The main reason is that RAID-Z is a variant mode of RAID5, and the RAID5 and RAID6 codes in Btrfs have security risks. Proxmox VE will not support the RAID-Z function until the security risks of RAID5 and RAID6 codes are resolved.
Select btrfs (RAID1), RAID1 requires at least two target hard disks (about the characteristics of RAID1, you can search for relevant content through search engines to understand), that is to say, the target hard disk in Figure 1 is a group of RAID1 hard disks, and the used space is only 1 hard disk, used as a boot disk, and another hard disk used as a backup disk.
In Proxmox VE 7.0, there are 3 types of soft RAID levels supported by the Btrfs file system, as shown in Figure 2.
btrfs(RAID0) : Also known as "striped" mode. In this mode, the capacity of the Btrfs volume is the sum of the capacity of all hard disks, but RAID0 does not provide any redundancy, and the failure of any hard disk in the Btrfs volume will cause the entire volume to be unavailable. In this mode, at least one hard disk is required.
btrfs(RAID1) : Also known as "mirror" mode. In this mode, data will be copied to all hard disks at the same time. In this mode, at least two hard disks with the same capacity are required, and the capacity of the entire volume is equal to the capacity of a single hard disk.
btrfs(RAID10) : This mode is a combination of RAID0 and RAID1. In this mode, at least 4 hard drives are required.
2.3.3. View Btrfs disk partition parameters
After the installation of Proxmox VE 7.0 is complete, let's take a look at the disk partitions of the Proxmox VE host, as shown in Figure 3, Figure 4 and Figure 6.
There are two system disks, one is the boot disk and the other is the backup disk, because we selected the RAID1 level for installation during installation. According to the RAID1 mechanism, the data on the backup disk is exactly the same as the data on the boot disk.
From Figure 4 and Figure 5, we can see that there are only hard disk devices and partition information actually stored like /dev/sda, and there are no logical devices like /dev/mapper.
Btrfs feature 1: Disable the default local storage local to support the btrfs feature of the local-btrfs subvolume
After Proxmox VE 7.0 is installed, the default local storage local will be disabled to support the btrfs feature of the local-btrfs subvolume, which is different from ext4, xfs and zfs. Another difference is that local-btrfs supports more types of storage formats in terms of storing content, and at the same time directly establishes storage paths in the WEB UI, as shown in Figure 6.
We can also view local and local-btrfs information through the storage configuration file of Proxmox VE 7.0, as shown in Figure 7.
Btrfs feature 2: WEB UI interface cannot directly create Btrfs storage, only through CLI commands
In the WEB UI interface of Proxmox VE 7.0, there is no Btrfs configuration function, and Btrfs storage cannot be created directly through the WEB UI interface, and can only be created through the background CLI command.
This may be because Proxmox feels that Btrfs is not stable and cannot be used in production, so Proxmox VE 7.0 does not add Btrfs configuration function in WEB UI. It is believed that as Btrfs becomes stable (it seems very far away), Proxmox VE will add the configuration function of creating Btrfs in the WEB UI interface, as shown in Figure 8.
In the WEB UI interface, LVM and LVM-Thin support the creation of ext4 and xfs storage, directories support the creation of ext4 and xfs storage, ZFS supports the creation of ZFS storage, but do not support the Btrfs configuration function.
2.2.3. Default storage location
In the Btrfs installation mode, Proxmox VE uses Btrfs as the storage method. After the target hard disk is divided into BIOS boot and EFI partitions during installation, a Btrfs storage pool named local-btrfs is automatically created in the third partition /dev/sda3. In local-btrfs, disk space can be divided to store virtual machines and container files, and storage paths can be created to store ISO images and container templates, etc. It seems that local-btrfs has "physical devices" and "directories". dual characteristics. Let's upload an ISO image, create a virtual machine, upload a container template, create a container, and see if that's the case.
● ISO image storage path: /var/lib/pve/local-btrfs/template/iso
In local-btrfs, select "ISO Image → Upload", and then upload an ISO image of ubuntu from the local computer, as shown in Figure 9.
After the ISO image is uploaded, the ISO image of ubuntu will appear on the "local-btrfs→ISO image" interface, as shown in Figure 10.
The storage path of the ISO image is located at /var/lib/pve/local-btrfs/template/iso, as shown in Figure 11.
● Virtual machine file storage path: /var/lib/pve/local-btrfs/images
Create a virtual machine in local-btrfs, select the "Create Virtual Machine" button, and enter the virtual machine information, as shown in Figure 12.
After the virtual machine is successfully created, the virtual machine file will appear in "local-btrfs→VM Disk", as shown in Figure 13.
When creating a virtual machine, the disk space is allocated from the local-btrfs storage pool to the hard disk of the virtual machine. That is to say, local-btrfs is equivalent to the disk pool of the virtual machine, and the hard disk of the virtual machine is equivalent to the block storage of local-btrfs.
The storage path of virtual machine files is /var/lib/local-btrfs/images, as shown in Figure 14.
● Container template storage path: /var/lib/pve/local-btrfs/template/cache
Method 1: Download the container template in the Proxmox VE interface
The advantage of using the Proxmox VE interface to download the container template is that it is convenient and simple, but the disadvantage is that the download is slow, and the container template can only be stored in the Proxmox VE host and cannot be migrated to other Proxmox VE hosts for reuse.
In the Proxmox VE 7.0 interface, select "local-btrfs→CT Template→Template", and select the container template to be downloaded in the pop-up template interface. Note that only one container template can be selected for download at a time, and multiple containers can be downloaded at the same time, as shown in Figure 15.
During the download process, the download speed is relatively slow, especially when a container template is downloaded at the same time. In the Proxmox VE 7.0 interface, the download progress cannot be seen. You can check the download progress in the storage path of the container template, but it is not very useful large, as shown in Figure 16.
After the container template is downloaded, the "Task" bar at the bottom of Proxmox VE 7.0 will prompt that the download is complete, but the download progress cannot be seen, as shown in Figure 17.
After downloading, in the CT template interface of Proxmox VE 7.0, the container template will automatically appear in the container template interface, as shown in Figure 18.
Method 2: Use Thunder to download the container template to the local, and upload the container template in the Proxmox VE interface
Using the Proxmox VE interface to directly download the container template, in addition to the advantage of convenience, there are too many disadvantages, especially the slow download and non-reusable use. It is recommended to directly use Thunder to download to the local computer to save, and then upload the container template through the Proxmox VE interface .
On the Proxmox Chinese community, address https://www.proxmox.wiki, you can directly download the container template, as shown in Figure 19.
In the container template download interface, select the container template you want, and then click Download, as shown in Figure 20. Use Xunlei or browser to download, the speed is very fast.
This is the container template downloaded to the local computer. The file format of the Proxmox container template is basically tar.gz and tar.xz, as shown in Figure 21.
Next, we upload the container template of the local computer to Proxmox VE 7.0, as shown in Figure 22. Note that in the Proxmox VE 7.0 interface, container templates cannot be uploaded in batches, and only one container template can be uploaded at a time.
After the container template is uploaded, the container template we just uploaded will appear in the container template interface, as shown in Figure 23.
The storage path of the container template is /var/lib/pve/local-btrfs/template/cache, as shown in Figure 24.
● Container file storage path: /var/lib/pve/local-btrfs/images
Step 1: Install the container
Create a container in local-btrfs, select the "Create CT" button, and enter the container information, as shown in Figure 25.
选择local-btrfs存储，选择您要安装的容器模板，然后点击“下一步”，继续按需填入容器的配置信息，如图26所示。
容器创建成功之后，在最底部会出现“TASK OK”字样，如图27所示。
容器创建过程中，会输出容器配置信息，如下所示：
Creating filesystem with 2097152 4k blocks and 524288 inodes
Filesystem UUID: 28b88c1f-a123-4867-91f9-b58861e23b43
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Use of uninitialized value $format in string eq at /usr/share/perl5/PVE/Storage/BTRFSPlugin.pm line 183.
Use of uninitialized value $format in string eq at /usr/share/perl5/PVE/Storage/BTRFSPlugin.pm line 189.
Use of uninitialized value $format in string eq at /usr/share/perl5/PVE/Storage/BTRFSPlugin.pm line 183.
Use of uninitialized value $format in string eq at /usr/share/perl5/PVE/Storage/BTRFSPlugin.pm line 189.
extracting archive '/var/lib/pve/local-btrfs/template/cache/debian-10-standard_10.7-1_amd64.tar.gz'
Total bytes read: 671672320 (641MiB, 4.9MiB/s)
Detected container architecture: amd64
Creating SSH host key 'ssh_host_dsa_key' - this may take some time ...
done: SHA256:dTHd/QKq1a0vd67wJkVjwMqkc+c7JsmTE+3KEw2Fr10 root@localhost
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
done: SHA256:gDTMIBUpIAUxpiEd8GhJ02VZMhCMBVVnwVhmkwkaEmw root@localhost
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
done: SHA256:JiLCYaJFnAJ99vkp+fdLlHitKNFbRDC0u+8fK41SVGE root@localhost
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
done: SHA256:xWhlh6tStLeaSC1ikxLIX4VIU45kJUEcReimy7uuG8E root@localhost
TASK OK
第二步：启动容器
选择刚才创建的容器，点击“启动”按钮，启动容器，容器启动成功之后，在控制台中，会出现登录入口，如图28所示。
输入默认用户root，输入在创建容器是设置的密码，然后就可以登录容器了，如图29所示。
第三步：容器文件存放路径
容器文件格式与虚拟机文件格式相同，都是以raw格式保存，如图30所示。
容器文件的存放路径与虚拟机文件的存放路径相同，都是/var/lib/pve/local-btrfs/images，如图31所示。
2.3.4. 新建Btrfs存储池
在默认情况下，备份、虚拟机和容器文件、ISO镜像、容器模板都存储在local-brtfs中，根据图4的默认分区，local-brtfs是创建在系统盘的第三分区/dev/sda3上的。把备份、虚拟机和容器文件、ISO镜像、容器模板等都保存在系统盘中的local-brtfs上，这样的规划是不合理：
- 系统盘的空间很快会被不断增长的虚拟机文件和容器文件占满空间，从而影响Proxmox VE系统的稳定性；
- 备份往往会占用比较大的空间，把备份存放到系统盘也不合理；
- 系统盘发生故障后，或者损坏之后，Proxmox VE系统无法启动，导致系统盘的虚拟机文件和容器文件无法提取出来，造成业务数据丢失。
除非是在演示场景或者测试场景，否则备份、虚拟机和容器文件不要保存在系统盘中。在生产场景中，强烈建议把备份、虚拟机和容器文件保存在非系统盘中。
第一步：查看Proxmox VE 7.0主机中有哪些未使用的业务盘
在创建新的Btrfs存储池之前，我们需要知道有哪些没有使用的硬盘，然后将这些硬盘汇聚起来，创建一个Btrfs存储池，如图32所示。
第二步：创建Btrfs之前，擦除硬盘数据和分区
在创建Btrfs之前，硬盘如果有数据或者其他分区，一定要先擦除掉。在Proxmox VE 7.0界面中，选择“Wipe Disk”按钮可以对硬盘进行逐个擦除，这里不能批量擦除。注意，一旦对硬盘进行了擦除，硬盘所有数据和分区都会被擦除掉。通过图32可以看到，4个未使用的硬盘并没有数据，也没有被GPT初始化（GPT初始化会产生分区），可以直接在这四个硬盘上创建Btrfs。
第三步：创建Btrfs文件系统
使用mkfs.btrfs命令创建Btrfs文件系统，-d和-m参数用于分别设置元数据和数据的配置文件，使用可选的-L参数，可以设置磁盘标签。Proxmox VE 7.0中，Btrfs支持以下模式：single、raid0、raid1、 raid10。
3.1）在带有标签 My-Storage的单个磁盘/dev/sdc上创建Btrfs文件系统：
root@pve-btrfs:~# mkfs.btrfs -m single -d single -L My-Storage /dev/sdc
btrfs-progs v5.10.1
See http://btrfs.wiki.kernel.org for more information.
Label: My-Storage #磁盘标签
UUID: 8866c15a-efa3-4396-836b-8630aaec482f
Node size: 16384
Sector size: 4096
Filesystem size: 20.00GiB
Block group profiles:
Data: single 8.00MiB
Metadata: single 8.00MiB
System: single 4.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Runtime features:
Checksum: crc32c
Number of devices: 1
Devices:
ID SIZE PATH
1 20.00GiB /dev/sdc
查看系统下磁盘的文件系统：
root@pve-btrfs:~# lsblk -fp
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
/dev/loop0 ext4 1.0 28b88c1f-a123-4867-91f9-b58861e23b43
/dev/sda
├─/dev/sda1
├─/dev/sda2 vfat FAT32 F141-2C9D
└─/dev/sda3 btrfs 21f64b58-1a6d-4ff7-a8ff-76a7b139efe6 31.3G 18% /
/dev/sdb
├─/dev/sdb1
├─/dev/sdb2 vfat FAT32 F141-9A86
└─/dev/sdb3 btrfs 21f64b58-1a6d-4ff7-a8ff-76a7b139efe6
/dev/sdc btrfs My-Storage 8866c15a-efa3-4396-836b-8630aaec482f
/dev/sdd
/dev/sde
/dev/sdf
/dev/sr0 iso9660 PVE 2021-07-05-19-15-41-00
在Proxmox VE 7.0界面中，也可以看到/dev/sdc变成了Btrfs文件系统格式了，如图33所示。
3.2）在/dev/sdd、/dev/sde、/dev/sdf和/dev/sdg这4个硬盘上创建一个 RAID10的Btrfs
我这里的虚拟机硬盘不够，稍等我再增加一块硬盘/dev/sdg，很快的哈，如图34所示。
root@pve-btrfs:~# mkfs.btrfs -m raid10 -d raid10 -L My-Storage /dev/sdd /dev/sde /dev/sdf /dev/sdg
btrfs-progs v5.10.1
See http://btrfs.wiki.kernel.org for more information.
Label: My-Storage
UUID: 8ca7cea3-298a-4039-a5b1-376db7707bb7
Node size: 16384
Sector size: 4096
Filesystem size: 80.00GiB
Block group profiles:
Data: RAID10 2.00GiB
Metadata: RAID10 512.00MiB
System: RAID10 16.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Runtime features:
Checksum: crc32c
Number of devices: 4
Devices:
ID SIZE PATH
1 20.00GiB /dev/sdd
2 20.00GiB /dev/sde
3 20.00GiB /dev/sdf
4 20.00GiB /dev/sdg
查看系统下磁盘的文件系统：
root@pve-btrfs:~# lsblk -fp
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
/dev/sda
├─/dev/sda1
├─/dev/sda2 vfat FAT32 F141-2C9D
└─/dev/sda3 btrfs 21f64b58-1a6d-4ff7-a8ff-76a7b139efe6 31.2G 19% /
/dev/sdb
├─/dev/sdb1
├─/dev/sdb2 vfat FAT32 F141-9A86
└─/dev/sdb3 btrfs 21f64b58-1a6d-4ff7-a8ff-76a7b139efe6
/dev/sdc btrfs My-Storage 889f5539-8144-418c-94c3-d450b862dd0c
/dev/sdd btrfs My-Storage 8ca7cea3-298a-4039-a5b1-376db7707bb7
/dev/sde btrfs My-Storage 8ca7cea3-298a-4039-a5b1-376db7707bb7
/dev/sdf btrfs My-Storage 8ca7cea3-298a-4039-a5b1-376db7707bb7
/dev/sdg btrfs My-Storage 8ca7cea3-298a-4039-a5b1-376db7707bb7
/dev/sr0 iso9660 PVE 2021-07-05-19-15-41-00
在Proxmox VE 7.0界面中，也可以看到/dev/sdd、/dev/sde、/dev/sdf、/dev/sdg变成了Btrfs文件系统格式了，如图35所示。
第四步：挂载 BTRFS 文件系统
然后可以手动挂载新的文件系统，如下所示：
root@pve-btrfs:~# mkdir /my-storage #在根目录下创建文件夹
root@pve-btrfs:~# mount /dev/sdc /my-storage #把/dev/sdc挂载到文件夹上
Note that the mount /dev/sdc /my-storage command is only a temporary mount. When Proxmox VE restarts, it will be automatically uninstalled. I use temporary mount here because it is a demonstration environment. If it is a production environment, please use the /etc/fstab file to complete the permanent mount.
About how to create a Btrfs storage pool of multiple hard disk devices, it will be discussed in the subsequent article on Proxmox VE back-end storage, which will not be discussed here due to space limitations.
After the mount is successful, the data access operation to the hard disk /dev/sdc can be realized through the /my-storage directory.
Step 5: Add the BTRFS file system to Proxmox VE 7.0
After selecting "Data Center → Storage → Add → BTRFS", the Proxmox VE 7.0 interface will automatically pop up the "Add: BTRFS" interface, as shown in Figure 36.
In the "Add: BTRFS" interface, enter the storage name, enter the storage path, select the storage content, and click the "Add" button, as shown in Figure 37.
"lisq-btrfs storage logo" appears in the navigation tree on the left, indicating that the lisq-btrfs storage is successfully enabled and can be used normally. Virtual machines and containers can be created in the lisq-btrfs storage, as shown in Figure 38.
2.3.5. Summary of PVE default storage
I won’t say any more about this part, just go to the form and serve it. I hope this form will be helpful for everyone to learn Proxmox VE, as shown in Figure 39.
Reference: "pve-admin-guide-7" user manual on Proxmox VE official website;