1. Configuring Mirroring
2. Use qemu to simulate nvdimm (install PM related kernel modules)
- running record
some problems encountered
- 1. The ext4 file system is damaged

Brief description of the process:
(Mirroring basic file system configuration)
1. Download the ubuntu-base mirroring
2. Mount the mirroring, configure the mirroring network, and install various software
(PM module configuration)
3. Compile the kernel module and copy it to In the image
4. Start qemu
5. Install the kernel module and configure related software

1. Configuring Mirroring

1. Use to qemu-imgcreate an image mirror (named here ubuntu-rootfs-raw-20G.image)

# [可选] sudo apt install qemu
qemu-img create -f raw ubuntu-rootfs-raw-20G.image 20G

Note: It needs to be rawa type, otherwise it cannot be formatted normally (because the qcow2 type does not pre-allocate space). The error message is as follows
$ sudo mkfs.ext4 ubuntu-rootfs-qcow2-20G.image
mke2fs 1.45.5 (07-Jan-2020)
ubuntu-rootfs-qcow2-20G.image: Not enough space to build proposed filesystem while setting up superblock

2. Format the image file as ext4 file system

# 格式化
sudo mkfs.ext4 ubuntu-rootfs-raw-20G.image
# 检查
file ubuntu-rootfs-raw-20G.image

3. Create an empty directory, the mount point of the mirror; then mount the mirror on it

mkdir ubuntu-rootfs-dir
sudo mount ubuntu-rootfs-raw-20G.image ubuntu-rootfs-dir

4. Download ubuntu-base-20.04.1-base-amd64.tar.gzthe image and unzip it into ubuntu-rootfs-dira folder (the essence is to insert files into the image, because the current image has been mounted to the folder)
(1) Download address, https://cdimage.ubuntu.com/ ubuntu-base/releases/20.04/release/ubuntu-base-20.04.1-base-amd64.tar.gz .
You can also choose other versions by yourself: https://cdimage.ubuntu.com/ubuntu-base/releases/，

wget https://cdimage.ubuntu.com/ubuntu-base/releases/20.04/release/ubuntu-base-20.04.1-base-amd64.tar.gz

(2) Unzip toubuntu-rootfs-dir

sudo tar -xzvf ubuntu-base-20.04.1-base-amd64.tar.gz -C ubuntu-rootfs-dir

5. Copy the network configuration information in the host to the image, so that it is convenient to install software with apt later

sudo cp /etc/resolv.conf ubuntu-rootfs-dir/etc/

6. Modify the source.list file by yourself, and set the apt mirror to accelerate

假设主机与客服机都是基于ubuntu 20.04，则可以直接拷贝主机的source.list,
否则需要自行配置
sudo cp /etc/apt/source.list ubuntu-rootfs-dir/etc/apt/source.list

7. Use chrootwill ubuntu-rootfs-dirbe temporarily set as the root directory (similar to docker), and start the terminal

sudo chroot ubuntu-rootfs-dir

8. Install the software

apt update\
&& apt upgrade\
&& apt install linux-image-kvm init vim -y

9. Set a password and the like

update-initramfs -u
echo root:root | chpasswd
echo ttyS0 > /etc/securetty
systemctl enable [email protected]

10. Exit and uninstall

sudo umount ubuntu-rootfs-dir

11. Subsequent expansion
(1) If you need to reinstall the software, you can repeat step 7, mount the mirror image, and then chrootenter.
(2) Image expansion operation

qemu-img resize ubuntu-rootfs-raw-20G.image +10G

2. Use qemu to simulate nvdimm (install PM related kernel modules)

If you want to run PM
1, install the software package (installed into the mirror)

apt install -y systemd numactl ndctl daxctl

2. When compiling the kernel, compile related modules

make -j16 vmlinux bzImage
make M=drivers/dax -j16
make M=drivers/nvdimm/ -j16
make M=drivers/dax/pmem -j16

3. Copy the compiled module to the image

# 1、挂载镜像
# 略

# 2、设置路径（路径根据需要进行修改）
image_dir_path=/home/my/my_images/ubuntu-base/ubuntu-rootfs-dir
module_path=$image_dir_path/root/my_modules
sudo mkdir $module_path

# 3、拷贝编译的内核模块到镜像中
cd linux-5.15.114
sudo cp -r ./drivers/dax $module_path/drivers/dax
sudo cp -r ./drivers/nvdimm $module_path/drivers/nvdimm

# 4、卸载镜像
# 略

3. Run qemu

qemu-system-x86_64 \
    -machine pc,nvdimm=on \
    -m 2G,slots=4,maxmem=32G \
    -nographic -kernel bzImage \
    -smp cores=4,threads=1,sockets=2 \
    -hda ubuntu_rootfs.ext4 \
    -object memory-backend-ram,id=mem0,size=1G  \
    -object memory-backend-ram,id=mem1,size=1G  \
    -numa node,memdev=mem0,cpus=0-3,nodeid=0 \
    -numa node,memdev=mem1,cpus=4-7,nodeid=1 \
    -numa node,nodeid=2 -numa node,nodeid=3 \
    -object memory-backend-ram,id=nvdimm1,size=4G\
    -device nvdimm,memdev=nvdimm1,id=nv1,unarmed=off,node=2 \
    -object memory-backend-ram,id=nvdimm2,size=4G\
    -device nvdimm,memdev=nvdimm2,id=nv2,unarmed=off,node=3 \
    -append "console=ttyS0 crashkernel=256M root=/dev/sda rootfstype=ext4 rw loglevel=8"

4. Install the module (note the dependencies between modules)

insmod /root/my_modules/drivers/dax/device_dax.ko
insmod /root/my_modules/drivers/dax/kmem.ko

insmod /root/my_modules/drivers/dax/pmem/dax_pmem_core.ko
insmod /root/my_modules/drivers/dax/pmem/dax_pmem.ko
insmod /root/my_modules/drivers/dax/pmem/dax_pmem_compat.ko

insmod /root/my_modules/drivers/nvdimm/nd_btt.ko
insmod /root/my_modules/drivers/nvdimm/nd_blk.ko
insmod /root/my_modules/drivers/nvdimm/nd_pmem.ko
insmod /root/my_modules/drivers/nvdimm/nd_virtio.ko
insmod /root/my_modules/drivers/nvdimm/virtio_pmem.ko

# check
lsmod

Note here: There may be dependencies between modules, which need to be in a certain order. If the order is wrong, the following error message may appear insmod: ERROR: could not insert module device_dax.mod: Invalid module format. At this point, use dmesg | tailto view detailed log information, and use modinfo ./dax_pmem.ko | grep depend to view module dependencies.
$ modinfo ./dax_pmem.ko | grep depend 
depends:        dax_pmem_core
# 可以看到dax_pmem.ko依赖dax_pmem_core模块

5. Configure nvdimm

# 首次配置
daxctl migrate-device-model
echo offline > /sys/devices/system/memory/auto_online_blocks
ndctl create-namespace -f --mode devdax --continue
daxctl reconfigure-device --mode=system-ram all

# 重启后重新配置
echo offline > /sys/devices/system/memory/auto_online_blocks
ndctl disable-namespace all
ndctl destroy-namespace all
ndctl create-namespace -f --mode devdax --continue
daxctl reconfigure-device --mode=system-ram all

6. Merge four or five steps,
and then create an init.sh file and put it in the image, which will run automatically at startup

# init.sh
# 用于image镜像中挂载内核模块等

insmod /root/my_modules/drivers/dax/device_dax.ko
insmod /root/my_modules/drivers/dax/kmem.ko

insmod /root/my_modules/drivers/dax/pmem/dax_pmem_core.ko
insmod /root/my_modules/drivers/dax/pmem/dax_pmem.ko
insmod /root/my_modules/drivers/dax/pmem/dax_pmem_compat.ko

insmod /root/my_modules/drivers/nvdimm/nd_btt.ko
insmod /root/my_modules/drivers/nvdimm/nd_blk.ko
insmod /root/my_modules/drivers/nvdimm/nd_pmem.ko
insmod /root/my_modules/drivers/nvdimm/nd_virtio.ko
insmod /root/my_modules/drivers/nvdimm/virtio_pmem.ko


# 首次配置
# daxctl migrate-device-model
# echo offline > /sys/devices/system/memory/auto_online_blocks
# ndctl create-namespace -f --mode devdax --continue
# daxctl reconfigure-device --mode=system-ram all

# 重启后重新配置
echo offline > /sys/devices/system/memory/auto_online_blocks
ndctl disable-namespace all
ndctl destroy-namespace all
ndctl create-namespace -f --mode devdax --continue
daxctl reconfigure-device --mode=system-ram all

running record

root@localhost:~/my_modules/drivers/nvdimm/nvdimm# echo offline > /sys/devices/system/memory/auto_online_blocks
root@localhost:~/my_modules/drivers/nvdimm/nvdimm# ndctl create-namespace -f --mode devdax --continue
{
  "dev":"namespace1.0",
  "mode":"devdax",
  "map":"dev",
  "size":"3.94 GiB (4.23 GB)",
  "uuid":"12f978b1-c1f4-4be8-a67d-aa076e0a4152",
  "daxregion":{
    "id":1,
    "size":"3.94 GiB (4.23 GB)",
    "align":2097152,
    "devices":[
      {
        "chardev":"dax1.0",
        "size":"3.94 GiB (4.23 GB)",
        "target_node":3,
        "mode":"devdax"
      }
    ]
  },
  "align":2097152
}
{
  "dev":"namespace0.0",
  "mode":"devdax",
  "map":"dev",
  "size":"3.94 GiB (4.23 GB)",
  "uuid":"0529e940-38b1-45fd-a0b1-43fcf95192ce",
  "daxregion":{
    "id":0,
    "size":"3.94 GiB (4.23 GB)",
    "align":2097152,
    "devices":[
      {
        "chardev":"dax0.0",
        "size":"3.94 GiB (4.23 GB)",
        "target_node":2,
        "mode":"devdax"
      }
    ]
  },
  "align":2097152
}
created 2 namespaces
root@localhost:~/my_modules/drivers/nvdimm/nvdimm# daxctl reconfigure-device --mode=system-ram all
[
  {
    "chardev":"dax0.0",
    "size":4225761280,
    "target_node":2,
    "mode":"system-ram",
    "movable":true
  }
]

Not first run:

root@localhost:~/my_modules/drivers/nvdimm# echo offline > /sys/devices/system/memory/auto_online_blocks
oy-namespace all
ndctl create-namespace -f --mode devdax --continue
daxctl reconfigure-device --moderoot@localhost:~/my_modules/drivers/nvdimm# ndctl disable-namespace all
=system-ram alldisabled 2 namespaces
root@localhost:~/my_modules/drivers/nvdimm# ndctl destroy-namespace all
destroyed 0 namespaces
root@localhost:~/my_modules/drivers/nvdimm# ndctl create-namespace -f --mode devdax --continue
{
    
    
  "dev":"namespace1.0",
  "mode":"devdax",
  "map":"dev",
  "size":"3.94 GiB (4.23 GB)",
  "uuid":"8259f939-0c3d-43c4-89cb-6e89653c290f",
  "daxregion":{
    
    
    "id":1,
    "size":"3.94 GiB (4.23 GB)",
    "align":2097152,
    "devices":[
      {
    
    
        "chardev":"dax1.0",
        "size":"3.94 GiB (4.23 GB)",
        "target_node":3,
        "mode":"devdax"
      }
    ]
  },
  "align":2097152
}
{
    
    
  "dev":"namespace0.0",
  "mode":"devdax",
  "map":"dev",
  "size":"3.94 GiB (4.23 GB)",
  "uuid":"41d3381f-cd6a-4c02-b8d1-c97cd2633cb4",
  "daxregion":{
    
    
    "id":0,
    "size":"3.94 GiB (4.23 GB)",
    "align":2097152,
    "devices":[
      {
    
    
        "chardev":"dax0.0",
        "size":"3.94 GiB (4.23 GB)",
        "target_node":2,
        "mode":"devdax"
      }
    ]
  },
  "align":2097152
}
created 2 namespaces
root@localhost:~/my_modules/drivers/nvdimm# daxctl reconfigure-device --mode=system-ram all
[
  {
    
    
    "chardev":"dax0.0",
    "size":4225761280,
    "target_node":2,
    "mode":"system-ram",
    "movable":true
  }
]
reconfigured 2 devices

some problems encountered

1. The ext4 file system is damaged

Problem: When the system starts, it encounters an ext4 error message

[ 3.630969] EXT4-fs error (device sda): htree_dirblock_to_tree:1080: inode #393220: comm systemd-tmpfile: Directory block failed checksum

This can also cause other problems, such as:

There are files that cannot be deleted

rm: cannot remove 'directory': Bad message

The file becomes a directory and cannot be deleted

# 提示双重目录（因为daxctl.conf变成了目录）
conf_files_filter_out: Directories inside directories are not supported: /etc/modprobe.d/daxctl.conf
# 删除不了
"cannot remove 'daxctl.conf': Directory not empty"

Solution:

Use e2fscktools to scan and repair the file system, and basically press yOK throughout the process. (According to the information I have seen, it seems that the ref of some files has not been updated, resulting in file errors)

e2fsck -y ubuntu-rootfs-raw-20G.image

Among them,
-fthe option is used to force the file system to be checked, even if the file system is mounted.
-yoption to automatically answer "yes" to all repair questions so that file system errors are automatically repaired.
examine:

$ e2fsck ubuntu-rootfs-raw-20G.image
e2fsck 1.45.5 (07-Jan-2020)
ubuntu-rootfs-raw-20G.image: clean, 13654/1310720 files, 321042/5242880 blocks

2. The kernel module was not loaded successfully

error message

$ ndctl create-namespace -f --mode devdax --continue
libkmod: ERROR ../libkmod/libkmod-module.c:838 kmod_module_insert_module: could not find module by name='dax_pmem'
原因：lsmod检查一下，应该是dax_pmem模块没有成功加载。

root@localhost:~# depmod -a
depmod: ERROR: could not open directory /lib/modules/5.15.114+: No such file or directory

原因：我一开始忘记把内核模块放到镜像里了。

3. qemu reports an error

# qemu环境中，执行关机命令
$ shutdown now
....
systemd-journald[202]: Failed to send stream file descriptor to service manager: Connection refused

Guess: It may be that there are unfinished files here, so qemu was closed by me. Maybe the previous file system error is because of this reason.

4. The host terminal cannot be opened normally

I don't know the specific reason (guess it is because I sometimes mounted the /dev directory, and then the terminal cannot be opened, and the terminal of the vscode remote link cannot be opened), and the error message is as follows:

There was an error creating the child proces for this terminalFailed to open PTY: No such device

insert image description here
Solution:
1. Find a way to enter the system, through safe mode or boot disk, or something, anyway, as long as you can run boot-repair
2. Download boot-repair
3. Run boot-repair to repair.

$ sudo apt install boot-repair
$ boot-repair

5. The module is inconsistent with the kernel version

There is an error during insmod, and then dmesg | tailyou can find the error message, which is roughly as follows:

device_dax: disagrees about version of symbol vmf_insert_mixed
 : unknown symbol kmalloc_caches (err -22)

Reason: In order to compile faster, my compilation command is make -j16 vmlinux, so only vmlinux has been updated, and other modules have not been updated.

tips (memo)
If there are directory a/c and directory b/c, now 1.txt is added under a/c, and I want to synchronize this 1.txt to b/c.
First of all, neither cp -r a/c b/cand cp -r a/c b/c/will work, they will copy a/c to b/c, and a folder of b/c/c will appear
Then, some solutions are proposed on the Internet: https://stackoverflow.com/questions/23698183/ how-to-force-cp-to-overwrite-directory-instead-of-creating-another-one-inside , but I think neither is very elegant. . .

First rm the old folder, then cp the new folder. Simple, but the disadvantage is that if the folder is very large, it will be very slow, right?

Use rsync to sync folders. Great, only downside: I'm not that familiar with rsync, and I'm not sure if rsync is installed by default on all servers

With -Targuments, treat the target directory as a file object. Disadvantages: This parameter is rarely used, and the function of treating folders as files, I am afraid that after I use it indiscriminately, I will not remember it clearly in the future.

cp -r a/c/* b/c/*. Disadvantage: I will definitely choose this when I usually use it, but it is not convenient to write it in the script, because the b/cfolder does not exist when it is run for the first time, then this command will report an error. Therefore, when I write scripts, I need to judge this situation, and then use different commands to deal with it.

Solution:
(1) Compile related modules at the same time when compiling the kernel

# 默认会编译所有代码
make -j16

(2) Replace the old module with the newly compiled module, and write a script here for convenience. The content of the script is to mount the image first, then copy the files, and finally uninstall the image. It’s just that for the sake of safety, some check codes are added (worried that some parameters are empty, and then the path becomes /something).

linux_kernel_root=.
image_root=/home/mingyang/my_images/ubuntu-base
image_path=$image_root/ubuntu-rootfs-raw-20G.image
mount_dir=$image_root/ubuntu-rootfs-dir

# check
if [[ ! -d $image_root ]]
then
    echo $image_root" not exists"
    exit
fi
if [[ ! -f $image_path ]]
then
    echo $image_path" not exists"
    exit
fi
if [[ ! -d $mount_dir ]]
then
    echo $mount_dir" not exists"
    exit
fi


# mount
sudo mount $image_path $mount_dir
ret=$?
if [[ ! ret ]]
then
    echo "mount $image_path to $mount_dir failed"
    exit
fi
echo "success mounting $image_path to $mount_dir"

# copy
module_path=$mount_dir"/root/my_modules"

sudo mkdir -p $module_path/drivers/dax
sudo mkdir -p $module_path/drivers/nvdimm

sudo rm -r $module_path/drivers/dax
sudo rm -r $module_path/drivers/nvdimm

sudo cp -r $linux_kernel_root/drivers/dax $module_path/drivers/dax
sudo cp -r $linux_kernel_root/drivers/nvdimm $module_path/drivers/nvdimm

echo "success copying modules to $module_path"

# umount
sudo umount $mount_dir
if [[ ! ret ]]
then
    echo "umount $mount_dir failed"
    exit
fi
echo "success umounting $image_path to $mount_dir"

Use qemu to create the ubuntu-base file system and install PM-related kernel modules

Table of contents