Use QEMU (x86) to simulate running the ARM64 architecture and perform kernel debugging

Emulate and debug ARM64 on x86 using the QEMU virtual machine.
Reference: Use QEMU to build ARM64 environment in ubuntu virtual machine

Preparation

• Free Host 20.04, Linux 5.16.0
• QEMU emulator version 7.2.92 (v8.0.0-rc2-23-gefcd0ec14b) ：qemu-7.2.29
• Suitable ARM64 kernel source: linux-4.14.221.tar.gz
• Install the cross-compiler tool:sudo apt-get install gcc-aarch64-linux-gnu

Install QEMU

Both command installation and source code installation do not support the Raspberry Pi 4B platform

1. Command to install qemu: sudo apt install qemu-system-arm,qemu-system-aarch64 --help
2. Install qemu-5.2.0/qemu-7.2 by compiling and installing.
3. Note the environment variables:export ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

However, command install does not support Raspberry Pi 4.

jiaming@jiaming-pc:~/Documents/jailhouse-rpi4$ qemu-system-aarch64 -M ?
Supported machines are:
...
raspi2               Raspberry Pi 2
raspi3               Raspberry Pi 3
realview-eb          ARM RealView Emulation Baseboard (ARM926EJ-S)
...

2. Install qemu-5.2.0/qemu-7.2 by compiling and installing.

Install ninja.

git clone https://github.com/ninja-build/ninja.git && cd ninja
./configure.py --bootstrap
cp ninja /usr/bin/

• qemu-5.2.0
• qemu-7.2.29

./configure --target-list=aarch64-softmmu,aarch64-linux-user --enable-debug
make -j4
make install

Use debootstrap to make root file system

install package

$ sudo apt-get install bison flex bc build-essential libncurses* libssl-dev
$ sudo apt-get install  qemu-user-static binfmt-support debootstrap

Initialize the root file system

$ sudo su - root
$ debootstrap --arch=arm64 --foreign buster linux_rootfs http://mirrors.ustc.edu.cn/debian/ # linux_rootfs：本地目录，最后制作好的文件系统会在此目录

$sudo cp /usr/bin/qemu-aarch64-static ./linux_rootfs/usr/bin/

$ sudo chroot linux_rootfs/ debootstrap/debootstrap --second-stage

Enter the root file system

$ chroot linux_rootfs/

Configure root user password

passwd root

Add user and set password

useradd -G sudo -m -s /bin/bash XXX
passwd XXX

set hostname

echo XXX-PC > /etc/hostname

set network

$ echo "auto lo" > /etc/network/interfaces
$ echo "iface lo inet loopback" >> /etc/network/interfaces
$ echo "allow-hotplug enp0s1" > /etc/network/interfaces
$ echo "iface enp0s1 inet dhcp" >> /etc/network/interfaces

install dependencies

$ apt update
$ apt install net-tools build-essential

Make ext4 file system

$ dd if=/dev/zero of=linux_rootfs.ext4 bs=1M count=2048

Format with the mkfs.ext4 command:

$ mkfs.ext4 linux_rootfs.ext4

Mount and copy content:

$ mkdir -p tmpfs
$ sudo mount -t ext4 linux_rootfs.ext4 tmpfs/ -o loop
$ sudo cp -af linux_rootfs/* tmpfs/
$ sudo umount tmpfs
$ sudo chmod 777 linux_rootfs.ext4

kernel compilation

$ cd linux-4.14.221/
$ export ARCH=arm64 ROSS_COMPILE=aarch64-linux-gnu-
$ make defconfig
$ make -j4

CONFIG_GDB_SCRIPTSTurn on and off when compiling the kernel CONFIG_DEBUG_INFO_REDUCED. CONFIG_FRAME_POINTERLeave it on if the architecture supports it .

After the compilation is successful, arch/arm64/boot/an Image file will be generated under the directory.

QEMU simulates ARM64 architecture startup

Start command:

qemu-system-aarch64 \
-m 1024 \
-cpu cortex-a57 \
-M virt -nographic \
-smp 4 \
-kernel linux-4.14.221/arch/arm64/boot/Image \
-append "noinintrd sched_debug root=/dev/vda rootfstype=ext4 rw crashkernel=256M loglevel=8" \
-drive if=none,file=linux_rootfs.ext4,id=hd0 \
-device virtio-blk-device,drive=hd0 

• qemu-system-aarch64: QEMU command, used to start an AARCH64 virtual machine.
• -m 1024: Set the memory size of the virtual machine to 1GB.
• -cpu cortex-a57: CPU model A57 used to execute the virtual machine.
• -M virt -nographic: Set QEMU to virtualization mode without displaying a graphical interface.
• -smp 4: Set the maximum number of threads supported by the virtual machine to 4.
• -kernel linux-4.14.221/arch/arm64/boot/Image: Specifies the Linux kernel file to load.
• -append "noinintrd sched_debug root=/dev/vda rootfstype=ext4 rw crashkernel=256M loglevel=8": Add kernel parameters to enable noinintrd sched_debug debug options and set root directory to /dev/vda in VDA.
• if=none,file=linux_rootfs.ext4,id=hd0: Specifies that the storage device to be used is the HD0 device on the IDE controller, and the file name is linux_rootfs.ext4.
• -device virtio-blk-device,drive=hd0 : Specifies the virtio blk device to use for storage devices.

At this point, the simulation environment is set up!

debugging

Cross compile strace and install

• strace

Generate compilation guide file

./bootstrap

arm64 cross compilation

./configure CC=aarch64-linux-gnu-gcc LD=aarch64-linux-gnu-ld --host=aarch64-linux --enable-mpers=no

compile

make LDFLAGS+="-static -pthread"

Add to root filesystem

sudo mount -t ext4 linux_rootfs.ext4 tmpfs/ -o loop
sudo cp -af strace/src/strace tmpfs/bin/
sudo umount tmpfs

strace use: linux strace command

Cross compile gdb and install

GUIs are not supported.

• gdb-8.0
• sudo apt-get install g++-9-aarch64-linux-gnu

export CC=aarch64-linux-gnu-gcc CXX=aarch64-linux-gnu-g++-9
cd gdb-8.0/
mkdir gdb-build
./configure --host=aarch64-linux-gnu --target=aarch64-linux-gnu --program-prefix=aarch64-linux- --prefix=/home/jiaming/Documents/jailhouse-rpi4/qemu-arm64/gdb-8.0/gdb-build
make
sudo make install
sudo cp /usr/aarch64-linux-gnu/lib/* /lib/

Compilation errors and solutions: https://blog.csdn.net/weixin_44602409/article/details/115716913

vim ~/.bashrc,Add to:

export PATH=$PATH:/home/jiaming/Documents/jailhouse-rpi4/qemu-arm64/gdb-8.0/gdb-build/bin

debug kernel

1. The QEMU command starts the kernel, adding -S -sparameters.

• -S is to suspend the gdbserver and allow the gdb tool to connect remotely.
• -s uses port 1234 for remote debugging by default

Kernel boot hangs, waiting for gdb to connect.

2. In a new command window, start the gdb command jiaming@jiaming-pc:~/Documents/jailhouse-rpi4/qemu-arm64/linux-4.14.221$ aarch64-linux-gdb vmlinuxand connect using the default port, target remote localhost:1234.

After exiting the gdb tool, the kernel continues to start and outputs the startup information.

postscript

1. When an ld error occurs during compilation, consider replacing the compiler version. make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- LD=/home/jiaming/gnu/aarch64/lin/aarch64-linux/bin/aarch64-linux-gnu-ld
   Or, specify all:

make ARCH=arm64 CROSS_COMPILE=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu- CC=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc LD=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-ld AR=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-ar install

2. The latest version of qemu does not support rpi4 platform emulation.
3. In the gparted tool, it is found that the disks are all gray, indicating that the disks have been mounted, and actually do not need to be mounted.

use busybox

• busybox-1.29.0
• Install the cross-compilation toolchain ( linaro-7.5 ), and the default 9.x version compiles incorrectly.

wget  https://busybox.net/downloads/busybox-1.29.0.tar.bz2
tar -xjf busybox-1.29.0.tar.bz2
cd busybox-1.29.0
make ARCH=arm64 CROSS_COMPILE=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu- CC=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc LD=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-ld AR=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-ar menuconfig
make ARCH=arm64 CROSS_COMPILE=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu- CC=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc LD=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-ld AR=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-ar
make ARCH=arm64 CROSS_COMPILE=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu- CC=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc LD=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-ld AR=/home/jiaming/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-ar install

dd if=/dev/zero of=busybox-1.29.0-rootfs_ext4.img bs=1M count=100 oflag=direct
mkfs.ext4 busybox-1.29.0-rootfs_ext4.img
mkdir rootfs
sudo mount busybox-1.29.0-rootfs_ext4.img rootfs/
sudo cp -raf busybox-1.29.0/_install/* rootfs/

cd rootfs
sudo mkdir -p proc sys tmp root var mnt dev
sudo mknod dev/tty1 c 4 1
sudo mknod dev/tty2 c 4 2
sudo mknod dev/tty3 c 4 3
sudo mknod dev/tty4 c 4 4
sudo mknod dev/console c 5 1
sudo mknod dev/null c 1 3
sudo cp -r ../busybox-1.29.0/examples/bootfloppy/etc/ .

cd ..
sudo umount rootfs

QEMU start command:qemu-system-aarch64 -m 1024 -cpu cortex-a57 -machine virt,gic-version=3,virtualization=on -kernel Image -append "console=ttyAMA0 root=/dev/vda init=/linuxrc rw" -hda busybox-1.29.0-rootfs_ext4.img -monitor stdio

To mount /dev/vda as the root file system, the following options need to be configured in the kernel

CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_PCI=y
CONFIG_EXT4_FS=y

Otherwise this error occurs:

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(254,0)

A new window pops up when QEMU starts

install sdl

Install sdl: https://blog.csdn.net/qq_22948593/article/details/109740372, no cross-compilation is required.
https://www.libsdl.org/release/SDL2-2.0.14.tar.gz

Reinstall QEMU

./configure --target-list=aarch64-softmmu,aarch64-linux-user --enable-debug --enable-sdl
make -j4
make install

There are -nographicparameters:

None -nographicParameters:

No -nographicparameter, add -monitor stdioparameter:

Press qto exit.