Use GDB, KGDB to debug application programs and kernel driver modules

2017-03-13 Sun slope build the GDB , the GDB KGDB , the GDB debugger double machine , KGDB debugging the application , using the GDB , nuclear drive module , compiled kernel , debug applications , configure the serial debug , configure Duplex Communications

http://blog.nsfocus.net/gdb-kgdb-debug-application/

Article Directory

Reading: 12,063

I have looked at a vulnerability in Linux kernel privilege escalation these days, which involves driver vulnerabilities and driver debugging content. Due to the different versions of Linux operating systems, if you can’t debug the driver on your own machine, you can say The POC source code that has been exploited cannot be successfully exploited at all. Because the exploitation of kernel vulnerabilities involves the POC structure of the instruction set, different kernel version modules load instruction addresses are different, so even if there is a POC, they cannot be used at all. Only by personally debugging in your own system can you make corresponding modifications to achieve kernel vulnerability exploitation Effect. This requires us to have an in-depth understanding of the debugging process and debugging methods of the Linux kernel driver. After two days of searching, configuring, and debugging, I finally figured out the basic method of kernel debugging, and provided technical support for kernel debugging and vulnerability analysis.

aims

It is often necessary to debug kernel modules during driver development or kernel vulnerability analysis. Under normal circumstances, the most direct way to debug the driver is to use the print debugging method. In the driver, add debugging information through printk. At the same time, the dynamic debugging of the drive can be realized by dynamically loading the module, which is also the simplest debugging method. As for the analysis of kernel vulnerabilities, since the Linux system is an open source project, regardless of the debugging of the application or the debugging of the kernel driver, the trigger point of the vulnerability can be found by viewing the source code.

So what if you want to do dynamic source code or debugging of the kernel driver like debugging a user-mode application, or to further debug the driver at assembly level or develop a kernel exploit program? Some people may say that there is generally no need for assembly-level debugging. However, in the process of exploiting the kernel vulnerabilities, it is often necessary to debug the kernel driver, and it is necessary to perform assembly instruction-level single-step tracking of the kernel driver, so as to determine the direction of the program. Or we need to construct a special instruction block to complete a certain function. This brings new challenges to our debugging kernel.

So how is it implemented in the kernel, and if it can be followed up inside the kernel for debugging?

This article is to solve this problem: while dynamically assembling and debugging user-mode applications, it can follow up the system call interface of the application, and debug the driver directly at source-level or assembly-level debugging (if there is no symbol table).

The program demonstrated in this article is to call a driver interface written by yourself through an application demo, and you can follow up and debug the driver when you debug the application. To build such an environment, we used the vmware virtual machine, which is widely used and simple to install. In order to be able to debug the program, a target machine and a client computer are required.

The target machine is used to install the driver while running the application, and the application will call the interface in the driver. At the same time, the target machine debugs the application program by itself (use GDB debugging in user mode).

The client is used to connect to the client and debug the driver in the target machine in the client (using GDB debugging).

Points to note:

In order to be able to debug the driver of the target machine, the target machine needs to support KGDB debugging.
In order to allow the client to communicate with the target machine, we configure the two machines in vmware to debug through serial communication.
If the client wants to support source-level debugging of the driver, it needs to load the symbol table of the driver into the debugger of the client.
The vmware virtual machine used in this article needs to install the target machine and the client in the virtual machine at the same time.
Since the article will include the operation process and debugging process of the client and the target machine, the green box will be used in this article to indicate the client-related operations, and the red box will be used to indicate the related operations on the target machine.

Basic environment construction

The figure below is a block diagram of a debugging program and a driver. Both operating systems are installed in a virtual machine, a client computer and a target computer. The client computer debugs the driver in the target computer through the serial port.

1.1 Resources and environment

Vmware virtual machine

Ubuntu operating system

Ubuntu source code

1.2 install vmware virtual machine and ubuntu operating system

The host computer win7 operating system used here, so I downloaded the virtual machine of VMWARE WORKSTATION 10.0.4 version directly on the Internet.

The process of installing the virtual machine is omitted. . . .

After installing the virtual machine, you need to install the ubuntu operating system in the virtual machine, and installing the ubuntu operating system in the virtual machine is also omitted. . . .

The above mentioned two operating systems, the target machine and the guest machine, and here we only install one operating system in the virtual machine. Don't worry, you will know later. Next, download a kernel source code from the official website, as shown below:

The download here is the 3.2.86 version of the X86-64 version (I forgot to mention it, the reason why the X86-64 version is used is because the ubuntu operating system installed on it is also of this architecture).

After the download is complete, copy the source code to the ubuntu virtual machine and decompress it, as shown in the following figure:

Compile the kernel

In order to be able to debug the driver, the operating system of the target machine needs to support the debug mode, so it is necessary to recompile the kernel so that the target machine supports the debug mode.

2.1 Configure kernel parameters

First enter the directory linux-3.2.86, and then execute the command make menconfig, as shown in the following figure:

The following graphical interface will appear:

Each item here is an item that needs to be selected before compiling the kernel, and different items can be compiled as needed. In order to enable the operating system to support kernel-level debugging, the KGDB debugging switch parameter needs to be turned on. The position of the debugging switch varies according to different ubuntu versions. As shown below:

In order to be able to support KGDB debugging, all of the above items need to be selected. In the process of kernel driver debugging, a breakpoint needs to be set in the driver, so that the kernel address needs to be written, so the following option needs to be removed

After the kernel parameter setting is completed, keep the set config file, the default save file name is .config file, keep it in the current directory. In order to ensure that we have disabled the write protection of the kernel, before starting to compile the kernel, check the config file again and open the .config file, as shown below:

After opening it, directly search for "RO" to find the following two items:

Make sure that the two items in the red box are in comment status. If they are not in comment status, you can directly modify them here and change their value to N. This basically completes the configuration of the kernel parameters.

2.2 Compile the kernel

After keeping the settings, compile the kernel:

Make

Make bzImage

Make modules

Make modules_install

Make install

At this time, a new kernel module vmlinux is generated in the current directory

At the same time, a new kernel system, symbol table and other information are generated in the /boot/ directory, as shown below

The red box is the new file, and the green box is the old kernel file.

The kernel is updated to the latest version, indicating that our kernel has been compiled successfully.

Configure dual machine communication

Driver debugging requires two virtual machines, the target machine and the client machine. A tricky method is used here, instead of installing two virtual machines, but directly cloning a copy of the ubuntu operating system compiled above in vmware, so that there are two ubuntu virtual machines, one as the target machine, One serves as a client. Of course, both systems support the kgdb debugging mode, and both use the same kernel.

3.1 Two-machine serial communication

Therefore, close the compiled ubuntu, and then use the vmware clone function to clone a copy (if you don't close the virtual machine, you can't clone).

The cloned virtual machine is used as the target machine, and the first compiled virtual machine is used as the client.

The above screenshots are not divided into the target machine, because the client is operating an ubuntu operating system, after the cloning is completed, there is a target machine and a client. Therefore, the following operations will use a green box to represent the client, and a red box to represent the target machine.

Configure the client, the two machines use through-port communication to configure the client serial port, as shown below:

It should be noted that when installing the virtual machine, the parallel port is installed by default, but there is no serial port. At this time, we need to delete the parallel port first, and then add the serial port, and the configuration serial port is shown in the installation diagram.

2. Configure the target machine, and the target machine should be configured as a server, as shown in the figure below:

It is also necessary to pay attention to the target terminal. If there is a parallel port, you need to delete the parallel port first, then add a serial port, and configure accordingly.

3.2 Verify serial communication configuration

After configuring the serial port, you can verify whether the configured serial port works. Start the client and target machine,

Input data to the serial port at one end and accept data at the other end. Here we select the target machine to input data, and the client computer receives the data. As shown below:

Of course, this process is to let the client open the receiving first, and then let the target machine send, so that the client can receive the data. As can be seen from the above figure, there is no problem with the normal data transmission through the serial port.

4 configure serial port debugging

After the above configuration is completed, it is equivalent to connecting a serial cable between the two virtual machines. If you want to debug the two systems through the serial cable, you also need to configure the serial debugging mode.

4.1 Client debugging configuration

Move the client system, log in to the system in root mode, and modify the grub startup configuration file. As shown below

The content in the yellow box indicates that a serial port connection is required. Of course, it can also be added to "GRUB_COMLINE_LINUX" in the following item.

After the configuration, you need to update grub to make the configuration take effect:

Update-grub

In this way, grub is updated, and serial communication will be loaded after restarting the device. After Grub updates the configuration, it will automatically modify the /boot/grub/grub.cfg file, as shown in the following figure:

Remember, it is not impossible to modify grub.cfg directly, but if /etc/default/grub is updated, if you run update-grub, grub.cfg will be updated again, causing the configuration directly in grub.cfg to fail (of course only Changing /etc/default/grub, and not running the Update-grub command does not invalidate grub.cfg).

In this way, the serial communication module of the client port is configured.

4.2 Target machine debugging configuration

The configuration of the target machine and the configuration of the client are basically the same, but there are some differences.

Start the client system, log in to the system in root mode, and modify the grub startup configuration file. As shown below

A parameter text is added here compared to the client configuration. This parameter means that the text interface is displayed instead of the graphical interface after the system is started (of course this is not necessary, but as the target machine, we can directly use the text interface interface to enter the system Up).

After the configuration, you need to update grub to make the configuration take effect:

Update-grub

After the update is completed, the /boot/grub/grub.cfg file is automatically modified, as shown in the figure below

The configuration has been changed in the figure. From the above picture, there is an additional "Ubuntu, with Linux 3.2.86—wait" option. This option is copied from the above option, and at the same time a new "kgdbwait" label is added to it, and a new option is added It is an extra startup item when grub is started. The "kgdbwait" parameter is added to enter the debugging mode when the system is just started. This parameter is not added to the above startup option, so debugging can only be carried out after the system is started. This target The computer supports two kinds of debugging, one is to debug the kernel when the system just starts, and the other is to debug the kernel or driver after the system starts.

So far our debugging mode has been established.

5 GDB dual machine debugging environment

Enter grub from the newly booted target machine, you will see the extra boot options as shown in the figure below.

If we choose the "Ubuntu, with Linux 3.2.86—wait" startup mode, the system will enter the mode shown below, waiting for the connection of the remote debugger, which is the connection of the GDB debugger on the client.

This shows that there is no problem with the configuration of the target machine, and you can start the client to connect and debug.

Because it is debugging the application program and driver program written by myself, and the driver program is loaded dynamically. So there is no need to debug the system when the system is just started. As long as the module is dynamically loaded after the system is started, it is not too late to debug the module. So restart the target machine, enter the "Ubuntu, with Linux 3.2.86" startup mode, and start the system directly.

5.1 driver code and compilation

Before debugging the code, let's take a look at the functions implemented by our application and driver. The program code is implemented on the target machine.

First look at the driver code. Since this article only explains driver debugging, only a small function is implemented for the driver code. The file opening function in the driver is as shown below:

The code is very simple, it is to generate a driver drv1, the driver implements an interface, that is, the opening function. When the system call of the driver is called in the application, two messages "device opened" and "device opened return!" are printed. If the driver is loaded successfully, it will display "drv1 init……", if the driver fails to load, it will display "Driver unloaded".

Compile the driver by writing Makefile

Directly compile the production drv.ko file:

5.2 Application and compilation

Finally, look at the application as shown below:

The application just calls the driver's opening function. We are concerned about how to build a debugging environment, so the demo does not provide more code. The default loading path of the driver is under /dev/, so the file opened by the application is /dev/drv1. The information is printed before and after the system call, and you can intuitively feel whether the printing is successful.

Compile the application

Gcc drv1_app.c –o drv1_app

5.3 Verify the driver

Before debugging the driver, verify the driver first to ensure that the application can call the driver's interface correctly.

Before calling the application system, make sure that the driver module has been loaded into the kernel.

From the above driver printing, it can be seen that after the driver is loaded, "drv1 init....." will be printed first, then load the driver and verify whether the driver is loaded successfully:

As shown in the figure above, first check the dmesg information, there is no information output before the driver is loaded. After the module is loaded, "/drv1 init..." is printed out, indicating that the driver has been loaded successfully, and then we run the application.

We already know from the application code that if the information can be printed before and after the system call, the system call interface open has been executed, and the information is printed correctly from the figure below.

Inside the driver, the call to the open function will also print related information, including the print information when the module is uninstalled, as shown in the following figure

From this, we can see that there is no problem with our drivers and applications.

6 Debug application and driver call interface

The next step is the most critical step, which is how to use the existing environment to debug the application while also being able to debug the driver.

6.1 Configure drive debugging

Restart the target machine, and load the driver after the start (the specific process has been demonstrated again).

Start the client, because our client and the target machine are the same version system. So the kernel modules are also consistent. Enter the path containing the kernel module, gdb debugs the kernel module, as shown in the following figure:

At the same time, set the serial port information (this step is not necessary, because our client has been set up for serial debugging when it is started. If the client is not set for serial debugging when it starts, it can be set here, but the target machine must be set in the serial port debugging mode when the system is started. , Of course, it is not necessary, enough to go around, unless you will set up the serial port after the target machine is booted into the system, otherwise you have to set up the serial port debugging when the target machine starts, how to set the serial port debugging after the target machine starts to enter the system, online I didn’t check it, and I don’t know), set up serial port debugging, as shown below:

After the setting is completed, you need to turn on the target machine debugging mode first (otherwise the client is debugging), and turn on the target machine debugging mode, as shown in the following figure:

After inputting this command and pressing Enter, the target computer will enter a suspended animation state, and the target computer has no response. This is a normal phenomenon, and it is waiting for the link of the debugger. At this time, we will return to the client to start serial debugging as shown below:

The above message will appear after the client starts debugging. At this time, the debugging environment of the client and the target machine is successfully established.

In this way, the driver can be debugged. In order to be able to view the symbol information in the driver, the symbol file needs to be loaded in the client. The client loads the symbol file as follows:

First of all, because what you want to debug is the driver of the target machine, you need to know the load address of the module in the target machine.

Since the target machine is still in the state of suspended animation in the debugging mode, the target machine cannot do any actions yet. In order to allow the target machine to continue to run, it is necessary to allow the target machine to continue to run in the client, so that in the GDB debugging of the client, the target level can continue to run as if debugging a normal application, as shown below:

At this time, the target machine returns to the operating mode, as shown in the figure below

At this time, you can view the load base address of the driver. Two viewing methods as shown below

The load base address can be found in both ways.

At this point, you can load the symbol table in the client. Because the target machine is currently running, you need to disconnect the target machine before you can put the client in the debug mode, so the target machine is disconnected again

In this way, the target computer is in a state of suspended animation, waiting for the client to debug, and the client enters the debugging mode, as shown in the figure below:

Load the symbol file and set a breakpoint for the Open function of the driver, as shown in the figure below

It should be noted that both the driver file and the driver source file in the figure should be located in the client machine (so, I forgot to introduce it before, and copy the driver file and source file compiled by the target machine to the client machine. ), it should be noted that the driver file copied from the client can be placed in any location, and the location of the source file should be consistent with the location of the source file in the target machine, so that source-level debugging can be performed (the source code in the yellow box above The location indicates the location in the target machine, so the source code of the client driver should also be placed in this location, so that the driver debugger can find the symbol location. If the driver is not debugged by source code, only assembly debugging is required without source File), and the load address is the base address of the target machine's driver load. Make no mistake about this.

After setting a breakpoint to the device_open function, the breakpoint points to the driver file source file drv1.c of the target machine, and enter the above figure.

So far, the driver is loaded and the symbol file is loaded. You can start the application and debug on the target machine, and debug the driver on the client machine.

The client let the debugger continue to run, as shown in the following figure:

6.2 Debug application

At this point, the suspended animation state has been removed from the target machine, and the gdb debugging application can be started on the target machine. As shown below.

Needless to say about application debugging. We set a breakpoint at the system call and run until the system call

Single step to 0x4005ac where the open function is called:

When it runs to 0x4005ac, it enters a suspended animation state, and the program can no longer run. This is because the device_open function in the driver of the target machine has been set on the client computer.

6.3 Debugging the driver

Back to the client, the client is already in debug mode as shown below:

This enters the debug mode of the driver. The driver can be debugged just like a normal application. Since the symbol file has been loaded before, you can view the driver source code and step through it, as shown in the figure below:

From the figure, it can be seen that the driver can not only view the source code, but also debug at the source and assembly level. Finally, continue to let the driver run. In this way, the application can run on the target machine again, knowing that the application exits successfully, as shown in the figure:

The formal end of the driver debugger and application debugging mode is as follows:

Run echo g >/proc/sysrq-trigger when the target machine is in debug mode (<gdb>shell echo g >/proc/sysrq-trigger if GDB is also debugging the application on the target machine)
At this time, the client computer is in the debugging mode, and directly exits <gdb> quit in the debugging mode. At this time, the client computer launches the debugging mode.
The target machine also launched the kernel debugging mode. The display content is:

. Driver debugging is over.

So far, the application and driver debugging environment and debugging methods are introduced.

Remarks

1. If you suddenly want to return to the driver debugging mode while debugging the application with gdb, you can directly enter shell echo g> /proc/sysrq-trigger in the application, so that the application debugger is in a suspended animation state, and the kernel debugging The GDB of the device can be debugged. If you want to return to the application debugger gdb, you can directly use the continue command in the GDB of the kernel debugger to return to the application debugger.

2. When debugging a driver or application that contains multiple functions or loop functions, you can also use the finish command to run directly to the end of the function, which is convenient for debugging.

If you need to know more, you can
join the QQ group: 570982169
direct inquiry: 010-68438880