Analysis of kernel opening mmu on arm linux

The version of the kernel I am using is 4.4. The processor is arm v7a core.

Where virtual addresses are enabled in the kernel

First find the place where virtual addresses are enabled in the kernel, the code is in arch/arm/kernel/head.S.

/*
 * Enable the MMU.  This completely changes the structure of the visible
 * memory space.  You will not be able to trace execution through this.
 * If you have an enquiry about this, *please* check the linux-arm-kernel
 * mailing list archives BEFORE sending another post to the list.
 *      
 *  r0  = cp#15 control register
 *  r1  = machine ID           
 *  r2  = atags or dtb pointer 
 *  r9  = processor ID         
 *  r13 = *virtual* address to jump to upon completion                                                                    
 *      
 * other registers depend on the function called upon completion                                                          
 */     
    .align  5
    .pushsection    .idmap.text, "ax"
ENTRY(__turn_mmu_on)           
    mov r0, r0
    instr_sync
    mcr p15, 0, r0, c1, c0, 0       @ write control reg 
    mrc p15, 0, r3, c0, c0, 0       @ read id reg                                                                         
    instr_sync
    mov r3, r3
    mov r3, r13
    ret r3
__turn_mmu_on_end:             
ENDPROC(__turn_mmu_on)
    .popsection

Observe this code and find that there are two cp15 operations in it. "mcr p15, 0, r0, c1, c0, 0" is to write r0 to register 1 of cp15. Here is a link related to cp15 operation for memorandum. http://blog.sina.com.cn/s/blog_858820890102v1gc.html
From here, the kernel has entered the era of virtual addresses. At this time, the value of the PC is the address of the instruction mcr p15, 0, r0, c1, c0, 0 and then +4. At this time, the PC is a virtual address, which physical address will it be converted to? The answer is the physical address of the next instruction! If it were not for this result, the kernel would fetch an unexpected instruction, causing the program to crash.

virtual address = physical address

How does the kernel achieve this?
Take a look at __create_page_tables

/*
 * Setup the initial page tables.  We only setup the barest
 * amount which are required to get the kernel running, which
 * generally means mapping in the kernel code.
 *
 * r8 = phys_offset, r9 = cpuid, r10 = procinfo
 *
 * Returns:
 *  r0, r3, r5-r7 corrupted
 *  r4 = physical page table address
 */
__create_page_tables:
    pgtbl   r4, r8              @ page table address
    ...
        /*
     * Create identity mapping to cater for __enable_mmu.
     * This identity mapping will be removed by paging_init().
     */
    adr r0, __turn_mmu_on_loc
    ldmia   r0, {r3, r5, r6}
    sub r0, r0, r3          @ virt->phys offset
    add r5, r5, r0          @ phys __turn_mmu_on
    add r6, r6, r0          @ phys __turn_mmu_on_end
    mov r5, r5, lsr #SECTION_SHIFT
    mov r6, r6, lsr #SECTION_SHIFT
    ...
ENDPROC(__create_page_tables)    

adr r0, __turn_mmu_on_loc This command loads the physical address of the label __turn_mmu_on_loc into r0, and then reads 3 words from this memory into r3, r5, and r6. So what is stored in these 3 words?
It can be known from the System.map file that the virtual address of the __turn_mmu_on_loc label is c0008138
c0008138 t __turn_mmu_on_loc

From the result of objdump of vmlinux, we can know that there are 3 addresses stored in this place, which are the virtual addresses of __turn_mmu_on_loc, __turn_mmu_on, __turn_mmu_on_end,
c0008138 <__turn_mmu_on_loc>:
c0008138: c0008138 andgt r8, r0, r8, lsr r1
c000813c : c0008280 andgt r8, r0, r0, lsl #5
c0008140: c00082a0 andgt r8, r0, r0, lsr #5

Then the meaning of sub r0, r0, r3 @ virt->phys offset is very clear, that is, to find the offset between the physical address and the virtual address of the label __turn_mmu_on_loc. Then find the physical address of __turn_mmu_on according to this offset, and then put this physical address into the address mapping table. In this way, when the MMU starts to work, the physical address and the virtual address are the same, which will not cause the problem of illegal addresses obtained after PC+4.

Transfer to 0xc0000000 virtual address

Before executing __turn_mmu_on, the value of r13, which is sp, has been set to the virtual address of __mmap_switched. In the final stage of __turn_mmu_on, the virtual address of __mmap_switched is loaded into the PC through the ret r3 command, and when the ret command returns, it will directly return to the __mmap_switched label, thus embarking on the golden road of virtual addresses. After this, the kernel runs in the virtual address space above 0xc0000000. So when is r13 loaded with virtual addresses? In the stext section, __mmap_switched is loaded into r13 before __mmu_enable is called.

     *
     * The processor init function will be called with:
     *  r1 - machine type
     *  r2 - boot data (atags/dt) pointer
     *  r4 - translation table base (low word)
     *  r5 - translation table base (high word, if LPAE)
     *  r8 - translation table base 1 (pfn if LPAE)
     *  r9 - cpuid
     *  r13 - virtual address for __enable_mmu -> __turn_mmu_on
     *
     * On return, the CPU will be ready for the MMU to be turned on,
     * r0 will hold the CPU control register value, r1, r2, r4, and
     * r9 will be preserved.  r5 will also be preserved if LPAE.
     */
    ldr r13, =__mmap_switched       @ address to jump to after
                        @ mmu has been enabled
    badr    lr, 1f              @ return (PIC) address
#ifdef CONFIG_ARM_LPAE
    mov r5, #0              @ high TTBR0
    mov r8, r4, lsr #12         @ TTBR1 is swapper_pg_dir pfn
#else
    mov r8, r4              @ set TTBR1 to swapper_pg_dir
#endif
    ldr r12, [r10, #PROCINFO_INITFUNC]
    add r12, r12, r10
    ret r12
1:  b   __enable_mmu
ENDPROC(stext)

So far, the task of opening the MMU and jumping to the virtual address space during the Linux startup process is completed.