"In-depth understanding of Android ART virtual machine" notes

Dex file format, command code

OneClass file corresponds to one Java source code file, and one Dex file can correspond to multiple Java source code files . When a developer develops a Java module (whether it is a Jar package or an Apk):

• On the PC platform, each Java source code file included in this module will generate a corresponding file with the same file name (excluding the suffix).class. These files are finally packaged into a compressed package (i.e.Jarpackage).

• On the Android platform, the contents of these Java source code files will eventually be compiled and merged into a file named classes.dex. However, from the perspective of the compilation process, Java source files will actually be compiled into multiple .class files, and then merged into Jar packages or < /span> file in the package. Apkclasses.dex

What are the benefits of this approach to Dex files? The author can think of at least the following two advantages:

• AlthoughClass files can reduce the redundancy of information such as strings through indexing, there may still be duplicate characters between multipleClass files. String information. Since classes.dex contains the contents of multiple Class files, duplicate information can be further removed.

• If oneClass file depends on anotherClass file, the virtual machine needs to read the other oneSince one file contains all the information, the number of I/O operations will be relatively reduced. Classclasses.dex

Endianness

On the Java platform, the byte order is Big Endian. Therefore, the content of the Class file also uses Big Endian byte order to organize its content. The default byte order of Dex files on the Android platform is Little Endian (this may be because ARM CPU (including X86 CPU) also uses Little endian byte order. Bar).

Overview of the Dex file format

Figure 3-3 shows an overview of the Dex file format. Its various members are explained below.

• The first is the Dex file header, which is very important. The type isheader_item.
• string_ids: Array, element type is string_id_item, which stores information related to strings.
• type_ids: array, element type is type_id_item. Information about the storage type (described by TypeDescriptor).
• field_ids: Array, element type is field_id_item, stores member variable information, including variable name, type, etc.
• method_ids: Array, element type is method_id_item, stores member function information including function name, parameters and return value type, etc.
• class_defs: Array, element type is class_def_item, storage class information.
• data: The important data contents of the Dex file are stored in thedata area. Some data structures will point to a certain location in the file through member variables such as xx_off. Starting from this location, the contents of the corresponding data structure are stored, and xx_off's The location generally falls in the data area.
• link_data: Theoretically, it is a reserved area and has no special function.

Compared with the and Class file formats, the characteristics of the Dex file format are as follows:

• has a file header, which is crucial to correctly parsing the entireDex file.
• There are severalxxx_ids arrays, includingstring_ids (related to strings), type_ids (related to data types), proto_ids (the main function is to describe the parameters and return value types of member functions, and also contains ShortyDescriptor information), field_ids (related to member fields) and method_ids (member function related).
• dataThe area stores most of the content, and the parsing of the data area depends on header and related data items.

Dex script introduction

The number of Dex instruction codes is similar to that of Class instruction codes, neither exceeding 255. However, the insns array (located in the code_item structure) that stores function content in the Dex file is larger than the code that stores function content in the Class file. The array (located in the Code property) is more difficult to parse. One of the reasons is thatThe Android virtual machine does not require an operand stack when executing the instruction code. All parameters either follow the instruction code directly like the Class instruction code, or are stored in registers< /span> In addition, although the official document details the format and meaning of all Dex command codes, it uses a special syntax to describe them, so beginners may find it difficult to understand when reading the official document. For instructions whose parameters are located in registers, the instruction code needs to carry some information to indicate which registers need to be operated when the instruction is executed. .

• insns_size and insns arrays: The length of the script array and the content of the script. The length of the JVM script in the Dex file format is 2 bytes, while the length of the JVM script in the Class file is 1 byte.

From Figure 3-9 we can see:

• The length of the Dex instruction code is still 1 byte, so the number of instruction codes will not exceed 255. But unlike the Class instruction code, the Dex instruction code is mixed with the first parameter to form a two-byte element stored in insns. In this double byte, the lower 8 bits are the instruction code and the higher 8 bits are the parameters. The author calls this double-byte element [parameter + opcode combination].

• [Parameter + opcode combination]ushort The next double-byte element after . [Pure parameter combination], or [Parameter + operation code combination] can be a new set of

• The format of the parameter combination also has requirements. Different characters represent different parameters, and the bit length of the parameter is determined by the number of characters. For example, AA represents a parameter, which occupies 8 bits, and each A represents a 4-bit length.

Tips:
Regarding the parameter format in Figure 3-9, according to the official documentation, readers need to understand the following points.
(1) Different characters represent different parameters. For example, A, B, and C represent three different parameters.
(2) The length of the parameter is determined by the number of corresponding characters. One character occupies 4 bits. For example: A represents a 4-bit parameter, AA represents an 8-bit parameter, and AAAA represents a 16-bit parameter.
(3) represents a special parameter, which has a value of 0. For example, ØØ represents such a parameter. The length of this parameter is 8 bits, and the value of each bit is 0.

In Figure 6-57:

• On the right is the C/C++ source code compiled into target machine code. The target machine code is compiled, and the runtime is only related to the target operating system and related libraries.

• Although Dex/Java bytecode is also compiled into target machine code, its compilation and operation not only rely on the operating system, but also rely on the specific virtual machine implementation. For example, if Dex bytecode is compiled into machine code, it relies on the ART virtual machine.

This truth is easy to understand, but it is also easy to ignore. Many people think that after Java bytecode is compiled into machine code, it can run directly on the OS without any restrictions like those machine codes compiled by C/C++. However, in the process of compiling Java bytecode into machine code, The virtual machine adds some necessary and special instructions, so that the resulting machine code is actually inseparable from the control of the virtual machine during operation. Here we might as well give an example to illustrate.

• Before the garbage collector of the Java virtual machine does object marking, it often sets a flag to indicate that it wants to mark the object.
• Other threads must check this flag frequently when running. When they find that this flag is true, these threads have to wait so that the garbage collector can safely mark the object. Otherwise, while the garbage collector is marking objects, other threads will create objects or change the reference relationships between objects at the same time, which will lead to inaccurate object marking and affect garbage collection.

Obviously, programmers will not actively add this check mark in the code. In fact, this is done proactively by the compiler. It adds tag check instructions in two places, one in the Entry basic block and the other in the loopheader basic block.

The machine code obtained by Java bytecode compilation is inseparable from the virtual machine, and its compilation also depends on the specific virtual machine implementation.

In the Dalvik era, Android adopted the more mature Just-In-Time (JIT) compilation scheme in Java virtual machine technology. JIT will convert the bytecode of hot Java functions into machine code, which can improve virtualization. The operating speed of the machine. After the Android virtual machine was replaced by ART, Google initially radically abandoned JIT and adopted the Ahead-Of-Time (precompilation, abbreviated as AOT) compilation solution. AOT causes the system to try to convert the bytecode of most Java functions in the APK into machine code when installing the application. It makes every possible effort to improve the running speed of the virtual machine with good intentions. However, AOT has brought about a series of side effects that affect the user experience, such as the application installation time being too long and the oat files generated by compilation being too large. To this end, Android has transformed the ART virtual machine in 7.0 (Nougat), using a combination of JIT and AOT compilation solutions to solve the shortcomings of pure AOT while also achieving the expected goals.

ELF file

corresponds to .class and .dex files, and the .oat file is the "executable file" on the Android ART virtual machine . Although Android officials have not clearly explained what oat means, through the relevant source code and some tools we found that it is actually an ELF file customized by Android. ELF files are the basis of oat files and are more difficult. Let’s learn ELF first in this chapter.

Overview

ELF is the abbreviation of Executable and Linkable Format, which is the most common binary file format on Unix (including Unix-like platforms such as Linux). Programmers who develop using Native languages ​​such as C/C++ deal with ELF files almost every day, such as:

• .oThe (or .obj) file obtained after compiling the C/C++ file is the ELF file.
• The dynamic library.so file is an ELF file.
• .oThe binary executable file obtained after linking the file and .so file is also an ELF file.

Tips
.oat is a customized ELF file, so the EFL file is the basis of the oat file, but the content contained in the oat file is closely related to the art virtual machine.

The executable file of the traditional Java virtual machine is the .class file, the executable file of the Dalvik virtual machine is the .dex file, and the executable file of the ART virtual machine is The executable file is the.oat file.

Introduction to ELF file format

As you can see from the previous content, ELF is the abbreviation of Executable and Linkable Format. The "Executable" and "Linkable" in its name indicate that ELF files have two important characteristics.

• Executable: Executable. ELF files will participate in the execution of the program (Execution). Including the running of binary programs and the loading of dynamic library.so files.
• Linkable: Linkable. ELF files are important participants in compilation and linking work. Let’s look at the contents of the ELF file format, as shown in Figure 4-1.

Figure 4-1 shows that if we observe ELF from different angles (View), we will see different information.

• Linking View: Linking View, which observes what an ELF file should contain from the perspective of compilation and linking.
• Execution View: Execution View, which observes what information an ELF file should contain from the perspective of execution (executable file or dynamic library file).

Interventions for special purposes.text和.bssetc.section.

• .text section:Instructions for storing programs. To put it simply, the machine instructions of the program are placed in this section. According to the specification, the sh_type of the .text section is SHT_PROGBITS (the value is 1), which means Program Bits, which is completely determined by the application itself (the machine instructions of the program are of course determined by the program itself), and sh_flags is SHF_ALLOC (when the ELF file When loaded into memory, it means that the Section will allocate memory) and SHF_EXECINSTR (which means that the Section contains executable machine instructions).
• .bss section: bss is the abbreviation of block storage segment. In the ELF specification, the .bss section contains a memory area that will be created by the system when the ELF file is loaded into the process space and the content of this memory is set to 0int a = 0" in main.c, the generated main.o will contain a valid .bss section. . Note that the .bss section does not occupy any file space in the ELF file, so its sh_type is SHF_NOBITS (the value is 8). It only allocates a memory of the size specified by sh_size when ELF is loaded into the memory. The value of sh_flags of .bss must be SHF_ALLOC and SHF_WRITE (indicating that the memory in this area is writable. At the same time, because the area is to be initialized to 0, the memory in this area is required to be writable). What kind of data should belong to the .bss section? If the reader defines a global "
• .data section:.data is similar to .bss, but the data it contains will not be initialized to 0. In this case, the corresponding information needs to be included in the file. So the sh_type of .data is SHF_PROGBITS, but the sh_flags are the same as .bss. Readers can try to define a variable such as "char c = 'f'" in main.c to see the changes in the .data section.
• .rodata section: Information containing read-only data, for example, the strings in printf in main.c fall into this category. Its sh_flags can only be SHF_ALLOC.

Creation and startup of virtual machines

In the Android system, the Java virtual machine is created by the famous Zygote process. Zygote is the creator of the Java world, that is, all Java processes in Android are forked from the Zygote process , The Zygote process itself is started by the init process on the Linux system by parsing the configuration script. Assuming that the target device has a 32-bit CPU architecture, the configuration script file corresponding to the zygote process is system/core/rootdir/init.zygote32.rc. This file describes how init starts the zygote process, as shown in Figure 7-1.

Let’s look at the function of AndroidRuntime. start

The two important functions in the above code that are closely related to starting the ART virtual machine are as follows.

• JnilnvocationInit function: It will load the core dynamic library of the ART virtual machine.
• AndroidRuntime'sstartVm function: After the core dynamic library corresponding to the ART virtual machine is loaded into thezyogte process, this function will start the ART virtual machine.

JniInvocation Init function introduction

Let’s first look at the function of JniInvocation. The code is as follows. Init

As can be seen from the above code, we will take out and save the function pointers of three functions from libart.so:

• The code for these three functions is located in java_vm_ext.cc.
• The second functionJNI_CreateJavaVM is used to create a Java virtual machine, so it is the most critical.

AndroidRuntime startVm function introduction

Next, let’s look at the function of AndroidRuntime. The code is as follows. startVm

As the comments in the above code say, JNI_CreateJavaVM function is not the one that Jnilnovcation Init got from libart.so a>JNI_CreateJavaVM function. Instead, it is defined directly in AndroidRuntime.cpp, with the code shown below.

turned around many times and finally connected with libart.so. Let's take a look at the function in libart.so right away. The code is as follows. JNI_CreateJavaVM

In the above code of , we saw the incarnation of ART virtual machine (i.e. a>). Among them: class in the codeART virtual machine is represented by libart.soJNI_CreateJavaVMRuntimeRuntime

• Runtime::Create will create aRuntime object.
• Runtime::StartThe function will start thisRuntime object, which is to start the Java virtual machine.

Let’s first look at the creation of Runtime objects.

VM and Runtime:
The English word for virtual machine is Virtual Machine (abbreviated as VM). Runtime is another word commonly used to refer to virtual machines in the field of virtual machine technology. Runtime is also translated as runtime. In this book, the author uses virtual machine to represent it. In the ART virtual machine Native layer code, Runtime is a class. There is also a Runtime class in the JDK source code (located under the java.lang package). This Java Runtime class provides some APIs for the entire virtual machine level, such as exit (exit the virtual machine), gc ( Trigger garbage collection), load (load dynamic libraries), etc.

memory map

MemMapIt is an auxiliary tool class that encapsulates operations related to memory map.

• MemMapUsemmap、msync、mprotect and other system calls to complete specific memory mapping, set memory read and write permissions and other related operations. It can create file-based memory maps as well as anonymous memory maps.
• mmapThe return value of system call is just a pointer representing the address, while MemMap provides more member variables to help us use it bettermmapThe mapped memory obtained. For example, eachMemMapobject has a name.
• In addition, for non-x86_64 64-bit platforms, if you want to map memory to the low 2G space address of the process (that is, if you want to use it on non-x86_64 64-bit platformsmmap MAP_32BIT flag), MemMap needs some special processing.

Auxiliary classes related to thread synchronization

ART provides auxiliary classes such as Mutex, ReadWriteMutex, ConditionVariable to implement common functions such as mutex locks and condition variables. synchronization operation. They are defined in mutex.h, and the implementation of different platforms is slightly different. In addition, ART also uses a method called Lock Hierarchies to solve the problem of deadlock that often occurs during thread synchronization due to the different order of using locks (that is, threads should seize mutex locks in the same order, such as locking first Lock mutex A, and then lock mutex B, otherwise a deadlock will easily occur). Under the Lock Hierachies system, a mutex lock can be set with a priority. If a resource requires multiple locks to protect, the low-priority lock can only be preempted after the high-priority lock is obtained. If the order is reversed, the runtime can handle it by reporting an error or program exit. Note that LockHierachies only provides ideas for solving deadlock problems. Readers can combine the ART code and the following informationhttp://www.drdobbs.com/parallel/use-lock-hierarchies-to -avoid-deadlock/204801163to deepen your understanding of it.

OAT file

Introduction to OatFileManager

OatFileManagerFile used to manage virtual machine loadingoat. After dexbytecode is compiled into machine code, the relevant content will be stored in a file named .oat in the file with the suffix name. Let’s first briefly understand the format of OAT files.

Introduction to OAT file format

Figure 7-3 shows part of the OAT file.

Figure 7-3 shows part of the content and format of the OAT file.

• An OAT file contains aOatHeader header structure. Note that thisOatHeaderinformation is not stored in the header of the OAT file. The OAT file is actually a file in ELF format, and the relevant information is stored in ELFIn the corresponding paragraph.
• Where did the Oat file come from? It is an item jarapkdex in a or package named a> files or < /span>),< /span>. file, etc.the other various information in the verification of the file, and the position (offset) of file it corresponds to. The path, item stores some information, such as the item, and the files in all or Each items (so-called can contain multiple or ). (the The process is completed with the help of obtained after compilation and processing package) or files, regardless of whether they are separate . From now on we will collectively refer to them as or files packaged into , etc. are actually , classes.dex, classes2.dexclasses3.dexdexjarapkdex.dexjarapkdex2oatjarapkdexmultidexjarapkdexoatOatDexFileOatDexFiledexdexoat
• OatDexFileFollowing the area is theDexFile area. When generating oat files, of jar or apk (if there are multiple , then the content including , ) will be completely copied to the corresponding area in the Oat file. To put it simply, an item in an OAT file contains the entire contents of a file. By including the contents of the dex file in the OAT file, the ART virtual machine only needs to load the OAT file to obtain relevant information, without the need to open the dex file separately. Of course, this approach also makes the OAT file size larger. classes.dexclasses2.dexclasses3.dexDexFileDexFile.dex
• Looking back nowOatDexFile. EachOatDexFile corresponds to aDexFile item. There is a member in OatDexFile, which is used to specify the corresponding offset of in the OAT filedex_file_offsetDexFile . Of course, OatDexFile there are other similar members (suffixed with offset_) used to indicate the offset of other information in the OAT file.

Finally, the author will briefly talk about the system basic classes contained in the boot oat file. There is a file under frameworks/base, the content of which is the class name that you want to load into the process (defined in JNI format) , these classes are included in different boot oat files. Figure 7-5 shows some of these contents. preloaded-classeszygote

The “…” in Figure 7-5 is an ellipsis added by the author. Since zygote is the first process in the Java world, other APP processes (including system server< a i=4> process) are all derived from zygoteprocessfork. So:

• These classes loaded into the zygote process are also called preloaded classes, so-called preloaded classes.
• According to the Linux process fork mechanism, other APP processes will inherit these preloaded classes after forking from zygote.

Introduction to signal processing and SignalAction

signal processing

In a Linux system, a process can receive signals (Signals) sent from the operating system or other processes. To put it simply, a signal is an event, which means that an event has occurred, and the receiving process can handle these events in a targeted manner.

• The Linux system supports two major types of signals, standard and real-time in POSIX. ART only handles standard signals.
• A signal is uniquely identified by a signal ID, a positive integer. Each signal corresponds to a signal processing method. Signal handling method refers to how a process handles a signal when it receives it. Processes can set specific handling methods for certain signals. If not set, the operating system will use pre-specified methods to handle these signals, which is the so-called default processing.
• A process can block certain signals (block signals). Blocking means that these signals will still be delivered by the operating system to the signal queue of the target process as long as they occur, but the OS will not notify the process for processing. These blocked signals (pending signals) will be stored in the signal queue of the target process. Once the process unblocks them, the OS will notify the process to process them.

If you want to set a signal processing structure for a certain signal, you need to use the system callsigaction, which is defined as follows.

SignalAction class introduction

Now let’s take a look at the encapsulation class provided by ART based on the above content SignalAction. From the above code, we can know that we can be an SignalAction object:

• directly sets a special signal processing function. This step is accomplished with the help of the SetSpecialHandler function.
• Sets a signal processing structure. This step is accomplished with the help of SetAction.

If you use the first method, you must call the SetSpecialSignalHandlerFn function.

Introduction to FaultManager

Initialization of FaultManager

The initialization step of FaultManager involves the constructor and Init function of FaultManager. Let’s first look at the constructor of FaultManager. The code is as follows.

Next, let’s look at the Init function. The code is as follows.

thread

Introduction to Attach function

Let’s look atAttach’s code, as shown below.

Attach contains three key functions internally. Let’s look at the first one, the constructor of Thread.

Thread constructor

The constructor of Thread is not complicated. It mainly completes the initialization of certain member variables. Let's look at the code.

Introduction to Init function

The code for Thread Init is as follows.

InitStackHwm

This function is used to set the thread stack of the thread. Let's first review how the stack space of the next thread is set. On the Android platform, we can create a thread by calling pthread_create. Let’s take a look at the function of pthread_create, focusing on the processing of the thread stack. The code is as follows Show.

Here again, it is specially stated that the stack has only one entrance and exit, that is, the top of the stack, and the bottom of the stack does not move. The thread's stack space is allocated by allocate_thread.

Through the code abovepthread_create, we can know that the creation process of the thread stack on the Android platform is as follows.

• mmap gets a piece of memory, and its return value is the low address of the memory (stack_base).
• Set a certain area of ​​the memory starting from the lower address (defined byguard_size) as inaccessible.
• Get the high address of the memory segment and pass it to the clone system call as the bottom position of the thread stack.

To summarize the above, it can be seen that

• In the ART virtual machine, each Thread object represents a thread.
• Each thread will set the address of its own Thread object in GDT inInitCpu, and save the index of the associated GDT entry to the FS register. The purpose of this is that when this thread executes generatedcode, if you need to call functions provided by the virtual machine such as quick entrypoints, you can use the process shown in Figure 7-8 to jump.
• The necessary prerequisite to achieve this goal is that the content of FS will be switched accordingly as the thread switches. Because there is only one FS, and the Thread object of thread A and the Thread object of thread B will not be the same, the content of FS should be adjusted accordingly as the thread switches. Fortunately, this part of the work is done by the operating system.

Thread FinishSetup

Next, let’s look at the Thread FinishSetup function. The code is as follows

Thread CreatePeer

Before studying the code, the author will briefly introduce some background knowledge related to Java Thread.

• We know that in the Java world, the concept of threads is wrapped in the Thread class. Create a Thread instance and start it to start a thread in the operating system concept.

• From the above description, we can know that a Java Thread instance needs to be associated with a thread in the operating system. If there is only one JavaThread instance but no corresponding thread in the operating system to support it, then this Thread object is just a piece of memory at best.

There is a member variable named nativePeer in the Java Thread class. This variable is the thread of the operating system associated with the Thread instance. . Of course, due to management needs, nativePeer does not directly correspond to information such as thread ID in the operating system, but is set to different information according to the implementation of different virtual machines.

The function belowCreatePeer consists of two parts:

• Creates an instance of Java Thread.

• Associate the calling thread (the thread in the operating system sense, that is, the art Thread object here) to the Java Thread instance =3> members. nativePeer

To put it simply, when the ART virtual machine executes to this place, the operating system thread representing the main thread has been created, but the main thread Thread example in the Java layer is not yet ready. And this preparation work is completed by CreatePeer.

After the above code is executed, we summarize the relevant information in Figure 8-3.

In Figure 8-3:

• The Java ThreadnativePeer member (the data type of the member in parentheses) points to an ART Thread object.
• tlsPtr.opeerBoth and tlsPtr.jpeer in the . We will see later that these two member variables only have different usage scenarios. has type , and opeer has type ART Thread object point to the same Java Thread instance. mirror Object＊jpeerjobject

ThreadList 和 ThreadState

Multiple Java threads are often running in the Java virtual machine. In order to facilitate management, ART designed a ThreadList class to uniformly manage these Java threads. Readers’ attention here:

• Each Java thread corresponds to a Thread object in the ART virtual machine.
• A Native thread can turn itself into a Java thread through theJavaVM::AttachCurrentThread interface. And this will create a corresponding Thread object.

Heap

HeapBitmap related classes

When we create an object using memory allocation methods such as new or malloc, what we get is the address of the memory where the object is located, that is, a pointer. The length of the pointer itself is 32 bits long or 64 bits long depending on the CPU architecture. If 10,000 objects are created, the memory space occupied by the pointers of these 10,000 objects themselves will be considerable. How to reduce the memory space occupied by the pointer itself? The method adopted by ART is very simple, which is to convert the object pointer into an index in a bitmap. Each bit in the bitmap points to a unique pointer. Consider the example in Figure 7-12.

In Figure 7-12:

• The middle box is a bitmap with n bits (if calculated in bytes, the length of the bitmap is n/8 bytes long). The bitmap itself is a block of memory, represented by the base addresspbitmap. There are two boxes above and below it, representing two consecutive pieces of memory. The starting addresses are pbase1 and pbase2 respectively.
• Let’s first look at the memory block corresponding to pbase1. What is stored in this memory is a pointer, p0 points to object 0 (object0), and p1 points to object 1 (object1). The memory length occupied by p0 and p1 itself is sizeof(指针) bytes. Obviously, if there are many objects, memory block 1 will take up a lot of space. The optimization method is very simple, which is to calculate the values ​​of p0 and p1 with the help of bitmap index. For example, the address of the xth object is pbase1 + x * sizeof(指针).
• In addition to saving the pointer of the object, can also use bitmaps to store larger blocks of space. For example, the memory block corresponding to pbase2 can be subdivided into spaces in units of 4KB. Then, the starting position of the yth 4K memory space is pbase2 + y * 4KB.
• Whether it is the memory corresponding to pbase1 or pbase2, if we want to know whether the xth object exists , how to deal with it? The answer is simple, just set the value of the x-th index bit in the middle bitmap box. If the value of the xth index bit is 1, it indicates that the xth object exists, such as is free. pbase1 + x * sizeof(指针)pbase1 + x * sizeof(指针)

art file

.art file format introduction

A jar or apk file containing classes.dex items will actually be generated after compilation by dex2oat There are two result files, one is the .oat file and the other is the .art file. Figure 7-15 briefly shows this process.

In Figure 7-15, when dex2oat is used to compile a jar package or apk, the output file contains two files.

• One is the .oat file. It is worth pointing out again that the contents of classes.dex in the jar or apk will be completely copied to the oat file.
• The other file is the .art file. It is the Image file often mentioned in the ART virtual machine code. The format of the art file is not introduced in the official documentation, and there is very little relevant information. Therefore, it is relatively difficult to learn the art file format. art files and oat files are closely related.

According to the source of the art file (such as which jar package or apk package it is compiled from), Image is divided into boot mirror (boot image) andappmirror (app image).

• The art file derived from an apk is called an App image.
• The art files from the core jar packages under /system/framework in the Android system are collectively called boot images.
  These core jar packages contain the most basic and important classes of the Android system. Note that there are multiple system core jar packages, such as core-oj.jar (oj is the abbreviation of open jdk. Almost all the classes included in jdk are in it), framework.jar, < /span>. boot image files, etc. Since these core classes must be loaded when the ART virtual machine starts, they are called org.apache.http.legacy.jar, okhttp.jar

Why is it called Image?
The author has been very confused for a long time why the art file is called Image. As the research deepens, the author has a more superficial understanding of this issue. First of all, art files are loaded into the virtual machine through mmap. The location loaded into the memory is described in the ImageHeader structure of the art file. Secondly, the content layout of the art file is strictly organized, and these contents will be loaded into different locations in the memory. Finally, after this information is mapped from the file to the memory, it can be directly converted into the corresponding object. It's as if we store the object's information in a file in advance, and then just restore it from the file later.
In addition, generally speaking, compilation for the core library will generate the boot.art image file, while compilation for the app uses dex2oat related options to control whether to generate the corresponding art file.

As far as this chapter is concerned, the ImageHeader in the art file structure is the most critical, and Figure 7-16 shows some of its information.

Figure 7-16 shows part of the art file format, which is divided into three parts: left, middle and right.

• On the left is the structure of the art file. Only the key data structure ImageHeader located at the head of the file is drawn.
• The right side of is the member variables of the ImageHeader structure. magic_The array stores the magic number in the art file format. The value is ['a', 'r', 't', '\n'], and the version_ array is the version of the art file format. number, the value is ['0', '2', '9', '\0']. image_begin_ indicates where the art file expects itself to be mapped to in the memory, and image_size_ indicates how much space is mapped to the memory. sections_ is a very important member in ImageHeader. It is an array. The array size is fixed to kSectionCount (value is 9), and the data type of the array member is ImageSection. The art file contains 9 sections, each section stores different information. ImageSection is used to describe where a section is in memory (based on the offset of image_begin_) and how big the section is. storage_mode_ indicates whether the file content (except ImageHeader) is compressed storage.
• In the middle is the art file loaded into memory. image_begin_ is the starting location of this memory. Pay special attention to the fact that the content of ImageHeader is included in sections_[kSectionObjects] (the value is 0), that is, the section starts from image_begin_). In addition, image_size_ only covers sections_[kSectionImageBitmap-1], and the last element of sections_sections_[kSectionImageBitmap] starts from a position aligned with the page size after image_size_. Combined with the above introduction to HeapBitmap, readers can know that sections_[kSectionImageBitmap] should be a bitmap space.

JNI

JavaVMExt and JNIEnvExt

This section discusses the two most common classes in JNI, JavaVM and JNIEnv. According to the author's introduction to JNI knowledge in the book "In-depth Understanding of Android Volume 1":

• JavaVM represents the Java Virtual Machine in the JNI layer. It works a bit like Runtime. It's just that JNI, as a specification, must set a unified structure, which is JavaVM here. In different virtual machine implementations, the real virtual machine objects can be completely different. For example, the Runtime in the art virtual machine is a well-deserved virtual machine. In addition, a Java process has only one JavaVM instance, In the ART virtual machine, JavaVM actually represents the JavaVMExt class .
• JNIEnv represents the JNI environment. Each thread that needs to interact with Java (whether the Java layer enters the Native layer, or the Native layer enters the Java layer) has an independent JNIEnv object. Similarly, JNIEnv is a data structure specified in the JNI specification, and different virtual machines have different implementations. In the ART virtual machine, JNIEnv actually represents the JNIEnvExt class.

JavaVM follows the process, and JNIEnv follows the thread.

JavaVMExt

Now let’s look at the creation of JavaVMExt object, first review it in Runtime Init Code in .

In Figure 7-19:

• JavaVM is a structure (of course, in C++, a structure is also a class type). When the CPLUSPLUS macro is defined (compiled as C++), JavaVM also has a type alias, JavaVM. Therefore, the real data type of JavaVM is _JavaVM.
• JNIInvokeInterface is also a structure. Among them, the data types of member variables such as JNIInvokeInterface’s AttachCurrentThread and GetEnv are all function pointers (for convenience of writing, Figure 7-19 Their parameters are not shown in ).
• JavaVMThe first member variable of the structure points to an JNIInvokeInterface object.
• JavaVMExt is a class derived from JavaVM.

Tips
In JNI or runtime modules, a object is often referenced through a JavaVM * type pointer. From the above introduction, we can know that the real data type of the object in ART is . JavaVMJavaVMJavaVMExt

JNIEnvExt

JNIEnvExtThe idea of ​​ is similar to JavaVMExt. Let’s look at the code directly.

JNINativeInterface is somewhat similar to JNIInvokeInterface mentioned in the previous section, both are structures containing many function pointers.

Now look atJNIEnvExt, which is a derived class ofJNIEnv. Its creation is done through the Create function.

Let’s focus on the contentgJniNativeInterface.

Summarize

After understanding the above JavaVMExt and JNIEnvExt codes, if the outside world passes their base classes JavaVM and JNIEnv When operating JNI-related interfaces, we can easily find where the real function implementation is. The author summarizes it as follows:

• When operatesJavaVM related interfaces, it appears in the class of the java_vm_ext.cc file. If you need to check JNI, first process it through the corresponding function of the class of . Eventually the related functions of the class will be called. JITcheck_jni.ccCheckJITJIT
• When operates JNIEnv related interface, it actually appears in the JNI class of jni_internal.cc. In the same way, if you need to check JNI, you can also handle it first through the corresponding function of the class of check_jni.cc. CheckJNI

The relevant data structures and APIs in JNI are defined in the header filejni.h. Let’s take a look at the contents.

To summarize the above code, we can see:

• The basic data types in Java correspond to a certain basic data type in the native layer in the JNI layer.
• The reference type in Java corresponds to _jobject (note the underline) and derived classes in the JNI layer. However, JNI users can only use the pointer types of _jobject and its derived classes (that is, JNI users can only use jobject, jclass, jstring and other data types without underscore) to indirectly reference instances of these objects. However, given that the above code clearly defines _jobject as a class without any member variables and member functions. As you can imagine, jobject, jclass, etc. have similar functions to void*.
• Member variables or member functions in Java classes also have corresponding types in JNI. Comparing _jobject, readers can find that _jfieldID and _jmethodID do not even have actual definitions. However, since JNI users can only operate through pointer types (jfieldID and jmethodID are both pointers), compilation will not report an error. However, this also shows that jfieldID and jmethodID have the same effect as void*

So, who are behind these "void*"? The Java Virtual Machine Specification (the author refers to the 7th edition of "Java VirtualMachine Specification") leaves the answer to this question to the implementation of various virtual machines. So, how does the ART virtual machine handle it?

Let’s first look at another set of auxiliary classes commonly used in ART.

ScopedObjectAccess and other auxiliary classes

Figure 8-1 shows the ScopedObjectAccess auxiliary class family.

In the class family shown in Figure 8-1:

• ValueObjectIt is a class that does not have any members and does not allow the compiler to automatically create a constructor.
• ScopedObjectAccessAlreadyRunnable is the key class. It contains three important member variables, Self_ points to the thread object of the current calling thread (type is Thread), env_ points to the current thread JNIEnvExt object, and vm_ points to the JavaVMExt object representing the virtual machine.

We only need to look at the code of ScopedObjectAccessAlreadyRunnable, and the problems left in the previous section will be easily solved.

The above code very clearly shows the corresponding functions of jfieldID, jmethodID and jobject in the ART virtual machine implementation. Specific data type.

• jfieldIDIt is true ArtField *.
• jmethodID It is true ArtMethod *.
• jobject points to a mirror Object object, but what it is needs to be down-converted from mirror Object* to the specified type.

Finally, let’s look at a code snippet usingAddLocalReference, as shown below.

Introduction to commonly used JNI functions

FindClass

FindClass is the API in JNIEnv, used to find the class information of the specified class name. From the introduction in Section 7.7.2, we can know that the real implementation of this function (regardless of checkJni) is located in jni_internal.cc, and the code is as follows.

RegisterNativeMethods

RegisterNativeMethodsIt is used to associate the functions of the native layer with the functions marked as native in the Java layer. This function must be called before use by every JNI library (provided as an so file on the Linux platform).

The content of the above code is relatively simple, but there is a small point that needs explanation, that is, fast jni mode.

• When calling the Java layer from a function into the JNI layer, the virtual machine converts the state of the execution thread from Runnable to Native. If the Java layer related functions are called again in the JNI layer, the state of the execution thread must be changed from Native to Runnable.
• Switching thread states will waste a little execution time. Therefore, some JNI functions that particularly emphasize execution speed can be set to fast jni mode. In this mode, there will be no state switching when executing this native function, that is, the state of the execution thread is always Runnable. Of course, the use of this mode has some impact on GC, so it is best to use it in situations where the execution time is short and will not block. In addition, this mode is currently used by many java native functions inside the ART virtual machine. In order to distinguish it from other native functions, the signature information string of functions using fast jni mode must start with "!" (exclamation mark).

LocalRef, GlobalRef and WeakGlobalRef related functions

Although the code of the JNI layer is developed in native language (C++ or C), some features related to GC in Java are still reflected in JNI.

• The objects created in the JNI layerjobject are local references by default. When the function returns from the JNI layer, the Local reference object is likely to be recycled. Therefore, an object of LocalReference cannot be permanently saved in the JNI layer.
• Sometimes the JNI layer really needs to save an jobject object for a long time. But as the previous rule says, after the JNI function returns, the relatedjobject objects may be recycled. How to save an jobject object that needs to be used for a long time? The answer is simple, just convert this Local Reference object into a Global Reference (global reference) object. And the global reference object will not be recycled by GC, but the user needs to actively release it . Of course, in order to reduce memory usage,the total number of global reference objects that a process can hold is limited.
• If you feel that Global Reference objects are inconvenient to use (for example, you need to actively release them), you can change the local reference object into a so-called weak global reference object. Weak global reference objects may be recycled, so you need to call the JNIEnv before use. 5>Function, compares a weak reference object with . IsSameObjectnullptr

The following are three sets of APIs provided by JNI to operate these three reference types.

We focus on the code ofNewGlobalRef, as shown below.

Combined withJavaVMExt’sAddGlobalRef code, we can know:

• There is only oneJavaVMExt object in a Java process, representing the virtual machine itself.
• There is a member variable in aJavaVMExt object. This variable is a container that can store global reference objects created in the process. globals_
• Every global reference object added to the globals_ container will get a IndirectRef value. Although the type of this value is a pointer type (void *), its value and the address of the mirror Object object to be saved and IndirectReferenceTable It is related to the internal method of element management. The outside world needs to restore a value to the corresponding object through the function of IndirectReferenceTable. GetIndirectRefmirror Object

The above code is the creation of a global reference object, which is completed with the help ofJavaVMExtobject'sAddGlobalRef. Similar to:

• If a local reference object is created, it will be completed using the function of the JNIEnvExt object. AddLocalRef
• EachJNIEnvExt object contains a locals_ member variable used to store the data created in this JNIEnvExt environment Local reference object.

JavaVM and JNIEnv

As mentioned above, JNI is an interface that helps the Java layer and Native layer interact. There are two key data structures in JNI.

• JavaVM:It stands for Java Virtual Machine. Each Java process has a globally unique JavaVM object.
• JNIEnv:It is the meaning of JNI running environment. Each Java thread has a JNIEnv object. When a Java thread performs JNI-related operations, it needs to use the JNIEnv object corresponding to the thread.

JavaVM and JNIEnv are data structures defined in jni.h, which contain function pointer member variables. Therefore, these two data structures are somewhat similar to interface in Java. Different virtual machine implementations will derive actual implementation classes from them. In the ART virtual machine, JavaVM and JNIEnv create code as shown below.

Let’s look at the creation of ARTJNIEnv

Management of reference objects in JNI

Let's first review the creation and destruction process of objects in the Native layer and Java layer.

• Taking C++ as an example, if you want to create an object in the Native layer, you need to use the new operator to first allocate memory and then construct the object. If this object is no longer used, you need to use the delete operator to first destruct the object and then reclaim the memory occupied by the object.
• The Java layer also constructs an object throughnew operations. If it is no longer used later, you can explicitly set the value of the variable holding this object to null (you can also skip this step and let the garbage collection scan and mark the object whether it has been cited). The memory occupied by the object is reclaimed during the garbage collection process.

As the middle layer that interacts between the Java layer and the Native layer, the JNI layer has some features of both the Native layer and the Java layer, especially in the creation and recycling of reference objects.

• is similar to the new operator in C++ that can create an object. The JNI layer can use functions such as JNI NewObject to create a Java-meaning object (reference type object). The object that is New is a reference object of type Local.
• The JNI layer can release the type reference object throughDeleteLocalRef (equivalent to setting the value of the variable holding this object in the Java layer to a>). If is not called, according to the JNI specification, after the JNI function returns, the type object will also be marked and recycled by the virtual machine according to the logic of garbage collection. LocalnullDeleteLocalRefLocal
• In addition to Local type objects, the JNI layer can convert a type reference object with the help of JNI Global related functions into a type object. The recycling of type objects can only be done by explicitly calling the related functions first by the program, and then the virtual machine can use the garbage collection mechanism to recycle them. LocalGlobalGlobalGlobal

Mirror Object、ArtField、ArtMethod

Introduction to key classes

ClassLinker involves many key classes. Knowing them will greatly help subsequent code understanding. Let’s look at the Mirror Object family first.

Mirror Object Family

There is a subfolder in the ART source code folder calledmirror. The classes defined by the code in this mirror subfolder are all located in the mirror namespace. The Chinese meaning of mirror is mirror. So, what is inside and outside this mirror?

It turns out that in the implementation of the ART virtual machine, some Java classes also have corresponding C++ classes at the virtual machine layer, such as the Mirror Object class family diagram shown in Figure 7-20.

Figure 7-20 shows several main classes in the Mirror Object family. in:

• Object corresponds to Java's Object class, and Class corresponds to Java's Class class. By analogy, DexCache, String, Throwable, StackTraceElement, etc. correspond to Java classes of the same name.
• Array corresponds to the Java Array class. For arrays of basic data types, such as int[], long[], such Java classes correspond to PrimitiveArray<int> and in the figure. PrimitiveArray<long>. PointArray in Figure 7-20 can correspond to IntArray or LongArray in the Java layer. For other types of arrays, they can be described by template classes. ObjectArray<T>

Note that IfTable has no corresponding class in the Java layer.

ArtField, ArtMethod, etc.

We know that classes in Java source code can contain member variables and member functions. When the class is compiled and converted by dex2oat, the member variables and member function information of a class will be converted into the corresponding C++ class, that is, as shown in Figure 7-23. ArtField and ArtMethod

Figure 7-23 shows the ArtField and ArtMethod classes, which are used to describe the member variables and member function information of the class. Among them:

• declaring_class_Member variables point to the class that declares the member.
• access_flags_The member variable describes the access rights of the member, such as public or private, etc.
• ArtFieldfield_dex_idx_The of . The type of the elements of the array can be described by array. dex file is the index of the member in the field_idsfield_idsfield_id_item

Similarly, several member variables of ArtMethod are also closely related to the dex file format, such as dex_code_item＿offset_ is The offset of the bytecode corresponding to the function in the dex file, dex_method_idx_ is the offset of the member in the method_ids array in the dex file Index, the data type of the elements of this array is method_id_item.

Let’s take a look at the function of ArtField. If you understand the Dex file format, this code is almost not difficult. GetName

In Figure 8-6:

• DexCache, PointArray, IfTable and Class are all from the mirror Object family. Pay special attention here to IfTable, which has no corresponding class in the Java layer.
• LengthPrefixedArray is a template array container class. The number of its array elements and the size of each element (that is, the value of SizeOf) must be determined at the beginning of creation. The total number of elements is not allowed to be modified during use. The implementation of this class is quite simple and I will not introduce it.
• ArtField and ArtMethod are used to describe member variables and member functions of a class respectively in the ART virtual machine code.

Figure 8-7 shows the data organization structure of four key information. The first is the class_def structure that represents the basic information of the class. The key contents are as follows.

• class_idx: It is actually an index value through which the string representing the class name can be found. In some book terms, they are also called Symbol References. Similarly, superclass_idx represents the class name of the parent class of the class.
• interfaces_off: The data structure it points to is represented by type_list. type_list contains a type_item array. Each member of the array corresponds to the class name of an interface class implemented by the class (the class name can be found through type_idx of type_item).
• class_data_off: The data structure it points to is represented by class_data_item, which contains information about the member variables and member functions of this class.

Let’s look at the class_data_item structure. Among them:

• direct_methodsThe array and the virtual_methods array represent the methods defined by the class and the methods it inherits or implements. According to the description of the dex file format, direct_methods contains all static, private functions and constructors in this class, and virtual_methods includes functions in this class except static, final and the constructor, and does not include functions inherited from the parent class (if this class without overloading it).
• static_fields and instance_fields represent static members and non-static members of the class.

Finally, there are encoded_field and encoded_method structures representing class members. Among them:

• field_idx_diffIs the offset of the index value, through which the variable name, data type, and class name of the class in which it is located can be found.
• method_idx_diff is similar to field_idx_diff. It can find the function name, function signature information (composed of parameter type and return value type) of this member function, and the class name of the class in which it is located.
• encoded_methodcode_off in points to the dex instruction code content corresponding to the member method.

First introduction to ArtField and ArtMethod

Next, we will introduce ArtField and ArtMethod, two data structures that represent the member variables and member methods of the class respectively.

As mentioned above, an ArtField object represents a member variable in the class. For example, a Java class A has a long variable named a. Then, there is an ArtField object in the ART virtual machine to represent this a. However, readers must pay attention, the variable a needs to be used to store a long data (in Java, long data occupies 8 bytes ) where is the space? The member variables of ArtField shown above do not show where to store these 8 bytes. Yes, an ArtField object only represents a member variable of a Java class, but it does not itself provide space to store the contents of this Java member variable.

Tip: When we introduce Class LinkFields below, we will see where the storage space required for this Java member variable is.

Next, let’s look at the member variables of ArtMethod.

An ArtMethod represents a member method in a Java class. For a method (that is, a function), its entry function address is the core information. Therefore, ArtMethod can directly store this information through the relevant variables in the member ptr_size_fields_ structure.

First acquaintance with Class

Next, let’s look at the Class class and first focus on its member variables.

The above-mentioned Class has many member variables, and the following member variables are particularly worthy of readers' attention. We first introduce their situation, and their origins and functions will be analyzed in detail below.

• iftable_: Saves the interface information directly or indirectly implemented by this class. Direct implementation refers to an interface of the class's own implementations. Indirect implementation means that a grandfather class implements an interface on its inheritance tree. In addition, an interface information contains two parts. The first part is the Class object corresponding to the interface class, and the second part is the interface method in the interface class.

• vtable_: Similar to iftable_, it saves all directly or indirectly defined virtual method information of the class. For example, the Object class has 11 familiar virtual methods such as wait, notify, toString, etc. Therefore, any derived class (except the interface class) will contain these 11 methods.

• methods_:methods_ only contains direct methods, virtual methods directly defined by this class and those copied methods such as Miranda (which will be introduced below). Generally speaking, vtable_ contains much more content than methods_.

• embedded_imtable_, embedded_vtable_ and fields_ are implicit member variables. Among them, the first two variables only exist in classes that can be instantiated. Instantiation means that the corresponding class of this class in the Java layer can create an object through new. For example, basic data classes, abstract classes, and interface classes are classes that cannot be instantiated.

Next we introduce three small knowledge points.

Interface default method

Starting from Java 1.8, the default implementation of the interface function can be defined in the interface interface class (its English description is Javainterface default method). Let’s look at some sample code.

Note that the interface default/static method mentioned above is only supported on Java 1.8.

Miranda Methods

Next, let’s get to know Miranda methods. What is this? It turns out that the Miranda method is related to the Miranda rights (Chinese translated as Miranda rights or Miranda rules) in the United States. The Miranda Rules say that if you cannot afford an attorney, the court will provide you with one. Putting it into the Java world, the Miranda rule becomes that if a class does not define a certain function, the compiler will provide you with this function. Why is the Miranda method needed? This is related to a flaw in earlier versions of JavaVM.

Tips
Regarding this automatically generated Miranda method, the information says that it is generated by the compiler, but it does not say whether it is compiled and can be seen in the class file. On the Android platform, the Miranda method is not included in the .dex file. Instead, when the ART virtual machine sets up the virtual function table for the MirandaAbstract class, it will copy the inInterface from the Miranda-Interface interface to its own virtual function table (as you will see when introducing LinkClass below). This is the same as actively declaring the MirandaAbstract class in the code. inInterface function has the same effect.

Marker Interface

Generally speaking, relevant functional functions will be defined in the Interface, and then implemented by the implementation class. However, there is also a type of interface class in the Java library that does not provide any functional functions. You must be familiar with these interface classes, such as the two very common interface classes listed below.

Cloneable and Serializable do not define any functions in the interface class. Such an interface is also called Marker Interface, which is an interface that functions as a marker. It just means that the implementer supports Cloneable or Serializable, and the actual Clone or Serialize function is completed by other functions, such as the following code.

From the Object clone function, we can see that the Marker Interface is indeed just a marker.

In addition, readers will be curious why ART spares no effort to organize all virtual methods of the class into VTable? You know, this is a very important job of the LinkVirtualMethods function called in LinkMethods. We can ask this question in reverse, what will happen if this VTable is not saved in Class? for example:

• Suppose we want to call the wait method of class A (that is, one of the 11 virtual methods of Object). Search the methods_array of class A (do readers still remember it? It saves all methods clearly defined by this class), there is no wait method in it. Yes, it can't be found in class A because class A doesn't define this method directly.

• Then, we should search their methods_arrays all the way up along the derivation relationship or implementation relationship of class A. Obviously, this process is very time-consuming and unacceptable.

Finally, looking back at the above introduction to ArtMethod member variables, we can see that it has a member variable named method_index_. This parameter is very important. Here, its value is briefly summarized as follows:

• If this ArtMethod corresponds to a static or direct function, then method_index_ points to the index in methods_ of the class in which it is defined.

• If this ArtMethod is a virtual function, method_index_ points to the index in its VTable. Note that VTables of multiple classes may contain the ArtMethod object (such as the 11 methods of Object), so it is necessary to ensure that this method_index_has the same value in different VTables. This is also the reason why the three functions in LinkMethods are more complicated. .

As mentioned above, in addition to sizeof (Class), the size of Class also includes the space required by IMTable, VTable (if the class is instantiable) and the space for static variables. If you want to know the location where static variables are stored in a Class, you can use the following function to obtain it.

LinkFields code introduction

Let’s first look at how much memory a Java class and instances of this class require in the ART virtual machine implementation. As shown in Figure 8-13

Figure 8-13 shows a Java Class object and the memory size required by the corresponding instance of this class.

• The left side is the memory size required for Java Class objects. It consists of three parts, first is sizeof(Class). Then (if any) the space required by IMTable and VTable, and finally the space required by the static variables of this class. Note that the reference type comes first, followed by long/double type, int/float type, short/char type, and finally the space required by byte/boolean type variables.

• The right side is the space required for the instance object corresponding to a certain Java class. It contains two parts, first is the size of the parent class object, followed by the space required for non-static member variables. The memory layout is the same as that of static member variables in Class.

Tips
In OOP, one knowledge point we often mention is that the member functions and static member variables of a class are class attributes, that is, they belong to the properties of the class. The non-static member variables defined in the class belong to the corresponding instance object of the class. This knowledge point is confirmed in the memory layout of Java Class and Java Object shown in Figure 8-13.

Next, let’s review an important member variable of ArtField. Its meaning can now be explained clearly.