Origin
Due to work reasons, I had to copy the Android source code and start looking. This time, I went directly to Android 10.0. After importing the code into Source Insight, I was filled with emotion. I once felt that I was more familiar with TA than with my own body... After all, I have been touching it for 9 years... Fortunately, Android is still developing rapidly. For example, just by compiling an image that can be loaded by the simulator, I discovered a lot of new things in the process. Therefore, I think I may need a series of short and succinct articles like "Get to know, Android 10 XXX" to make up for the shortcomings in my heart.
Negative pain and positive pain
Let me start with a bowl of chicken soup that is not so poisonous, at least it still works a little bit for me. I recently read a book that mentioned a passage from Wang Guowei.
Wang Guowei believes that one of the most basic characteristics of the human mind is that it is always in motion. You can think east or west, but it is very difficult to think about nothing at all. The heart can only get pleasure when it is fully active. Once there is nothing to do, it will fall into pain-this is a kind of "negative pain", that is, boredom, eating and waiting for death. The opposite is "positive suffering". For example, if you succumb to sleep at night, you work hard, but you don't appreciate the leader, and you get hated by your colleagues. Although it is also uncomfortable, but the heart is always strenuous exercise, conforming to nature, so it still contains the element of happiness. In contrast, "negative pain" is even more unbearable due to the violation of morality. In order to avoid this suffering, people invented all kinds of "distractions" in their spare time after work, resulting in all kinds of hobbies. When Xiang Hongzuo, a poet of the Qing Dynasty, explained why he wrote the lyrics, he said: "If it's not for useless things, how can you send a living with you?"
Wang Guowei's passage comes from his "Study on Human Hobbies", which the author of this book translated into modern Chinese. In fact, the core meaning of the above passage is-people, don't die idle. Idle death is the most uncomfortable. It seems that I said earlier that "I would rather be idle than move around" is against human nature... That being the case, let's get started with Android 10. It will be painful for a while.
Introduction to Android 10 Build System
The Android 10 Build system has a name, it's called Soong. Before Soong, Android's build system also had a name, but it was a bit more earthy, called Make-meaning a build system based on Makefile files. Soong is actually not a stranger, and he has probably emerged in Android 7.
Soong's grand goal is to kill Make. But until Android 10, this seemingly easy little goal has not been fully achieved. why?
The language for writing Makefile files belongs to Languange (domain language) of Domain Specific, which is Turing complete. Therefore, Makefile can be very complicated.
AOSP itself is a very large and complex system. This results in thousands of Makefiles in AOSP. Various dependencies, combinations, etc.
OMG...So it is not easy for Soong to kill Make quickly.
.... Keep looking.
First look at the build directory in the Android 10 source code. It looks like this now:
There are two more important things to understand:
The original make stuff is placed in the make directory. In order to shield users from the difference in switching compilation systems, envsetup.sh and so on are still there.
An increase of the Blueprint , Kati and soong three directories. These three (plus a ninja ) together form the new Soong compilation system of AOSP. The figure below shows several important members of the Soong compilation system.
For users of Android system:
The method of use is the same as before. First introduce build/envsetup, sh, and then execute commands such as m/mm/mmm. But the current m/mm/mmm command will call the corresponding tool in Soong.
Soong will hand over the .bp file to the blueprint tool set for processing. The blueprint tool set is written in go language. The official document describes it as: a meta-build system . The input is a .bp file. Output as .ninja file
Soong handed over the .mk/Makefile to the kati/ckati tool for processing. kati is written in go language, and ckati is written in c++. Kati official documentation describes it as: kati is an experimental GNU make clone . In other words, kati is equivalent to the make command. It's just that kati does not perform specific compilation work, but generates ninja files.
Ninja is a magical thing, it means ninja Momotaro. Ninja itself is not a compiler, it is a tool to call a specific compiler. The real compilation work is still done by the compiler, such as gcc, clang, java, etc. Ninja reads the compilation configuration file at the end of .ninja, and then invokes the corresponding compiler.
Well, the above is the Big picture of the entire Soong compilation system. Now we need to understand some of these small issues.
what is ninja
Ninja is the ninja Momotaro. It originally came from the Google Chrome team. Because the engineer found that the original make system was slow. So consider optimization. As a digression, I still admire Google’s engineering culture. It’s hard to imagine any company in China that would allow a team to optimize a compilation system that works well, and I certainly didn’t know what it could be optimized to .... .
Anyway, ninja's thinking is still very far-sighted. If too many if/else are written in the Makefile, the complexity of this management (including modification) is greatly increased. I actually found this problem in my work. For example, the gradle file compiled by the Android APP is now more complicated. Because of different channels and debug/release reasons, code statements such as if/else also appear in the gradle file. I have considered whether there can be a higher-level tool that can generate gradle files specifically for debug versions, for example. In this way, the gradle file in the debug version does not need to be distinguished by if/else.
Unexpectedly, this idea of mine is consistent with ninja's. In summary, ninja's input file (.ninja) does not have branching logic like if/else. It contains pure compilation rules. As for the distinction between debug/release, it is written by higher-level tools, such as python, go and other languages that are more convenient for debugging.
Therefore, the .ninja file is called the assembly file in the compilation system. Since it is an assembly file, naturally you don't want you to modify the .ninja file, but please adjust the place where the .ninja file is generated. For example, blueprint or kati in the picture above. Therefore, ninja is a relatively low-level thing, it should be embedded in a large compilation system. Then the ninja file is generated by a higher-level tool and handed over to ninja for processing. This is the logic of ninja. Its official website address is: https://ninja-build.org
What is .bp and blueprint
Soong system uses .bp as the compilation configuration file, instead of the previous Makefile. Compared to a complicated Makefile, .bp is a file similar to json. bp is the abbreviation of blueprint, we will introduce blueprint later.
First of all, .bp is much refreshing in appearance, even gradle can't compare with it. The following is what a bp file looks like.
The content of the .bp file is obviously borrowed from Google's internal Bazel compilation system. However, the .bp in Android does not completely copy bazel, but omits the support for branch logic.
In addition, bp is relatively simple to write, but because formal documents are scarce, it is not easy to write it right. For example, what does tidy_checks above mean? What values can be taken? The people urgently need a formal and strict document! ! ! .
Who handles .bp files in Android? The answer is blueprint . blueprint reads the .bp and converts it into a .ninja file. blueprint itself is a more complex tool, it is written in go language. I don’t intend to introduce too much here. Keep in mind that our goal is to "get to know". Its official website address is https://github.com/google/blueprint
What is kati/ckati
Android 10 has not yet converted all .mk files into .bp files. Therefore, a tool is needed to convert .mk/Makefile files to .ninja. This is the new tool kati. Kati has less documentation. To understand it, the README.md and INTERNALS.md under build/kati in the source code are the best materials, especially INTERNALS.md, which is ruthlessly slapped with a little bit of go face
According to the above document, kati is the guy who is responsible for killing the make system. It’s just that it’s now a soong, so kati is downgraded to a tool for converting Makefile files into .ninja. It was in kati's documentation that I realized how complicated Make is.
Kati was first written in go, and then found that the speed was not good, and finally changed to C++. So there are two versions, kati and ckati . Why is the go version of kati not fast? The author said that the main reason is that GC is slow. The AOSP compilation system contains about 1 million variables and will not be released...
Of course, there must be no picture and no truth about the face slap:
Execution of m command
After we source build/envsetup.sh, we can compile the system or a certain module through m, mm or mmm. This usage is the same as before, but the internal implementation is switched to soong. Here is a brief look at how envsetup.sh hooks up with soong. Mainly understand the m command, look at the figure below.
m is a function defined in envsetup.sh, which will execute the script build/soong/soong_ui.bash. And the core of this soong_ui.bash is the following picture:
The last executed is the soong_ui command. The source code of this soong_ui is located in build/soong/cmd/soong_ui/main.go.
In addition,
In the past, lunch needed to be executed when compiling the system. No lunch is actually needed now. Just use m PRODUCT-aosp_x86_64-eng to compile the eng system of this device aosp_x86_64-eng.
In the past, a certain module had to be compiled, such as make libart. Now only need to use m libart. It seems to be a little more convenient.
Tips, you might as well use vscode to configure the go environment, then debug this main.go, and experience soong firsthand...
Other content
envsetup.sh provides more commands, hmm executes them to get all the information. As shown below:
such as,
allmod can get all modules under the compilation target
For example, pathmod libart will get the source directory where libart is located. The answer is art/runtime.
Commands such as allmod and pathmod actually parse the out/module-info.json file-by calling relevant python tools. From this point, it can be seen that the AOSP compilation system is actually a place where various comprehensive technologies are highly concentrated, and it is definitely not enough to write Makefiles in the past.
In addition, there are many little secrets of compilation in AOSP 10 hidden in some secretly generated files, such as the out/.module_paths directory in the following figure :
All mk files and bp file information encountered during compilation are stored in the corresponding .list. Of course, besides this .moduel_paths, there are more secrets waiting for everyone to discover.
Follow-up arrangements
The source code of AOSP 10 has been used for about five days, and I found that there are some knowledge that needs to be understood, such as APEX, ART, etc. Next, we will do a series of "understanding" about these things. Everyone is also welcome to provide some goals, so that our "Know Android 10" series can fly a little further.
last of the last
The result I expect is not what my friends have learned from my books, articles, and blogs and what they have done, but rather, Shennong, I stepped on your shoulders.
I have finished discussing the issue of learning. The following public account will learn and share some basic technologies and new technologies. Your contributions are also welcome. However, as I said in the "Contact Information" of the public account-Zheng Yuanjie has a sentence in the fairy tale King "Wisdom Teeth" that impressed me, to the effect that "I have the right to remain silent, but every word you say is May become my source of inspiration". Therefore, the impact is not one-way, it is possible that I learned more from you.
Collection of essays by Shennong and his friends
Long press to identify the QR code and follow us