AOSP compilation system evolution: technology upgrade from Make to Ninja (Android13)

introduction

Before Android 7.0, the Android compilation system mainly used GNU Make and Android.mk to describe and execute build rules. However, as the project scale expands, the Makefile organization method leads to problems such as increased compilation time.

In order to solve these problems, starting from Android 7.0, Google introduced the Soong build system, which provides the flexibility required for Android build.

The Soong build system was introduced in Android 7.0 (Nougat) and was designed to replace Make. It leverages the Kati GNU Make cloning tool and Ninja build system components to speed up Android builds.

ninja serves as a replacement for GNU Make and introduces the kati tool to convert Android.mk into a ninja build rules file. Subsequently, in Android 8.0, Google further introduced the Android.bp file as a pure configuration file replacing Android.mk.

The Android.bp file is essentially a configuration file based on json format and does not contain complex syntax such as control flow. It generates the corresponding ninja build rule file (build.ninja) through the Blueprint+soong conversion process, and then uses ninja to perform the actual build work.

Through this series of improvements, Google aims to speed up the construction process of Android projects and improve overall efficiency. The Soong build system consists of Kati GNU Make cloning tool and Ninja build system components, which further optimizes the build process of Android projects.

Note, the latest news, Google plans to fully migrate the AOSP compilation system to bazel in the next few years. This migration is currently in the early stages, but you can make some changes to the current build file to get started. Be prepared for Bazel. Once the migration is complete, Bazel will replace all existing build systems and build configuration systems in AOSP (Make, Kati, Soong, Make-based product configuration).

https://source.android.google.cn/docs/setup/build/bazel/introduction?hl=zh-cn

Build process

Compilation system composition

Android's compilation directory is located in /build. In the build directory in the Android 13 source code, there are several important folders:

blueprint: used to process Android.bp files and generate *.ninja files for ninja processing.
Kati: used to process Android.mk files and generate *.ninja files, which are also processed by ninja.
make: This folder contains the original make process, such as envsetup.sh, etc.
soong: The core part of the build system, including soong_ui.bash.

In the build directory, the core folder is linked to make/core, and envsetup.sh is linked to make/envsetup.sh. This is mainly done to shield users from the differences when switching compilation systems.

To sum up, the blueprint and kati folders in the build directory are used to process Android.bp and Android.mk files and generate corresponding *.ninja files for ninja to process, while the make folder still retains the original make process, and the soong file Clips are the core part of the build system.

Compilation process

The figure below shows the compilation process of soong

During the compilation process of Android, the Android.bp file will be collected into the out/soong/build.ninja.d directory. Based on this information, the blueprint tool will generate the out/soong/build.ninja file.

On the other hand, the Android.mk file will generate the out/build-aosp_arm.ninja file by the kati and ckati tools.

Finally, these two ninja files will be merged into the out/combined-aosp_arm.ninja file. This merged ninja file contains the complete compilation rules for Android and is used to perform actual build operations.

Compilation steps

source build/envsetup.sh
lunch aosp_arm-eng # 不清楚想要lunch的项目时，可以先执行lunch，然后在选择项目
make -j8

Compile initialization

envsetup.sh description

The first step in compilation is to execute the following command:

source build/envsetup.sh

Here are some of the things it does

lunch description

After the environment is initialized, we need to start a compilation target. For example, if we want to compile aosp_arm-eng, then we only need to execute the following command:

lunch 3
#或者
lunch aosp_arm-eng

If you don't know what target you want to compile, just execute a lunch command and all targets will be listed. Just press Enter and the aosp_arm-eng target will be used by default.

soong compilation process

When executingrunKatiBuild, an important step is to load the build/make/core/main.mk file. This main.mk file is the main control file of the Android Build system. Starting from main.mk, it uses the include command to include all required .mk files, and finally forms a collection of all compiled scripts in memory, which is equivalent to a huge Makefile. This Makefile may look huge, but it actually consists of three main types of content: variable definitions, function definitions, and target dependency rules. In addition, the mutual inclusion of mk files is also very important.

Make and Soong comparison

Make example

##Android.mk
LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)
LOCAL_MODULE := libxmlrpc++
LOCAL_MODULE_HOST_OS := linux

LOCAL_RTTI_FLAG := -frtti
LOCAL_CPPFLAGS := -Wall -Werror -fexceptions
LOCAL_EXPORT_C_INCLUDES := $(LOCAL_PATH)/src

LOCAL_SRC_FILES := $(call \
     all-cpp-files-under,src)
include $(BUILD_SHARED_LIBRARY)

Soong example

##Android.bp
cc_library_shared {
    
    
     name: “libxmlrpc++”,

     rtti: true,
     cppflags: [
           “-Wall”,
           “-Werror”,
           “-fexceptions”,
     ],
     export_include_dirs: [“src”],
     srcs: [“src/**/*.cpp”],

     target: {
    
    
           darwin: {
    
    
                enabled: false,
           },
     },
}

Tool chain relationship

The relationship between the Android.mk file, Android.bp, kati, Soong, Blueprint and Ninja is as follows:

Android.bp --> Blueprint --> Soong --> Ninja
- The Android.bp file is generated and parsed by the Blueprint tool, and then passed to the Soong tool for processing, and finally the Ninja file is generated.
Makefile或Android.mk --> kati --> Ninja
- The Makefile or Android.mk file is converted by the kati tool to generate a Ninja file.
Android.mk --> Soong --> Blueprint --> Android.bp
- Android.mk files can be converted to Android.bp files through the androidmk tool provided by Soong, but it is only suitable for simple configuration.

During the compilation process, the existing Android.mk file and the existing Android.bp file are converted into corresponding Ninja files respectively. The out/build-<product_name>.ninja file is generated from Android.mk and other Makefile files, while the out/soong/build.ninja file is generated from the Android.bp file. In addition, a smallerout/combined-<product_name>.ninja file will be generated to combine the two as the execution entry.

Ultimately, Ninja files are tools that directly control source code compilation. They contain the rules and instructions required to build Android systems.

Soong module compilation tips

We generally use themm command to compile modules, but if we use mm every time we modify it, then the compilation time will make you doubt your life, because every time When mm is executed, a global check will be performed and the *.ninja global file will be regenerated. This speed is very slow, so the following is recommended: :
If there are changes to Android.bp or Android.mk, or if the source code files are added or deleted, use the mm command.

Use the ninja command when there is no Android.bp or file addition or deletion, and use scripts to encapsulate some shortcut commands, so that you can skip global checks and save compilation time.

Reference link

[Soong Build System] https://android.googlesource.com/platform/build/soong/+/refs/heads/master/README.md
[Android Make Build System] https://android.googlesource.com/platform/build/+/master/README.md
[Build Android platform with Bazel] https://docs.bazel.build/versions /master/bazel-overview.html?hl=zh-cn