Android OTA upgrade column guide

Article directory

Android OTA upgrade column guide

Since 2017, I have written some articles related to Android OTA upgrades, mainly introducing the A/B upgrade system introduced from Android 7.1, including the earlier A/B system, to the later dynamic partition, and to the recent virtual A/B partition, and some recent Android OTA topics.

It is no exaggeration to say that this is currently the most systematic and comprehensive column on Android OTA upgrades on the entire network.

Starting from the beginning of this year, I decided to devote more energy to Android OTA. I will analyze the new features of Android Q(10) and Android R(11) in detail, and track the latest Android version as soon as possible within the year.

Please forgive me for changing some columns from free to paid, because I put a lot of energy into each article, and each article tries to explain the cause and effect of the problem clearly. If it is completely free, there is really no motivation to update. Thank you for your understanding and tolerance~~ I am also very grateful to the friends who have subscribed for their support and recognition.

If you have subscribed to the paid column, please be sure to add me on WeChat, and pull you into the "Dynamic Partition & Virtual Partition Column VIP Q&A Group".

Reply "wx" in the backstage of the official account "Rocky Watching the World" to get personal WeChat.

So far, I have written articles related to Android OTA upgrades, including the following series:

Basic introduction: "Android A/B System" series
Core module: "Android Update Engine Analysis" series
Dynamic Partition: "Android Dynamic Partition" series
Virtual partition: "Android Virtual A/B Partition" series
Related tools: "Android OTA related tools" series

Note:

The two series "Android A/B System" and "Android Update Engine Analysis" have been completed, and the rest are still being updated.

"Android Dynamic Partition" and "Android Virtual A/B Partition" are paid columns, the initial price is 29.9, and the price will be adjusted as the content increases

The current price of "Android Dynamic Partition" is 49.9,

At present, the price of "Android Virtual A/B Partition" is 49.9, gradually rising, and finally adjusted to 299.

"Android OTA Related Tools" mainly includes the analysis and discussion of existing Android OTA related tools, as well as publishing some OTA tools written by myself.

This article summarizes all the articles in these series into this article and explains them one by one to facilitate selective reading and systematic learning.

This article is mainly divided into two parts:

The first part is a quick entry, which provides links to all articles, click to jump directly;
The second part makes a brief introduction to the content of each article, and you can choose topics of interest to read according to your needs.

1. Quick Entry

The basic introduction "Android A/B System" series has been completed.

Column address: https://blog.csdn.net/guyongqiangx/category_12140293.html

The core code "Android Update Engine Analysis" series has been completed.

Column address: https://blog.csdn.net/guyongqiangx/category_12140296.html

Dynamic Partition "Android Dynamic Partition" series, being updated.

Column address: https://blog.csdn.net/guyongqiangx/category_12140166.html

Virtual A/B "Android Virtual A/B Partition" series, being updated.

Column address: https://blog.csdn.net/guyongqiangx/category_12121868.html

The upgrade tool "Android OTA related tools" series is being updated.

2. Brief introduction

1. Basic introduction: "Android A/B System" series

The "Android A/B System" series is based on the early code AOSP 7.1.1_r23 of the A/B system. Although the code version is relatively old, some content is no longer applicable after the introduction of dynamic partitions in Android 10(Q), but most of the content or operations Still relevant after Android 11®.

As an introductory series of the entire A/B system, if you have not been exposed to the Android A/B system before, it is recommended to read the articles in this column first to have a general understanding of the A/B system.

Column address: https://blog.csdn.net/guyongqiangx/category_12140293.html

Android A/B System OTA Analysis (1) Overview

Link: https://blog.csdn.net/guyongqiangx/article/details/71334889

Mainly introduced:
- What is an A/B system
- Partitioning of the A/B system
- The state of the A/B system and
- Escalation switching of the A/B system
Android A/B System OTA Analysis (2) Generation of System Image

Link: https://blog.csdn.net/guyongqiangx/article/details/71516768

Mainly introduced:
- The difference between A/B system mirroring and mirroring content under traditional OTA upgrade mode
- A/B system related Makefile variables
- A/B system image file generation, including recovery.img, boot.img, system.img, userdata.img, cache.img, vendor.img
Android A/B System OTA Analysis (3) Communication between the main system and bootloader

Link: https://blog.csdn.net/guyongqiangx/article/details/72480154

Mainly introduced:
- How does the Android main system communicate with the bootloder during traditional OTA upgrades
- Detailed analysis of the boot_control HAL interface of the Android main system communicating with the bootloader in the A/B system
  - Interface definition and implementation of boot_control
    - Implementation of Google Platform Brillo
    - Implementation of edison on Intel platform
    - Implementation of the QualComm Platform
    - Implementation of the Broadcom STB platform
  - boot_control's test tool bootctl
  - The call of boot_control
    - bootloader calls boot_control
    - Android main system boot_control_android calls boot_control
    - update_verifier calls boot_control
Android A/B System OTA Analysis (4) System Startup and Upgrade

Link: https://blog.csdn.net/guyongqiangx/article/details/72604355

Mainly introduced:
- The bootloader reads and checks the process of boot_control
- How does the linux system start and hang on the Android system partition
- How the Android main system and recovery system are started
- Production of A/B system upgrade package
  - Production of full packages
  - Creation of incremental/difference packages
- Upgrade Sample Log for Update Engine
Android A/B System OTA Analysis (5) Client Parameters

Link: https://blog.csdn.net/guyongqiangx/article/details/122430246

Mainly introduced:
- Parameters supported by update_engine_client client
- How the parameters of the update_engine_client client segment are parsed and passed to the update engine service
- How to upgrade using remote and local files
- How to set the offset and size parameters when upgrading
Android A/B System OTA Analysis (6) How to get the offset and size of the payload

Link: https://blog.csdn.net/guyongqiangx/article/details/122498561

Mainly introduced:
- Format of the zip file
- 3 ways to obtain the offset and size of payload.bin in the A/B system upgrade package update.zip
  - Android O starts to automatically generate offset and size data
  - Use zipinfo to manually calculate offset and size data
  - Use the zip_info.py script tool to calculate offset and size data

2. Core module: "Android Update Engine Analysis" series

The "Android Update Engine Analysis" series is based on the early code AOSP 7.1.1_r23 of the A/B system. Although the code version is relatively old, the content of the Update Engine partition processing part is in Android 9§, Andrid 10(Q) and Android 11® and subsequent There are updates one after another, but the core process of Update Engine itself has not changed, so even if you work on the Android version based on dynamic partition or virtual A/B, the core process of Update Engine still has reference value. Compared with the latest code, A The early code for the /B system was simpler and easier to understand.

For example, the overall operation of the entire A/B system, compiling and packaging, using the Update Engine Client to update, and the Update Engine Service to receive data, parse the payload, and write the data to the partition is still the same on Android 10(Q) and Android 11®.

The difference is that when Android 10(Q) starts to update the manifest after receiving the payload, the dynamic partition will first use the data in the manifest to update the partition table (lpmetadata) of the dynamic partition in the head of the super device. After the partition table is updated, according to the partition The data in the table maps some parts of the super device to another slot, and then performs the same update as before based on these two slots.

For the virtual A/B system starting from Android 11®, after receiving the manifest of the payload and starting to update, the virtual A/B partition will first use the data in the manifest to update the partition table (lpmetadata) of the dynamic partition in the head of the super device, and then from The free area of the super device, or allocate space from the data partition for building and mapping another slot.

Therefore, unlike the dynamic partition, the other slot of the virtual A/B does not exist in the system (it actually exists during the dynamic partition), and is mapped through the free space of the super device or the files of the data partition , which is why it is called virtual A/B, because only one set actually exists on the device.

After the virtual device is mapped, the system performs the same update as the previous A/B system based on a real slot and a virtual slot. After the update is completed, the system restarts and starts from the virtual slot. If the virtual slot starts successfully, the data in the virtual slot starts to be merged into the real slot. If the virtual slot fails to start, the system falls back to the real slot.

Therefore, except for the different processing of partitions (virtual A/B will also have a merge operation), the core process of Update Engine is basically the same from Android 7(N) to the latest version.

Column address: https://blog.csdn.net/guyongqiangx/category_12140296.html

Analysis of Android Update Engine (1) Makefile

Link: https://blog.csdn.net/guyongqiangx/article/details/77650362

This article mainly analyzes the Makefile of Update Engine line by line based on the code of Android 7.1.1, so as to obtain the module structure of the entire update engine code, including which libraries and executable files are generated, and which debugging tools are available.

In Android 10(Q), the Android.mk of the Update Engine module is replaced by Androi.bp. The original analysis of Android.mk is no longer applicable, but I still recommend you to read the Androi.bp file to understand the entire Update Engine's module structure.
Android Update Engine analysis (2) Protobuf and AIDL files

Link: https://blog.csdn.net/guyongqiangx/article/details/80819901

Mainly introduced:
- The update_metadata.proto file describing the payload structure, before the introduction of dynamic partitions in Android 10(Q), this file has some small updates, but the key points have not changed much.
- Describe the IUpdateEngine.aidl file of the Update Engine service interface, and analyze the hierarchical relationship between the Update Engine binder service and the interface based on the file compiled from the interface.
Regarding how to compile and transform AIDL, and the implementation relationship with the binder service, it is more complicated. I personally think that it is not necessary to know how to relate it. There is no need to invest too much time in this part.
Android Update Engine Analysis (3) Client Process

Link: https://blog.csdn.net/guyongqiangx/article/details/80820399

Mainly analyze the update_engine_client around the Update Engine client, including:
- Dependencies for the update_engine_client file
- Parameter handling of update_eingine_client
  - How to parse command line arguments
  - How to handle suspend, resume, cancel, reset_status, follow, update and other operations, and further call the Update Engine Service service
Android Update Engine Analysis (4) Server Process

Link: https://blog.csdn.net/guyongqiangx/article/details/82116213

Mainly analyze the Update Engine server process update_engine, including:
- Dependencies for Update Engine server files
- Line-by-line analysis of the entry file main.cc
- How update_engine_daemon is initialized and the binder service is created
- How Update Engine implements callback notification
Analysis of Android Update Engine (5) Action Mechanism of Server Core

Link: https://blog.csdn.net/guyongqiangx/article/details/82226079

Mainly introduced:
- The building blocks of the Action mechanism, including Action, Action Processor and Action Pipe
- Analyzed the processing operations of the Action mechanism, including starting and stopping processing, pausing and resuming processing, and finishing work at the end
- Hierarchical relationship of multiple Actions
  - AbstractAction, Action
  - InstallPlanAction, DownloadAction, FilesystemVerifierAction, PostinstallRunnerAction
- After ApplyPayload receives the upgrade request, how to build ActionPipe and start the upgrade
- How to create and execute actions such as suspend, resume, cancel and resetStatus
Analysis of Android Update Engine (6) Detailed Action of the Server Core

Link: https://blog.csdn.net/guyongqiangx/article/details/82390015

Detailed analysis of the implementation code of the 4 core Actions of Update Engine:
- InstallPlanAction
- DownloadAction
- FilesystemVerifierAction
- PostinstallRunnerAction
Analysis of Android Update Engine (7) FileWriter of DownloadAction

Link: https://blog.csdn.net/guyongqiangx/article/details/82805813

Further detailed analysis of the DownloadAction operation, where FileWrite is the core of the entire upgrade write data:
- When does DownloadAction write received data?
- Detailed structure of Payload file
- How is the received upgrade data written? (Write operation of DeltaPerformer)
  - How to update the data receiving progress information?
  - How to parse the header data of the upgrade package to get the DeltaArchiveManifest data?
  - How to verify the signature?
  - Such as extracting partition information from DeltaArchiveManifest?
  - How do I update the upgrade status information?
  - How to extract the InstallOperation of each partition and check the hash of the payload data?
  - Execute the update operation of InstallOperation in detail
  - How to extract the signature of the upgrade data?
This part of the code is located in the function DeltaPerformer::Write(), and the importance of this function cannot be overemphasized. Therefore, this article is the top priority of the entire Update Engine and is worth reading repeatedly.

Many changes in the following dynamic partition and virtual A/B are Write()located in the processing flow of .

So far, the process analysis on the Android device is finished.
Analysis of Android Update Engine (8) Upgrade package creation script analysis

Link: https://blog.csdn.net/guyongqiangx/article/details/82871409

Mainly introduced:
- How to make full and incremental upgrade packages?
- Line by line analysis upgrade package creation tool ota_from_target_files
- Analyze the payload line by line to generate the script brillo_update_payload
  - How to generate payload file
  - How to generate a hash of payload data and metadata data
  - How to write the signature of the payload and metadata back to the payload file
  - How to extract the properties data of the payload file
If you want to know the detailed upgrade package production process, this article is recommended to read in detail. And at the end of the article, the command line steps of using delta_generator to generate payload are summarized.
Android Update Engine Analysis (9) 6 operations of the delta_generator tool

Link: https://blog.csdn.net/guyongqiangx/article/details/122351084

It mainly introduces the delta_generator tool that generates differential data at compile time:
- Review the steps to generate payload files using delta_genertor
- Main program source code analysis of delta_generator
- 6 operations supported by delta_generator
Android Update Engine Analysis (10) Generate payload and metadata hash

Link: https://blog.csdn.net/guyongqiangx/article/details/122393172

It mainly introduces how delta_generator generates payload and metadata hash:
- How to calculate the hash of payload and metadata
- How to write the calculated hash back to the payload file
- How to manually calculate the hash of payload and metadata on the command line
- How to manually parse the header data of the payload
- How to manually calculate and verify the hash of the payload
Analysis of Android Update Engine (11) Update payload signature

Link: https://blog.csdn.net/guyongqiangx/article/details/122597314

Mainly introduced:
- Summarize the generation and processing of payload data
- How to update payload file signature
Android Update Engine Analysis (12) Verify payload signature

Link: https://blog.csdn.net/guyongqiangx/article/details/122634221

Mainly introduced:
- How to generate the public key used for authentication
- How to verify payload signature using public key
- Payload signature verification process
- How to manually verify signatures using command line tools
- How to use protobuf tools to restore metadata signature data
Android Update Engine Analysis (13) Extracting payload property data

Link: https://blog.csdn.net/guyongqiangx/article/details/122646107

Mainly introduced:
- How delta_generator extracts properties data from payload
- How to manually extract the attribute data of the payload
Android Update Engine Analysis (14) Generate payload data

Link: https://blog.csdn.net/guyongqiangx/article/details/122753185

Mainly introduced:
- Reviewed the process of payload generation
- Source code analysis of delta_generator generating payload
- Introduced the payload_config structure that generates payload data
- How is the payload data finally assembled?
Android Update Engine Analysis (15) FullUpdateGenerator Strategy

Link: https://blog.csdn.net/guyongqiangx/article/details/122767273

It mainly introduces the generation strategy of the full package and the entire calling process.
Android Update Engine Analysis (16) ABGenerator Strategy

Link: https://blog.csdn.net/guyongqiangx/article/details/122886150

It mainly introduces the generation strategy and calling process of the differential package. For the strategy of the full package, the generation process of the differential package is more complicated. It is recommended to follow the article to read the source code in detail.
Android Update Engine Analysis (17) Generation and Application of 10 Types of InstallOperation Data

Link: https://blog.csdn.net/guyongqiangx/article/details/122942628

The 10 operations used to upgrade data generation and writing in the payload file are analyzed in detail:
- ZERO, REPLACE_XZ, REPLACE_BZ and REPLACE operations
- REPLACE_XZ, REPLACE_BZ and REPLACE operations
- ZERO operation
- DISCARD operation
- MOVE and SOURCE_COPY operations
- BSDIFF and SOURCE_BSDIFF operations and IMGDIFF operations

3. Dynamic Partition: "Android Dynamic Partition" series

The Android A/B system introduces dynamic partitions starting from Android 10(Q). In the original A/B system, each partition in the two slots is pre-allocated with a fixed size. After supporting dynamic partition, the two slots of A/B system are included in the super device, and the slot partition table information of A/B system is expressed by the LpMetadata data in the header of the super device. One advantage of this is that the size of the partition in the slot can be dynamically adjusted. The problem is that the partition table information on the super device needs to be processed during the upgrade, and the partition on the super device needs to be correctly analyzed and loaded when the system starts.

The "Android Dynamic Partition" series is based on the "Android A/B System" and "Android Update Engine Analysis" series, based on the android-10.0.0_r47 code, and mainly focuses on the changes after the A/B system introduces dynamic partitions.

Column address: https://blog.csdn.net/guyongqiangx/category_12140166.html

Detailed explanation of Android dynamic partition (1) 5 pictures let you understand the principle of dynamic partition

Link: https://blog.csdn.net/guyongqiangx/article/details/123899602

Mainly introduced:
- The essence of dynamic partitioning is to add, delete, modify and check partition table data LpMetadata
- The principle of the device mapper mechanism that supports dynamic partitioning
- Android dynamic partition and description data layout of LpMetadata
  - Macro: super partition
  - Middle view: metadata data
  - Micro: Gemoetry and metadata data structure
- The core data structure LpMetadata of dynamic partition
- Provides an example of Android Dynamic Partition Mapping
- Compilation and generation of super.img
- Analysis of super.img
- Mapping of super.img
If you want to get a general understanding of the basic principles of dynamic partition composition through an article, then reading this article is enough.
Detailed Explanation of Android Dynamic Partition (2) Introduction of Core Modules and Related Tools

Link: https://blog.csdn.net/guyongqiangx/article/details/123931356

Mainly introduced:
- Core modules related to dynamic partitioning
  - liblp (logic partition lib)
  - libdm (device mapper lib)
  - libfs_mgr (filesystem manager lib)
  - libsparse (sparse image lib)
- Dynamic partition related debugging tools
  - lpmake, lpdump, lpflash, lpunpack
  - dmctl, dmuserd
  - simg2img, img2simg, append2simg, simg_dump.py
If you want to debug dynamic partitions or view dynamic partition information, lpdump and dmctl are the most commonly used tools.
Detailed explanation of Android dynamic partition (3) Dynamic partition configuration and generation of super.img

Link: https://blog.csdn.net/guyongqiangx/article/details/124052932

Mainly introduced:
- The configuration option switch of dynamic partition, including the configuration of native dynamic partition and the configuration of retrofit dynamic partition
- Configuration example analysis of dynamic partition
  - crosshatch (Pixel 3 XL)
  - bonito (Pixel 3a XL)
  - Emulator cuttlefish
- Parameter checking for dynamic partitions
- Parameter handling and tracking of dynamic partitions in Makefiles
  - Where are the dynamic partition related parameters finally set?
- Analysis of compilation and generation of native dynamic partition super.img
  - super.img in dist mode
  - super.img in debug mode
  - super_empty.img
  - build_super_image.py script analysis
If you are not sure how to configure dynamic partitions and how these configurations take effect, please refer to the analysis in this article.
Detailed explanation of Android dynamic partition (4) Processing of dynamic partition in OTA

Link: https://blog.csdn.net/guyongqiangx/article/details/124224206

Mainly introduced:
- Dynamic partition data analysis in payload
- How to Make an Upgrade Package for Dynamic Partitions
- When making an upgrade package
  - How to package dynamic partition data
  - How to output dynamic partition data to payload file
  - Packaging process of dynamic partition data
- When updating
  - How to process the received manifest data
  - How to update the dynamic partition data of the super device
  - Update and map process for dynamic partitions
This article mainly focuses on how to output the dynamic partition information to the payload file when making the upgrade package; how to parse the dynamic partition information from the payload and use it for dynamic partition upgrade when updating and upgrading.
Detailed explanation of Android dynamic partition (5) Why is super.img not generated?

Link: https://blog.csdn.net/guyongqiangx/article/details/128005251

It mainly introduces why the super.img is not generated when compiling the AOSP reference device, although the dynamic partition is enabled
Detailed Explanation of Android Dynamic Partition (6) The underlying mechanism of dynamic partition

Link: https://blog.csdn.net/guyongqiangx/article/details/128305482

Mainly introduced:
- The dual meaning of dynamic partition, dynamic partition in user mode
- device mapper principle
- The principle of linear device mapping as the basis for dynamic partitioning
  - Example of manual partition linear device mapping
- dmsetup create parameter explanation
- Manually use the dmsetup tool to map the super device
It mainly uses some linux command line tools to simulate dynamic partition mapping, so as to deepen the understanding of dynamic partition mechanism.
Android dynamic partition detailed explanation (seven) overlayfs and adb remount operation

Link: https://blog.csdn.net/guyongqiangx/article/details/128881282

Mainly introduced:
- The underlying principle of overlayfs
- How to compile and configure overlayfs under Linux, and how to check overlayfs at runtime
- Why introduce overlayfs in Android system
- Close dm-verity on Android and execute remount process
- Open dm-verity on Android and execute the process of unmuount
- OTA upgrade problem caused by executing remount in Android
  - OTA upgrade fails after adb remount
  - adb remount partition, causing OTA upgrade super partition resize fail
It is mainly aimed at the analysis and summary of a series of problems caused by remount that are often asked in OTA discussion groups.

4. Virtual partition: "Android Virtual A/B Partition" series

Column address: https://blog.csdn.net/guyongqiangx/category_12121868.html

Detailed Explanation of Android Virtual Partition (1) Recommended References

Link: https://blog.csdn.net/guyongqiangx/article/details/128071692
Detailed Explanation of Android Virtual Partition (2) Virtual Partition Layout

Link: https://blog.csdn.net/guyongqiangx/article/details/128167054
Detailed Explanation of Android Virtual Partition (3) Partition Status Change

Link: https://blog.csdn.net/guyongqiangx/article/details/128517578
Detailed Explanation of Android Virtual Partition (4) Compile Switch

Link: https://blog.csdn.net/guyongqiangx/article/details/128567582
Detailed Explanation of Android Virtual Partition (5) Changes of BootControl Interface

Link: https://blog.csdn.net/guyongqiangx/article/details/128824984

5. Upgrade tools: "Android OTA related tools" series

6. Other articles

Linux snapshot (snapshot) principle and practice (1) Basic principle of snapshot

Link: https://blog.csdn.net/guyongqiangx/article/details/128494795
Linux snapshot (snapshot) principle and practice (2) Snapshot function practice

Link: https://blog.csdn.net/guyongqiangx/article/details/128496471

3. Other

If you have already subscribed to the dynamic partition and virtual partition paid column, please be sure to add me on WeChat, note the subscription account, and pull you into the "Dynamic Partition & Virtual Partition Column VIP Q&A Group". I will answer your questions about A/B systems, dynamic partitions, virtual partitions, various OTA upgrades and signatures at a convenient time.

In addition, I have an Android OTA upgrade discussion group, which now has 400+ friends, mainly discussing OTA upgrade topics for various devices such as mobile phones, car machines, TVs, set-top boxes, and tablets. If you are engaged in OTA upgrade work, welcome Join the group to communicate together, please indicate "Android OTA Discussion Group" when adding me on WeChat. This group is limited to Android OTA developers~

Reply "wx" in the backstage of the official account "Rocky Watching the World" to get personal WeChat.