I. Introduction
The first three articles of Baidu APP iOS package volume optimization series focus on the overall solution of package volume optimization, picture optimization and resource optimization. Picture optimization is in-depth optimization from three angles of useless pictures, Asset Catalog and HEIC format. Resource optimization includes Large resource optimization, useless configuration files and duplicate resource optimization. This article focuses on code optimization. In Baidu APP practice, code optimization includes useless class optimization, useless module slimming, useless method slimming, repeated code reduction, tool class slimming and AB experiment curing . In the code optimization process, Mach-O and Link Map need to be analyzed. In the previous article, we have analyzed the Mach-O file. This article first introduces the Link Map file, and then introduces the code optimization scheme in detail.
Review of a series of articles on Baidu APP iOS package volume optimization practice:
"Baidu APP iOS terminal package size 50M optimization practice (1) overview"
"Baidu APP iOS terminal package size 50M optimization practice (2) picture optimization"
"Baidu APP iOS terminal package size 50M optimization practice (3) resource optimization"
2. Detailed explanation of Link Map file
2.1 Introduction
Link Map is an auxiliary file for binary files in Mach-O format. It describes the overall picture of the executable file, including the information of each object file after compilation and their code segment and data segment storage details in the executable file. . Through the Link Map file, we can know the path of the executable file, CPU architecture, target files, symbols and other information, analyze which class or library in the executable file occupies a relatively large size, and slim down the installation package. In addition, we can clearly understand the executable file. The internal structure of the executable file and the positional relationship of each object file in it, which is very helpful for analysis and debugging.
2.2 Generate linkMap file
Xcode -> Project -> Build Settings -> Write Link Map File option value is set to yes, Path to Link Map File is set to the storage location of the specified LinkMap file.
2.3 LinkMap file structure analysis
2.3.1 Basic Information
# Path: /Users/richard/Desktop/demo/DerivedData/demo/Build/Products/Debug-iphoneos/demo.app/demo
# Arch: arm64
Path is the path to the executable and Arch is the architecture type.
2.3.2 Object file list
The Object file list lists all compiled object files, including .o files and dylib libraries. Each object file has a corresponding number. The first column in the above figure is the number that can be used to correspond to a specific class. This number will also be used in the later Symbols section.
2.3.2 Section table
The Section segment table describes the offset position and size of each segment in the final compiled executable file, including the code segment (TEXT) and data segment (DATA). The first column in the segment table is the offset position of the data in the file, the second column is the size occupied by the Section, the third column is the Segment type, and the fourth column is the Section type. About Segment and Section, the previous article explains in detail about Mach-O It has already been introduced, so I won’t repeat it here.
2.3.4 Symbols
The Symbols module gives the specific memory conditions of the methods in the class. in
-
The first column is the starting address of the method, through which we can check the segment table above;
-
The second column is the size, through which the size occupied by the method can be calculated;
-
The third column is the belonging class (.o), the value is the specific number, and the corresponding class can be known by checking the target file list;
-
The fourth column is the method name.
Through the Symbols module, we can analyze the size of the corresponding method of each class.
3. Code optimization
3.1 Useless class slimming
3.1.1 Static detection to obtain useless classes
An Introduction
The so-called static detection is to analyze the linkmap file and the Mach-o file. The __DATA __objc_classlist section in the Mach-o file records the addresses of all classes, and the __DATA __objc_classrefs section records the addresses of referenced classes. The unused classes can be obtained by taking the difference The address, and then symbolize, you can get the unreferenced class information.
- First, get all class addresses, command: otool -v -s __DATA __objc_classlist.
otool -v -s __DATA __objc_classlist /Users/ycx/Desktop/demo.app/demoContents of (__DATA,__objc_classlist) section0000000100008238 00009980 00000001 000099d0 000000010000000100008248 00009a48 00000001 00009a98 000000010000000100008258 00009ac0 00000001 00009b38 00000001
- Second, get the address of the referenced class, command: otool -v -s __DATA __objc_classrefs.
otool -v -s __DATA __objc_classrefs /Users/yangchengxu/Desktop/demo.app/demoContents of (__DATA,__objc_classrefs) section000000010000990000000000 00000000 00000000 00000000000000010000991000000000 00000000 000099d0 00000001000000010000992000000000 00000000 00000000 00000000
-
Third, take the difference, subtract the address of the referenced class from the addresses of all classes, and get the address information of the unused class.
-
Fourth, symbolization, traversing the Linkmap and Mach-O files can obtain the specific class name corresponding to the address information, establish the mapping relationship between the class and the address, and reverse-analyze the class name through the address.
Advantages and disadvantages
Advantages: The detection method is simple and easy.
shortcoming:
-
For a class that has a reference relationship but will not be called at all, it cannot be judged as a useless class. With the version iteration and the handover of new and old employees, the entry of many functions no longer exists, and the related classes will not be called at all, but the reference relationship is still retained. This situation cannot be detected by means of static detection.
-
Static detection cannot be applied to scenarios where classes and methods are called through reflection. Because static detection cannot perceive the runtime environment, it cannot predict which classes or methods will be called by reflection. Therefore, in this case, static detection will not be able to accurately detect useless classes or useless methods.
3.1.2 Dynamic detection to obtain useless classes
An Introduction
We know that the OC class structure has an isa pointer, which points to the original class meta-class of the class. By reading the objc source code, we found a flag flag in the class_rw_t structure of the meta-class, flags 1<<29 bits Identify whether the current class has been initialized at runtime, refer to the source code path:
Values
for
// These are not emitted by the compiler and are never used in class_ro_t.
// Their presence should be considered in future ABI versions.
// class_t->data is class_rw_t, not class_ro_t
#define RW_REALIZED (1<<31)
// class is unresolved future class
#define RW_FUTURE (1<<30)
// class is initialized
#define RW_INITIALIZED (1<<29)
// class is initializing
#define RW_INITIALIZING (1<<28)
// class_rw_t->ro is heap copy of class_ro_t
#define RW_COPIED_RO (1<<27)
// class allocated but not yet registered
#define RW_CONSTRUCTING (1<<26)
// class allocated and registered
#define RW_CONSTRUCTED (1<<25)
// available for use; was RW_FINALIZE_ON_MAIN_THREAD
// #define RW_24 (1<<24)
// class +load has been called
#define RW_LOADED (1<<23)
From this, the method to detect whether a class is initialized looks like this:
#define W_INITIALIZED (1<<29)bool isinitialized() { return getMeta() -›data()-›flags & W_INITIALIZED;}
Advantages and disadvantages
advantage:
-
Not intrusive to line-of-business code;
-
No performance loss;
-
It can be tested against the actual online operating environment;
shortcoming:
Only OC is supported, and Swift, C, and C++ cannot be covered.
3.1.3 Technical solutions adopted by Handbai
Accuracy is improved by combining both dynamic and static methods. We will introduce the detailed plan later in the article.
3.2 Slimming down useless modules
Useless modules that do not have any dependencies, that is, will not be referenced by other libraries, are easier to identify. There is also a type of useless modules that need attention. Although they still have code reference relationships, they are no longer logically used. For example, some outdated event codes (such as the Beijing Winter Olympics, etc.), old codes after business revisions, and abandoned open source libraries. With the iteration of the version, the number of such useless modules will gradually increase.
In Baidu APP package size optimization practice, we use the index of useless class ratio to quickly identify modules that are no longer used. Specifically, this metric is calculated by counting the number of all useless classes in the module, dividing it by the number of total classes in the module, and multiplying by 100%. If the value of this indicator is relatively high, it means that there are more useless codes in the module, which need to be optimized and cleaned up.
After the previous useless class thinning link, we already know all the useless classes in Baidu APP, so we can obtain the specific classes and quantities contained in each module from the LinkMap file, and calculate the proportion of useless classes in each component. If the proportion of a library is 100%, it means that the library is no longer used at all and can be taken offline directly. For libraries that account for more than 90%, you can properly shut down and transfer, delete useless classes, keep only a few useful classes and migrate them to other libraries, thereby reducing the number of components.
The specific classes contained in each component can be obtained by traversing the Object files field in the LinkMap file. You can refer to the following script code to achieve this function:
def find_class(base_link_map_file): link_map_file = open(base_link_map_file, 'rb') reach_files = 0 reach_sections = 0 reach_symbols = 0 files_map = {} while 1: line = link_map_file.readline() line = line.decode('utf-8', errors='ignore') if not line: break if line.startswith("#"): if line.startswith("# Object files:"): reach_files = 1 if line.startswith("# Sections"): reach_sections = 1 if line.startswith("# Symbols"): reach_symbols = 1 else: if reach_files == 1 and reach_sections == 0 and reach_symbols == 0: index = line.find("]") if index != -1: tmpfile = line[index + 2:-1] file = tmpfile.split("/")[-1] frameworkIndex = file.find("(") if frameworkIndex!= -1: frameworkName = file[0: frameworkIndex] className = file[frameworkIndex + 1:len(file)-1] if files_map: if frameworkName in files_map: files_map[frameworkName] = files_map[frameworkName] + " , " + className else: files_map[frameworkName] = className else: files_map[frameworkName] = className link_map_file.close() return files_map
3.3 Useless methods to lose weight
Regarding useless method checking, the commonly used method in the industry is to combine Mach-O and LinkMap files, and analyze the structure of the two to obtain useless methods. First, __objc_selrefs in Mach-O represents the collection of all referenced methods, and __objc_classlist in Mach-O represents the Objective-C class list, and then disassemble its structure to obtain the data in BaseMethods, InstanceMethods and ClassMethods as a collection of all methods , and finally make a difference with the reference method obtained in the first step to obtain the useless method. In order to obtain the package volume income of each useless method, it is also necessary to combine the linkmap file for analysis. This is a feasible solution seen so far, but the problem with this solution is that the accuracy rate is not high, and the actual measurement does not exceed 40%, which is why some large manufacturers give up useless methods to lose weight.
Baidu has made some technological innovations in this regard, solving this industry problem from the perspective of compilation. Simply put, first write an LLVM plug-in to obtain all methods in the static compilation phase, and then obtain the method call relationship. Doing diff can initially obtain useless methods, and then exclude them as follows Special case: classification method identification exception, inheritance chain subclass calling parent class method, implementation of system class protocol method, protocol inheritance chain method identification problem, hard-coded call problem, reflection call problem, notification call method identified as useless method problem, detailed solution we There will be an introduction to the article later.
3.4 Simplify repetitive code
In the process of software development, especially in projects developed by multiple people in different departments, there are codes copied and pasted, and there are some special cases, such as when the project is refactored, in order not to affect the existing logic, the programmer will copy an old code and then use it in the Redeveloping on this basis, it is more likely that there will be a large number of duplicate codes at this time. With the iteration of the version, the above situation will intensify and lead to more and more duplicate codes. Therefore, whether it is to reduce the package size or reduce the historical burden, it is very effective to streamline the duplicate code necessary.
In the Baidu APP package volume optimization solution, we use the open source tool PMD to scan duplicate codes, and then combine the actual situation to logically reconstruct these codes to achieve the purpose of streamlining. PMD is an open source tool, the official website address: https:/ /pmd.github.io/, easy to use, code errors can be known through static analysis, and errors can be reported without running the program. The CPD tool attached to PMD can directly detect duplicate code, supporting Java, C, C++, C# , Groovy, PHP, Ruby, Fortran, JavaScript, PLSQL, Objective C, Matlab, Python, Go, Swift languages, and detection rules can be customized freely, usage reference: https://pmd.sourceforge.io/pmd-5.5. 1/usage/cpd-usage.html
Install with brew command
brew install pmd
Use the following command to do duplicate code detection
//其中，--files 用于指定文件目录，--minimum-tokens 用于设置最小重复代码阈值，--format 用于指定输出文件格式，支持 xml/csv/txt 等格式，这里建议使用 xml，方便查看 //生成的 XML 文件内容如下，根据 file 标签信息就能定位到重复代码位置。pmd cpd --files 扫描文件目录 --minimum-tokens 70 --language objectivec --encoding UTF-8 --format xml > repeat.xml
检测结果如下所示，其中duplication标签中的lines表示重复内容的行数，file标签表示从那一行开始重复及具体重复文件路径，codefragment标签表示重复的代码。
<pmd-cpd> <duplication lines="16" tokens="162"> <file begintoken="16933" column="33" endcolumn="4" endline="28" endtoken="17094" line="13" path="path1"> <file begintoken="23979" column="47" endcolumn="4" endline="26" endtoken="24140" line="11" path="path2" /> <codefragment> *************************** </codefragment> </duplication></pmd-cpd>
3.5 Tools and methods to lose weight
In daily development projects, we all use various tools and methods. The commonly used implementation methods are as follows
-
Implement Category of system classes, such as NSDate, UIImage, NSArray, and NSDictionary classification methods;
-
Independent packaging;
App has a commonTools module in the initial development stage, which is used to store various tools and methods. However, with version iterations and personnel changes, the business becomes more and more complicated. New students do not know that the underlying modules have implemented similar methods. In order to develop It is convenient to integrate another set in its own module, which leads to repeated construction of tools and methods. The main purpose of this module slimming is to excavate and optimize repeated tools and methods. In the practice of Baidu APP, it mainly starts from the following two angles.
- Traverse the LinkMap file and dig out duplicate Categories. Refer to the following script code to realize this function:
def get_files_map(base_link_map_file): link_map_file = open(base_link_map_file, 'rb') reach_files = 0 reach_sections = 0 reach_symbols = 0 files_map = {} while 1: line = link_map_file.readline() line = line.decode('utf-8', errors='ignore') if not line: break if line.startswith("#"): if line.startswith("# Object files:"): reach_files = 1 if line.startswith("# Sections"): reach_sections = 1 if line.startswith("# Symbols"): reach_symbols = 1 else: if reach_files == 1 and reach_sections == 0 and reach_symbols == 0: # files index = line.find("]") if index != -1: symbol = {"file": line[index + 2:-1]} key = int(line[1: index]) files_map[key] = symbol pass link_map_file.close() return files_map
- For non-category tools and methods, check and merge, and finally sink into the unified tool library.
3.6 AB experimental fixed line
In the process of APP development, in order to more effectively verify the actual effect of newly developed functions, we will conduct AB experiments, usually separating the experimental group and the control group, and performing certain operations in the experimental group, while not in the control group. To perform this operation, we will observe the effect of this operation on the experimental variable to determine whether the operation has a significant impact on the experimental results.
For an application with over 100 million daily active users like Baidu APP, there will be about 10 AB experiments in each version, and 240 AB experiments a year. With the long-term version iteration, a large number of AB experiment codes will be accumulated, but in fact there is only one The code of the branch is effective online, and the code of the other branch will not be executed. Therefore, promoting the AB experiment to solidify and removing the code of the invalid branch can achieve the purpose of reducing the package size.
Baidu APP promotes the solidification of the AB experiment in three steps. First, obtain the solidified switch from the AB experiment platform; second, the development tool judges whether the switch corresponding to the experiment exists in the code; third, distribute it to the responsible developer Students solidified the AB experiment and deleted unused codes.
The implementation of the second step is very critical, which is to judge whether a switch still has the corresponding code logic. The solution adopted by Baidu APP is to obtain a set of all possible switch strings, and then judge whether the switch obtained in the first step is in the set. , if the corresponding experiment to illustrate the switch needs to be cured.
In the .h and .m files of Objective-C, we often use the following code to define an AB switch, and then refer to it in subsequent codes.
#define kFaceverifyResourceOptimizeABTestKey @"face_verify_resource_optimize_enable"
For the contents of the .h and .m files of Objective-C, use regular filtering, and the matching expression is @"(.*?)", and you can get all the string sets that may load switches.
In the same way, in Swift files, we usually define an AB switch through the following code, and then refer to it in subsequent codes. The loading method is exactly the same. For Swift files, the regular expression should be "(.?)".
static let verifyResourceOptimizeABTestKey: String = "face_verify_resource_optimize_enable"
Four. Summary
Code optimization is also the highlight of package volume optimization, but compared with image and resource optimization, code modification has a wider impact, and OC language has a variety of dynamic calling methods, which makes code deletion more likely to cause quality problems, so It is relatively difficult to optimize the revenue landing. Baidu APP has unearthed 20M revenue in the process of optimization practice. After two quarters, it only landed about 8M, and the rest still needs to be promoted.
This article first introduces the LinkMap file format in detail, and then makes a systematic explanation of the Baidu APP code optimization scheme (useless class optimization, useless module slimming, useless method slimming, streamlining repetitive code, tool class slimming, and AB experiment solidification). It will introduce its principle and implementation in detail for other optimizations, so stay tuned.
——END——
References:
[1], PMD introduction: https://pmd.github.io/
[2] How to use PMD CPD: https://pmd.sourceforge.io/pmd-5.5.1/usage/cpd-usage.html
[3], XNU source code: https://github.com/apple/darwin-xnu
[4], objc source code: https://github.com/apple-oss-distributions/objc4/tags
Recommended reading:
Construction and application practice of long connection components on Baidu iOS terminal
Baidu App Startup Performance Optimization Practice
Application practice of light sweeping motion effect on mobile terminal
Android SDK security hardening issues and analysis