From data to large model application, on November 25th, Hangzhou Yuanchuang Conference was held to share development tips.

Author | Liu Junqi

Introduction

In software development, we often encounter some code problems, such as complex logical structure, confusing dependencies, redundant code, and difficult-to-read naming. These problems may lead to a decrease in the maintainability of the code, increase maintenance costs, and also affect development efficiency. At this time, the existing code structure is usually improved and optimized through refactoring. In the refactoring work, most of the work is done manually, which is a time-consuming and error-prone process. For R&D personnel, it is a great challenge to complete the reconstruction of existing functions with quality and efficiency without changing the functions and behaviors of the software. This series uses Python to automate tools to support the practice of the code refactoring process.

The full text is 5529 words, and the estimated reading time is 14 minutes.

In the previous article " Supporting OC code reconstruction practice through Python scripts (1): Module calling relationship analysis ", the focus was on using Python to analyze module calling relationships, and outputting it as a .csv format file and importing it into excel Evaluate the impact of reconstruction and the reconstruction method of each data item, and clarify the specific work and benefits after reconstruction before starting the reconstruction coding work. This will play a key role in the decision-making of human investment in project establishment. Especially when team resources are scarce, it's important to make clear what you're going to do.

At the same time, it was also mentioned in the previous article that at the technical level, a mechanism for communicating data items between modules (referred to as data paths in this series of articles ) is first implemented. The supported data items do not need to be made public, and can also be used by modules in other components. Read and write. Based on the implementation of the data path , connecting the XXXSetting module to the data path can solve the problem of incompatible interface changes caused by changes in data items in the XXXSetting module, and also reduce the number of secondary releases of the upper relying party components. Indirectly improve the R&D efficiency of the R&D needs related to the XXXSetting module data items.

During the process of connecting the data items of the XXXSetting module to the data path , hundreds of data items need to be reconstructed one by one according to the standards of the data path . The manual reconstruction method is costly and has a high probability of error. It is necessary to test each item one by one. The verification cost is high. The Python script we use generates the code that connects to the data path . It can accurately generate the code segment for each data item to connect to the data path , making this reconstruction work zero in the testing and launch phases. Bug.

The content of this article first briefly introduces the basic functions of the data path , and then explains how to use automated tools written in Python. The XXXSetting module is integrated into the data path as a data item provider , and the implementation ideas for automatic code generation are discussed.

01 Data path technology implementation and access

Based on the goal of this configuration data item reconstruction work and the reuse of the data path, the realization goal of the data path is to support the access of different modules. As shown in Figure 1, there are two types of modules related to the data path.

△Figure-1

1.1 Definition and introduction of data item interaction module

Data items are mainly divided into two categories according to the supply and demand relationship, data item provision modules and data item usage modules.

1. Data item providing module : The data provider (such as XXXSetting mentioned in this article) follows the data reading and writing protocol agreed in the system and provides interoperable data item support for the system. There is an n:1 relationship between data items and data item providing modules. The data path supports multiple data items to provide module access, which is a 1:n relationship.

2. Data item usage module : The data user (such as the module in XXXLib in the previous article), based on the capabilities provided by the data path, makes data read and write calls, and obtains and updates the values of the dependent data items. There is a 1:n relationship between the data path and the data item usage module.

1.2 Data path implementation and module introduction

The main implementation idea of the data path is to provide a unified interface, support different data items to provide module access, and manage the access data item providing modules in the data path . When the data item uses the module to read and write data, it is provided according to the data path. interface for data synchronization of data items. It is mainly divided into data item providing module interface layer, data item providing module management and data item reading and writing service module .

1. Data item providing module interface layer : It is agreed that the data item provider wants to implement the data reading and writing capabilities. Only modules implemented in accordance with this standard can be accessed as data providing modules.

2. Data item provision module management : All data item provision modules in the management system provide an interface for registration. The data item provision module can be called to register and access data items that need to be managed in the data path . At the same time, when receiving a data item read and write request, the read and write of the data item associated in the module is distributed.

3. Data item reading and writing service module : Provides stable data reading and writing capabilities, globally accessible, searches the data item providing module according to the key , and calls the interface of the data item providing module to realize the reading and writing of data items.

02 Implementation of data provider’s access to data channel

2.1 The main work of connecting data items to the data path

1. The data providing module accesses the data path : According to the interface layer convention of the data item providing module , there are two main steps to realize the reading and writing of data items:

The information of the readable and writable data items registered with the data path is an array. The array stores the Key of each data item. The naming rule of the Key provides the module class name_data item name for the data item . This part of the code uses Python The script is automatically generated.
For the reading and writing of data items, the data item provides a module to implement the interface for reading and writing numbers. The data item is matched according to the key, and then the data item is read and written. This part of the code is also automatically generated using a Python script.

2. Data items use modules to access the data path : the original direct calling method is changed to the indirect calling method through the data path . The detailed implementation will be introduced in the next article, so stay tuned.

2.2 Organizing data items that need to be reconstructed

In the content of the previous article " Supporting OC code reconstruction practice through Python scripts (1): Module calling relationship analysis ", in the section 3.1.2 Extracting variable types and variable names , after preprocessing, all The type of data item and the name of the data item.

At the same time, combined with the section 3.3.2 Pre-analysis statistical output of data items in the previous chapter , the data items used by multiple components are taken and determined as the data items that need to be reconstructed this time.

Calculate the intersection of these two parts of data to obtain the complete set of data item types and data item names that need to be reconstructed , which can be used when generating data item reading and writing codes. Below is an example of the dataset.

// 数据项类型 数据项名称;
NSString value1;
NSString value2;
BOOL value3;
...

2.3 Data item list generation of data item providing module

The data path itself does not generate data, but only serves as a bridge for data reading and writing. When the data item provider module is connected to the data path, it needs to know which data items the data path supports reading and writing.

The specific implementation is to inform the data path through the agreement of the data item providing module interface layer , which is called by the data path and returns a list of data items supported by the data item providing module . The data structure of the data item list is an array, and each data item in the array is The key, the key generation format provides the module class name_data item name for the data item . The conversion code implemented in Python is as follows:

# 原代码行示例 NSString value1; 参考2.2小节中的代码
matchObj = re.match(r"(.*)\s+(.*);", line, re.M|re.I)
if matchObj:
    # value = matchObj.group(2) -- value1
    key = '         @\"' + className + '_' + matchObj.group(2) + '\",\n'
    # key = '         @"className_value1",\n'  
    # key 按OC的写法，每一行一个key，按NSArray的方式初始化多个key

2.4 Data item reading code generation

A data item that can be read and written through the data path is registered. When the data path needs to read or write the data item, the data item provider implements the reading and writing of the data item in accordance with the standard.

2.4.1 Data item reading code example

The data path supports reading of basic data types. Each data type corresponds to a different reading interface. The data provider implements different types of data reading according to the type of data item. For the same data type, the data provider implements reading according to the type of data item. The corresponding data item value returned by key, the OC code generated by the target is as follows:

// 数据项是 NSString类型
- (NSString *)stringForKey:(NSString *)key {
    if ([key isEqual:@"className_value1"]) {
        return self.value1;
    }
// 如有多个数据项，自动也合到同一个函数
    if ([key isEqual:@"className_value2"]) {
        return self.value2;
    }
    return nil;
}

// 数据项是 BOOL类型
- (BOOL)boolForKey:(NSString *)key {
    if ([key isEqual:@"className_value3"]) {
        return self.value3;
    }
    return NO;
}
// 其它...

2.4.2 Data item reading and implementation generation

Because different types of data items need to be read using different interfaces, during code conversion, the converted code lines will be stored in different data variables according to the type of data items. Each data variable will During initialization, the function header is added, and after the conversion is completed, the function tail is added.

Function header example, taking the data item as NSString type as an example

# NSString 类型的数据读接口，函数头字串由变量保存
funName = '- (NSString *)stringForKey:(NSString *)key {'

Function body example, each data item generates corresponding code, and stores the reading of each data item in turn.

# 原代码行示例 NSString value1; 参考2.2小节中的代码
matchObj = re.match(r"(.*)\s+(.*);", line, re.M|re.I)
if matchObj:
    funbody = '    if ([key isEqual:@\"'
    funbody += 'className_' + matchObj.group(2) + '\"]) {\n'
    funbody += '        return self.' + matchObj.group(2) + ';\n'
    funbody += '    }\n\n'
    # funbody 为转换之后的读取某个数据项的部分代码，匹配key，之后再返回对应的值，增加一些空格及换行，代码按规范对齐
    #    if ([key isEqual:@"className_value1"]) {
    #        return self.value1;
    #    }

Function tail example, taking NSString type as an example, after all data items are converted, add

funEnd = '    return nil;\n'
funEnd += '}\n\n'

Different data types are converted in sequence. After all data items are converted, they are combined into one file. The content of the file can be directly copied to the project and used directly.

2.5 Data item update

2.5.1 Data item update code example

The data path supports the update of basic data types. Each data type corresponds to a different update interface. The data provider uses it to implement different types of data updates based on the type of data item. For the same data type, the data provider updates based on the key. The corresponding data item value, the OC code generated by the target is as follows:

// 数据项是 NSString类型
- (void)updateString:(NSString *)value forKey:(NSString *)key {
    if ([key isEqual:@"className_value1"]) {
        self.value1 = value;
        return;
    }
// 如有多个数据项，自动也合到同一个函数
    if ([key isEqual:@"className_value2"]) {
        self.value2 = value;
        return;
    }
}

// 数据项是 BOOL类型
- (void)updateBool:(BOOL)value forKey:(NSString *)key {
    if ([key isEqual:@"className_value3"]) {
        self.value3 = value;
        return;
    }
}

// 其它...

2.5.2 Data item update implementation generation

Because the types of data items are different, different interfaces need to be used to update the data items. Therefore, during code conversion, the converted code lines will be stored in different data variables according to the type of the data items. Each data variable The function header will be added during initialization, and the function tail will be added after the conversion is completed.

Function header example

# NSString 类型的数据读接口，函数头字串由变量保存
funName = '- (void)updateString:(NSString *)value forKey:(NSString *)key {'

Function body example, each data item generates corresponding code, and stores the reading of each data item in turn.

# 原代码行示例 NSString value1; 参考2.2小节中的代码
matchObj = re.match(r"(.*)\s+(.*);", line, re.M|re.I)
if matchObj:
    funbody = '    if ([key isEqual:@\"'
    funbody += 'className_' + matchObj.group(2) + '\"]) {\n'
    funbody += '        self.' + matchObj.group(2) + ' = value;\n'
    funbody += '        return;\n'
    funbody += '    }\n\n'
    # funbody 为转换之后的更新某个数据项的部分代码，匹配key，之后再返回对应的值，增加一些空格及换行，代码按规范对齐
    #    if ([key isEqual:@"className_value1"]) {
    #        self.value1 = value;
    #        return;
    #    }

Function tail example, after all data items are converted, add

funEnd += '}\n\n'

03 Summary

The content of this article is based on the analysis and conclusion of the previous article. It is a practice to automatically generate data items used by multiple components into the data path code using Python scripts.

Because there are many data items involved, it is necessary to select the data items that need to be reconstructed among all data items, generate a data item key list, and connect to different types of read and write interfaces according to the type of data items. It is difficult to ensure the quality of the process of connecting data items to the data path using manual coding , and it is also difficult to verify the integrity of data item migration.

The use of Python script implementation tools to support the code generation of data items connected to the data path can automatically and accurately generate the code for each data item connected to the data path , which can reduce the investment in R&D and testing manpower and indirectly improve R&D efficiency. .

In the next article, we will introduce how to use Python scripts to support the adaptation of data items using modules to access the data path . Interested students can continue to pay attention.

Welcome to join the Baidu search front-end team and continue to recruit iOS/Android/Web front-end R&D engineers

Resumes are welcome to be submitted to [email protected]

——END——

Recommended reading

Talk to InfoQ about Baidu’s open source high-performance search engine Puck

A brief discussion on search presentation layer scenario technology-tanGo practice

First introduction to search: Baidu search product manager’s first lesson

Application of intelligent question and answer technology in Baidu search

Support OC code reconstruction practice through Python script (1): module calling relationship analysis