Introduction : This article first introduces the open source ecosystem of Baidu Mini Programs, and then encounters vendor browser cooperation scenarios in the development path, which leads to the pain points and challenges of Mini Program distribution guarantee, and then from distribution channels, capability detection, capability matching and capability intervention. The specific scheme and design are discussed in detail, and finally the design ideas and thinking in the construction road are summarized.

The full text is 5543 words, the estimated reading time is 14 minutes

1. Business introduction

1.1 Overall Introduction of Mini Program Open Source

Baidu has created a truly open source and open ecosystem from the first day of making smart applets. Our vision is: to define the best experience in the mobile era, build industry standards for smart applets, break down silos, and jointly build an open source, open, and prosperous small program. Program industry ecology. There are three types of ecological players in Baidu Smart Mini Programs: alliance partners, developers and users. For alliance partners, to meet the partners' demands for ecological co-construction, traffic sharing, and business win-win, so that the missing scenes in the partner's app can be efficiently supplemented, and the partner's app users can use the time to share the monetization income of the applet. For developers, it satisfies the developer's requirements for one-time development and multi-terminal operation, which makes the developer's traffic acquisition more convenient and efficient, and increases the developer's business income. For users, it can satisfy users' demands for one-stop service without jumping out. Baidu Search is a very important distribution portal for small programs. By building a unified small program industry ecology, it improves the user experience and service closed loop in Baidu's search ecology.

How does the open source architecture of Baidu's smart applet support the operation of the entire alliance ecosystem?

As shown in the figure, the entire business architecture of the Mini Program Open Source Alliance is divided into four layers. Including: alliance host layer, external alliance enabling layer, intermediate service layer, infrastructure layer.

Alliance host layer. Baidu smart applet has access to a very rich alliance hosts, including OPPO browser, VIVO browser, Xiaomi browser, iQiyi, Baidu Express Edition and so on.
External alliance enabling layer, Baidu Smart Mini Program Open Source Alliance mainly provides hosts with access capabilities, operating capabilities, and resource output capabilities through the open source alliance platform to support the entry, configuration, integration, access detection, and small program information of members of the open source ecosystem. , material information, sub-commission, order information, operation and other demands. After the host obtains the data of the applet and the material through the resource output module, it can explore the landing scene and distribute the resources in the infield of its own app.
The internal intermediate service layer is mainly a linking service, encapsulating basic services, and supporting the acquisition of basic business data by the external alliance enabling layer. Mainly include: Mini Program open source host management, capability asset management, distribution management, order and commission management, data management, etc.
The infrastructure layer mainly includes the implementation of small program open source framework, data service, small program/material service, package service, and security reinforcement.

1.2 Introduction of Mini Program Open Source and Vendor Cooperation

With the continuous development of the Baidu Mini Program Open Source Alliance, it has also attracted leading mobile phone manufacturers. Mini Program Open Source and manufacturers have jointly built a wealth of business scenarios, including not only the browser scenarios of the manufacturers, but also various scenarios such as desktop lock screen, negative one screen, and global search. In the construction of such a variety of scenarios, the common challenges we face are: how to ensure the quality of online distribution of manufacturers, and how to build a full-process distribution control channel from the manufacturer's browser to Baidu search to the applet, so that accurate and real-time control can be achieved. Small program distribution for vendor browsers.

How exactly do we think about and build? It will be described in detail below.

2. Introduction to the overall technical construction ideas

2.1 Key technical points for open source distribution of small programs

1. Background introduction

The core object of open source assurance is the host and the applet, and the core purpose is to ensure the distribution quality of the applet on the host.

Why are there distribution quality issues? After entering the open source, the host needs to integrate the open source SDK and develop related capabilities. After these processes are completed, the host obtains the corresponding host capabilities. The applet is developed based on the swanjs development framework. The bottom layer of the swanjs runtime framework also relies on the open source SDK to run. Therefore, after the applet is developed, the applet itself also obtains or relies on related capabilities. Because different hosts support different capabilities, the same applet may encounter a certain host that does not support a certain capability used by the applet when the same applet is distributed on different hosts, and thus the applet cannot be opened normally on the host. Or downgrade to H5 to open, resulting in impaired user experience.

Therefore, it is necessary to have an open source guarantee scheme to detect the mismatch of capabilities between the host and the applet in advance, so as to ensure that the system can freely control the whole process of distributing the applet on the host.

2. Technical points

In order to realize the construction of the distribution channel of the applet on the host, the core involves three major sections: the opening of the terminal process, the opening of the capability and grading flow, and the opening of the final distribution flow, as shown in the following figure:

1) The end process includes the host-end process and the applet-end process.

a. Host-side process:

When the host is settled in open source, the host needs to integrate the open source SDK and development after the successful settlement. After the host development is completed, the host app can be packaged and released; at the same time, the host package needs to be tested by the host before the release. capacity support.

b. Mini program process:

The developer develops the applet. After the development is completed, the operation of the applet package version is released online. After that, the applet open source guarantee module will subscribe to the applet release action, and at the same time, pull the applet package information to perform the applet capability scanning test, and produce a small program. Program capability usage.

2) Capability and grading flow

After the host and the applet complete the access and capability scanning respectively, the open source assurance system can analyze the results of the host and the applet capability scan respectively, and produce and store the final host and applet capability usage. At the same time, the open source security system can rank the host and the applet respectively based on the host's ability and the applet's ability, calculate which level the host and the applet can currently reach, and mark the host and the applet with rating marks respectively.

3) Distribution stream

It is mainly implemented by the applet distribution terminal. It mainly calculates whether the applet meets the distribution conditions on the host based on the distribution strategy configured by the user in the open source security system.

3. Explanation of terms

SWANJS : Small program front-end runtime framework.

Capability : Capability refers to the general designation of various APIs, components, frameworks and other system capabilities encapsulated and supported in Baidu APP or open source SDK in the Mini Program open source system. For details, please refer to the Mini Program Developer Documentation https:// smartprogram.baidu.com/docs/develop/api/apilist/ For more information, there are currently more than 1,000 open-source Mini Program capabilities.

CTS test : CTS stands for Compatibility Test Suite, a compatibility test suite; it provides a set of compatibility tests to ensure the stable operation of the smart applet in the host APP. Before the host APP goes online, it needs to pass the CTS test.

AST scan : Abstract Syntax Tree scan is a source code scanning tool for applet applications. It scans and analyzes the source code of the applet by means of the abstract syntax tree. Currently, it supports the capabilities and capability attributes used by the applet. , capability parameters, capability return values, and capability-related critical paths.

L-level : According to the use of the ability on the applet side, some basic capabilities, characteristic capabilities and low-frequency capabilities are distinguished. At the same time, the host can implement on-demand according to these capability standards. The levels are divided as follows:

L1: The applet comes with it, without the additional cooperation of the host (framework implementation).

L2: It is recommended to complete the capabilities to ensure the basic experience, and the internal matrix must be realized (including the necessary capabilities including the unique capabilities of the hands).

L3: Featured capabilities, strong dependent capabilities are met, and the host implements on demand.

L4: Low-frequency capability, does not pursue satisfaction, the host implements it on demand.

2.2 Host Capability Detection Mechanism

1. Use the host tool for CTS proficiency test

After the host integrates the open source SDK and completes development, it is packaged. The host tool can be downloaded from the open source host platform to test the CTS capability of the package body. , CTS supports the host to perform full-capacity automatic tests and supplementary tests. The supplementary test can perform batch tests on the abilities that fail in the full-capacity test, and upload the automatic test report after the test is completed. There are also some capabilities that cannot realize CTS automatic testing, and need to perform manual testing. After the testing is completed, the host uploads the manual testing report on the mobile phone.

2. The host tool uploads the test report

After the host executes the automatic test and manual test, it needs to upload the automatic test report and the manual test report to the open source security system respectively. At the same time, because the same host version may repeat the test process multiple times, multiple test reports will be generated, so the host needs to Select a final automatic test report and a manual test report respectively, so that the open source assurance system can parse the final confirmed report.

3. Report analysis output host capability

After receiving the host's confirmation report, the open-source system starts to perform report analysis, calculates the host's capability support, and produces the final list of host capabilities and stores them. The host ability level is obtained and marked; the output host ability and host level will be used in the distribution strategy when the subsequent host distributes the applet.

Capability testing service availability guarantee:

After the user completes the test report confirmation action, the open source assurance system performs report parsing and capability analysis related operations in real time, and completes it in one process. If the host capability detection fails due to abnormal system execution, the open source assurance system is designed to retry the task to get the bottom of it. , to ensure that all hosts can complete the ability test.

2.3 Mini Program Capability Detection Mechanism:

1. Mini Program Package Publishing and Publishing News Subscription

The applet capability detection channel depends on the developer to complete the whole process of applet development and release in the developer center. After that, the open source security system is responsible for subscribing to the applet release behavior and automatically pulling the package body information of the applet, and finally pushes it to the AST applet capability Scan tool to perform scanning operation.

2. Mini Program Capability Scan

The core of applet capability scanning relies on the AST applet capability scanning tool, which mainly scans and analyzes the package body code of the applet through the abstract syntax tree, and supports scanning which open source capabilities are used by the applet, the parameters corresponding to the capabilities, and the return value. Values and attributes, as well as the critical path of the Mini Program involved in the capability. After the above information is successfully scanned, the result will be sent back to the open source assurance system.

3. Analysis of Mini Program Capability

After receiving the AST applet capability scan results, the open source assurance system will analyze the scan results, extract and store the capabilities and other key information used by the applet, and trigger the applet rating task to perform the applet level calculation. , output and store the level of the current mini-program package version for use in subsequent assurance processes.

Capability testing service availability guarantee:

When the open source security system subscribes to the release information of the applet, it starts to push the AST scanning and subsequent processes. The whole process is a real-time stream, but there are multiple sub-processes such as cross-system interaction. In order to ensure that the applet capability detection process must be able to All the executions are successful. The open source assurance system is designed to retry the task to get the bottom line, and re-push AST scanning and capability analysis for the applet records that fail the ability test. At the same time, the intermediate steps are guaranteed to be idempotent, ensuring that all applet programs can complete the ability test.

2.4 Distribution matching mechanism:

In order to obtain the compatibility between the host App and the applet, the applet open source business system has established a grading mechanism. The applet and the host app are rated as L1 to L4 from low to high, and based on the L level, which applet can be distributed is determined. Specific steps include:

1. Capability analysis of the host app and the applet: perform the capability test of the host app to be matched to obtain all the capabilities supported by the host app; scan the capability of the applet to be matched to obtain all the capabilities required for the applet to run.

2. Rating of host apps and mini programs: According to the importance of capabilities, the capabilities of the host apps and mini programs are graded and aggregated to assess the ratings of apps and mini programs.

3. The distribution matching between the host App and the applet: Determine the matching applet set according to the host App level. The L1-level host App can only match the L1-level applet, and the L2-level host App can match the L1 and L2. For Mini Programs, the highest level of the host App is L4, that is, it has the ability to run all Mini Programs.

2.5 Distributing intervention mechanisms:

In order to obtain the distributable applet on the host App, the applet open source business system combines the internal distribution matching mechanism and the administrator intervention mechanism to determine the set of applet that the host App can open. Specific steps include:

1. Mini Program administrator control: The first level of Mini Program distribution control, based on the administrator's privatization of the Mini Program, App blacklist, and the request for the Mini Program to go offline, to ensure that the App cannot be opened privately, blocked, or downloaded. Line applet.

2. L-level matching mechanism control: The second-level applet distribution management and control, through the L-level matching mechanism, obtains the set of applets that can run normally, so that by default, the host App can only open the L-level matching applet.

3. Intervention by the host App administrator: In addition to the matching mechanism, the App administrator can submit a list of the Mini Programs to be distributed to express distribution demands according to the requirements of the Mini Program running data and the application scenarios of the App.

3. Summary and thinking

Breaking the silos and building an open source, open and prosperous applet industry ecology has always been the goal of the applet open source alliance. The improvement of the basic capability framework, the exchange of data from all walks of life, and the friendly and active cooperation model between developers, hosts and Baidu are the keys to accomplishing such goals. The planning of the technical architecture must be carried out around the common goals and the demands of partners, with practical tracking and service as the concept, and the use of reasonable technical means to empower products and ecology, so as to reflect the ecological value. In the construction of the open source business architecture of the Mini Program, the following conclusions and thoughts are summarized and considered on the service-oriented structure and the whole process of troubleshooting:

3.1 Design ideas of service-oriented architecture

1. Cohesion of basic data and convergence of services

Data is the foundation of services. Whether it is a developer platform, a tp platform or a host platform, there will be usage scenarios for basic data. Unified management of basic data can well ensure the unity of basic information on each platform, and the unified convergence of basic data Provide services that can be effectively managed and operated.

2. Self-developed rpc framework empowers microservice architecture and governance

The rpc framework is often an inseparable component of the microservice architecture. The internal service calls in the overall framework use the rpc framework. While improving the efficiency of network interaction, the operation and maintenance functions of the rpc framework are used to effectively improve service stability.

3. The principle of single responsibility splits the original heavy modules, and effectively implements the micro-service architecture

With the rapid development of open source business year by year, the original function aggregation and boundary division of many modules are no longer in line with the existing scenarios. Based on the trend of business development and the concept of single responsibility of microservices, dismantling services can not only effectively reduce the cost of services coupling, and more scalable and robust services.

4. Log collection, monitoring and alarm service stability guarantee

Distributed log collection and comprehensive monitoring and alarming are important means to ensure service stability, as well as common ways to troubleshoot and solve problems. However, not all places require log collection and monitoring and alarming, and reasonable and effective log reporting and Monitoring alarm indicators can effectively find and solve problems at the first time.

3.2 Troubleshooting the whole process

The open source link of the applet is relatively long. When encountering an online bad case, if the core log data is incomplete and the log data link is not connected, the upstream and downstream data links cannot be queried. The troubleshooting cycle is long, or even impossible. Problems, and then lead to stability problems can not be well resolved and dealt with, based on such problems, online log construction is very important.

1. http request log link

Trace maintenance : Based on the server mesh concept, using the java agent capability and the openTrace standard, the publishing platform manages the generation and transmission of traces in a unified manner, which is not only non-invasive, but also requires little access cost to the business.
Trace link : The publishing platform provides the desired trace link topology, which can effectively check the calling scenarios and time spent under the traceId, and help developers find and troubleshoot problems in a timely manner.
Trace log : The unified distributed log collection service will collect the logs submitted by the business side, compatible with the trace information managed by the publishing platform, and retain the specific log details of the trace link, which is convenient for more detailed troubleshooting in specific problem scenarios.

2. Material message log link

Material unique identification : The material service will manage and store material data in a unified manner. Materials in different scenarios have their own unique identification systems, and use the material unique representation to connect the link logs of material messages.
Log collection and retrieval : The company's unified collection service provides business identification definition, retrieval and sorting functions. For long-link material message delivery and distribution scenarios, the entire asynchronous link can be well connected.

Recommended reading:

Baidu Mini Program Package Streaming Download and Installation Optimization

Front-end engineering FaaS SSR solution

A brief discussion on the realization of the log middle platform that is neither heavy nor lost

Design and practice of Baidu ToB vertical account authority platform

Input Visualization Methods in Visual Transformer

In-depth understanding of WKWebView (rendering) - DOM tree construction

In-depth understanding of WKWebView (Introduction) - WebKit source code debugging and analysis