Ecological evolution of SREWorks front-end low-code components: monorepo architecture reconstruction and remote component loading practice

Author: Wang Wei (Diqian)

Article structure

• Background of the project
• evolution analysis
• Evolution of monorepo architecture
• • Webpack and Rollup
  • How to Migrate Smoothly
  • build optimization
• Extensible and pluggable components
• Evolution summary
• Version Dynamics

Background of the project

SREWorks is an open source cloud-native digital intelligence operation and maintenance platform for enterprise-level complex businesses. It is the tempering and precipitation of years of engineering practice by the big data SRE team. As an important part of the platform, the front-end unified hosting project (frontend) provides a set of configurable front-end low-code technology solutions with serverless experience: low-code and configuration are the basic features of front-end low-code solutions.

The frontend project adopts the technical framework of React+antd, and designs a set of component mapping, arrangement, analysis, and rendering engineering systems: the antd component is used as the free editing granularity, and the user can use visual interaction or json editing in the front-end designer. According to the actual usage scenario of the operation and maintenance work, perform attribute configuration/component nesting assembly on the components; at the same time, according to the target requirements of the usage scenario, arrange the layout of the page components, bind the data source, and insert Dynamic Logic at the appropriate point to complete the page The node design work forms the node model nodeModel, which is parsed and rendered by the template parsing engine. Since a detailed introduction to the architecture design has been made before, I will not repeat it here. For details, please refer to this article https://mp.weixin.qq.com/s/_kqItPbivVmIrOVXvEaVlg

Our vision of open-sourcing this front-end project is to accumulate more usage scenarios, integrate more user needs, and work with the community to build a rich ecosystem of front-end operation and maintenance components. In the past six months, in order to better evolve the ecology of front-end components, frontend has upgraded its architecture for the two key points of " extensible and easy to plug and unplug ":

1. At the architectural level, the monorepo mode has been reconstructed;
2. Front-end components support remote dynamic loading;

In the text, we have made a staged induction and summary of some problems encountered one after another during the entire iteration process, as well as the selection and thinking of technical solutions.

evolution analysis

Students who pay attention to our open source dynamics should know that our first version of open source frontend has nearly 100,000 codes. Without the help of reliable and detailed documents, it is still necessary to quickly understand the design concept, structure mechanism of the entire project and participate in the contribution. There is a certain degree of difficulty. At the same time, from the perspective of the internal subdivision of the project, the designer, model layer, and component layer each have very different update frequencies. A small change during development requires the entire project to be built.

On the other hand, frontend comes from the version of the company's internal engineering practice. Although the two have the same source, due to the rapid functional evolution of the company's internal and open source scenarios, the originally designed sharing mechanism for the public framework layer and component layer has been somewhat difficult. It laid the groundwork for the subsequent iteration of evolution. Combined with our two goals of creating an open source ecosystem of " extensible and easy to plug and unplug ", we need to solve the following problems comprehensively:

• The base level supports the loading of runtime remote components to solve the diverse scenarios and demands of users
• Fine-grained splitting of large and comprehensive projects structurally
• Extract the framework layer and widget component layer and build them separately
• share-tools tools can be shared
• Internal business code stripping
• Upgrade each dependent version on the premise of adapting to internal business operations to facilitate the introduction and evolution of new technologies
• Build tool upgrades and configuration tuning to effectively improve build efficiency and reduce build volume

In view of the above problems to be solved, we conducted technical research on the evolution plan, and also referred to and adopted the suggestions of community students. We decided to adopt the following two solutions to solve the problems mentioned above:

1. At the architectural level, the monorepo mode is used for refactoring to extract several sub-dependent packages such as the framework layer, widget component layer, and shared-tools, and the optimized construction is carried out with webpack5 (main application package) + rollup (sub-dependent package) as the construction tool ;
2. For components that cannot enter the code base, provide remote component scaffolding, support packaging remote components in umd format and dynamically import and remove them in the form of dynamic script tags, so as to achieve runtime loading and expansion;

Let's share the implementation of these two programs in detail:

Monorepo Architecture Evolution

Monorepo is a single warehouse (repository) and multiple packages (package). Large-scale front-end engineering projects use this model for development and management, which can bring many development and management conveniences:

• Clearer module structure and dependencies
• Fine-grained independent building units facilitate collaborative development and separate releases of sub-packages with different update frequencies
• More efficient code reuse, etc.

In the v1.4 version, the technical solution of lerna + yarn workspace was used to implement the Monorepo architecture practice: split the original project into @sreworks/app main package application, and @sreworks/components, @sreworks/widgets, @sreworks/framework , @sreworks/shared-utils four npm sub-dependent packages. The directory structure changes as shown in the following figure:

Through the lerna+yarnworkspace solution, configure each sub-package into the workspace space, and the update of each sub-dependency package in the workspace space will be synchronized to the node_module of the main application package in real time. There is no need to publish npm, and you can choose to release npm separately for specific sub-packages The version or each package releases a new version synchronously, which can update the main application dependencies at a smaller granularity, which is convenient and efficient.

Webpack and Rollup

After designing the split of subpackages, we started to transform the file structure, change the way components are imported and mounted, optimize the processing of remote component loading, migrate theme styles, etc. Students who are interested in handling details or want to discuss and communicate can join the communication group at the end of the article).

After processing the project structure code, we started to build the project: the choice of construction tools, about construction tools, webpack, gulp, rollup and later vite, based on our actual situation, finally selected Webpack and Rollup as alternatives Solution: Both Webpack and Rollup are essentially escaping and packaging non-ES5 code. A powerful compiler function reads the target file through the configuration entry, and then outputs the escape file; to complete the packaging of the entire project, babel- Loader, React, Vue and other loader processing and a series of plugin timely mounting processing can complete the processing of types of files such as pictures, css files, JSX and Vue templates, and the work of js mounting html.

Features of Webpack and Rollup:

Based on the comparison of the above characteristics and the practice of excellent open source projects in the industry, frontend chose to use Webpack5 as the construction tool for the main package application, and Rollup as the construction tool for the sub-package application. The main package application HMR is just needed for daily development, so Webapck was chosen; For subpackage dependencies, more convenient configuration and smaller output are better choices.

How to Migrate Smoothly

The whole project with such a large volume cannot run without complete and accurate disassembly and construction at the code level. A small mistake will cause the entire project to throw an error. Moreover, it is useless to use sourcemap in the second-party package. After the main package is built, it is difficult to find out where the problem is, so it has to be reinvented... In the real-time process at the beginning, it took a lot of time. The modification and investigation of sub-packages, the construction of the main application package and other verification cycles are very long, and this is the difficulty of exploratory transformation.

So can you verify and migrate at a smaller granularity? The loading of remote components gives inspiration. Try to package the component package @SREWorks/widgets into esm format and directly modify node_modlues in the original large and complete project to import the dependent package package file, and the corresponding loading mechanism, to verify each sub-dependent package on the project that can run, and cooperate with sourcemap, troubleshooting instantly speeds up. In this way, the verification of other packages such as @SREWorks/framework was carried out in turn, and the blindfold construction troubleshooting problem was solved.

The style problem is quite a headache. The solution adopted here is to keep the general style in the main package, and the sub-package has fewer scenes covered by the style reset, so the css-module method is used for isolation and construction.

build optimization

After the smooth verification of each sub-dependency package on the original project, when it came to the construction of the main application package, the construction volume reached an astonishing 5.5M, which is still the volume after gzip compression:

By analyzing this picture, the build of this version has the following problems:

• Dependencies with the same name appear multiple times, and each sub-dependency package has duplicate dependencies
• Some dependent packages are too large to build, such as BizCharts

In view of the above existing problems, optimize @sreworks/app in three dimensions as a whole:

First, optimize to 2.8M by unifying subpackage dependency checking and merging dependency versions

const namespace = {
  appRoot: path.resolve('src'),
  appAssets: path.resolve('src/assets'),
  // 减少子依赖包内部重复依赖
  '@ant-design': path.resolve(process.cwd(), 'node_modules', '@ant-design'),
  'js-yaml': path.resolve(process.cwd(), 'node_modules', 'js-yaml'),
  'ace-builds': path.resolve(process.cwd(), 'node_modules', 'ace-builds'),
  'brace': path.resolve(process.cwd(), 'node_modules', 'brace'),
  'lodash': path.resolve(process.cwd(), 'node_modules', 'lodash')
}
...
  resolve: {
    alias: paths.namespace,
    modules: ['node_modules'],
    extensions: ['.json', '.js', '.jsx', '.less', 'scss'],
  },

Second, extract some large dependent packages to CDN, as follows in the externals configuration item; optimize the volume to 1.6M. But considering some dedicated cloud usage scenarios, external CDN cannot be used. Therefore, a custom build script is used to migrate the target dependencies from node_modules to the output folder and load them into html, reducing the number of large dependent packages involved in the build, while ensuring the normal use of the proprietary cloud environment.

  externals: {
    // 剥离部分依赖，不参与打包
    'react': 'React',
    'react-dom': 'ReactDOM',
    "antd":"antd",
    ...
  },

Third, adjust the key component path and 1.48M, reduce the volume by 70%, and optimize the build time from 74 seconds in the V1.3 version to 23 seconds, an increase of 68%.

Extensible and pluggable components

Although frontend has built-in basic components commonly used in operation and maintenance scenarios, there are more than 50 components such as chart components, landing components, and layout components. According to user feedback after open source, users still have common demands for customization and scalability: generally speaking, they can be roughly divided into two categories:

1. The front-end framework is also React, which has its own customized usage scenarios. The built-in components cannot meet the current needs and need to be expanded
2. The front-end technology stack is Vue, and there are many historical components accumulated, and the cost of refactoring all React is too high

For problem 1 , frontend originally provided JSXRender, which supports users to customize and expand simple static rendering components with JSX, but does not support advanced features such as attribute configuration, data source, and dynamic business logic processing. Front-end plug-in, it is easy to think of the introduction of npm package, but this can only be adapted in the scenario of engineering code development, to use and remove the runtime, you need to find another solution. Going back further, front-end development has developed from the jQuery era to today's Agular, React, Vue troika and the participation of various engineering construction tools, but the essence has not changed, and it is still mounted in the script tag in html js code for rendering and loading. Therefore, it is natural to think of loading our remote components in the form of script tags, but here we need to achieve dynamic, batch loading, and removable, that is: package the remote components in umd format and publish them to the cloud, and obtain the corresponding accurate path. Introduced in the form of tags:

(function (){
  let script = document.createElement('script')
  script.type = 'text/javascript'
  script.src = url // 目标组件url
  document.getElementById('targetDomId').appendChild(script)
})()
script.addEventListener('load',callback,false) // 嵌入逻辑

If you want to batch load multiple words, namely:

const loadRemoteComp = async () => {
  let remoteCompList = ["url_a","url_b","url_c",...];
  try {
    remoteComList.forEach(item => {
      pros.push(Promise.resolve(loadSingleComp());
    })
    window['REMOTE_COMP_LIST'] = await Promise.all(pros);
  } catch (error) {
    console.log(error);
  }
}
loadRemoteComp()

Of course, it also involves browser terminal adaptation, fault tolerance and other details. Here, frontend uses a relatively mature systemjs package to load components, and handles the above details properly, making reasonable use of resources, saving time and efficiency.

For problem two, the technology stack is different, that is, the loading of heterogeneous components. At present, frontend only does heterogeneous compatible rendering for Vue components, and uses Vue components in React. At the beginning, I thought of using a conversion tool to manually convert the Vue component into a React component, and then paste and build it, but this method has a big flaw: the APIs of different versions are quite different, and manual transcoding generally requires conversion. Manual secondary inspection and adjustment of the code requires developers to be familiar with the properties of the new and old versions of the two frameworks. Secondary confirmation of non-compliant code or updated hooks will virtually increase the threshold for use.

Inspired by the Docker container, thinking that although React and Vue belong to different technology stack systems, they are different from the differences between Java and Golang. Vue and React are essentially the encapsulation of native js objects, so theoretically speaking, they can be used in React Containerized rendering of Vue components: the essence is the operation of binding and mounting Vue objects:

createVueInstance (targetElement, reactThisBinding) {
  const { component, on, ...props } = reactThisBinding.props
  reactThisBinding.vueInstance = new Vue({
    el: targetElement,
    data: props,
    ...config.vueInstanceOptions,
    render (createElement) {
      return createElement(
        VUE_COMPONENT_NAME,
        {
          props: this.$data,
          on,
        },
        [wrapReactChildren(createElement, this.children)]
      )
    },
    components: {
      [VUE_COMPONENT_NAME]: component,
      'vuera-internal-react-wrapper': ReactWrapper,
    },
  })
}

Through the frontend remote component scaffold @sreworks/widget-cli, package the React component and Vue component and publish it to CDN, and then edit it in the material development department to easily load and remove the remote component runtime, which solves the problem. Question 2, the goals of " scalability " and " pluggability " have been achieved .

Evolution summary

So far, a series of problems listed in the beginning have been basically solved, and a co-construction path has been laid for the construction of the front-end operation and maintenance component ecology, which can be done:

• Develop from @sreworks/widgets package, JSXRender custom components, use @sreworks/widget-cli to develop remote components Three-dimensional extension component application richness
• Clearer structural dependencies, lowering the threshold for learning and contributing to this set of low-code projects
• With smaller granularity update unit and shorter construction time, it is convenient for daily collaborative development
• After the sub-package is split, conditions are provided for the subsequent small-scale step-by-step introduction of TS

Version Dynamics

We will continue to improve, optimize and upgrade the functions according to the rhythm of the work project. Currently, it is mainly the output of front-end low-code functions. Subsequent API low-code editing and arrangement have been included in the version planning to cover the whole link low-code use. Everyone has better Suggestions are welcome to raise more issues, and more developers are welcome to participate in our ecological construction (@小友手, you can also directly front-end @地谦)

SREWorks open source address:

https://github.com/alibaba/sreworks/paas/frontend

{{o.name}}

{{m.name}}