Suggestions on speeding up Nodejs application compilation and construction | JD Cloud technical team

The overall process of compiling and building

1. Pull the compiled image

2. Pull the cache image

3. Pull project source code

4. Mount the cache directory

5. Execute compilation command (user-defined)

6. persistent cache

7. Upload compiled image

Why is it fast to build locally, but very slow on the compiler

Each build environment on the editing machine is brand new, and it takes a few more steps to complete a build than locally:

1. Ready-made global package cache VS reconstruction cache : We can simply understand it as the global cache directory when we use npm. The editor needs to prepare a persistent cache environment, including downloading, mounting, and rebuilding the cache. If the cache content If it is too large, the time will be relatively longer, and the local build directly uses a stable local file system;

2. Incremental installation dependencies VS full installation dependencies : the process of installing is not often required locally. Even if it is needed, because there is a persistent node_modules directory, full installation is not required, but the editor environment needs to reinstall everything needed for this project every time. rely;

3. Incremental build VS full build : The local build will put the build cache in the node_modules directory by default, and these builds can be used during the second build, making subsequent builds faster, but the default cache location for this build is edited It will not be persisted on the machine, that is, it needs to be fully built every time.

4. Network environment : Some dependent package installations rely on external networks or even overseas networks. The local network environment is relatively smooth, but the network of the editing machine does not guarantee access to overseas networks.

5. Advantages that are difficult to use : multi-core and large memory, most of the work in the construction of nodejs projects is executed on one thread, it is not easy to directly use the multi-core advantages of the compiler

6. Additional steps : The compiler needs to download the image, make and upload the running image, and persist the cached content, while the local is generally only the output package.

So starting from the above perspective, we can optimize the construction speed based on some ideas like this:

1. Optimize image size;

2. Make good use of persistent cache to achieve incremental builds (editing opportunities to persistently cache the contents of the /cache/ directory)

3. Take full advantage of multi-core advantages:

   For example, the type verification of ts-loader can be executed in a separate thread through other plug-ins, and eslint-loader also supports multi-threading (but currently has bugs and is not recommended).

   For another example, we can decouple the functional modules of the project and split them into multiple builds at the same time.

4. Reduce unnecessary builds:

   For example, reasonably configure exclude to streamline the scope of the build file;

   For infrequently changing files, take them out and build them once, and reuse them next time.

5. Determine whether there may be other ways to remove packages that depend on the external network

How to analyze build speed

1. Check /cache/ directory size:
2. Add: to the compilation command du -sh /cacheto view the directory size through the build log
3. Add before and after the overall compilation command date, you can see the time-consuming construction process of your own project, that is, the execution time of the compilation command
4. Add in front of each line of the main compilation command time, eg: time npm installYou can see the actual time consumption of the install process, and the build process is the same.
5. Compare the overall build time (task time directly displayed on the webpage) with the compilation command execution time (date time at the end - date time at the beginning), if the overall time exceeds the compilation command execution time by a lot (> 1min30s), it may be the /cache/ directory Or the image is too large.

The following is a detailed introduction:

Use a smaller running image

If there is a larger image, it is recommended to contact the operation and maintenance for optimization.

Make good use of persistent caching

Cache can speed up application construction, but if the cache directory continues to grow, it may slow down the speed to a certain extent.

Understand the caching mechanism:

1. 缓存目录: /cache/

2. 默认行为: 对于 nodejs 的应用, 目前持久缓存会为 npm, pnpm 提供安装包的缓存, 以加快 npm install / pnpm install 的过程

3. 工作原理: 

    3.1 /cache/ 目录下的内容会构建成功后自动上传到服务器进行存储, 并在下次构建任务执行前进行挂载

    3.2 /cache/ 与 当前工作目录(即 './', 拉取的源码存放位置) 不在同一个文件系统(相当于是缓存在C盘而源码在D盘), pnpm install的行为将从 hark link回退为文件复制(硬链接的方式相对于大量小文件的拷贝, 速度要快很多)

    3.3 /cache/ 的工作涉及上传、下载过程, 如果过大也将会影响整个构建过程的速度

Exclude the impact of global cache on build speed

Check the size of /cache/, you can add in the compilation command: du -sh /cache, check the log, if the folder exceeds 1G (for reference only), it is recommended to contact J-one to clean up the application cache

Solve the performance loss caused by cache cross-disk

Main idea : Make the source code and /cache/ in the same file system. Currently, this method is recommended for pnpm applications.

Principle : Keep the source code and /cache/ in the same file system, which can make the hard link method of pnpm take effect. Compared with copying tens of thousands of small files in node_modules, the execution efficiency will be greatly improved. Reference: Is it possible for Pnpm Work across multiple drives or file systems?

Method : Copy the code of the current working directory to /cache/ and then execute the install and build commands.

Reference command :

    # 记下当前工作目录
    CUR_WORKSPACE=`pwd`
    # 存放源码
    # 咱统一用 /cache/source 放源码就好, 虽然也可以改成其它目录的名字
    mkdir -p /cache/source
    # 拷贝当前目录的代码, 到 /cache/source 下
    rsync -r ./ /cache/source --exclude=node_modules --exclude=.git
    # 切换 workspace
    cd  /cache/source
    ########## 这里替换成自己需要的内容  ###########
    # 执行 install
    pnpm i
    # 执行 build
    pnpm run build
    ########## 这里替换成自己需要的内容  ###########

    # 将构建结果拷贝到抽包地址
    ########## 如果不是 dist, 请根据需要换成其它目录, 就是你项目构建完生成的目标代码目录
    cp -r ./dist/* ${CUR_WORKSPACE}/.build
    # 删除不需要被缓存的文件
    cd ../ && rm -rf /cache/source

The above compilation commands are simplified based on the Xingyun deployment front-end project itself.
Please modify it according to your own needs on the basis of understanding the principles and ideas.

Cache build results

webpack and its plug-ins will cache the build results. We can use the persistent cache of /cache/ to implement code build caching. Other build tools can also be configured by referring to related documents.

If you use webpack4 or build tools that rely on webpack4, such as @vue/cli-service, etc., you usually use cache-loader to cache the build results, and babel-loader also has its own build cache, but they are placed in node_modules/ by default. Under the cache directory, it is recommended to refer to relevant documents to set the cache directory to /cache/build (or other subdirectories of /cache/)

For webpack5, the cache function has been integrated, and plug-ins such as cache-loader can be deleted to reduce unnecessary work. Reference: webpack cache

If it is a monorepo application, sub-project level caching can also be implemented, such as using nx to manage monorepo, you can configure NX_CACHE_DIRECTORY to set the cache address, eg:

export NX_CACHE_DIRECTORY=/cache/jdos3-console-ui/.nx

eslint is also a very time-consuming operation. It also supports caching, but it is not enabled by default. You can also enable caching if necessary, but the caching strategy needs to use 'content', because the createTime of each build file will change, and the metadata strategy will fail. . Reference: eslint cache

Usually we need to be compatible with both local development and cloud deployment, which can be achieved through environment variables. Take webpack5 as an example:

Cache configuration for webpack5:

{
    cache: {
        type: 'filesystem',
        profile: true,
        cacheDirectory: process.env.BUILD_CACHE_DIRECTORY,
        compression: 'gzip',
    },
}

At the same time, add to the compilation command of Xingyun deployment:

export BUILD_CACHE_DIRECTORY=/cache/.webpack

Another way to use cache: cache node_modules

(The compilation team proposed this idea, I have not tried it yet, and the general solution for this idea on the product is being explored)
Main idea : Simulate local builds (local builds will permanently retain the node_modules directory)
Benefits :
1. Accelerated install 2. Use code to build
the cache: webpack5 or babel-loader generally store the build cache in the node_modules/.cache directory, which is why many applications build faster locally. Of course, the .cache directory will It continues to grow and needs to be cleaned up regularly. If you are interested, you can check whether this directory exists in the local code and how much space it takes up.

Reference command :
It is roughly the same as the above process of "resolving the performance loss caused by cross-disk cache", except that the last rm process keeps the node_modules directory for next use

    ####### 与上面 解决缓存跨盘造成的性能损失 一致 #########
    # 记下当前工作目录
    CUR_WORKSPACE=`pwd`
    # 存放源码
    mkdir -p /cache/source
    # 拷贝当前目录代码到 /cache/ 下
    rsync -r ./ /cache/source --exclude=node_modules --exclude=.git
    # 切换 workspace
    cd  /cache/source
    # 执行 install
    npm i
    # 执行 build
    npm run build
    # 将构建结果拷贝到抽包地址
    cp -r ./dist/* ${CUR_WORKSPACE}/.build
    
    ####### 差异: 删除时排除 node_modules 目录 #########
    # 删除不需要被缓存的文件
    ls -A | grep -vE "^\.$|^\.\.$|^node_modules"|xargs rm -rf

Reduce source code

Avoid submitting node_modules and various large binary files in coding

Optimize the compilation process

Optimize the process of installing dependent packages

1. Some projects rely on image-minimizer-webpack-plugin, which is a tool for compressing images. The cwebp-bin and other resources that this resource relies on need to be downloaded from overseas websites. This process may be slow or even fail. If possible, It is recommended to directly submit the compressed image to the code base, and remove the reference to this plugin.
2. You can add time before the compilation command, for example, time pnpm installto observe the time-consuming of this step. If this step is very long, you can check whether there are dependent packages that can be removed, or disable the installation of optional dependent packages, and sometimes upgrade the build Tooling also enables package dependencies to be optimized.

Optimize the build process

1. For applications built by webpack, check whether the exclude is correctly set for rules and plugins (if supported) to reduce unnecessary file construction
2. Enable the build cache (but the continuous growth of the cache still needs attention, and the problem of excessive cache may be optimized from the product level later)
3. ts-loader can usually enable transpileOnly: true, and perform type checking through fork-ts-checker-webpack-plugin
4. The optimization of eslint can optimize the rules. Some verification rules are very time-consuming, but at the same time the benefits are not very great. You can consider turning them off. Specifically, you can do this:

4.1 Setting the __TIMING__ environment variable can enable the performance analysis of each eslint rule; export TIMING = 14.2
Execute the build normally locally, detect the output of eslint rule performance, analyze the rules that take a long time, and confirm whether it is necessary

Supplement :

• Regarding the multi-threading problem of eslint: After multi-threading is enabled for eslint, the rule exceptions found in the build process will not be thrown, causing the rules to actually fail. Refer to Issue for this problem. This problem has been around for a long time and has not been effectively resolved.
• At the same time, you can also consider implementing the eslint verification as a git hook to avoid submitting non-standard codes. At this time, this step can be omitted during the build process.

5. For the process of code minify, it is recommended to use esbuild, which can be configured in webpack.

{
   optimization: {
       minimize: true,
       minimizer: [
           new TerserPlugin({
               minify: TerserPlugin.esbuildMinify,
           }),
       ],
   }
}

6. For parts that do not change frequently, it is recommended to compile them in advance or optimize them through DllPlugin . For example, Xingyun Deployment project itself relies on monaco editor, but it is time-consuming to build its source code every time, so directly compile the pre-compiled code Submitted, and will be used directly later.

7. Pay attention to avoid a project being built multiple times, for example:
7.1 For applications using vue-cli-service, starting from v5.0.0-beta.0, different packages may be generated according to the browser list configuration, which will lead to multiple builds
7.2 Some projects require micro-frontend access, and may use different entries for independent runtime and sub-application modes, thus building twice. For example, users of JModule , because the very early version of webpack-jmodule-plugin cannot customize the entry file , usually build twice, it is recommended to upgrade to the latest @jmodule/plugin-webpack, and use the same entry file to build once.

8. If it is a relatively simple application, you can consider switching to other build tools, such as esbuild, swc, and the performance difference brought by the programming language can indeed form a blow to dimensionality reduction.

9. If possible, analyze the dependencies between project codes and split them into multiple builds for parallel execution. The biggest advantage of the compiler is multi-core, which we can make full use of.

10. Upgrading webpack and other build plugins can usually bring a certain degree of speed improvement, and the compilation of our jci project has benefited from the upgrade.

Supplement :

1. For more detailed optimization of webpack, please refer to https://webpack.docschina.org/configuration/cache/
2. Similarly, you can also consider adding time before the build command, for example time npm run build, to facilitate the observation of the time of this step.
3. You can also use 'speed-measure-webpack-plugin' to analyze the construction time of webpack.

The acceleration of front-end construction is a relatively complex and detailed project. Currently, the product is continuously tracking the slow-build applications and trying to optimize the compilation speed. However, the front-end itself has a relatively free technical environment, and there is no unified construction tool and process. In addition, The execution efficiency of the language itself and the single-threaded construction are not good enough for the compiler to exert its maximum capabilities, so the current global general optimization methods are still relatively limited, and it still depends on the optimization of the project itself. I hope everyone will work together to build a better tomorrow.

Author: JD Technology Lin Guanghui

Content source: JD Cloud developer community

{{o.name}}

{{m.name}}