Rspack - A high-performance Rust-based web build tool

A high-performance Rust-based web build tool

Introduction
Rust has been rapidly occupying the field of front-end infrastructure with its high performance and reliability in recent years. "Any application system that can be implemented with Rust will eventually be implemented with Rust." In March of this year, ByteDance open sourced the Rust-based construction tool Rspack. Unlike other Rust construction tools such as turbopack, Rspack took a route compatible with the webpack ecology, which means that businesses can migrate from webpack to Rspack at low cost.

Introduction to Rspack

Let's quote the introduction of Rspack official website

Rspack (pronounced /'ɑrespæk/,) is a high-performance build engine based on Rust, which is interoperable with the Webpack ecosystem, can be integrated by Webpack projects at low cost, and provides better build performance.
Rspack has completed compatibility with webpack's main configuration and adapted to webpack's loader architecture. At present, you can seamlessly use various loaders you are familiar with in Rspack, such as babel-loader, less-loader, sass-loader and so on. Our long-term goal is to fully support the loader feature. In the future, you can use those more complex loaders in Rspack, such as vue-loader. At present, Rspack's support for caching is relatively simple, and only supports memory-level caching. In the future, we will build stronger caching capabilities, including migratable persistent caching, which will bring more room for imagination, such as different in monorepo Rspack's cloud cache can be reused on any machine to improve the cache hit rate of large projects.

Webpack-compatible

As one of the most popular front-end construction tools, Webpack is mainly due to the rich Loader and Plugin ecology in its community. Rspack is compatible with the main configuration of Webpack, and is adapted to support the Loader architecture, which facilitates gradual upgrades and low-cost migration from Webpack to Rspack. It is worth noting that the goal of Rspack is not fully compatible with the Webpack API. Rspack still lacks a lot of webpack plug-in Hooks and some APIs, and some configurations will affect the build product. It is still being completed and cannot be seamless switch.

Different from other radical Rust build tools such as Turbopack, Rspack compatibility with Webpack will bring some performance loss. Compared with other Rust build tools in the industry, there is also a performance gap, but the balance between performance and migration costs is worthy of everyone's attention. Rspack provides the industry with A more moderate solution. At present, the Rspack team has established a cooperative relationship with the Webpack team, and may explore the integration of Rspack into Webpack in the future.

In addition to the webpack-based construction method, Rspack is also compatible with the Rollup-based construction method, which can meet the needs of different application scenarios

Performance improvements brought by Rust

Rspack benefits from Rust's high-performance compiler support, and the native code generated by Rust compilation is usually more efficient than JavaScript. JavaScript's garbage collection mechanism will continue to find and release objects and variables that are no longer used when the program is running. Rust's memory management is to manage memory through the ownership system and check it according to a series of rules at compile time, making Rust amazing. memory utilization. In addition, the Rust language has good support for parallelization, enabling Rspack to execute multi-threaded parallel execution in the stages of module diagram generation and code generation, so as to make full use of the advantages of multi-core CPUs and greatly improve compilation performance.

Rspack core logic and core plug-ins are implemented in rust

The performance bottleneck of Webpack mainly comes from its js Plugin | Loader, while Rspack rewrites most of Webpack's Loader and Plugin based on Rust, and has built-in common building capabilities such as Typescript, JSX, CSS, CSS Modules, Sass, etc., reducing the js Plugin | The performance problems and communication overhead costs brought by the Loader ensure its construction speed. At the same time, Rspack supports custom plug-ins and custom Loaders to expand more capabilities, with strong pluggability and freedom.

Loader compatible

Loader	compatibility	replace	Remark.
babel-loader	compatible	Most of the conversion functions of babel-loader are implemented internally using SWC	Babel-loader has a great impact on performance, please use it with caution
sass-loader	compatible
less-loader	compatible
postcss-loader	compatible
yaml-loader	compatible
json-loader	compatible
stylus-loader	compatible
@mdx-js-loader	compatible
@svgr/webpack	compatible
raw-loader	compatible	type: “asset/source”
url-loader	compatible	type: “asset/resource”
css-loader	compatible	type: “css”

Plugin compatible

Plugin	compatibility	replace	Remark
html-webpack-plugin	There is an alternative	builtins.html or @rspack/plugin-html	Currently does not support childCompiler related functions of webpack
DefinePlugin	built in	builtins.define
copy-webpack-plugin	partially compatible	builtins.copy	Not compatible with the latest version, but compatible with copy-webpack-plugin@5 version
mini-css-extract-plugin	built in	type=“css”
terser-webpack-plugin	built in	builtins.minifyOptions
progressPlugin	built in	builtins.progress
webpack-bundle-analyzer	built in		Built-in support, --analyze enabled
webpack-stats-plugin	compatible
tsconfig-paths-webpack-plugin	built in	resolve.tsConfigPath

It should be noted that at present, custom Plugin and Loader still need to be developed in Javascript, and Rust development is not supported for now. For some highly customized businesses, the performance loss caused by js Plugin | Loader communication is also worthy of attention. The Rspack team will explore and support businesses to use Rust to develop custom plugins and Loaders in the future.

Caching and incremental compilation

Through the file-level caching mechanism, Rspack can share the cache between different code versions, reduce repeated compilation, and improve construction speed and development efficiency. At present, Rspack's support for caching is relatively simple. It only supports memory-level caching. In the future, a migratable persistent cache will be built to realize the reuse of Rspack's cloud cache in different businesses. Rspack also adopts a more efficient incremental compilation strategy in the HMR stage, and only recompiles the modified code. HMR is extremely fast and greatly shortens the build time on large projects.

Rspack implementation

Rspack itself is a very large system, including many functions and modules. The following will introduce the implementation of Rspack from the main process of Rspack

Reminder: Rspack is still in the rapid iteration stage, and some functions may be adjusted in the future

initialization

Read parameters from repack.config.js, configuration files, and Shell commands to generate configuration parameters and create Compiler objects.
(Like webpack, rspack supports MultiCompiler modules, running different configurations in different compilers)

const compiler = await cli.createCompiler(rspackOptions, "serve");
const compilers = cli.isMultipleCompiler(compiler) ? compiler.compilers : [compiler];

The Compiler object will create a binding instance based on napi-rs during initialization (napi-rs encapsulates most commonly used N-API interfaces into Safe Rust interfaces), through N-API (N-API calls will bring very large overhead) to communicate with native addons written in Rust.

Create a Rust @rspack/core instance through node-binding, and subsequent Rspack hooks, build, etc. are processed through this instance

#[napi]
impl Rspack {
  #[napi(constructor)]
  pub fn new(
    env: Env,
    options: RawOptions,
    js_hooks: Option<JsHooks>,
    output_filesystem: ThreadsafeNodeFS,
  ) -> Result<Self> {
    init_custom_trace_subscriber(env)?;
    // rspack_tracing::enable_tracing_by_env();
    Self::prepare_environment(&amp;env);
    tracing::info!("raw_options: {:#?}", &amp;options);

    let disabled_hooks: DisabledHooks = Default::default();
    let mut plugins = Vec::new();
    if let Some(js_hooks) = js_hooks {
      plugins.push(JsHooksAdapter::from_js_hooks(env, js_hooks, disabled_hooks.clone())?.boxed());
    }

    let compiler_options = options
      .apply(&amp;mut plugins)
      .map_err(|e| Error::from_reason(format!("{e}")))?;

    tracing::info!("normalized_options: {:#?}", &amp;compiler_options);

    let rspack = rspack_core::Compiler::new(
      compiler_options,
      plugins,
      AsyncNodeWritableFileSystem::new(env, output_filesystem)
        .map_err(|e| Error::from_reason(format!("Failed to create writable filesystem: {e}",)))?,
    );

    let id = COMPILER_ID.fetch_add(1, Ordering::SeqCst);
    unsafe { COMPILERS.insert_if_vacant(id, rspack) }?;

    Ok(Self { id, disabled_hooks })
  }
}

#[napi(
	catch_unwind,
	js_name = "unsafe_build",
	ts_args_type = "callback: (err: null | Error) => void"
)]
pub fn build(&amp;self, env: Env, f: JsFunction) -> Result<()> {
	let handle_build = |compiler: &amp;mut _| {
		// Safety: compiler is stored in a global hashmap, so it's guaranteed to be alive.
		let compiler: &amp;'static mut rspack_core::Compiler<AsyncNodeWritableFileSystem> =
			unsafe { std::mem::transmute::<&amp;'_ mut _, &amp;'static mut _>(compiler) };

		callbackify(env, f, async move {
			compiler
				.build()
				.await
				.map_err(|e| Error::new(napi::Status::GenericFailure, format!("{e}")))?;
			tracing::info!("build ok");
			Ok(())
		})
	};
	unsafe { COMPILERS.borrow_mut(&amp;self.id, handle_build) }
}

For the native addon written in Rust, different systems and CPU architectures will be distributed through different npm packages. Here, all native packages are used as optionalDependencies, and the native addon package corresponding to the current system environment requirements is determined in binding.js.

switch (platform) {
  case 'linux':
    switch (arch) {
      case 'x64':
        nativeBinding = require('./rspack.linux-x64-musl.node')
        break
      case 'arm64':
        nativeBinding = require('./rspack.linux-arm64-musl.node')
        break
      default:
        throw new Error(`Unsupported architecture on Android ${arch}`)
    }
    break;
	...
}
module.exports.default = module.exports = nativeBinding

Initialize the server service

@rspack/dev-server starts the server service, where @rspack/dev-server directly inherits webpack-dev-server

import { RspackDevServer } from "@rspack/dev-server";
server = new RspackDevServer(compiler.options.devServer ?? {}, compiler);
await server.start();

Start @rspack/dev-middleware, generate middleware bound to Rspack's compiler, and then call this middleware in the server service started in the previous step. Monitor resource changes through watch mode, and then automatically package and write them into memory and send them to the server service.
Enable the devServer.devMiddleware middleware in the configuration

import rdm from "@rspack/dev-middleware";
private setupDevMiddleware() {
	// @ts-expect-error
	this.middleware = rdm(this.compiler, this.options.devMiddleware);
}

Under the server, the file content is obtained from the memory through the node_binding api
(now the memory fs will cause problems with the outputFileSystem, Rspack temporarily disables getRspackMemoryAssets, which means that each change will pack the new file to the local instead of going through the memory, here will be Time-consuming will affect)

import { getRspackMemoryAssets } from "@rspack/dev-middleware";
if (Array.isArray(this.options.static)) {
	this.options.static.forEach(staticOptions => {
		staticOptions.publicPath.forEach(publicPath => {
			compilers.forEach(compiler => {
				if (compiler.options.builtins.noEmitAssets) {
					middlewares.push({
						name: "rspack-memory-assets",
						path: publicPath,
						middleware: getRspackMemoryAssets(compiler, this.middleware)
					});
				}
			});
		});
	});
}

Lazy-compilation (Lazy-compilation), only compiled when the user visits

compilers.forEach(compiler => {
	if (compiler.options.experiments.lazyCompilation) {
		middlewares.push({
			middleware: (req, res, next) => {
				if (req.url.indexOf("/lazy-compilation-web/") > -1) {
					const path = req.url.replace("/lazy-compilation-web/", "");
					if (fs.existsSync(path)) {
						compiler.rebuild(new Set([path]), new Set(), error => {
							if (error) {
								throw error;
							}
							res.write("");
							res.end();
							console.log("lazy compiler success");
						});
					}
				}
			}
		});
	}
});

Build phase (Make)

Find all the entry files according to the entry in the configuration, and save the entry_dependencies and module_graph corresponding to the entry

pub fn setup_entry_dependencies(&mut self) {
	self.entries.iter().for_each(|(name, item)| {
		let dependencies = item
			.import
			.iter()
			.map(|detail| {
				let dependency =
					Box::new(EntryDependency::new(detail.to_string())) as BoxModuleDependency;
				self.module_graph.add_dependency(dependency)
			})
			.collect::<Vec<_>>();
		self
			.entry_dependencies
			.insert(name.to_string(), dependencies);
	})
}

Generate modules according to the entry dependencies, and traverse the module_graph in the hmr stage to find the modules that depend on modification, create a build_queue to call the loader to translate the module into standard JS content, call the JS interpreter to convert the content into an AST object, and find out the modules that the module depends on. Then recurse this step until all the files that the entry depends on have been processed by this step

// move deps bindings module to force_build_module
for dependency_id in &force_build_deps {
  if let Some(mid) = self
    .module_graph
    .module_identifier_by_dependency_id(dependency_id)
  {
    force_build_module.insert(*mid);
  }
}

let mut need_check_isolated_module_ids = HashSet::default();
let mut origin_module_issuers = HashMap::default();
// calc need_check_isolated_module_ids & regen_module_issues
for id in &force_build_module {
  if let Some(mgm) = self.module_graph.module_graph_module_by_identifier(id) {
    let depended_modules = mgm
      .all_depended_modules(&self.module_graph)
      .into_iter()
      .copied();
    need_check_isolated_module_ids.extend(depended_modules);
    origin_module_issuers.insert(*id, mgm.get_issuer().clone());
  }
}

Generation phase (Seal)

TreeShaking

if option.builtins.tree_shaking {
  let (analyze_result, diagnostics) = self
    .compilation
    .optimize_dependency()
    .await?
    .split_into_parts();
  if !diagnostics.is_empty() {
    self.compilation.push_batch_diagnostic(diagnostics);
  }
  self.compilation.used_symbol_ref = analyze_result.used_symbol_ref;
  self.compilation.bailout_module_identifiers = analyze_result.bail_out_module_identifiers;
  self.compilation.side_effects_free_modules = analyze_result.side_effects_free_modules;
  self.compilation.module_item_map = analyze_result.module_item_map;

  // This is only used when testing
  #[cfg(debug_assertions)]
  {
    self.compilation.tree_shaking_result = analyze_result.analyze_results;
  }
}
self.compilation.seal(self.plugin_driver.clone()).await?;

According to the dependency relationship between the entry and the module, it is assembled into a Chunk containing multiple modules, and then each Chunk is converted into a separate file and added to the output list. This step is the last chance to modify the output content
. After outputting the content, determine the output path and file name according to the configuration, and write the file content to the file system

#[instrument(name = "compilation:seal", skip_all)]
pub async fn seal(&mut self, plugin_driver: SharedPluginDriver) -> Result<()> {
  use_code_splitting_cache(self, |compilation| async {
    build_chunk_graph(compilation)?;
    plugin_driver.write().await.optimize_chunks(compilation)?;
    Ok(compilation)
  })
  .await?;
  plugin_driver
    .write()
    .await
    .optimize_chunk_modules(self)
    .await?;

  plugin_driver.write().await.module_ids(self)?;
  plugin_driver.write().await.chunk_ids(self)?;

  self.code_generation().await?;

  self
    .process_runtime_requirements(plugin_driver.clone())
    .await?;

  self.create_hash(plugin_driver.clone()).await?;

  self.create_chunk_assets(plugin_driver.clone()).await;

  self.process_assets(plugin_driver).await?;
  Ok(())
}

Rspack currently only supports incremental builds in the make phase, not yet in the seal phase

how to use

Replace the webpack related package with the corresponding package of Rspack, and replace the called code

• webpack -> @rspack/core
• webpack-dev-server -> @rspack/dev-server
• html-webpack-plugin -> @rspack/plugin-html

Replace the configuration that Rspack has built-in support for

• Style-loader, css-loader, MiniCssExtractPlugin and other CSS-related functions have built-in support in Rspack
• Code compression functions such as TerserWebpackPlugin and CssMinimizerPlugin have built-in support in Rspack, and they are enabled by default in production mode
• DefinePlugin can be replaced by builtins.define in Rspack
• ReactRefreshWebpackPlugin has built-in support in Rspack, which can be turned on or off through builtins.react.refresh

Remove configurations that are not fully supported by Rspack

• cache Currently Rspack only supports memory caching, and it will be enabled by default in development mode
• resolve.plugins is currently not supported by Rspack
• CaseSensitivePathsPlugin, ESLintPlugin, etc. are currently not supported in Rspack

const path = require('path');
module.exports = {
  context: __dirname,
  mode: 'development',
  entry: {
    main: ['./src/index.jsx'],
  },
  builtins: {
    html: [{}],
    define: {
      'process.env.NODE_ENV': '\'development\'',
    },
  },
  module: {
    rules: [
      {
        test: /.less$/,
        use: ['less-loader'],
        type: 'css',
      },
    ]
  },
  output: {
    path: path.resolve(__dirname, 'dist')
  }
};

Webpack comparison

From the previous introduction to the Rspack construction process, we can find that most of the Rspack architecture refers to Webpack, and also reuses many webpack capabilities such as webpack-dev-server, webpack-dev-middleware, etc. So, how much performance improvement can Rspack bring compared to Webpack?

From the data on Rspack’s official website, we can see that the construction speed of Rspack is really amazing compared to Webpack, which is mainly due to the parallel architecture and high performance brought by Rust, as well as the optimization of Rspack’s incremental compilation and caching. Although Webpack can implement multi-threading through happypack and run some Rust SWC-loader through N-API, there is still a big gap in performance due to its own architecture problems.

In addition, Rspack has also established a cooperative relationship with the Webpack team. When Rspack reaches a certain maturity, the webpack team will try to integrate Rspack into webpack in an experimental manner.

Summarize

At present, Rspack is still in the early stage, and there are still many small problems (such as Cache cleaning, memory io, etc.) to be resolved. In the process of digging into the source code, many optimization spaces have been found (such as support for incremental compilation in the seal stage, etc.), If you consider using it in a production environment, you may need to be more cautious. It is recommended to wait for the subsequent stable version.

At present, Byte has begun to promote internally and continues to improve, and we look forward to seeing more complete products in the future.