Recently, the package managers of several projects have been switched from npm to pnpm. The migration experience is very good, and it can be regarded as the best tool migration in personal experience. The following is my intuitive feeling of using pnpm:
-
The experience is excellent, the dependency installation speed is extremely fast, and the disk space is small.
-
It is easy to get started, and most npm / yarn projects can be migrated at low cost, and the official also has more detailed Chinese documents.
-
The way pnpm organizes the node_modules directory is compatible with native Node, works well with packaging tools, and can be safely used in production environments.
-
Although pnpm dependent access is strict, the rules are clear and the boundaries are clear. It is no longer prone to dependency conflicts as before. Instead, it reduces the mental burden when using it and corrects some of my previous misunderstandings.
Combined with the learning before use and the feelings during use, the following will introduce the precautions for using pnpm and the advantages of pnpm as a modern package manager.
Reference article:
Deep Thoughts on Modern Package Managers - Why do I now recommend pnpm over npm/yarn?
About dependency management
Switching from npm to pnpm can be confusing without a basic understanding of how package managers manage dependencies.
We know that the project's dependencies are declared in the package.json file of each project, and there are three types, dependencies, devDependencies, and peerDependencies.
On the Internet, there are the following common sayings about dependency types dependencies and devDependencies:
Dependencies are formal dependencies, which are the packages that the project product depends on.
devDependencies are development dependencies, packages that are only used for local development and testing.
This kind of statement cannot be said to be completely wrong, but at least it is not clear enough, so it is difficult for us to truly understand them, so we often step on the pit of dependent package versions in our daily work.
There is even a simplified and more widely circulated saying: dependencies = production dependencies, devDependencies = development dependencies, which is even more misleading to us.
Are dependencies related to the "production environment"?
Let's create the simplest vite & vue project:
npm create vite@latest my-vue-app -- --template vue
vite and related plugins are dependencies of the local development environment, so we won't mention them for now. But vue is obviously the main dependency of the application running, and it must be running in the production environment. If we move it into devDependencies, will it not be able to be packaged?
{
"name": "my-vue-app",
"private": true,
"version": "0.0.0",
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview"
},
"dependencies": {
- "vue": "^3.2.25"
},
"devDependencies": {
"@vitejs/plugin-vue": "^2.3.3",
"vite": "^2.9.9"
+ "vue": "^3.2.25"
}
}
After modification, execute the installation, packaging and preview commands:
npm i
npm run build
npm run preview
It can be seen that there is no impact at all, and we can even draw the following conclusions:
When developing a web application, even if all dependencies are declared in devDependencies, it will not affect the successful construction, packaging and operation of the application.
So dependencies = production dependencies, devDependencies = development dependencies is one-sided. The "production environment" and "development environment" we often say are behaviors during construction. Construction is not the responsibility of the package manager, but the work of tools such as webpack, rollup, and vite. At this time, the role of the package manager is only to execute Script only. The behavior of various package managers to deal with the differences between dependencies and devDependencies all occurs during the dependency installation period, that is, the process of npm install.
The difference between dependencies and devDependencies
Suppose we have project a, whose package.json structure is as follows:
{
"name": "a",
"dependencies": {
"b": "^1.0.0"
},
"devDependencies": {
"c": "^1.0.0"
}
}
The dependencies of a and the dependencies of b and c are as follows:
// node_modules/b/package.json
{
"name": "b",
"dependencies": {
"d": "^1.0.0"
},
"devDependencies": {
"e": "^1.0.0"
}
}
// node_modules/c/package.json
{
"name": "c",
"dependencies": {
"f": "^1.0.0"
},
"devDependencies": {
"g": "^1.0.0"
}
}
We use solid lines to represent dependencies dependencies, and dashed lines to represent devDependencies dependencies. The dependency tree of project a is as follows:
After executing npm install, the final content of the node_modules directory of a is as follows
node_modules
├── b // a 的 dependencies
├── c // a 的 devDependencies
├── d // b 的 dependencies
└── f // c 的 dependencies
We noticed that the installed packages are all flattened into the node_modules directory, which is the solution adopted by the previous generation of package managers such as npm and yarn to solve the excessively deep dependency hierarchy. However, this solution will bring other confusions, and pnpm has been optimized for these problems, which will be discussed in the following article - the file structure of traditional package managers.
It can be seen that the package manager will use the project's package.json as a starting point to install all dependencies and dependencies declared in devDependencies. But for the deeper-level dependencies of these first-level dependencies, only the dependencies in dependencies will be installed during the deep traversal process, and the dependencies in devDependencies will be ignored. Therefore, the devDependencies of b and c - e and g - are ignored, while their dependencies - d and f are installed.
Why is this so? Because the package manager thinks: as a package user, of course we don't need to care about their dependencies during development and construction, so devDependencies will be ignored for us. The dependencies are the content that the package product depends on to work properly, and of course it must be installed.
Back to the scenario of web application development, the product of web application is often deployed to the server, and will not be released to the npm warehouse for other users to use, and the package manager will install all the first-level dependencies, regardless of dependencies or devDependencies. In this case, dependencies and devDependencies may really only serve as semantic conventions.
peerDependencies
peerDependencies declares the synchronization dependencies of the package. However, the package manager will not automatically install dependencies for users like dependencies. When users use a package, they must follow the peerDependencies of the package to install the corresponding dependencies synchronously, otherwise the package manager will prompt an error.
The usage scenario of peerDependencies is generally the peripheral plug-ins of the core library, such as vue for vuex, or vite for @vitejs/plugin-vue2. Plug-ins generally cannot work independently of the core library.
The following demonstrates an example plugin that uses peerDependencies correctly. This plugin is suitable for vite, and its role is to parse the template files of vue 2.7 and above, so the versions of vite and vue are limited.
// @vitejs/plugin-vue2 的 package.json
{
"name": "@vitejs/plugin-vue2",
// ...
"peerDependencies": {
"vite": ">=2.5.10",
"vue": "^2.7.0-0"
},
// dependencies、devDependencies 与其他字段 ...
}
Compared with the default automatic installation of dependencies, peerDependencies can guide users to install core dependencies correctly through the prompt information during installation, which can avoid some dependency version conflicts to a certain extent.
The file structure of a traditional package manager
Continue to look at the example above - the difference between dependencies and devDependencies.
For the above dependency tree, according to the generation rules of node_modules, the directory is as follows:
node_modules
├── b // a 的 dependencies
| └── node_modules
| └── d // b 的 dependencies
├── c // a 的 devDependencies
| └── node_modules
| └── f // c 的 dependencies
It is conceivable that if d and f have their own dependencies, the generated directory structure will be too deep, and the file system of some operating systems will be difficult to support.
Our commonly used npm and yarn, in order to solve the problem of too deep dependency levels, solve the problem by flattening dependencies. All dependencies are flattened into the node_modules directory, and there is no deep nesting relationship anymore.
node_modules
├── b // a 的 dependencies
├── c // a 的 devDependencies
├── d // b 的 dependencies
└── f // c 的 dependencies
In the above example, suppose a adds a dependency d, because b's dependency d has been flattened to node_modules. When the require() method does not find node_modules in b, it will continue to look for node_modules in the upper directory, and can find the flattened dependencies, so the package manager does not need to install d repeatedly.
Therefore, another benefit of flattening dependencies is: when installing new packages, the package manager will also keep looking for the upper-level node_modules. If a package of the same version is found, it will not be reinstalled, and a large number of packages will be solved at the same time. Problem with repeated installations.
Although npm / yarn has solved many problems, there is still a lot of room for optimization:
- Flattening relies on complex algorithms and consumes more performance, and there is still room for speed-up in dependency installation.
A large number of files need to be downloaded repeatedly. On the one hand, the utilization rate of the disk space is insufficient. In addition, a large number of decompression and IO operations will further reduce the execution efficiency. - Although flat dependency solves many problems, it immediately brings the problem of illegal access to dependencies. In some cases, project code can be used in code that is not defined in package.json
- In the package, this situation is what we often call ghost dependencies.
pnpm advantage - hard links save disk space
Due to my limited understanding of the file system of the operating system, here is a reference to the article about the in-depth thinking of modern package managers - why do I now recommend pnpm instead of npm/yarn? The relevant description basically clearly shows that pnpm uses disk space advantages.
Internally, pnpm uses a content-based addressing file system to store all files on the disk. The outstanding features of this file system are:
- The same package will not be installed repeatedly. When using npm / yarn, if 100 projects depend on lodash, then lodash is likely to be installed 100 times, and this part of the code is written in 100 places on the disk.
But using pnpm will only be installed once, and there is only one place to write in the disk, and the hardlink will be used directly when used again later (hard link, for those who are not clear, please refer to this article Linux soft link and hard link ).- Even if there are different versions of a package, pnpm will greatly reuse the code of the previous version. For example, if lodash has 100 files, and one more file is added after the update version, then the disk will not rewrite 101 files, but keep the hardlink of the original 100 files, and only write the new
one document.
pnpm advantage - soft link optimization dependency management
Let's take the installation of vue as an example, first use npm to install:
npm i vue -S
As you can see, npm's flat dependency management leads to a lot of messy things in node_modules.
Try again with pnpm:
pnpm i vue -S
How pure! How pleasing to the eye!
At this time, some students will start to wonder, there is only vue in the node_modules of the root directory, and there is no node_modules in the vue directory, so the dependencies required by vue are missing? That's the beauty of pnpm! When we expand the .pnpm directory, we will find something different. The necessary dependencies are originally here.
But this kind of directory structure does not conform to the rule that require() keeps searching upwards for dependencies in node_modules. How does vue get these resources?
Observe carefully, we will find that the vue of node_modules is actually just a soft link (students who often use Windows can understand it as a shortcut).
What it actually points to is the corresponding package in the .pnpm directory.
It can be seen that the vue in .pnpm is where the "prime spirit" is, and the one in node_modules is just the "incarnation".
For vue under the [email protected]/node_modules/ directory, you can naturally find several dependencies in @vue/ from the upper node_modules, and the dependency loss problem we were worried about before is easily solved! Ingeniously, these dependencies are actually the "incarnation" of soft links, and their bodies are also installed in .pnpm with the same structure. The figure below simply marks the dependencies and links.
pnpm puts the package itself and dependencies under the same node_modules, achieving compatibility with native require(). Dependencies are introduced in the form of soft links, and their ontology is also organized in the same structure. As a result, the dependency file structure of all packages is consistent with the declarations in package.json, which is no longer as confusing as before.
pnpm advantage - more secure access to dependencies
By default, ghost dependencies are prohibited, which is the benefit of pnpm's soft link-based dependency management mode.
The dependency file structure of pnpm is consistent with the declaration in package.json, so we will no longer be able to access packages not declared in package.json. This solves the ghost dependency problem that npm / yarn has been relying on, and improves the security of dependency access.
Let’s take a scenario where ghost dependencies are generated. In the previous section, use npm to install dependent projects as an example. We write the following code.
The code can run successfully:
PS D:\learning\npm> node a.js
{ asyncWalk: [AsyncFunction: asyncWalk], walk: [Function: walk] }
The dependencies of estree-walker here are as follows:
estree-walker -> @vue/complier-core -> @vue/complier-dom -> vue
We only declare vue in our package.json, but we can use packages that have three layers of dependencies with vue.
On the surface, there is no problem, but if Vue is updated one day and no longer depends on estree-walker, then our code will report an error, which is the risk of illegal access to dependencies. Of course, this behavior obviously doesn't work in pnpm. The package you want to use in the project code must be honestly and correctly declared in package.json.
Although projects currently under development are prohibited from accessing ghost dependencies, due to historical reasons, many released packages have more or less ghost dependencies. In order to be compatible with them and reduce users' migration and usage costs, pnpm will upgrade all dependent packages to .pnpm/node_modules by default.
This part involves pnpm's dependency promotion strategy, which can be modified by configuring the .npmrc file in the root directory of the project, and even allows pnpm to support the wayward behavior of accessing ghost dependencies. For details, please refer to the official document.npmrc | Dependency promotion settings
Basic use of pnpm
If you used to be an npm / yarn user, migrating pnpm has basically no cost in terms of command usage. In this regard, there are also very detailed introductions in the official documents .
Next, we will actually migrate a vue2 ancestral project to pnpm. The dependencies declared in the package.json of the ancestor project are as follows:
{
// ...
"dependencies": {
"axios": "^0.21.0",
"cropperjs": "^1.5.11",
"echarts": "^4.8.0",
"echarts-liquidfill": "^2.0.6",
"element-ui": "^2.13.2",
"file-saver": "^2.0.5",
"highlight.js": "^9.0.0",
"js-base64": "^3.7.2",
"lodash": "^4.17.19",
"marked": "^1.2.7",
"moment": "^2.24.0",
"qs": "^6.10.2",
"save": "^2.4.0",
"sortablejs": "^1.13.0",
"v-viewer": "^1.5.1",
"video.js": "^7.10.2",
"vue": "^2.6.11",
"vue-bus": "^1.2.1",
"vue-clipboard2": "^0.3.1",
"vue-contextmenu": "^1.5.10",
"vue-cropper": "^0.5.5",
"vue-py": "0.0.4",
"vue-qr": "^4.0.9",
"vue-router": "^3.4.3",
"vue-ueditor-wrap": "^2.4.4",
"vuedraggable": "^2.24.3",
"vuex": "^3.4.0",
"xlsx": "^0.16.9",
"xss": "^1.0.10"
},
"devDependencies": {
"@types/echarts": "^4.9.12",
"@types/file-saver": "^2.0.4",
"@types/lodash": "^4.14.178",
"@types/node": "^17.0.16",
"@types/qs": "^6.9.7",
"@types/sortablejs": "^1.10.7",
"@typescript-eslint/eslint-plugin": "^5.10.1",
"@typescript-eslint/parser": "^5.11.0",
"@vitejs/plugin-legacy": "^1.7.1",
"cross-env": "^7.0.3",
"eslint": "^7.32.0",
"eslint-config-airbnb-base": "^15.0.0",
"eslint-config-airbnb-typescript": "^16.1.0",
"eslint-plugin-vue": "^8.4.0",
"prettier": "^1.18.2",
"sass": "~1.32.13",
"stylelint": "^14.3.0",
"stylelint-config-recess-order": "^3.0.0",
"stylelint-config-recommended-vue": "^1.1.0",
"stylelint-config-standard-scss": "^3.0.0",
"typescript": "^4.4.4",
"vite": "^2.8.6",
"vite-plugin-html-env": "^1.1.1",
"vite-plugin-vue2": "^1.9.3",
"vue-eslint-parser": "^8.2.0",
"vue-template-compiler": "^2.6.11",
"vue-tsc": "^0.31.1"
},
// ...
}
First, delete the package-lock.json file and the node_modules directory. Ensure that pnpm is installed through npm i -g pnpm, and execute pnpm install to install all dependencies.
Similar to npm, pnpm uses the following commands to install and uninstall dependencies:
# 根据 package.json 中的依赖声明安装全部依赖
pnpm install
# 安装指定依赖，并在 dependencies 中声明依赖
pnpm install -S xxx
# 安装指定依赖，并在 devDependencies 中声明依赖
pnpm install -D xxx
# 卸载指定依赖
pnpm uninstall xxx
After installation, pnpm really reported a warning:
ERR_PNPM_PEER_DEP_ISSUES Unmet peer dependencies
.
├─┬ eslint-config-airbnb-base
│ └── ✕ missing peer eslint-plugin-import@^2.25.2
├─┬ eslint-config-airbnb-typescript
│ └── ✕ missing peer eslint-plugin-import@^2.25.3
├─┬ stylelint-config-recommended-vue
│ ├── ✕ missing peer postcss-html@^1.0.0
│ └─┬ stylelint-config-html
│ └── ✕ missing peer postcss-html@^1.0.0
├─┬ stylelint-config-standard-scss
│ └─┬ stylelint-config-recommended-scss
│ └─┬ postcss-scss
│ └── ✕ missing peer postcss@^8.3.3
└─┬ echarts-liquidfill
└── ✕ missing peer zrender@^4.3.1
Peer dependencies that should be installed:
eslint-plugin-import@">=2.25.3 <3.0.0" postcss-html@">=1.0.0 <2.0.0" postcss@^8.3.3 zrender@^4.3.1
This is because pnpm does not automatically install peerDependencies for us, just follow the prompts to install all peerDependencies:
pnpm i -D eslint-plugin-import postcss-html postcss
pnpm i -S zrender@^4.3.1
Consistent with npm, pnpm also executes the script through pnpm run, executes the following command to run the application:
pnpm run dev
After running the application, an error occurs:
This is a typical problem of illegal access to ghost dependencies. We can check the dependencies in pnpm-lock.yaml and find that viewerjs is a dependency of v-viewer. Further open the node_modules directory to confirm.
// node_modules/v-viewer/package.json
{
"name": "v-viewer",
// ...
"dependencies": {
"throttle-debounce": "^2.0.1",
"viewerjs": "^1.5.0"
}
}
npm has access to viewerjs due to its reliance on flattening (see: file structure of traditional package managers). After switching to pnpm, access to undeclared dependencies is not allowed by default, so we need to install viewerjs additionally.
pnpm i -S viewerjs
This time, we successfully started the project and the migration is complete:
Of course, ghost dependencies can also be raised to the root node_modules by creating a .npmrc file in the root directory and configuring the public-hoist-pattern or shamefully-hoist field in it directory to solve. Reference: Dependency Boost Settings
# .npmrc
# 提升含有 eslint(模糊匹配)、prettier(模糊匹配)、viewerjs(精确匹配) 的依赖包到根 node_modules 目录下
public-hoist-pattern[]=*eslint*
public-hoist-pattern[]=*prettier*
public-hoist-pattern[]=viewerjs
# 提升所有依赖到根 node_modules 目录下，相当于 public-hoist-pattern[]=*，与上面一种方式一般二选一使用
shamefully-hoist=true
Of course, it is highly not recommended to use this method to solve the dependency problem. It does not take full advantage of the security advantages of pnpm dependency access, and goes back to the old path of npm / yarn.
For most projects, the transition from npm to pnpm can be basically smooth according to the above ideas, and the official FAQs are detailed enough to solve most of the problems in the migration process.