Front-end engineering practice - javaScript handwritten rollup

Webpack packaging is very cumbersome and the packaging volume is large. rollup mainly packages the js library. Vue/react/angular are all using rollup as a packaging tool.

First experience of rollup project

Add folder rollupTest

Initialize the project: npm init -y

install dependencies

npm install rollup -D

Modify the configuration file package.json

{
  "name": "rollupTest",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "build": "rollup --config"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "rollup": "^2.79.1"
  }
}

 Add src/main.js file

console.log('hello')

New rollup configuration file rollup.config.js

export default {
    input:'./src/main.js',//打包文件入口
    output:{
        file:'./dist/bundle.js',//指定打包后存放的文件路径
        format:'cjs',//打包文件输出格式，输入用require，输出用module.exports
        name:'bundleName'//打包输出文件的名字
    }
}

View packaging results

Execute the command: npm run build

The result is as follows

The package file dist is generated

Treeshaking first experience

Add src/msg.js file

export var name = 'zhangshan'
export var age = '18'

Modify src/main.js

/*//例子1
console.log('hello')
*/

//treeshaking例子
import {name,age} from './msg'
console.log(name);

Execute the command npm run build

View the packaged file of dist/bundle.js, only name has no definition of age, this is because age is introduced but not used

Rollup handwriting - pre-requisite knowledge

magic-string

magic-string is a tool for manipulating strings and generating source-maps. magic-string is a library about string manipulation written by the author of rollup.

Install command:

npm i magic-string -D -S

Here's an example from github:

var MagicString = require('magic-string');
var magicString = new MagicString('export var name = "beijing"');
//类似于截取字符串
console.log(magicString.snip(0,6).toString()); // export
//从开始到结束删除字符串(索引永远是基于原始的字符串，而非改变后的)
console.log(magicString.remove(0,7).toString()); // var name = "beijing"

//很多模块，把它们打包在一个文件里，需要把很多文件的源代码合并在一起
let bundleString = new MagicString.Bundle();
bundleString.addSource({
    content:'var a = 1;',
    separator:'\n'
});
bundleString.addSource({
    content:'var b = 2;',
    separator:'\n'
});
/* let str = '';
str += 'var a = 1;\n'
str += 'var b = 2;\n'
console.log(str); */
console.log(bundleString.toString());
// var a = 1;
//var b = 2;

which introduces the method

const MagicString = require('magic-string');

The advantage of magic-string is that sourcemap will be generated

AST

Abstract Syntax Tree (AST), or Syntax tree for short, is an abstract representation of the grammatical structure of source code. It represents the grammatical structure of the programming language in the form of a tree, and each node on the tree represents a structure in the source code. By manipulating this tree, we can accurately locate declaration statements, assignment statements, operation statements, etc., and implement operations such as code analysis, optimization, and changes.

AST splits a piece of code through lexical analysis and syntax analysis.

Lexical analysis: word segmentation based on spaces

Grammatical analysis: according to the meaning of each word, the role is separated

You can simply understand that it is a tree-structured representation of the code you write. The cores of tools such as webpack, UglifyJs, and lint are all implemented through the ast abstract syntax book, which enables code inspection and analysis. The bottom layer is called js parser to generate abstract syntax tree.

AST workflow

• Parse (analysis) converts the source code into an abstract syntax tree, and there are many estree nodes on the tree
• Transform (transformation) transforms the abstract syntax tree
• Generate (code generation) Generate new code from the converted abstract syntax tree in the previous step

acorn

Acorn is used to parse the program into an ast syntax tree structure

npm i acorn -d -s

const acorn = require('acorn');

The default usage of acorn is very simple. Just parse the code string and get the AST structure:

let acorn = require("acorn");

console.log(acorn.parse("for(let i=0;i<10;i+=1){console.log(i);}", {ecmaVersion: 2020}));

Node {
  type: 'Program',
  start: 0,
  end: 39,
  body: [
    Node {
      type: 'ForStatement',
      start: 0,
      end: 39,
      init: [Node],
      test: [Node],
      update: [Node],
      body: [Node]
    }
  ],
  sourceType: 'script'
}

It can be seen that the type of this AST is program, indicating that this is a program. body contains the AST child nodes corresponding to all the following statements of the program.

Each node has a type type, such as Identifier, indicating that this node is an identifier;

The code can be converted into a syntax tree in real time through the online website astexplorer

Code directory structure

Node traversal walk.js function

walk receives two parameters, the first parameter is a node, and the second parameter is an object

The first value of the object is the enter node to enter the function, and the second value of the object is the leave node to exit the function

The depth-first search method implemented by the walk function traverses nodes, and traverses child nodes first, and then traverses sibling nodes after the child nodes have been traversed.

Object.keys() is used to obtain an array of object property names, which can be used in conjunction with array traversal

walk.js

/*
walk.js AST节点遍历函数
*/
//walk接收两个参数，参数一是节点，参数二是一个对象
//对象第一个值是enter节点进入函数,对象第二个值是leave节点退出函数
//enter和leave函数在调用时传入，不在此定义
function walk(ast,{enter,leave}){
    //调用visit方法 第二个参数是父节点，顶级语句没有父节点
    visit(ast,null,enter,leave);
}
/**
 * 访问此node节点
 * @param {*} node 
 * @param {*} parent 
 * @param {*} enter 
 * @param {*} leave 
 */
function visit(node,parent,enter,leave){
    //1.如果调用了enter方法 先执行此节点的enter方法
    if(enter){
        enter(node,parent)
    }
    //2.根据深度优先搜索，先遍历该节点的子节点
    //观察语法树可以发现，子节点所在的节点必然是个对象，因此只需要遍历是对象的节点
    // Object.keys(node)获取node对象里所有属性的Key值，在结合filter过滤器使用，筛选出符合条件的
    let childKeys = Object.keys(node).filter(key=>typeof node[key]==='object')
    childKeys.map(childKey=>{
        let value = node[childKey]
        //分情况讨论
        //2.1.如果value是数组 对数组每项进行遍历
        if(Array.isArray(value)){
            //递归调用visit，将当前节点val、父节点node、进入函数和退出函数传进去
            value.map(val=>{
                visit(val,node,enter,leave)
            })
        } else {
            //2.2不是数组则调用一次visit
            visit(value,node,enter,leave)
        }

    })
    //3.如果调用了leave方法再执行离开的方法
    if(leave){
        leave(node,parent)
    }
}
module.exports = walk;

Test walk function - add test file ast.json

{
    "type": "Program",
    "start": 0,
    "end": 25,
    "body": [
      {
        "type": "ImportDeclaration",
        "start": 0,
        "end": 23,
        "specifiers": [
          {
            "type": "ImportDefaultSpecifier",
            "start": 7,
            "end": 8,
            "local": {
              "type": "Identifier",
              "start": 7,
              "end": 8,
              "name": "$"
            }
          }
        ],
        "source": {
          "type": "Literal",
          "start": 14,
          "end": 22,
          "value": "jquery",
          "raw": "'jquery'"
        }
      }
    ],
    "sourceType": "module"
  }

Test walk function - add ast.js file

/*ast.js
实现功能：
1.将源代码解析为AST语法树
2.调用walk函数遍历AST语法树
*/
//引入acorn 
//webpack和rollup都是使用acorn模块把源代码转成抽象语法树AST
let acorn = require('acorn');
let walk = require('./walk');
//代码也可以从文件中导入，不必写死
let astTree = acorn.parse(`import $ from 'jquery';`,{
    locations:true,ranges:true,sourceType:'module',ecmaVersion:8
});

//定义缩进，padding每次填充ident个空格
let ident = 0;
const padding = ()=>" ".repeat(ident)
console.log(astTree.body)
//从ast在线网站https://astexplorer.net/可以发现，有意义的语句开始在body中
astTree.body.map(statement=>{
    //遍历AST语法树种的每个语句，由walk遍历这条语句
    //遍历时采用深度优先方式
    walk(statement,{
        enter(node){
            //我们只关心有type的节点
            if(node.type){
                console.log(padding() + node.type);
                ident+=2;
            }
        },
        leave(node){
            if(node.type){
                ident-=2;
                console.log(padding()+node.type)
            }
        }
    })
})

print result

Scope mock

Scope: In js, scope is a rule used to specify the scope of variable access

Scope chain: The scope chain is composed of a series of variable objects in the current execution environment and the upper-level execution environment, which guarantees the orderly access of the current execution environment to variables and functions that meet the access rights.

The role of Scope is very simple, it has a

The names attribute array is used to save the variables in this AST node. rollup according to this

The Scope chain builds the scope of variables.

Scope.js

/**
 * scope.js模拟作用域类
 */
class Scope{
    /**
     * 定义构造函数，入参是options对象，默认值是{}
     */
    constructor(options ={}){
        this.name = options.name//给作用域起个名字
        this.parent = options.parent//父作用域
        this.names = options.params || []//此作用域中有哪些变量
    }
    /**
     * add添加方法，添加name到作用域对象
     */
    add(name){
        this.names.push(name)
    }
    /**
     * findDefiningScope查找方法，查找name属于哪个作用域对象
     */
    findDefiningScope(name){
        if(this.names.includes(name)){
            return this//如果当前对象的names包含name则返回当前对象
        }else if(this.parent){
            return this.parent.findDefiningScope(name)
        }else{
            return null
        }
    }
}
module.exports = Scope;

test scope --testScope

let Scope = require('./scope');
let a = 1;

function one() {
  let b = 2;

  function two(age) {
    let c = 3;
    console.log(a, b, c, age);
  }

  two();
}

one();
let globalScope = new Scope({
  name: 'globalScope', params: [], parent: null
});
globalScope.add('a');
let oneScope = new Scope({
  name: 'oneScope', params: [], parent: globalScope
});
oneScope.add('b');
let twoScope = new Scope({
  name: 'twoScope', params: ['age'], parent: oneScope
});
twoScope.add('c');

let aScope = twoScope.findDefiningScope('a');
console.log(aScope.name);

let bScope = twoScope.findDefiningScope('b');
console.log(bScope.name);

let cScope = twoScope.findDefiningScope('c');
console.log(cScope.name);

let ageScope = twoScope.findDefiningScope('age');
console.log(ageScope.name);

let xxxScope = twoScope.findDefiningScope('xxx');
console.log(xxxScope);

Implementation of Mini-rollup

Add analyse.js

/**
 * analyse.js 找出当前模块使用到了哪些变量
 * 还要知道哪些是当前模块产生的，哪些是引用别的模块产生的
 * @param {*} ast ast语法树
 * @param {*} MagicString 源代码
 * @param {*} module 属于哪个模块
 */
const Scope = require('./scope')
const walk = require('./walk')
function analyse(ast,MagicString,module){
    let scope = new Scope()
    //遍历当前语法树的所有的顶级节点
    ast.body.map(statement=>{
        //建立作用域链
        function addToScope(declaration){
            let name = declaration.id.name//获得这个声明的变量
            scope.add(name)//将变量添加到当前的全局作用域中
            if(!scope.parent){//当前是全局作用域
                statement._defines[name] = true//在全局作用域下声明一个全局的变量say
            }
        }
        //通过Object.defineProperties属性为statement添加_source属性，方便日后取源代码数据
        Object.defineProperties(statement,{
            _defines:{value:{}},//存放当前模块定义的所有全局遍历
            _dependsOn:{value:{}},//当前模块没有定义但是使用的变量，即引入的外部的变量
            _included:{value:false,writable:true},//此语句是否已包含在打包中，防止同一条语句被多次打包
            _source:{value:MagicString.snip(statement.start,statement.end)}
        })
        //构建作用域链
        walk(statement,{
            enter(node){
                let newScope
                switch(node.type){
                    case 'FunctionDeclaration'://节点类型为函数声明,创建新的作用域对象
                        const params = node.params.map(x=>x.name)
                        addToScope(node)
                        newScope = new Scope({
                            parent:scope,//父作用域就是当前的作用域
                            params
                        })
                        break
                    case 'VariableDeclaration':
                        node.declarations.forEach(addToScope)//这样写不会有问题码？？？？
                        break
                }
                if(newScope){//说明当前节点声明了一个新的作用域
                    //如果此节点生成一个新的作用域，那么会在这个节点放一个_scope，指向新的作用域
                    Object.defineProperty(node,'_scope',{value:newScope})
                    scope = newScope
                }
            },
            leave(node){
                if(node._scope){//离开的节点产生了新的作用域，离开节点时需要让scope回到父作用域
                    scope = scope.parent
                }
            }
        })
    })
    console.log('第一次遍历结束',scope)
    ast._scope = scope
    //找出外部依赖_dependsOn
    ast.body.map(statement=>{
        walk(statement,{
            enter(node){
                if(node._scope){//该节点产生了新的作用域
                    scope = node._scope
                }
                if(node.type === 'Identifier'){//是个标识符
                    //从当前的作用域向上递归，照这个变量在哪个作用域中定义
                    const definingScope = scope.findDefiningScope(node.name)//查看该name是否已经被调用过
                    if(!definingScope){
                        statement._dependsOn[node.name]=true //表示这是一个外部依赖的变量
                    }
                }
            },
            leave(node){
                if(node._scope){
                    scope = scope.parent
                }
            }
        })

    })
}
module.exports = analyse;

Add module.js

Each file is a module, and there is a one-to-one correspondence between modules and module instances.

let MagicString = require('magic-string');
const { parse } = require('acorn');
const analyse = require('./ast/analyse');

//判断一下obj对象上是否有prop属性
function hasOwnProperty(obj, prop) {
  return Object.prototype.hasOwnProperty.call(obj, prop);
}

/**
 * 每个文件都是一个模块，每个模块都会对应一个Module实例
 */
class Module {
  constructor({ code, path, bundle }) {
    this.code = new MagicString(code, { filename: path });
    this.path = path;//模块的路径
    this.bundle = bundle;//属于哪个bundle的实例
    this.ast = parse(code, {//把源代码转成抽象语法树
      ecmaVersion: 7,
      sourceType: 'module'
    });
    this.analyse();
  }

  analyse() {
    this.imports = {};//存放着当前模块所有的导入
    this.exports = {};//存放着当前模块所有的导出
    this.ast.body.forEach(node => {
      if (node.type === 'ImportDeclaration') {//说明这是一个导入语句
        let source = node.source.value;//./msg 从哪个模块进行的导入
        let specifiers = node.specifiers;
        specifiers.forEach(specifier => {
          const name = specifier.imported.name;//name
          const localName = specifier.local.name;//name
          //本地的哪个变量，是从哪个模块的的哪个变量导出的
          //this.imports.age = {name:'age',localName:"age",source:'./msg'};
          this.imports[localName] = { name, localName, source }
        });
        //}else if(/^Export/.test(node.type)){
      } else if (node.type === 'ExportNamedDeclaration') {
        let declaration = node.declaration;//VariableDeclaration
        if (declaration.type === 'VariableDeclaration') {
          let name = declaration.declarations[0].id.name;//age
          //记录一下当前模块的导出 这个age通过哪个表达式创建的
          //this.exports['age']={node,localName:age,expression}
          this.exports[name] = {
            node, localName: name, expression: declaration
          }
        }
      }
    });
    analyse(this.ast, this.code, this);//找到了_defines 和 _dependsOn
    this.definitions = {};//存放着所有的全局变量的定义语句
    this.ast.body.forEach(statement => {
      Object.keys(statement._defines).forEach(name => {
        //key是全局变量名，值是定义这个全局变量的语句
        this.definitions[name] = statement;
      });
    });

  }

  //展开这个模块里的语句，把些语句中定义的变量的语句都放到结果里
  expandAllStatements() {
    let allStatements = [];
    this.ast.body.forEach(statement => {
      if (statement.type === 'ImportDeclaration') {return}
      let statements = this.expandStatement(statement);
      allStatements.push(...statements);
    });
    return allStatements;
  }

  //展开一个节点
  //找到当前节点依赖的变量，它访问的变量，找到这些变量的声明语句。
  //这些语句可能是在当前模块声明的，也也可能是在导入的模块的声明的
  expandStatement(statement) {
    let result = [];
    const dependencies = Object.keys(statement._dependsOn);//外部依赖 [name]
    dependencies.forEach(name => {
      //找到定义这个变量的声明节点，这个节点可以有在当前模块内，也可能在依赖的模块里
      let definition = this.define(name);
      result.push(...definition);
    });
    if (!statement._included) {
      statement._included = true;//表示这个节点已经确定被纳入结果 里了，以后就不需要重复添加了
      result.push(statement);
    }
    return result;
  }

  define(name) {
    //查找一下导入变量里有没有name
    if (hasOwnProperty(this.imports, name)) {
      //this.imports.age = {name:'age',localName:"age",source:'./msg'};
      const importData = this.imports[name];
      //获取msg模块 exports imports msg模块
      const module = this.bundle.fetchModule(importData.source, this.path);
      //this.exports['age']={node,localName:age,expression}
      const exportData = module.exports[importData.name];
      //调用msg模块的define方法，参数是msg模块的本地变量名age,目的是为了返回定义age变量的语句
      return module.define(exportData.localName);
    } else {
      //definitions是对象,key当前模块的变量名，值是定义这个变量的语句
      let statement = this.definitions[name];
      if (statement && !statement._included) {
        return this.expandStatement(statement);
      } else {
        return [];
      }
    }
  }
}

module.exports = Module;

Add bundle.js

const fs = require('fs');
const MagicString = require('magic-string');
const Module = require('./module');
const path = require('path')
class Bundle {
  constructor(options) {
    //入口文件的绝对路径，包括后缀
    this.entryPath = options.entry.replace(/\.js$/, '') + '.js';
    this.modules = {};//存放着所有模块 入口文件和它依赖的模块
  }

  build(outputFileName) {
    //从入口文件的绝对路径出发找到它的模块定义
    let entryModule = this.fetchModule(this.entryPath);
    //把这个入口模块所有的语句进行展开，返回所有的语句组成的数组
    this.statements = entryModule.expandAllStatements();
    const { code } = this.generate();
    fs.writeFileSync(outputFileName, code, 'utf8');//写文件，第一个是文件名，第二个是内容
  }

  //获取模块信息
  /**
   * 
   * @param {*} importee //入口文件的绝对路径
   * @param {*} importer 由哪个入口导入的
   * @returns 
   */
  fetchModule(importee,importer) {
    let route;
    if(!importer){
      route = importee
    } else {
      if(path.isAbsolute(importee)){//如果是绝对路径
          route = importee
      }else if(importee[0]=='.'){//如果是相对路径
          route = path.resolve(path.dirname(importer),importee.replace(/\.js$/,'')+'.js')
      }
    }
    if (route) {
      //从硬盘上读出此模块的源代码
      let code = fs.readFileSync(route, 'utf8');
      let module = new Module({
        code,//模块的源代码
        path: route,//模块的绝对路径
        bundle: this//属于哪个Bundle
      });
      return module;
    }
  }

  //把this.statements生成代码
  generate() {
    let magicString = new MagicString.Bundle();
    this.statements.forEach(statement => {
      const source = statement._source;
      if(statement.type === 'ExportNamedDeclaration'){
        source.remove(statement.start,statement.declaration.start)//将单词export去掉
      }
      magicString.addSource({
        content: source,
        separator: '\n'//分割符
      });
    });
    return { code: magicString.toString() };
  }
}

module.exports = Bundle;

Add rollup.js

let Bundle = require('./bundle');
function rollup(entry,outputFileName){
    //Bundle就代表打包对象，里面会包含所有的模块信息
    const bundle = new Bundle({entry});
    //调用build方法开始进行编译
    bundle.build(outputFileName);
}
module.exports = rollup;

Add debugger.js

const path = require('path');
const rollup = require('./lib/rollup');
//入口文件的绝对路径
let entry = path.resolve(__dirname,'src/main.js');
let output = path.resolve(__dirname,'src/dist/bundle.js')
rollup(entry,output);

Rollup process analysis

Run debugger.js->entry entry file-"rollup.js calls the rollup method to pass in the entry address and exit file name-"--"bundle.js creates a bundle instance according to the entry entry-"rollup.js calls the bundle.build method, Pass in the export address--"bundle.js build method calls the fetchModule method, pass in the entry file address entryPath--"bundle.js fetchModule method reads the code source code according to the entry file, and creates a module object instance--"module.js call The constructor method generates the ast syntax tree, and calls the analyze method--"analyse.js method adds the source code_source attribute to each statement--"returns bundle.js and calls the expandAllStatements method--"module.js expandAllStatements method gets it All statements--"bundle.js calls the generate method to put each line together.

execute debugger.js

node .\debugger.js