Use of Node.js Buffer

1. Understanding Buffer

1.1. Binary data

All content in the computer: text, numbers, pictures, audio, and video will eventually be represented in binary.

JavaScript can directly process very intuitive data: such as strings, which is what we usually display to users.

No, isn't JavaScript also able to process pictures?

In fact, on the web page, we have always handed over pictures to the browser for processing;
JavaScript or HTML is only responsible for telling the browser the address of a picture;
The browser is responsible for obtaining the image and finally rendering the image;

But it's different for the server:

There are relatively many local file types to be processed by the server;
For example, a certain file that saves text is not utf-8encoded by use, but by use GBK, then we must read their binary data, and then convert it into the corresponding text through GKB;
For example, what we need to read is a piece of image data (binary), and then perform secondary processing on the image data (cropping, format conversion, rotation, and adding filters) by some means. There is a Sharp library in Node, which is Read the picture or pass in the Buffer of the picture and then process it;
For example, in Node, a long connection is established through TCP. TCP transmits a byte stream. We need to convert the data into bytes before passing it in, and we need to know the size of the transmitted bytes (the customer service needs to judge how much to read based on the size. content);

We will find that for front-end development, it is seldom to deal with binary, but for server-side to do a lot of functions, we must directly operate its binary data;

Therefore, in order to facilitate developers to complete more functions, Node provides us with a class Buffer, and it is global.

1.2. Buffer and binary

As we said before, Buffer stores binary data, so how is it stored?

We can think of Buffer as an array storing binary;
Each item in this array can hold 8-bit binary:00000000

Why 8 bits?

In computers, in rare cases we will directly operate a bit of binary, because the data stored in a bit of binary is very limited;
So usually 8 bits are combined together as a unit, which is called a byte;
That is 1byte = 8bit, 1kb=1024byte, 1M=1024kb;
For example, the int type in many programming languages is 4 bytes, and the long type is 8 bytes;
For example, TCP transmits a byte stream, and the number of bytes needs to be specified when writing and reading;
For example, the values of RGB are 255 respectively, so they are essentially stored in one byte in the computer;

In other words, Buffer is equivalent to an array of bytes, and each item in the array has a size of one byte:

If we want to put a string into the Buffer, what is the process?

const buffer01 = new Buffer("why");

console.log(buffer01);

The process of string storage buffer

Of course, we don't want us to do this now:

VSCode warnings

Then we can use another creation method:

const buffer2 = Buffer.from("why");
console.log(buffer2);

What if it is Chinese?

//创建Buffer(字符串中包含中文)
const buffer3 = Buffer.from("你好");
console.log(buffer3);
// <Buffer e4 bd a0 e5 a5 bd>
const str = buffer3.toString();
console.log(str);
// 你好

If encoding and decoding are different:

const buffer3 = Buffer.from("你好", 'utf16le');
console.log(buffer3);

const str = buffer3.toString('utf8');
console.log(str); // �s�~CQ

2. Other uses of Buffer

2.1. Other creation of Buffer

There are many ways to create Buffer:

Buffer creation

Take a look Buffer.alloc:

We will find that an 8-bit length Buffer has been created, and all data in it defaults to 00;

// 1.创建一个Buffer对象
// 8个字节大小的buffer内存空间
const buffer01 = Buffer.alloc(8);

console.log(buffer01); // <Buffer 00 00 00 00 00 00 00 00>

We can also manipulate it:

buffer01[0] = 'w'.charCodeAt();
buffer01[1] = 100;
buffer01[2] = 0x66;
console.log(buffer01);

You can also use the same method to get:

console.log(buffer01[0]);
console.log(buffer01[0].toString(16));

2.2. Buffer and file reading

Reading of text files:

const fs = require('fs');

fs.readFile('./test.txt', (err, data) => {
  console.log(data); // <Buffer 48 65 6c 6c 6f 20 57 6f 72 6c 64>
  console.log(data.toString()); // Hello World
})

Reading of image files:

fs.readFile('./zznh.jpg', (err, data) => {
  console.log(data); // <Buffer ff d8 ff e0 ... 40418 more bytes>
});

Reading and conversion of image files:

Convert a read picture into a 200x200 picture;
Here we can sharp do it with the help of the library;

const sharp = require('sharp');
const fs = require('fs');

sharp('./test.png')
  .resize(1000, 1000)
  .toBuffer()
  .then(data => {
    fs.writeFileSync('./test_copy.png', data);
  })

Three. Buffer memory allocation

In fact, when we create a Buffer, we do not frequently apply for memory from the operating system. It will first apply for a memory of 8 * 1024 bytes in size by default, which is 8kb

node/lib/buffer.js：135行

Buffer.poolSize = 8 * 1024;
let poolSize, poolOffset, allocPool;

const encodingsMap = ObjectCreate(null);
for (let i = 0; i < encodings.length; ++i)
  encodingsMap[encodings[i]] = i;

function createPool() {
  poolSize = Buffer.poolSize;
  allocPool = createUnsafeBuffer(poolSize).buffer;
  markAsUntransferable(allocPool);
  poolOffset = 0;
}
createPool();

If we call Buffer.from to apply for Buffer:

Here we take creating from a string as an example
node/lib/buffer.js：290行

Buffer.from = function from(value, encodingOrOffset, length) {
  if (typeof value === 'string')
    return fromString(value, encodingOrOffset);
 
 // 如果是对象，另外一种处理情况
  // ...
};

Let's look at the call fromString:

node/lib/buffer.js：428行

function fromString(string, encoding) {
  let ops;
  if (typeof encoding !== 'string' || encoding.length === 0) {
    if (string.length === 0)
      return new FastBuffer();
    ops = encodingOps.utf8;
    encoding = undefined;
  } else {
    ops = getEncodingOps(encoding);
    if (ops === undefined)
      throw new ERR_UNKNOWN_ENCODING(encoding);
    if (string.length === 0)
      return new FastBuffer();
  }
  return fromStringFast(string, ops);
}

Then we look at fromStringFast:

What is done here is to judge whether the remaining length is enough to fill the string;
If not enough, then by createPool creating a new space;
poolOffsetIf it is enough, use it directly, but the offset change to be made later ;
node/lib/buffer.js：428行

function fromStringFast(string, ops) {
  const length = ops.byteLength(string);

  if (length >= (Buffer.poolSize >>> 1))
    return createFromString(string, ops.encodingVal);

  if (length > (poolSize - poolOffset))
    createPool();
  let b = new FastBuffer(allocPool, poolOffset, length);
  const actual = ops.write(b, string, 0, length);
  if (actual !== length) {
    // byteLength() may overestimate. That's a rare case, though.
    b = new FastBuffer(allocPool, poolOffset, actual);
  }
  poolOffset += actual;
  alignPool();
  return b;
}

4. Stream

4.1. Understanding Streams

What is a stream?

Our first reaction should be running water, which flows continuously;
The stream in the program has a similar meaning. We can imagine that when we read data from a file, the binary (byte) data of the file will be continuously read into our program;
And this series of bytes is the stream in our program;

So, we can understand streams like this:

It is a manifestation and abstract concept of continuous bytes;
The stream should be readable as well as writable;

When learning to read and write files before, we can read and write files directly through readFileor writeFilemethods, why do we need streams?

Although the way of directly reading and writing files is simple, it cannot control some detailed operations;
For example, where to start reading, where to read, and how many bytes to read at a time;
After reading a certain position, suspend reading, resume reading at a certain time, etc.;
Or the file is very large, such as a video file, it is not appropriate to read it all at once;

In fact, many objects in Node are implemented based on streams:

Request and Response objects of the http module;
process.stdout object;

Official: In addition, all streams are instances of EventEmitter:

We can take a look at the operation in the Node source code:

Stream and EventEmitter relationship

Classification of Stream:

Writable: A stream (for example) to which data can be written fs.createWriteStream().
Readable: A stream (for example) from which data can be read fs.createReadStream().
Duplex: Readablestreams Writablethat are both and net.Socket.
Transform: A stream (for example ) Duplexthat can modify or transform data as it is written and read .zlib.createDeflate()

Here we use the operation of fs to explain Writable and Readable, and you can learn the other two by yourself.

4.2. Readable

Before we read information from a file:

fs.readFile('./foo.txt', (err, data) => {
  console.log(data);
})

This way is to read all the contents of a file into the program (memory) at one time, but this way of reading will cause many problems we mentioned before:

The file is too large, the read position, the end position, and the size of a read;

At this time, we can use it createReadStream. Let's look at a few parameters. For more parameters, please refer to the official website:

start: the position where the file reading starts;
end: the end position of file reading;
highWaterMark: The length of bytes read at one time, the default is 64kb;

【start，end】

const read = fs.createReadStream("./foo.txt", {
  start: 3,
  end: 8,
  highWaterMark: 4
});

How do we get the data?

The read data can be obtained by listening to the data event;

read.on("data", (data) => {
  console.log(data);
});

We can also listen to other events:

read.on('open', (fd) => {
  console.log("文件被打开");
})

read.on('end', () => {
  console.log("文件读取结束");
})

read.on('close', () => {
  console.log("文件被关闭");
})

We can even suspend pause and resume reading resume at a certain moment:

read.on("data", (data) => {
  console.log(data);

  read.pause();

  setTimeout(() => {
    read.resume();
  }, 2000);
});

4.3. Writable

Previously we wrote to a file like this:

fs.writeFile('./foo.txt', "内容", (err) => {
  
});

This method is equivalent to writing all the content to the file at once, but this method also has many problems:

For example, we hope to write the content a little bit, and accurately write the position of each write, etc.;

At this time, we can use it . Let's look at a few parameters. For more parameters, please refer to the official website: createWriteStream

flags: The default is w, if we want to write additionally, we can use aor a+;
start: the location to write;

We do a simple write

const writer = fs.createWriteStream("./foo.txt", {
  flags: "a+",
  start: 8
});

writer.write("你好啊", err => {
  console.log("写入成功");
});

If we wish to listen for some events:

writer.on("open", () => {
  console.log("文件打开");
})

writer.on("finish", () => {
  console.log("文件写入结束");
})

writer.on("close", () => {
  console.log("文件关闭");
})

We will find that we cannot listen to close the event:

This is because the write stream is not automatically closed after it is opened ;
We have to close it manually to tell Node that the writing is over;
and will emit an finish event;

writer.close();

writer.on("finish", () => {
  console.log("文件写入结束");
})

writer.on("close", () => {
  console.log("文件关闭");
})

Another very common method is end:

endThe method is equivalent to doing two steps: write the last content to the file, and close the file

writer.end("Hello World");

4.4. pipe method

Under normal circumstances, we can 输入流manually put the read 输出流into it for writing:

const fs = require('fs');
const { read } = require('fs/promises');
// 创建可读流和可写流
const reader = fs.createReadStream('./foo.txt');
const writer = fs.createWriteStream('./bar.txt');

reader.on("data", (data) => {
  console.log(data);
  writer.write(data, (err) => {
    console.log(err);
  });
});

We can also do this with pipes:

reader.pipe(writer);

writer.on('close', () => {
  console.log("输出流关闭");
})