Abstract: Let's look at an example that shows the huge advantages of using stream programming ideas in terms of memory consumption.
This article is shared from Huawei Cloud Community " A Concrete Example of Using Node.js Stream API to Reduce Server-side Memory Consumption ", author: Jerry Wang.
The HTTP response object (res in the code above) is also a writable stream. This means that if we have a readable stream representing the contents of big.file, we can join the two to each other and get almost the same result without consuming ~400 MB of memory. Node's fs module can provide us with a readable stream of any file using the createReadStream method. We can pipe it to the response object.
Let's look at an example that shows the huge advantages of using stream programming ideas in terms of memory consumption.
Let's create a large file first:
const fs = require('fs');
const file = fs.createWriteStream('./big.file');
for(let i=0; i<= 1e6; i++) {
file.write('this is a big file.\n');
}
file.end();The fs module can be used to read and write files using the stream interface. In the example above, we write data to the big.file via a loop that writes 1 million rows to the writable stream.
Running the code above produces a file of about 400 MB.
Here's a simple Node web server designed to serve big.file exclusively:
const fs = require('fs');
const server = require('http').createServer();
server.on('request', (req, res) => {
fs.readFile('./big.file', (err, data) => {
if (err) throw err;
res.end(data);
});
});
server.listen(8000);The server starts with an initial memory consumption of around 8 MB.
After accessing the server with a browser, the memory consumption jumped to 434.8 MB.
We basically put the entire big.file content in memory before writing it to the response object. This is very inefficient.
The HTTP response object (res in the code above) is also a writable stream. This means that if we have a readable stream representing the contents of big.file, we can join the two to each other and get almost the same result without consuming ~400 MB of memory.
Node's fs module can provide us with a readable stream of any file using the createReadStream method. We can pipe it to the response object:
const fs = require('fs');
const server = require('http').createServer();
server.on('request', (req, res) => {
const src = fs.createReadStream('./big.file');
src.pipe(res);
});
server.listen(8000);We now visit the above reimplemented server and see that the memory consumption is greatly reduced.
This is because, when a client requests that large file, we stream it one chunk at a time, which means we don't buffer its entire huge file content in memory at all. Memory usage increased by about 25 MB and that's it.
We can make the test scenario a bit more extreme: regenerating big.file with 5 million lines instead of 1 million would push the file to over 2 GB, which is actually larger than the default buffer limit in Node.js.
If you try to feed the file using fs.readFile, you will get an out of memory error by default.
But using fs.createReadStream, streaming 2 GB of data to the requester is no problem at all, and most importantly, the process memory usage is about the same.
Click to follow and learn about Huawei Cloud's fresh technologies for the first time~