Using NodeJS streams

What is stream?

A stream is a sequence of elements that become available over time. You can process chunks of a big file without loading the whole file at once into memory.

In NodeJS streams are a core feature and understanding them is a good way to make your life easier as a NodeJS developer.

NodeJS provides us with 4 types of Streams:

• Writable
  • you can write chunks of data, like writing chunks to a file
• Readable
  • you can read chunks of data, like reading chunks of a file
• Duplex
  • both Readable and Writable like using a Socket
• Transform
  • These are like Duplex streams and can transform data as it is written and read

Streams have the advantages of keeping data in memory at minimum, making operations performed with streams much more effiecient and with a lower footprint on memory.

When using in context of a file, you can choose to read parts of a file, without reading the whole file at once in memory.

fs module provides an API for interacting with the file system and has some built in functionality for using streams.

All the functions from fs module have synchronous and asynchronous forms.

• Asynchronous
  • The functions that are asynchronous take a callback as the last argument.
• Synchronous
  • You need to wrap your code in a try...catch statement when working with synchronous fs functions as it might throw errors and they will propagate up.

But as normal, you can use util module with promisify function and convert them to promises, which makes it very easy to use in combination with async and await.

Write a file using fs.createWriteStream

To create a writable stream we can use createWriteStream function provided by the fs built in module.

Create a file and paste the following code:

  const fs = require('fs'); 

  const buildData = () => {
      return [
          {
              name: 'Alex',
              age: 18
          },
          {
              name: 'Daniel',
              age: 20
          }
      ]
  }

  const names = buildData().map((person) => {
      return person.name;
  });

  const file = fs.createWriteStream('names.txt');
  names.forEach(name => {
      file.write(`${name}\n`);
  })

  file.end();

The important things here are:

  const fs = require('fs'); 
  ...
  const file = fs.createWriteStream('names.txt');
  names.forEach(name => {
    file.write(`${name}\n`);
  })

  file.end();

Use const fs = require('fs'); to import the built in node module. const file = fs.createWriteStream('names.txt'); will create our writable stream by returning a fs.WriteStream object.

Next, things get clear, we just need to use the fs module and createWriteStream function to open a writable stream and we are prepared to begin writing our chunks to the file.

To write a chunk of data to the stream, we need to use the write function available on the file object. Calling end method on our stream will signal stream closing.

Calling the write() method after calling end() will raise an error as we are trying to write something to a closed stream.

events.js:292
    throw er; // Unhandled 'error' event
      ^

    Error [ERR_STREAM_WRITE_AFTER_END]: write after end

Hope you understood how basic streams work in NodeJS. As always, to learn more you have to get your hands dirty with code.