Read a big file using NodeJS streams

Lets say that you have a file with 100.000 rows and you need to read it all. How should you proceed doing this task using a Readable stream in NodeJS.

Create a file with 100.000 rows from linux command;

Keep the path /tmp/100krowsfile as it will get deleted if you somehow forget to delete the file.

  yes "NodeJS Book" | head -n 100000 > /tmp/100krowsfile

Solution

Now let’s write our NodeJS application that will read chunks of data from 100krowsfile file

  const fs = require('fs');
  const file = fs.createReadStream('/tmp/100krowsfile', {
      encoding: "utf-8"
  });

  file.on('data', function(line) {
      console.log(`Chunk length:  ${line.length} \nLine from file: ${line.split('\n')}`);
  })

  file.on('end', function() {
      console.log('finished reading');
  });

How it works?

Breaking down the code:

  const fs = require('fs');
  const readStream = fs.createReadStream('/tmp/100krowsfile', {
      encoding: "utf-8"
  });

We are using const fs = require('fs'); to import the fs module and to create the readable stream from the file.

  const readStream = fs.createReadStream('/tmp/100krowsfile', {
      encoding: "utf-8"
  });

encoding: "utf-8" is used to specify the kind of encoding used for the file, this way NodeJS knows how to decode the content of the file.

  readStream.on('data', function(line) {
      console.log(`Chunk length:  ${line.length} \nLine from file: ${line.split('\n')}`);
  })

  readStream.on('end', function() {
      console.log('finished reading');
  });

The important things here are:

• readStream.on('data', callback)
• readStream.on('end', callback)

What are those data and end tokens ? They are stream events and are a way to handle various scenarios when working with streams

The data event emits either a Buffer object or a string if encoding was previously set on the stream. The end event emits when the stream gets EOF. It indicates that no other data events will happen.

Hope you understood how to read a big file using streams in NodeJS. As always, to learn more you have to get your hands dirty with code. Cheers!