Node.js — Filter Data in Streams

Using streams in Node.js is a powerful mechanism to process data. Especially when working with large data sets where you might want to filter chunks that don’t match a given criteria.

Node.js Series Overview

  1. String Replace All Appearances
  2. Remove All Whitespace From a String in JavaScript
  3. Generate a Random ID or String in Node.js or JavaScript
  4. Remove Extra Spaces From a String in JavaScript or Node.js
  5. Remove Numbers From a String in JavaScript or Node.js
  6. Get the Part Before a Character in a String in JavaScript or Node.js
  7. Get the Part After a Character in a String in JavaScript or Node.js
  8. How to Check if a Value is a String in JavaScript or Node.js
  9. Check If a String Includes All Strings in JavaScript/Node.js/TypeScript
  10. Check if a Value is a String in JavaScript and Node.js
  11. Limit and Truncate a String to a Given Length in JavaScript and Node.js
  12. Split a String into a List of Characters in JavaScript and Node.js
  13. How to Generage a UUID in Node.js
  14. Reverse a String in JavaScript or Node.js
  15. Split a String into a List of Lines in JavaScript or Node.js
  16. Split a String into a List of Words in JavaScript or Node.js
  17. Detect if a String is in camelCase Format in Javascript or Node.js
  18. Check If a String Is in Lowercase in JavaScript or Node.js
  19. Check If a String is in Uppercase in JavaScript or Node.js
  20. Get the Part After First Occurrence in a String in JavaScript or Node.js
  21. Get the Part Before First Occurrence in a String in JavaScript or Node.js
  22. Get the Part Before Last Occurrence in a String in JavaScript or Node.js
  23. Get the Part After Last Occurrence in a String in JavaScript or Node.js
  24. How to Count Words in a File
  25. How to Shuffle the Characters of a String in JavaScript or Node.js
  26. Append Characters or Words to a String in JavaScript or Node.js (Coming soon)
  27. Check if a String is Empty in JavaScript or Node.js (Coming soon)
  28. Ensure a String Ends with a Given Character in JavaScript or Node.js (Coming soon)
  29. Left-Trim Characters Off a String in JavaScript or Node.js (Coming soon)
  30. Right-Trim Characters Off a String in JavaScript or Node.js (Coming soon)
  31. Lowercase the First Character of a String in JavaScript or Node.js (Coming soon)
  32. Uppercase the First Character of a String in JavaScript or Node.js (Coming soon)
  33. Prepend Characters or Words to a String in JavaScript or Node.js (Coming soon)
  1. Filter Data in Streams
  1. Get Number of Seconds Since Epoch in JavaScript
  2. Get Tomorrow’s Date in JavaScript
  3. Increase a Date in JavaScript by One Week
  4. Add Seconds to a Date in Node.js and JavaScript
  5. Add Month(s) to a Date in JavaScript or Node.js
  6. Add Week(s) to a Date in JavaScript or Node.js
  7. Get the Current Year in JavaScript or Node.js
  8. How to Get a UNIX Timestamp in JavaScript or Node.js
  9. How to Convert a UNIX Timestamp to a Date in JavaScript or Node.js
  10. Add Days to a Date in JavaScript or Node.js
  11. Get Yesterday's Date in JavaScript or Node.js
  12. Add Minutes to a Date in JavaScript or Node.js (Coming soon)
  13. Add Hours to a Date in JavaScript or Node.js (Coming soon)
  14. Check If a Date Is Today in JavaScript or Node.js (Coming soon)
  15. Check If a Date is Tomorrow in JavaScript or Node.js (Coming soon)
  16. Check If a Date is Yesterday in JavaScript or Node.js (Coming soon)
  17. How to Format a Date YYYY-MM-DD in JavaScript or Node.js (Coming soon)

Example: Loading Lines From File to Database

Imagine the following example: you want to load a file with rows of data into a database. Your requirement is that each row has a value in the “name” column. If there’s no value, you want to skip it.

your-example-file.csv

name      , department  
-------------------------
Marcus    , streaming  
          , joking        <-- filter this line
Norman    , programming  
Christian , programming  

The task is to filter the second row because of the missing name.

How to Filter Data in Streams

Transform streams are a way to filter data in streams. Implement the _transform() function to pass through chunks that match your condition. In case the condition doesn’t match, omit the current chunk and proceed with the next one.

Within the _transform method, you can either use this.push() to pass through data or use the second parameter of the next callback. Here’s a code snippet that uses the next callback:

const { Transform } = require('stream')

class Filter extends Transform {

  constructor() {
    super({
      readableObjectMode: true,
      writableObjectMode: true
    })
  }

  _transform(chunk, encoding, next) {
    if (this.has(chunk.name)) {
      return next(null, chunk)
    }

    next()
  }

  has(value) {
    return !!value
  }
}

module.exports = Filter  

Notice that the chunk passes through your transform if it matches the condition. If not, you’re notifying Node.js to process the next chunk by calling next().

The object mode configuration in the constructor ensures that the chunks in _transform are JavaScript objects and not buffers. It also improves the readability of the example 😃

Usage

A way of integrating the filter in your stream is to .pipe your input to the transformer and from there to the destination stream:

const Fs = require('fs')  
const Filter = require('./your-filter-transform')

const transformer = new Filter()  
const input = Fs.createReadStream(source)  
const output = Fs.createReadStream(destination)

input.pipe(transformer).pipe(output)  

That’s it! A transform stream is a great way to receive data from an input source, read the chunk, process it, and pass the changed value to the destination.

Enjoy coding & make it rock!

Explore the Library

Find interesting tutorials and solutions for your problems.