Node.js — Run Async Functions in Batches

When working with arrays of data, you may need to run an asynchronous operation on each item. The Promise.all() method provides a convenient interface to run a list of promises in parallel, but it has some limitations.

Using Promise.all() may overload the available resources, especially when reaching out to the Internet or interacting with the database. Depending on your list size, you may send out thousands of requests parallelly into the wild.

This tutorial shows you how to use a promise pool to run a batch of promises in parallel and the batches themselves in sequence.

Node.js Series Overview

  1. String Replace All Appearances
  2. Remove All Whitespace From a String in JavaScript
  3. Generate a Random ID or String in Node.js or JavaScript
  4. Remove Extra Spaces From a String in JavaScript or Node.js
  5. Remove Numbers From a String in JavaScript or Node.js
  6. Get the Part Before a Character in a String in JavaScript or Node.js
  7. Get the Part After a Character in a String in JavaScript or Node.js
  8. How to Check if a Value is a String in JavaScript or Node.js
  9. Check If a String Includes All Strings in JavaScript/Node.js/TypeScript
  10. Check if a Value is a String in JavaScript and Node.js
  11. Limit and Truncate a String to a Given Length in JavaScript and Node.js
  12. Split a String into a List of Characters in JavaScript and Node.js
  13. How to Generage a UUID in Node.js
  14. Reverse a String in JavaScript or Node.js
  15. Split a String into a List of Lines in JavaScript or Node.js
  16. Split a String into a List of Words in JavaScript or Node.js
  17. Detect if a String is in camelCase Format in Javascript or Node.js
  18. Check If a String Is in Lowercase in JavaScript or Node.js
  19. Check If a String is in Uppercase in JavaScript or Node.js
  20. Get the Part After First Occurrence in a String in JavaScript or Node.js
  21. Get the Part Before First Occurrence in a String in JavaScript or Node.js
  22. Get the Part Before Last Occurrence in a String in JavaScript or Node.js
  23. Get the Part After Last Occurrence in a String in JavaScript or Node.js
  24. How to Count Words in a File
  25. How to Shuffle the Characters of a String in JavaScript or Node.js
  26. Append Characters or Words to a String in JavaScript or Node.js (Coming soon)
  27. Check if a String is Empty in JavaScript or Node.js (Coming soon)
  28. Ensure a String Ends with a Given Character in JavaScript or Node.js (Coming soon)
  29. Left-Trim Characters Off a String in JavaScript or Node.js (Coming soon)
  30. Right-Trim Characters Off a String in JavaScript or Node.js (Coming soon)
  31. Lowercase the First Character of a String in JavaScript or Node.js (Coming soon)
  32. Uppercase the First Character of a String in JavaScript or Node.js (Coming soon)
  33. Prepend Characters or Words to a String in JavaScript or Node.js (Coming soon)
  1. Get Number of Seconds Since Epoch in JavaScript
  2. Get Tomorrow’s Date in JavaScript
  3. Increase a Date in JavaScript by One Week
  4. Add Seconds to a Date in Node.js and JavaScript
  5. Add Month(s) to a Date in JavaScript or Node.js
  6. Add Week(s) to a Date in JavaScript or Node.js
  7. Get the Current Year in JavaScript or Node.js
  8. How to Get a UNIX Timestamp in JavaScript or Node.js
  9. How to Convert a UNIX Timestamp to a Date in JavaScript or Node.js
  10. Add Days to a Date in JavaScript or Node.js
  11. Get Yesterday's Date in JavaScript or Node.js
  12. Add Minutes to a Date in JavaScript or Node.js (Coming soon)
  13. Add Hours to a Date in JavaScript or Node.js (Coming soon)
  14. Check If a Date Is Today in JavaScript or Node.js (Coming soon)
  15. Check If a Date is Tomorrow in JavaScript or Node.js (Coming soon)
  16. Check If a Date is Yesterday in JavaScript or Node.js (Coming soon)
  17. How to Format a Date YYYY-MM-DD in JavaScript or Node.js (Coming soon)

Run Async Functions/Promises in Batches

Let’s say you have a list of 500 items and you need to perform an asynchronous operation. You don’t want to run the processing for all 500 items in parallel. At the same time, you also don’t want to waste resources by running all 500 items in sequence.

The idea: chunk the list with 500 items into smaller lists with 20 items each and run 20 items in parallel.

Use a Promise Pool

We recommend a promise pool to run batches of items concurrently. A promise pool runs the asynchronous operation at most X items in parallel. Precisely, it handles async pooling for you and tries to run the maximum allowed number of parallel operations.

The @supercharge/promise-pool package provides map-like, concurrent promise processing.

Here’s an example using a promise-pool to process a list of 50,000 users. The pool runs at most 20 operations concurrently:

const PromisePool = require('@supercharge/promise-pool')

const users = [  
  { id: 1, name: 'Marcus' },
  { id: 2, name: 'Norman' },
  { id: 3, name: 'Christian' },
  …,
  { id: 50000, name: 'Future Studio' }
]

/**
 * Process the list of 50,000 users with a concurrency of 20 items.
 * The promise pool takes the next task from the list as soon
 * as one of the active tasks in the pool finishes.
 */
const { results, errors } = await PromisePool  
  .for(users)
  .withConcurrency(20)
  .process(async data => {
    const user = await User.createIfNotExisting(data)

    return user
  })

The promise pool iterations through all items in the list, even though an operation may cause an error. The pool collects all errors and results and returns them in an object. You may use destructuring to access the result and errors properties directly.

Try to Avoid Promise.all()

JavaScript’s global Promise class provides the static all method which accepts an array of promises. The downsides of this approach: it runs all promises in the list simultaneously and waits for every promise to finish.

You may chunk your original list of items into smaller lists and use Promise.all on these smaller lists. This leads to the shortcoming that all promises must resolve before processing the next batch. This is where the promise-pool shines: it starts takes the next task from the list of operations as soon as one in the batch finishes.


Mentioned Resources

Explore the Library

Find interesting tutorials and solutions for your problems.