2. Tutorials
  3. Node.js — Run Async Functions in Batches

Node.js — Run Async Functions in Batches

When working with arrays of data, you may need to run an asynchronous operation on each item. The Promise.all() method provides a convenient interface to run a list of promises in parallel, but it has some limitations.

Using Promise.all() may overload the available resources, especially when reaching out to the Internet or interacting with the database. Depending on your list size, you may send out thousands of requests parallelly into the wild.

This tutorial shows you how to use a promise pool to run a batch of promises in parallel and the batches themselves in sequence.

Node.js Series Overview

  1. Increase the Memory Limit for Your Process
  2. Why You Should Add “node” in Your Travis Config
  3. Create a PDF from HTML with Puppeteer and Handlebars
  4. Create Your Own Custom Error
  5. Retrieve a Request’s IP Address in Node.js
  6. Detect the Node.js Version in a Running Process or App
  7. How to Base64 Encode/Decode a Value in Node.js
  8. Check if a Value Is Null or Undefined in JavaScript or Node.js
  9. How to Fix “Uncaught SyntaxError: Cannot use import statement outside a module”
  10. Fix „Socket Hang Up“ Errors
  11. Nested Destructuring in JavaScript or Node.js
  12. Cache Computation Results in a Local Variable
  13. ESM Bypass Cache for Dynamic Imports
  14. How to Fix ESM Error “Directory import is not supported resolving ES modules imported”
  1. String Replace All Appearances
  2. Remove All Whitespace From a String in JavaScript
  3. Generate a Random ID or String in Node.js or JavaScript
  4. Remove Extra Spaces From a String in JavaScript or Node.js
  5. Remove Numbers From a String in JavaScript or Node.js
  6. Get the Part Before a Character in a String in JavaScript or Node.js
  7. Get the Part After a Character in a String in JavaScript or Node.js
  8. How to Check if a Value is a String in JavaScript or Node.js
  9. Check If a String Includes All Strings in JavaScript/Node.js/TypeScript
  10. Check if a Value is a String in JavaScript and Node.js
  11. Limit and Truncate a String to a Given Length in JavaScript and Node.js
  12. Split a String into a List of Characters in JavaScript and Node.js
  13. How to Generage a UUID in Node.js
  14. Reverse a String in JavaScript or Node.js
  15. Split a String into a List of Lines in JavaScript or Node.js
  16. Split a String into a List of Words in JavaScript or Node.js
  17. Detect if a String is in camelCase Format in Javascript or Node.js
  18. Check If a String Is in Lowercase in JavaScript or Node.js
  19. Check If a String is in Uppercase in JavaScript or Node.js
  20. Get the Part After First Occurrence in a String in JavaScript or Node.js
  21. Get the Part Before First Occurrence in a String in JavaScript or Node.js
  22. Get the Part Before Last Occurrence in a String in JavaScript or Node.js
  23. Get the Part After Last Occurrence in a String in JavaScript or Node.js
  24. How to Count Words in a File
  25. How to Shuffle the Characters of a String in JavaScript or Node.js
  26. Append Characters or Words to a String in JavaScript or Node.js
  27. Check if a String is Empty in JavaScript or Node.js
  28. Ensure a String Ends with a Given Character in JavaScript or Node.js
  29. Left-Trim Characters Off a String in JavaScript or Node.js
  30. Right-Trim Characters Off a String in JavaScript or Node.js
  31. Lowercase the First Character of a String in JavaScript or Node.js
  32. Uppercase the First Character of a String in JavaScript or Node.js
  33. Prepend Characters or Words to a String in JavaScript or Node.js
  34. Check if a String is a Number
  35. Convert a String to Buffer
  36. Prevent Line Breaks in String Template Literals
  37. How to Implement a Custom `toString` Method
  38. What Is `Symbol.toStringTag` and How to Use It (Coming soon)
  1. Filter Data in Streams
  1. Get Number of Seconds Since Epoch in JavaScript
  2. Get Tomorrow’s Date in JavaScript
  3. Increase a Date in JavaScript by One Week
  4. Add Seconds to a Date in Node.js and JavaScript
  5. Add Month(s) to a Date in JavaScript or Node.js
  6. Add Week(s) to a Date in JavaScript or Node.js
  7. Get the Current Year in JavaScript or Node.js
  8. How to Get a UNIX Timestamp in JavaScript or Node.js
  9. How to Convert a UNIX Timestamp to a Date in JavaScript or Node.js
  10. Add Days to a Date in JavaScript or Node.js
  11. Get Yesterday's Date in JavaScript or Node.js
  12. Add Minutes to a Date in JavaScript or Node.js
  13. Add Hours to a Date in JavaScript or Node.js
  14. Check If a Date Is Today in JavaScript or Node.js
  15. Check If a Date is Tomorrow in JavaScript or Node.js
  16. Check If a Date is Yesterday in JavaScript or Node.js
  17. How to Format a Date YYYY-MM-DD in JavaScript or Node.js
  18. Convert a Date to UTC in JavaScript or Node.js
  19. Get Number of Days in a Month
  1. How to Run an Asynchronous Function in Array.map()
  2. How to Reset and Empty an Array
  3. for…of vs. for…in Loops
  4. Clone/Copy an Array in JavaScript and Node.js
  5. Get an Array With Unique Values (Delete Duplicates)
  6. Sort an Array of Integers in JavaScript and Node.js
  7. Sort a Boolean Array in JavaScript, TypeScript, or Node.js
  8. Check If an Array Contains a Given Value in JavaScript or Node.js
  9. Add an Item to the Beginning of an Array in JavaScript or Node.js
  10. Append an Item at the End of an Array in JavaScript or Node.js
  11. How to Exit and Stop a for Loop in JavaScript and Node.js
  12. Split an Array Into Smaller Array Chunks in JavaScript and Node.js
  13. How to Get an Index in a for…of Loop in JavaScript and Node.js
  14. How to Exit, Stop, or Break an Array#forEach Loop in JavaScript or Node.js
  15. Retrieve a Random Item From an Array in JavaScript or Node.js
  16. How to Reverse an Array in JavaScript and Node.js
  17. Sort an Array of Strings in JavaScript, TypeScript or Node.js
  18. Sort an Array of Objects in JavaScript, TypeScript or Node.js
  19. Check If a Value Is an Array in JavaScript or Node.js
  20. Join an Array of Strings to a Single String Value
  21. Create and Fill an Array Containing N Items
  1. Human-Readable JSON.stringify() With Spaces and Line Breaks
  2. Write a JSON Object to a File
  3. Create a Custom “toJSON” Function in Node.js and JavaScript
  4. Securely Parse JSON
  5. How to JSON Stringify Errors
  1. Check If a Value Is Iterable in JavaScript or Node.js
  2. Check If a Value Is Async Iterable in JavaScript or Node.js
  3. What is `Symbol.iterator`?
  4. How to Implement an Iterator
  1. Extend Multiple Classes (Multi Inheritance)
  2. Retrieve the Class Name at Runtime in JavaScript and Node.js
  3. Deprecate a Method in JavaScript and TypeScript
  1. Generate a Random Number in Range With JavaScript/Node.js
  2. Ensure a Positive Number in JavaScript or Node.js
  3. Check if a Number Is Infinity
  4. Check If a Number has Decimal Places in JavaScript or Node.js
  5. Use Numeric Separators for Better Readability
  6. Convert Boolean to Number
  7. Convert Number to Boolean
  8. Check if a Value is a Number
  1. How to Merge Objects
  2. How to Check if an Object is Empty in JavaScript or Node.js
  3. How to CamelCase Keys of an Object in JavaScript or Node.js
  4. How to Snake_Case Keys of an Object in JavaScript or Node.js
  5. How to Destructure a Dynamic Key in JavaScript or Node.js
  6. How to Get All Keys (Including Symbols) from an Object in JavaScript or Node.js
  7. How to Delete a Key From an Object in JavaScript or Node.js
  8. Iterate Through an Object’s Keys and Values in JavaScript or Node.js
  9. How to Convert URLSearchParams to Object
  10. Check If a Value Is an Object in JavaScript or Node.js
  11. Conditionally Add Properties to an Object in JavaScript or Node.js
  12. How to Lowercase Keys of an Object in JavaScript or Node.js
  13. Unset an Object Property
  1. Get a File’s Created Date
  2. Get a File’s Last Modified or Updated Date of a File
  3. How to Create an Empty File
  4. Check If a Path or File Exists
  5. How to Rename a File
  6. Check If a Path Is a Directory
  7. Check If a Path Is a File
  8. Retrieve the Path to the User’s Home Directory
  9. How to Touch a File
  10. Read File Content as String
  11. Check If a Directory Is Empty
  12. How to Create a Directory (and Parents If Needed)
  13. Get a File‘s Extension
  14. Get the Size of a File
  15. Get a File Name (With or Without Extension)
  16. Read a JSON File
  17. Delete a File From Disk
  18. Check if a Path is a SymLink (Symbolic Link)
  19. Check if a Path is a Socket
  20. Check if a Path is a FIFO File (Named Pipe)
  21. Check if a Path is a File URL
  22. Handling `fs.Stats` Constructor is Deprecated
  23. What is `Fs.realpath`?
  1. Create From Object
  2. Transform to an Object
  1. Determine the Node.js Version Running Your Script
  2. Show Your NODE_MODULE_VERSION
  1. Check if a Value is a Symbol in JavaScript or Node.js
  1. Detect if Running on Linux
  2. Detect if Running on macOS
  3. Detect if Running on Windows
  4. Check if Running on 64bit or 32bit Platform
  5. Constant for Platform-Specific Newline
  6. Determine the Current Operating System
  7. Retrieve the OS Directory for Temporary Files
  1. How to Download a File
  2. Strict Type for Request Headers with TypeScript
  1. Calculate an MD5 Hash
  2. Retrieve the List of Supported Hash Algorithms
  3. Calculate a SHA256 Hash
  4. Calculate a SHA512 Hash
  1. Using the Styletext Utility for Terminal Text Formatting

Run Async Functions/Promises in Batches

Let’s say you have a list of 500 items and you need to perform an asynchronous operation. You don’t want to run the processing for all 500 items in parallel. At the same time, you also don’t want to waste resources by running all 500 items in sequence.

The idea: chunk the list with 500 items into smaller lists with 20 items each and run 20 items in parallel.

Use a Promise Pool

We recommend a promise pool to run batches of items concurrently. A promise pool runs the asynchronous operation at most X items in parallel. Precisely, it handles async pooling for you and tries to run the maximum allowed number of parallel operations.

The @supercharge/promise-pool package provides map-like, concurrent promise processing.

Here’s an example using a promise-pool to process a list of 50,000 users. The pool runs at most 20 operations concurrently:

const PromisePool = require('@supercharge/promise-pool')

const users = [  
  { id: 1, name: 'Marcus' },
  { id: 2, name: 'Norman' },
  { id: 3, name: 'Christian' },
  …,
  { id: 50000, name: 'Future Studio' }
]

/**
 * Process the list of 50,000 users with a concurrency of 20 items.
 * The promise pool takes the next task from the list as soon
 * as one of the active tasks in the pool finishes.
 */
const { results, errors } = await PromisePool  
  .for(users)
  .withConcurrency(20)
  .process(async data => {
    const user = await User.createIfNotExisting(data)

    return user
  })

The promise pool iterations through all items in the list, even though an operation may cause an error. The pool collects all errors and results and returns them in an object. You may use destructuring to access the result and errors properties directly.

Try to Avoid Promise.all()

JavaScript’s global Promise class provides the static all method which accepts an array of promises. The downsides of this approach: it runs all promises in the list simultaneously and waits for every promise to finish.

You may chunk your original list of items into smaller lists and use Promise.all on these smaller lists. This leads to the shortcoming that all promises must resolve before processing the next batch. This is where the promise-pool shines: it starts takes the next task from the list of operations as soon as one in the batch finishes.

Mentioned Resources

Marcus Pöhls

Marcus is a fullstack JS developer. He’s passionate about the hapi framework for Node.js and loves to build web apps and APIs. Creator of Futureflix and the “learn hapi” learning path.

Explore the Library

Find interesting tutorials and solutions for your problems.

Android

Node.js

Server

Miscellaneous