Parallel HTTP requests and race conditions

Questions
1

Let us define a workflow which has 5 HTTP requests in serial. Nodes 1, 2 and 3 can be ran in parallel. Nodes 4 and 5 depends on results of 1, 2 and 3. How can we can parallelize this workflow to improve performance in a n8n idiomatic way?

Share the output returned by the last node

Information on your n8n setup

core

  • n8nVersion: 1.80.5
  • platform: docker (self-hosted)
  • nodeJsVersion: 20.18.3
  • database: postgres
  • executionMode: scaling
  • concurrency: -1
  • license: enterprise (production)
  • consumerId: 94f874ca-d834-43c9-8d19-475eb62ba0ed

storage

  • success: all
  • error: all
  • progress: false
  • manual: true
  • binaryMode: memory

pruning

  • enabled: true
  • maxAge: 336 hours
  • maxCount: 10000 executions

client

  • userAgent: mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/133.0.0.0 safari/537.36
  • isTouchDevice: false

Generated at: 2025-02-28T18:09:12.796Z

1 Like
2

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
3

You can use subworkflows to make parallel executions.
When you call a subworkflow, you can set it to NOT WAIT for the execution to finish.

image
image643×400 9.95 KB

As for the other nodes, you can build them normally, since their data depend of previous nodes.

If my reply answers your question, please remember to mark it as a solution.

4

Well, nodes 1, 2 and 3 are parallelizable, but we still need their output to proceed to 4 and 5.

5

By building a sub-workflow, the last executed node always sends the data to the parent-workflow.

You can aggregate the data from nodes 1,2,3 and send it to the parent workflow to continue the process.

6

I’m facing a similar issue and I don’t think there’s a clear way to solve this. I have an HTTP node that makes an API call that takes quite a bit to execute. The input to this node is usually 8-12 items. I’d like to make all those 8-12 HTTP calls concurrently. Not one after the other. Calling another workflow and not waiting means this workflow just moves on. So how does that help? I imagine that’s why @brunolnetto replied by emphasising that the outputs of the callee still need to be aggragated, so the caller workflow must wait. The problem is not waiting, the problem is that the calls are made one after the other, and not all at the same time.

7

I understand the issue. The previous answer was indeed not completely right.

There is loads of workarounds to be found but the easiest and most stable would be to have a table somewhere or for example redis to keep track of the requests/subflows done in parallel and then have that trigger a last workflow that can aggregate the data and continue processing. Aggregating I normally do by sending the output to a redis cache and grabbing it from there.

There is not really a native way to run certain things in parallel and then wait for them all to finish and continue the workflow.

8

@mig82 You can try the workflows on URL [1]. To test, take the below steps on a linux machine:

  1. Copy each json content on a workflow;
  2. Activate the workflows;
  3. Run workflow for parallel manager in test mode;
  4. Change permission of script main.sh with command run chmod +x main.sh;
  5. Change the variables base_test_url and base_prod_url content to your base URLs;
  6. Run script with command ./main.sh.

The output will be something like:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1656  100   957  100   699    218    159  0:00:04  0:00:04 --:--:--   377

real    0m4.385s
user    0m0.068s
sys     0m0.000s
ID: 1 -> Decoded: {"y8mHorj7":5}
ID: 2 -> Decoded: {"NLlrdOBN":96}
ID: 3 -> Decoded: {"Zcg29onM":96}
ID: 5 -> Decoded: {"a":1,"b":4,"c":9,"y8mHorj7":25,"NLlrdOBN":9216,"Zcg29onM":9216}
ID: 4 -> Decoded: {"a":1,"b":4,"c":9,"y8mHorj7":25,"NLlrdOBN":9216,"Zcg29onM":9216}

[1] assets/n8n at main · alloyha/assets · GitHub