Setting up a Queue or list to process API calls and avoid

Questions
1

I have a sequence of workflows that fire when a user signs up or creates and account for the first time. One of these workflows does a high volume of API calls (can be 1000+) through loops to an API that allows 240 calls per min. I have tested the process with creating 5 users back to back and I ended up exceeding my API call limit and erroring the workflow. I’m thinking of scalability. I have played with timing between calls but that can double or triple what can already be a long process (it happens on the back end so I guess that is ok) But I was hoping someone could recommend a way to build a queue system. I have looked into queue mode but since my limitations are API related and not system based I wasn’t sure if that would be a good route. As I was brain storming I was thinking of instead of actually starting the workflow with the signup maybe add the user to a list, and then always have a process that it checking and processing that list (executing the workflow) one at a time?!? Could that be done? Thank you for any insight.

Information on your n8n setup

  • n8n version: Version 0.227.1
  • Database (default: SQLite): MariaDB
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app): npm
  • Operating system: Ubuntu Server 22.04
2

One way would be to use an actual queue like RabbitMQ.
You can then have one workflow which push a message for each user that signs up.
And another workflow which listens to those messages and is configured to only process one at a time.

2 Likes
3

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.