Handling Rate Limits Across Parallel Workflows Without Losing Throughput

Questions
1

Hi everyone, I’m dealing with a more advanced scaling issue and could use some guidance.
I have multiple workflows running in parallel (triggered by webhooks and cron) that all call the same external API, which has strict rate limits (e.g., 100 requests per minute). As usage grows, I’m starting to hit 429 errors, and retries are making things worse by creating spikes.
Current setup looks like:
Webhook/Cron → Split In Batches → HTTP Request → Process
Right now:
Each workflow handles its own retries
No shared awareness of rate limits
Occasional bursts exceed API limits
Example retry logic:
if ($json.statusCode === 429) {
await new Promise(resolve => setTimeout(resolve, 1000));
return $input.all();
}
The problem:
Multiple workflows retry at the same time → thundering herd effect
Slowing down one workflow doesn’t fix others
I need a global rate limit control, not per-workflow
I’m considering:
Using a queue (Redis / Bull)
Centralizing API calls into a single workflow
Building a custom rate limiter with shared state

Describe the problem/error/question

What’s the best pattern in n8n to handle global rate limiting across multiple workflows while still keeping good throughput?

What is the error message (if any)?

Please share your workflow

(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)

Share the output returned by the last node

Information on your n8n setup

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
3

Hi @Roseline You’re running into a classic rate limit problem and your current setup (per-workflow retries) will always cause spikes. Each workflow acts independently, so retries happen at the same time → 429 storms.

Try to Centralize the API calls

Instead of letting every workflow hit the API:

All workflows → Queue → Single Worker → HTTP Request → Process

You can Create one “API worker” workflow
• Other workflows send requests to it (Webhook / Execute Workflow)
• Worker processes requests one by one or in controlled batches

Or use Use Redis / queue system
• Push jobs into queue
• Worker pulls at a fixed rate (e.g. 100/min)

4

Thanks for the suggestions i can see is a global rate limit

5

Thanks for the suggestions @Niffzy

6

The centralized workflow approach is definitely the cleanest solution. One thing worth adding on top of that pattern: use a Wait node with a calculated delay instead of a fixed one.

If your API allows 100 req/min, you can compute the delay per request as (60 / 100) * 1000 = 600ms. Then each call to the central dispatcher just waits 600ms before passing through. No thundering herd, no shared Redis state needed, and throughput stays predictable.

For even more resilience, you can also add a “circuit breaker” pattern - if you get 3 consecutive 429s, stop sending for a full 60s before resuming. This protects you during any unexpected burst.

Hope that helps!

– Nguyen Thieu Toan (Jay Nguyen) - n8n Verified Creator