Handle API rate limits and retries in n8n without creating duplicate requests

Questions
1

Hi guys I have an n8n workflow that processes customer records and sends data to an external API. When the API starts returning 429 Too Many Requests, n8n retries the requests, but during heavy traffic I sometimes end up with duplicate API calls.
My current workflow is basically:
PostgreSQL → Loop Over Items → HTTP Request → Update PostgreSQL
I’m considering storing the API request state in PostgreSQL: CREATE TABLE api_jobs (
id BIGSERIAL PRIMARY KEY,
customer_id BIGINT NOT NULL,
status TEXT NOT NULL DEFAULT ‘pending’,
attempts INT NOT NULL DEFAULT 0,
last_attempt_at TIMESTAMP,
external_id TEXT
);
And before sending the request: UPDATE api_jobs
SET
status = ‘processing’,
attempts = attempts + 1,
last_attempt_at = NOW()
WHERE id = $1
AND status = ‘pending’
RETURNING *;
Would you recommend using n8n’s retry settings, a PostgreSQL-backed retry queue, exponential backoff, or a combination of these?
And how would you make the API calls idempotent so that a retry doesn’t accidentally create the same customer/order twice?

Describe the problem/error/question

What is the error message (if any)?

Please share your workflow

(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)

Share the output returned by the last node

Information on your n8n setup

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
2

Hey @anon20761130, while you wait for a response, here are some things that might help:

Suggested resources

Automatically matched to your question.

Docs:

Forum:

@Niffzy - you’ve helped with similar issues before, can you take a look?

Automatically suggested by n8n’s community bot. It’s a pilot - please share feedback here.

3

@anon20761130 For this scenario, I’d use PostgreSQL job tracking + exponential backoff + API idempotency keys.

The main goal is to retry failed requests without creating duplicates.

  1. Track the job

UPDATE api_jobs
SET
status = ‘processing’,
attempts = attempts + 1,
last_attempt_at = NOW()
WHERE id = $1
AND status = ‘pending’
RETURNING *;

Only the worker that successfully claims the job should process it.

  1. Retry intelligently
    For 429 and temporary 5xx errors, use exponential backoff: 2s → 4s → 8s → 16s

Respect Retry-After when the API provides it.

  1. Prevent duplicates
    If the API supports idempotency keys, send a stable key such as: customer-123-order-456

That way, if n8n retries the same request, the API can recognize it as the same operation.

4

@anon20761130 on your loop over items node add a wait node inside the loop

5

Hi @anon20761130
Retry On Fail reruns the node inside the same execution, so those attempts all finish before Update PostgreSQL runs and attempts never increments. The queue has no record the request went out, which is how the same customer gets sent twice. Run one retry layer, not two: turn Retry On Fail off on the HTTP Request node and set On Error to “Continue (using error output)”. The 429 then arrives as an item on the error branch, Update PostgreSQL records the attempt, and the next scheduled run does the backoff.
The second duplicate source is the claim query. WHERE status = 'pending' leaves a row stuck on ‘processing’ forever if n8n restarts mid-request, so reclaim on age rather than widening the status filter:

WHERE id = $1
  AND (
    status = 'pending'
    OR (status = 'processing' AND last_attempt_at < NOW() - INTERVAL '5 minutes')
  )