Syncing large amounts of data

Questions
1

Hello I have a workflow that pulls thousands of records from an external API and processes them in batches. Occasionally, the API returns 429 Too Many Requests, which causes some items to fail.
I’m thinking about using the Loop Over Items node with smaller batches and adding a delay between requests.
Would you recommend handling the retries directly in n8n, or using a queue/Redis approach for a larger production workflow?

Describe the problem/error/question

What is the error message (if any)?

Please share your workflow

(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)

Share the output returned by the last node

Information on your n8n setup

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
2

Hey @Amelia_Carter, while you wait for a response, here are some things that might help:

Suggested resources

Automatically matched to your question.

Docs:

Forum:

@Niffzy - you’ve helped with similar issues before, can you take a look?

Automatically suggested by n8n’s community bot. It’s a pilot - please share feedback here.

3

Hi @Amelia_Carter For a smaller workflow, I’d keep it inside n8n. Use loop over items with a small batch size and a wait node between batches.

For example:

Get Data

↓

Loop Over Items

↓

API Request

↓

429?

├─ No → Continue

└─ Yes → Wait → Retry

I’d also check the API’s Retry-After header if it provides one:

const retryAfter = Number($json.headers?.['retry-after'] || 5);

return [{

json: {

retryAfter

}

}];

For high-volume production workloads, I’d move the processing to a queue/worker setup so one rate-limit response doesn’t hold up the entire workflow.

4

Thanks for the reply @Emmas

I will check it out

5

For thousands of records I would stay inside n8n until the 429s become frequent enough to stall the whole run. Loop Over Items with a small batch size, a Wait between batches, and a retry branch that reads Retry-After when the API sends it is usually enough. Persist a cursor or last-successful id so a partial failure does not restart from page one. Once you are regularly hitting limits or several workflows share the same API budget, move the worker side to a queue so one rate-limit response cannot block unrelated jobs.

6

I’d keep this inside n8n until you actually need independent workers, but I’d add two things before scaling batch size: checkpointing and idempotency.

For 429s:

  • honor Retry-After when the API sends it

  • otherwise use bounded exponential backoff + a little jitter

  • retry the individual page/item batch, not the entire sync

  • cap attempts and send exhausted items to an exception queue/table instead of silently dropping them

For recovery, persist a checkpoint such as:

source + page/cursor + last_confirmed_item

and make the destination write idempotent with a stable source ID/upsert key.

That way a worker crash or retry does not mean “start thousands of records again.”

I’d move to Redis/a queue only when you need things like:

  • multiple workers consuming in parallel

  • backpressure independent of one workflow execution

  • jobs surviving worker restarts cleanly

  • different retry schedules per job

For one API sync, a DB-backed work table plus small n8n batches is often simpler and easier to operate than introducing Redis early.

7

Hi @Amelia_Carter, I can help make the sync recover cleanly when the API slows down or limits requests, so you don’t have to check failed records by hand. We could start with a copy of one workflow and a small test batch before touching the live setup. Did the changes suggested here solve the issue, or are records still failing?