Hi everyone,
We are running n8n in queue mode (main + 3 workers, Docker Compose) and trying to implement a robust centralized error-handling pattern. We want to validate whether our current approach is correct or if there is a better way.
What we expect:
- Any workflow step fails → Error Trigger workflow fires
- Handler detects retry count, retries from the failed step
- After max retries (3), routes to a dead-letter / alert path
What we are currently doing:
We have a dedicated Error Trigger workflow. Inside it:
- Call
GET /api/v1/executions/{id} to fetch failed execution details
- Walk the
retryOf chain to derive current retry depth
- If
retryCount < 3, call:
POST /api/v1/executions/{id}/retry with body { "loadWorkflowExecution": true }
- If limit reached → dead-letter branch (Slack / email alert placeholder)
This works, but we have a few open questions:
1. Is this API-based retry pattern the recommended production approach, or is there a better native mechanism?
2. When retrying with loadWorkflowExecution: true, is it guaranteed to resume from the failed node only, or can earlier nodes re-execute in some cases?
3. For workflows with side effects (DB inserts, external API calls), what is the recommended way to make retries safe and avoid duplicate data?
4. Code node + large list iteration scenario (most important for us):
We have a Code node that iterates a large list (e.g., 5000 items). If a failure happens mid-way (say item 700), when retry happens:
- Does n8n resume from item 700, or does it start the Code node from item 1 again?
- If it restarts from item 1, should we externalize progress state and make each item idempotent ourselves?
- Is there any built-in checkpointing support for this kind of iterative workload?
5. Is there an official or community-recommended pattern for long-running iterative workflows where partial progress must be preserved across retries?
Any guidance, reference workflows, or links to docs would be very helpful.
Thanks!
Hey Devdatt_99,
For question 3, I can share something from my own experience: idempotency on retries is the core issue, not the retry mechanism itself. What works for me: Before every DB insert or external API call, add a check to see if the record already exists, based on a unique ID that’s generated before the first attempt. That way, the node can be retried as many times as you want without creating duplicates.
For question 4, unfortunately I can’t contribute anything reliable — I don’t work with iterations over lists of that size in n8n. My gut feeling would be that the code node completely restarts on a retry (no internal checkpointing), but that should be confirmed by someone with direct queue mode experience.
Your error trigger workflow architecture sounds solid in principle. I’m curious what the community has to say about question 1 — whether the API-based retry approach is really the recommended production method or if there’s a native way that I’m not aware of.
1 Like
Good, detailed question. I run n8n in queue mode in production, here’s how I’d answer each.
-
Two layers, not one. Use node-level Retry On Fail (maxTries + wait) for transient failures like network blips and 429s, which handles most cases in place. Keep your Error Trigger workflow for what’s left: alerting, dead-letter, recording the failure. The API retry-from-failed is fine for controlled re-runs, but treat it as “re-run the execution,” not a guaranteed resume. You’re essentially rebuilding orchestration n8n doesn’t guarantee, so lean on idempotency to make re-runs safe rather than trying to make resume perfect.
-
Don’t rely on it as a clean checkpoint. A retry replays using saved data for already-completed nodes, but it’s at-least-once, not exactly-once. Any node with side effects downstream of the failure point can re-execute. Design as if anything after a non-idempotent step may run again.
-
This is the real lever, more than the retry mechanism. Pre-generate a deterministic key for each unit of work (derived from the business data, not a random per-run id) and gate every side effect on it: a unique constraint + INSERT … ON CONFLICT DO NOTHING for DB writes, the provider’s idempotency key for external APIs (Stripe etc.), and a small dedupe/ledger table keyed by that id so a re-run is a no-op. I keep a ledger table exactly for this, so retries can’t double-charge or double-insert.
-
The Code node has no checkpointing. On retry it re-runs from item 1, not item 700. n8n doesn’t checkpoint inside a node. So for 5000 items, don’t iterate them inside one Code node. Either (a) make each item idempotent and keep a processed-ledger so the re-run skips the ~700 already done, or (b) restructure to item/batch level with Split In Batches (or a sub-workflow called per batch) so a failure is scoped to one batch and partial progress lives in your own state, not in n8n’s execution.
-
The durable worklist pattern. Externalize state: a table of items with a status column (pending / done / failed). Drive the loop off WHERE status = pending, process in batches, mark each item done idempotently as it completes. Any retry just picks up the pending items, so resume is automatic and you never reprocess a done one. Combine with node-level Retry On Fail for transient per-item errors and a failed status as your dead-letter.
Net: stop trying to make n8n resume mid-node perfectly, and instead make every item idempotent + externalize progress. Then retries, native or API, become safe by construction.
1 Like
Thanks @syed_noor, this fills exactly the gaps I left open. The durable worklist pattern (status column, WHERE pending, mark idempotent) is far more robust than anything I’d have tried to build with the native retry mechanisms. Will use this approach for larger iterations going forward, appreciate the detailed breakdown.
Glad it helped, Rene — happy to. It really comes into its own on the bigger iterations: once progress lives in your own table instead of n8n’s execution, resume takes care of itself. Good luck with it.