Hi
I’m building a multi-tenant automation platform with n8n, and some workflows can run for several minutes because they interact with external APIs, AI services, or file processing.
Webhook
↓
Fetch Data
↓
AI Processing
↓
External API
↓
Update Database
My concern is what happens if a workflow times out or a worker crashes halfway through processing.
I’m trying to design a system that’s resilient and can recover without creating duplicate work or losing progress.
I’m considering:
Breaking large workflows into smaller jobs
Queue-based processing
Saving checkpoints after each step
Retry logic with idempotency
For those running n8n in production:
How do you handle long-running workflows?
Do you split them into smaller workflows or keep them as a single execution?
Describe the problem/error/question
What is the error message (if any)?
Please share your workflow
(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)
How do you resume processing after a failure without starting from the beginning?
I would love to hear how others are handling this in production, especially in high-volume environments.
For long-running workflows, the best approach is usually to avoid making one huge workflow handle everything.
Recommended pattern
Break the process into smaller steps:
Webhook
↓
Create Job
↓
Queue
↓
Process Step 1
↓
Save Progress
↓
Process Step 2
↓
Complete
Store workflow progress/state in a database or shared storage.
Make each step retry-safe (idempotent).
Use queues for heavy processing.
Add checkpoints so failed jobs can resume instead of restarting.
For example
Instead of:Download File → Process 1M Records → AI Analysis → Upload Result
Use:Download File → Save Job → Process Chunks → Update Status
Try to Avoid
Very long executions that hold resources for hours
Relying on workflow memory for progress
Restarting everything after a small failure
Instead of one giant workflow, split your process into a Parent (Orchestrator) and several Child (Worker) workflows.
The Orchestrator: Handles the Webhook, validates the request, and creates a “Job” record in your database (e.g., PostgreSQL or n8n Data Table) with a status of PENDING.
The Workers: Each major step (Fetch → AI → API → DB) becomes its own workflow called via the Execute Workflow node.
Why this works: If the “AI Processing” worker crashes, the Orchestrator knows exactly where it stopped because the Job record in your DB hasn’t been updated to AI_COMPLETED
For high-volume multi-tenant platforms, do not process the logic inside the Webhook response window.
The Pattern: Webhook → Push to Queue (RabbitMQ, Redis, or even a simple DB table) → Respond 202 Accepted to the client.
The Consumer: A separate n8n workflow triggered by a Poll or a Queue Trigger that picks up the job and processes it.
Benefit: This decouples the ingestion from the processing. If your AI service is slow or hitting rate limits, the jobs simply sit in the queue rather than timing out the HTTP connection.
To avoid starting from the beginning after a failure, implement External State Tracking.
The Checkpoint Table: Maintain a table with: job_id, current_step, payload_snapshot, and status.
The Logic: At the start of every worker workflow, check the current_step. If the job is marked as FETCH_DATA_COMPLETE, the workflow skips the fetch step and jumps straight to AI processing.
Resumption: You can build a “Recovery Workflow” that runs every hour, looks for jobs stuck in PROCESSING for more than X minutes, and re-triggers them.
To prevent duplicate work (e.g., charging a customer twice or creating duplicate DB entries), you must implement Idempotency Keys.
The Key: Generate a unique request_id at the Webhook level. Pass this ID to every external API call.
The API Side: Ensure the external APIs you use support idempotency keys (many AI and Payment APIs do). If they don’t, perform a “Lookup” before “Create” (e.g., Check if record with this request_id already exists in the target DB).
n8n Node Settings: Use the Error Handling tab on critical nodes. Set “Retry on Fail” with an exponential backoff to handle transient network blips without failing the whole workflow.
@Selena_Gloria Great breakdown above everyone. One practical addition: watch out for n8n’s own execution/queue mode settings, not just your app-level design, EXECUTIONS_TIMEOUT and EXECUTIONS_TIMEOUT_MAX cap how long a single execution can run before n8n itself kills it, separate from any timeout logic you build. If you’re on queue mode, also check worker concurrency (N8N_CONCURRENCY_PRODUCTION_LIMIT) a stuck long-running job can otherwise block a worker slot and starve other tenants’ jobs in a multi-tenant setup.
Hi @Selena_Gloria
Splitting into sub-workflows does not shorten anything by itself. The Execute Sub-workflow node has “Wait for Sub-Workflow Completion” turned on by default, so the parent execution stays open until the last child returns and the whole chain is still a single long execution on the canvas only. Turn that option off on the steps whose output the parent does not need, and each child becomes an independent execution while the parent finishes at the handoff.
The setting that decides what survives a kill is EXECUTIONS_DATA_SAVE_ON_PROGRESS, which is false by default. An execution that dies part-way records nothing node by node, so after the fact there is no way to see how far it actually got. Turning it on gives you that, at the cost of a database write after every node.
