N8n Workflow Fails to Resume Safely After Partial Execution (Idempotency & Checkpointing Issue)

Questions
1

Hi everyone, I’m running into a more advanced issue with an n8n workflow that processes large datasets, and I’m struggling to make it resumable without causing duplicates or data loss.
My workflow pulls data from an API using pagination, processes it in batches, and writes to a database:
HTTP Request (paginated) → Split In Batches → Function → Database
The problem happens when the workflow fails midway (e.g., API timeout or rate limit). When I rerun it:
• Some records get duplicated
• Some records are skipped
• I can’t reliably continue from where it stopped
Right now, I’m just looping through pages like this:
let page = 1;
while (true) {
const res = await this.helpers.httpRequest({
url: https://api.example.com/data?page=${page},
json: true
});
if (!res.data.length) break;
page++;
}
But there’s no real checkpointing, so if it crashes at page 20, I have to restart from page 1.
I’ve considered:
• Storing the last processed page in a DB
• Using upserts to avoid duplicates
• Breaking workflows into smaller executions
But I’m not sure what the best production pattern is in n8n for:
• Idempotent processing
• Checkpointing progress
• Safe retries without duplication

Describe the problem/error/question

How are people handling large, failure-prone workflows like this in n8n? Is there a recommended architecture or pattern for making workflows safely resumable?

What is the error message (if any)?

Please share your workflow

(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)

Share the output returned by the last node

Information on your n8n setup

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
3

Hi @Keira_Becky Your workflow has no state, so when it fails, it restarts blindly duplicates and skips.

Try to

  1. Store progress (checkpointing)
    Save the last processed page (or cursor) in a DB:

{
“last_page”: 20
}

On restart, start from that page, not 1

  1. Make writes idempotent (very important)
    Use upsert or unique keys in DB:

INSERT INTO users (id, name)
VALUES (…)
ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name;

This prevents duplicates if the same data runs again.

  1. Process in small chunks
    Instead of one long run:
    • Trigger per page (or small batch)
    • Save progress after each success
  2. Retry safely
    If a batch fails:
    • Retry only that batch
    • Don’t restart everything
4

Do you mean I should Get last_page → Fetch page → Process → Save last_page → Repeat

5

Yeah @Keira_Becky Get last_page → Fetch next page → Process → Save last_page → Repeat

Simple explanation
• Get last_page → read from DB (e.g. 20)
• Fetch page 21 → continue from where you stopped
• Process → handle and store data
• Save last_page = 21 → update progress
• Repeat → move to next page

If the workflow crashes at page 21:
• Next run starts from 21, not 1
• No duplicates, no skipping

6

@Niffzy i got it fix thanks :blush: