I built a practice workflow to explore how to make batch processing hold up when things go wrong. Sharing it in case it’s useful, and I’d love feedback.
What it does
Takes a batch of records and validates each one. Sends invalid records down a separate path and logs them Removes duplicates Calls an API with retry handling . Logs successful calls and API errors to Google Sheets. Ends with a run summary (total, invalid, duplicates removed, successes, errors)
What I learned
Splitting success and error paths early makes debugging much easier. Logging every failure means nothing disappears silently. Final summary shows at a glance whether a run was healthy.
I’m still improving it. Next I plan to add a failure alert (email or Slack) and run it on a schedule. If you have ideas for making it more robust, please let me know. I’m also open to small contract projects with n8n, APIs, webhooks and AI agents.
One edge case worth testing is when the API actually completes the action but the response times out. The retry can create the same record twice, even if you already cleaned duplicates earlier in the batch
I’d use a stable idempotency key wherever the API supports it, and keep a persistent record of which operations have already finished. For anything uncertain, checking the external system before retrying makes the whole workflow a lot safer
Thanks, that’s a great edge case. I hadn’t tested a timeout where the API finishes but the response is lost, so my retry could create a duplicate. I’m going to add a processed-records log in Google Sheets and check it before each call, and use an idempotency key where the API supports it. I’ll post an update once it’s done.
Update: thanks for the edge case, Mahdi. I added a processed-records log and a check before each API call. Re-running the same batch now sends 0 records, and a new record (ID 6) went through while the old ones were skipped. Screenshots attached. Next I’m adding failure alerts
Nice progress, Muskan. The rerun test is a solid check that already-processed records get skipped while new ones still go through.
One test I’d keep on the list is a timeout that happens after the external API has already completed the action but before the success log is updated. That’s exactly where the processed-records check alone can miss a duplicate on retry.
The failure alerts sound like a useful next step too — especially if they can distinguish a confirmed failure from an uncertain outcome
Nice structure. A few things that have saved me on similar client workflows:
- For the failure alert, use a separate error workflow. Build a small workflow that starts with an Error Trigger, sends the Slack or email message, and set it as the Error Workflow in this workflow’s settings. That way you still get alerted when the whole run crashes, not only when a single API call fails inside your error path.
- Keep dedup keys between runs. Removing duplicates inside one batch is good, but on a schedule the same record often shows up again the next day. Storing processed IDs (a Data Table works well for this) and checking against it before the API call stops repeat processing.
- Write to Google Sheets once per run, not once per item. With bigger batches you can hit the Sheets API write limits. Collecting the log rows and appending them in one call at the end is faster and avoids random 429 errors in the logging step itself.
- Control the pace of the API calls. Loop Over Items with a small batch size plus a short Wait between batches usually keeps you under the API’s rate limit, so you rely less on retries.
Mahdi’s point about idempotency keys is the big one for anything that creates records.
