Handling HTTP Requests errors (retries mechanism after X delay)

Questions
,
1

Hi,

I did some research and haven’t found how to implement an automatic reply mechanism after X delay when an HTTP request fails.

It is common when we are dealing with API requests to get a different status code than “200”. An API request can fail for diverse reasons like:

  1. API server currently down
  2. Rate limit exceeded
  3. Etc…

So what I am trying to do in my workflow is to set a retry mechanism but so far I haven’t been able to successfully implement it.

In my use case, I process multiple items (e.g. 100 different URLs) and to build my final report, this is critical to have the data of these 100 URLs.

So in the HTTP Request node, in the settings, I have set the error "Continue (using error output) which opens a branch “error” when one or multiple items fail.

I tried a few things, like adding a wait node in the error branch and then sending it back to the same HTTP request node (which is part of a Loop), but it doubled my items. For example, if I have initially 4 items and one fails, after it waits and loops back to the HTTP request node, I end up having 8 items.

What I need is a way to reprocess only the failed items and then complete the loop only when all my items have been successfully processed.

Scenario: I process 1000 URLs, 400 URLs processed correctly, and then, the API service goes down for like 15 minutes… then it retries those 600 URLs left every 5 minutes until the API service goes up again, so it retries in a loop until API go up. When the API goes up again, then the 600 URLs continue to be processed as normal. When done, all the 1000 items can exit the loop node as normal like there were no API errors.

This is a very common scenario so I’m quite surprised to have not seen any information on how to handle this. I would highly appreciate your help on that.

Thanks!

Describe the problem/error/question

Can’t handle API errors with a failover mechanism to retry the failed items in my loop until they are all processed properly.

What is the error message (if any)?

No error but when I send back the items failed to the HTTP request node in my loop, the amount of items double, instead to only reprocess the failed items.

Please share your workflow

Information on your n8n setup

  • n8n version: n8n 1.62.5
  • Database (default: SQLite): PostgreSQL
  • n8n EXECUTIONS_PROCESS setting (default: own, main): default?
  • Running n8n via (Docker, npm, n8n cloud, desktop app): Package Version: [email protected]
  • Operating system: Ubuntu 24.04
2

Hey @John_Stevenson,

What you have looks about right to me, Do you have some same data and some screenshots that show what happens when it doesn’t work?

In theory the Stats Extraction is going to run again with the output of the wait node which would be the error data, Although it looks like you are referenceing data from prebious nodes instead of taking the $json route so it could be that you need another loop over items node if the Stats Extraction is running more than once so you have a fixed point to reference.

3

Hey @Jon

I put the wrong API key to trigger the error and did a test with just one URL. So as you can see in my screenshot, the HTTP request node processes my URL… then moves that item to the error branch due to API error, tries to reprocess it… and now I move to 2 items (duplicate).

So with the current error handling, I end up duplicating my items every time it pass by the error branch.

items failed
items failed1484×641 37.5 KB

items failed 2
items failed 21207×610 37.4 KB

4

Hey @Jon ,

Did you have a chance to look at my post above? thanks :slight_smile: