Overview: A company I work with uses a 3rd party service to manage all their reservations and rental vehicles. This company likes to have everything on a Google Spreadsheet. So I use Webhooks from the 3rd party service to send data to the Company’s Spreadsheet via N8N. 95% of the time this works perfectly.
Challenge: Once every few days, the 3rd party service runs through every rental vehicle (whether it updates the vehicle or not) and sends out 100s of webhooks at once, this causes the Google API to limit out and thus N8N fails each execution.
Question: Is there a way I can setup N8N to basically capture all the Webhooks data and then process through it slowly. So house the data in a database and then it casually processes each row of data say every 30 seconds? Now of the data is time sensitive so if it takes an hour or so to process through the data that’s okay. It becomes an issue when the data doesn’t show up though until I manually re-run the failed execution.
The Webhook node will be the starting point of the workflow. In your case, I’m not sure how you’ll process Webhook later or it’s an API?
If I’m working on this, Here is what I’ll do.
Use the wait node to slow down the process before inserting it into Google Sheet. (Recommended)
(Optional) Also, make sure to run everything in the main process (Check here). Please note that this method comes with Pros and Cons. Please look at the documentation for more information.
If it’s an API, This needs one more extra setup to do that.
You need three columns for example. Store the whole list of API Urls somewhere like Airtable and process that later.
Date
Webhook URL
done
16/04/22
https://
yes
16/04/22
https://
yes
16/04/22
https://
no
You have to compare the date and fetch a single webhook and make sure to mark it as done everytime. Once everything is completed. You can make an automation to delete all the rows of data.
Wouldn’t using the wait node just move everything out for however long the wait is set to? Example being: If I have 100 calls to the webhooks and then the 100 calls hits the wait (Say 1 min) node it’d just wait 1 min on each call and then process all 100 calls 1 min out, wouldn’t it?
There Is a few ways to deal with your issue. Ill list 2 I can think of.
You could set up a database/table between the webhook and the actual processing to try and throttle the requests to Google. You would not delete the record to be processed when the google API sends you a response and try again later. Add a wait node when an error occurs and your problem should be fixed.
You could also add a trigger in this workflow which is called by the workflow when an error occurs.
It would look like this: Google sheet call → if Error → Wait → trigger the same workflow
Its a bit of a dirty solution but should work. Make sure to add some kind of counter to the item So you can raise the wait if it fails again. so wait time = some number * counter. Or Some number * anothernumber*Counter.
hope I gave you some ideas on how to deal with this.
