I have data i’m pulling in from a google sheet (~1500 rows and 34 columns). The data contains various information on specific accounts. There is one column that contains a URL for the account, i.e. www.something.com, that I want to use in an HTTP GET node.
My goal is to use the data I get back from the HTTP node in a Code node.
I’m currently on the second tier of the Pro plan.
What is the error message (if any)?
This issue I’m facing “might” be with N8N cloud memory or the amount of requests I’m sending out at one time. Whatever I do, my workflow doesn’t execute successfully for the full data set.
Is there a way to optimize my workflow here? Any help is appreciated!
What happens when the workflow fails to run, What you have now looks like it should be ok but it depends on how much data that http request node is pulling back in. Have you tried using a subworkflow for the http request and code nodes to see if that helps?
Thanks for your response! When the workflow fails to run it seems to point to an execution error on the HTTP node or straight up tells me N8N has run out of memory.
The data depends entirely on the site which can be as large as 5mb sometimes.
With regards to using a subworkflow, wouldn’t I run into the same situation of the HTTP node timing out considering the input size is the same (~1500 sites to loop through and grab the data)?
Hey @Zaid_Ret just following up from our email thread. Have you considered an approach similar to this one? I’m sure @Jon could come up with something nicer but this is potentially an option to move the memory utilization into subworkflows.
I would recommend splitting the subworkflow out into its own workflow. I only included it in the same workflow here for legibility.
Hey @Ludwig ! Thanks so much for the solution! I think this is getting me closer to an optimal solution. Using your example, my workflow can handle a few hundred sites easily however, it still fails to do a batch of 500 or more. Do you think that might be related to the cloud memory or can I optimize is further?
