I am building a Website Scrapper that is connected to google sheets with 1000+ rows of URLS. My subworkflow handles the HTTP Request and HTML parsing to get the required data. In my original iteration, I was using N8N’s implicit looping and setting a batch size directly on the HTTP Request Node, however, since I am processing 1000s of URLs, the workflow crashes when memory limitations kick in. To mitigate for this, I set up Loop Over Items, with Reset turned on with a batch size of 50, so that 50 rows of enriched data get passed to Google Sheets before the Loop’s reset kicks in. The workflow is working but is suboptimal for edge cases - when I only have 1 row of data in google sheets, the Loop still runs 50 iterations of the same row.
How can I mitigate for this? I understand that I could connect the done output of the Loop directly to Update Sheets node, but then I would have to turn Reset off?
when I only have 1 row of data in google sheets, the Loop still runs 50 iterations of the same row.
I am not sure this is accurate, can you explain how you came to this conclusion? I tried in my workflow and it looks like if the batch size is greater than the input size, the loop over executes twice - one for input data and once to go into done branch.
An example I ran is below. In the example you can see three flows:
One where the input is 26 and the batch size is dynamic
(equal to 13 if input is > 13 or equal to input size if input size is <= 13)
You can see that the “loop over items” ran 3 times (because batch size was 13 - twice to go over data and once after coming back from the last batch
One where the input is 10 and the batch size is dynamic
(equal to 13 if input is > 13 or equal to input size if input size is <= 13)
The “loop over items” ran 2 times (because batch size was 10) - once to go over data and once after coming back from the last batch
One where the input is 10 and the batch size is static = 50
Here in spite the fact that the batch size is large (compared to the input), the loop only executed twice again, once to go over data and once after coming back from the last batch.
If you still think that the loop over items executes many times even if there is only one input, you can use the same technique as in the first two examples here - set it dynamic based on the input size with an expression.